A central monitoring system (CMS) for a truly distributed and cloud based video conferencing solution could only mean one thing: Humungous amounts of data. Consider some simple graphs, on the monitoring systems’ web based dashboard, plotting basic statistics such as call quality, video resolution, bitrate, etc. This data is collected every 10 seconds for each participant. That makes it 360 * 4 rows per participant in a one hour conference (4 being the number of channels open – Audio In, Audio Out, Video In and Video Out). Now multiply that into 5 (number of participant in the conference) and multiply that into 100 (number of conferences per day). I think you get the idea. But this colossal volume of data is not the only problem we face here and it certainly is not the biggest. Unconventional solutions like mongoDB helped us get around the problem that the huge amount of data poses, with relative ease.
The second more telling issue was identifying patterns in the data. Patterns which would tell the support team about customers facing quality issues on a particular conference and point them towards the reasons behind these issues. This would make the whole process of addressing customer grievances more proactive. A simple rule evaluator is something we zeroed in on initially which would simply pick rules specified as strings in a configuration file and run them against records which are being pulled from the database.
Is this really a good solution though? Save loads and loads of data and then mine through it to find interesting patterns. What if we turn this on its head? I mean pre determine the queries and let the data pass through it. Instead of storing the data and running queries against stored data, we store queries and run the data through. This is exactly the essence of Event Stream Processing (ESP) or Complex Event Processing (CEP).
Once we decided to go in the CEP direction, Esper was our man. Esper is capable of triggering custom actions written as Plain Old Java Objects (POJO’s) when event conditions occur among event streams. It is designed for high volume event correlation where millions of events coming in would make it impossible to store them all to later query them using classical database architecture. A tailored Event Processing Language (EPL) allows expressing rich event conditions, correlation, possibly spanning time windows, thus minimizing the development effort required to set up a system that can react to complex situations. EPL statements derive and aggregate information from one or more streams of events, to join or merge event streams, and to feed results from one event stream to subsequent statements. EPL is similar to SQL in its use of the select clause and the where clause. However EPL statements instead of tables use event streams and a concept called views. Similar to tables in an SQL statement, views define the data available for querying and filtering. Views can represent windows over a stream of events. Views can also sort events, derive statistics from event properties, group events or handle unique event property values. Esper is available for Java as Esper, and for .NET as NEsper.
Let’s just look at a basic example which reflects on the ease and power of leveraging Epser within your application. Following is a POJO representing statistics of an Endpoint at a point in time:
public class EndpointStats {
private long id;
private int callQuality;
private String …..
public void setId() { … }
public long getId() { … }
public void setCallQuality() { … }
public int getCallQuality() { … }
}
The CMS gets this POJO as an event every few seconds (say 10). We want to be able to trigger an alert on the CMS if the call quality for a user is less than 3 for 2 minutes or more. Sending an event to the Esper engine is a matter of a one line statement: epServiceProvider.getEPRuntime().sendEvent(event);
Now to select the number of endpoint stats with call quality less than 3 over a 2 minute window we would use the following EPL statement:
String str = “select count(*) as mycount from endpointstats(callQuality < 3).win:time(120 sec)”;
And add an update listener for the above EPL statement:
ep = epService.getEPAdministrator().createEPL(str);
ep.addListener(new UpdateListener() {
@Override
public void update(EventBean[] arg0, EventBean[] arg1) {
System.out.println(arg0[0].get(”mycount”));
}
});
Every time the condition in the EPL statement is met, the update listener is called with the selected parameters. Further processing can be done in the update listeners.
To concur, the CEP/ESP model needs to be given a serious consideration while dealing with huge data and events, and Esper is a powerful tool with some very useful features like an IO event simulation package called EsperIO which allows throughput and latency testing in your target environment, support for multiple types of event representations, a fairly vast Event Processing Language and a lot more. Refer to http://esper.codehaus.org/ for additional information.
- Parmeet