Azure Stream Analytics – A Simple Proof-Of-Concept

In my last post I gave a high level overview of the CallStats proof-of-concept app that I built using Azure Stream Analytics, in this post I’m going to explain how to get it running for yourself.

Prerequisites

You will need an Azure subscription with the Stream Analytics preview enabled.

Setting up Service Bus

The first step is to configure some Event Hubs, these will be the ingest and output for Stream Analytics.

  1. From the Azure portal (http://manage.windowsazure.com) create a new Service Bus namespace, (make a note of this, as you will need it when configuring the projects).
  2. Create a new Event Hub called CallStatsIn with 8 partitions
    1. Create a shared access policy with the Manage, Send and Listen permissions
    2. Make a note of the connection string for this policy
  3. Create another Event Hub, this time called CallStatsOut with 8 partitions
    1. Create a shared access policy with the Manage, Send and Listen permissions
    2. Make a note of the policy name and the primary key

Setting up Stream Analytics

The next step is to configure a new Stream Analytics job

  1. From the Stream Analytics section in the Azure portal create a new job, you can call it whatever you like
  2. Select the Input tab, and create a new event hub input called CallStatsIn specifying JSON as the serialization format
  3. Select the Output tab and create a new event hub output called CallStatsOut, specifying JSON as the serialization format
  4. Select the Query tab, and paste the following query:
SELECT
    ConnectionName, 
    DateAdd(second,-5,System.TimeStamp) as WinStartTime, 
    System.TimeStamp as WinEndTime, 
    Sum(CASE WHEN EventType=1 THEN 1 ELSE 0 END) as Started,
    Sum(CASE WHEN EventType=2 THEN 1 ELSE 0 END) as Ended,
    Sum(CASE WHEN EventType=3 THEN 1 ELSE 0 END) as Dropped,
    Sum(CASE WHEN EventType=4 THEN 1 ELSE 0 END) as NotAnsweredBusy
FROM 
    callstatsin
GROUP BY 
    HoppingWindow(second, 5, 1), ConnectionName

Setting up the Solution

Download my sample solution from GitHub (https://github.com/kzhen/StreamAnalytics/archive/master.zip).

  1. Open in Visual Studio (as an administrator)
  2. Open the App.config file in the Consumer project
    1. Set ServiceBus.Namespace to the namespace you created above
    2. Set ServiceBus.KeyName to the name of the shared access policy for CallStatsOut Event Hub
    3. Set ServiceBus.Key to the Primary Key for the CallStatsOut Event Hub
  3. Open the App.config file in the CallProducer project
    1. Set Microsoft.ServiceBus.ConnectionString to the connection string for the CallStatsIn Event Hub

Running the Solution

Build the solution then start each project. You should see output in each of the console projects and the graph on the webpage should start updating

Advertisements

Azure Stream Analytics

There have been a number of new Azure services announced in recent weeks, one that caught my eye was Azure Stream Analytics. Stream Analytics is a cloud based event processing engine, which ingests real time events from various sources, runs a temporal query and then outputs the results for you to consume or to store for later analysis.

The feature set currently available is a simplified version of StreamInsight, the on-premise predecessor. I’ve written previously about using StreamInsight to visualize telephone calls in real-time. So I thought I would redo this using Stream Analytics.

The pieces of the puzzle

There are a number of different components to the overall solution

stream-analytics-telephony

What they do:

  • Fake Telephony App Server – this publishes events to the Azure Event Hub (CallStatsIn)
  • Event Consumer – this subscribes to the events which are published by Stream Analytics and broadcasts the events to a SignalR hub
  • Web Dashboard – hosts the SignalR hub and a web page which listens to the SignalR hub and updates a real-time chart as data comes in

Events and Queries

The events which the Fake Telephony App Server produces aren’t very complicated:

public class PhoneCallEvent
{
  public string ConnectionName { get; set; }
  public Guid Id { get; set; }
  public int EventType { get; set; }
}

On a random schedule it will generate a number of CallStarted, CallEnded, CallDropped, CallNotAnsweredBusy events and publishes them to the Event Hub (CallStatsIn)

In Stream Analytics I have a job which is subscribed to the CallStatsIn Event Hub and aggregates this data with the following temporal query:

SELECT
    ConnectionName,
    DateAdd(second,-5,System.TimeStamp) as WinStartTime,
    System.TimeStamp as WinEndTime,
    Sum(CASE WHEN EventType=1 THEN 1 ELSE 0 END) as Started,
    Sum(CASE WHEN EventType=2 THEN 1 ELSE 0 END) as Ended,
    Sum(CASE WHEN EventType=3 THEN 1 ELSE 0 END) as Dropped,
    Sum(CASE WHEN EventType=4 THEN 1 ELSE 0 END) as NotAnsweredBusy
FROM
    callstatsin
GROUP BY
    TumblingWindow(second, 5), ConnectionName

Web Dashboard

The above query will stream the results to the CallStatsOut Event Hub which my consumer is subscribed to. The consumer will then broadcast results as they are received to a SignalR Hub. My web dashboard is subscribed to the SignalR hub and will update my line chart in real-time.

2-live-chart

This is the output from the Fake Telephony Server App, showing the total number of events being published to the CallStatsIn Event Hub.

4-producer

This is the raw data from the CallStatsOut Event Hub, which is then broadcast to the SignalR hub which the dashboard is subscribed to

3-consumer

Wrapping Up

My first impressions from working with Stream Analytic are overall very positive. It is an easy service to get going with and has lots of potential.

So far the only downside that I’ve come across is that you’re only allowed one output per job. This means if you wanted to store the results for later processing you would need to setup two duplicate jobs with the same query but with different outputs (PITA to maintain).

I’ll put my code up on GitHub in the coming days for anyone who is interested.