Azure real-time insights and visualisations
This is the 3rd article in our Azure series. We have been building an architecture that will work for a simple node application but will also scale to micro-services and/or IoT, and then into a fully fledged solution platform running on Azure. In the previous two articles, we focused on rapid application development on Microsoft Azure , and how to collect fast Pino logs in an Azure Event Hub . To complete the first part of our series, we will investigate how we can process and gain insights from the data we are generating.
![architecture](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/3186e35065-1722602897/architecture3-1600x.png)
Real-time insights: Stream Analytics
This post assumes you have a Web App that pipes logs into an Event Hub. (If not instructions are included in the previous article .) Now we want to be able to gain real-time insights from that data.
Azures Stream Analytics processes data in real-time. In Microsoft's words:
“Using Stream Analytics, you can examine high volumes of data flowing from devices or processes, extract information from the data stream, and look for patterns, trends, and relationships. Based on what’s in the data, you can then perform application tasks. For example, you might raise alerts, kick off automation workflows, feed information to a reporting tool such as Power BI, or store data for later investigation.”
With Stream Analytics we want to be able to extract information from our log stream, eg,
- Query response times
- 90th, 95th and 99th percentile response times
- Warnings on a 404 requests
- Alerts on Denial-of-Service attacks
For this article, we will simply look at query response times.
Real-time dashboard: Power BI
The last piece of our minimal architecture would be to visualise the information extracted from our logs. Azures Power BI Embedded allows the creation of a real-time dashboard by embedding components in your application. However, for the simplicity of the article, we will be using Power BI where a dashboard is ready for use.
Caveat Emptor: Even though the Power BI dashboard uses real-time data, it has to be manually refreshed to show it.
Initial setup
To start, please provide a basic application with logging, as per Collect fast Pino logs in an Azure Event Hub .
Provision Power BI
First, we want to set up our Power BI account, because Stream Analytics would need it as a target output.
Power Bi UI is not completely integrated into Azure UI yet. Do not set up your Power BI account from within Azure. Instead, open https://powerbi.microsoft.com/en-us/get-started/ Click Try Free (in the middle-left of the page) under Power BI Cloud collaboration and sharing, then register with your existing Microsoft login.
Now we want to create a workspace from the menu on the left:
![01a_create_workspace](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/b34435d5e3-1722602894/01a-create-workspace-1600x.png)
![02_power_bi_try_pro_free](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/50aa0844d1-1722602895/02-power-bi-try-pro-free-1600x.png)
Now we can proceed to create a workspace named pinoPowerBI
:
[
![02b_create_workspace](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/9fa833dd93-1722602895/02b-create-workspace-1600x.png)
][1]
Register your app on Azure Active Directory
Some of the online advice says that it is a requirement to register your app on Azure Active Directory (AAD). However, this only relevant if you want to embed Power BI into your app. We will be viewing our graphs directly on https://powerbi.microsoft.com/ , so we can ignore this.
Provision Stream Analytics
Create an input and an output
Microsoft uses the words input and source, and output and sink interchangeably. In this article, we will stick to input and output.
Go back to https://portal.azure.com/ . Search for Stream Analytics jobs
:
![03_sa_search](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/bb741bb269-1722602895/03-sa-search-1600x.png)
Add a New Stream Analytics Job
, named responseTimesAll
:
![04_sa_create_new](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/e2882e77f3-1722602895/04-sa-create-new-1600x.png)
This should result in a success screen like this:
![05_sa_create_success](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/152bde7199-1722602895/05-sa-create-success-1600x.png)
Before we create the input. Go create an Event Hub policy under pinoEventHub
called listenPinoEvents
with listen
privileges:
![05a_add_poilcy](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/daa1a55372-1722602895/05a-add-poilcy-1600x.png)
Now, click on Inputs
under Job Topology
:
![06_sa_to_add_input](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/4e33565817-1722602895/06-sa-to-add-input-1600x.png)
Add a new Input and connect it to listenPinoEvents
with alias inputFromPinoEventHub
:
![07_sa_new_input](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/410fa700d5-1722602895/07-sa-new-input-1600x.png)
![08_sa_input_added](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/bcd647b2e5-1722602895/08-sa-input-added-1600x.png)
Click on Outputs
under Job Topology
, and choose to add a new Output
:
![10_sa_to_add_output](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/37dbe1dfc9-1722602896/10-sa-to-add-output-1600x.png)
Name the output alias dashboard
, and choose Power BI
under Sink
:
![11_sa_output_create](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/6b7daa0dcc-1722602896/11-sa-output-create-1600x.png)
Now click authorize, and sign in with the Power BI user you enabled earlier:
![12_power_bi_login](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/2be2a0bd66-1722602896/12-power-bi-login-1600x.png)
Now we can create the Output named dashboard
, in Group Workspace pinoPowerBI
, with Dataset Name pino
and Table Name `responseTimes :`
![13_sa_output_create](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/e6b2a81eb0-1722602896/13-sa-output-create-1600x.png)
![14_sa_output_added](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/677dc80980-1722602896/14-sa-output-added-1600x.png)
Now we are all set to connect input and output.
Real-time query language
The real power of Stream Analytics is the Stream Analytics Query Language . We will only be doing a simple example, but with this language queries can done as data is received from the input.
To create our query we will select Query
under Job Topology
:
![15_sa_overview](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/2e2a62735b-1722602896/15-sa-overview-1600x.png)
We want to create a time-series graph showing response times of queries against time. So, we need to SELECT
responseTime
and time
. We take those values from the input inputFromPinoEventHub
and send them into the output dashboard
, using FROM inputFromPinoEventHub
and INTO dashboard
.
However, Pino formats time in unix time, and Power BI needs it in ISO 8601 format. In Stream Analytics every event has an associated time stamp. We can override that time stamp and convert our time
value into ISO 8601 format, using TIMESTAMP BY DATEADD(millisecond, time, 1970-01-01T00:00:00Z)
. Lastly we need to change our SELECT
to pick the event time stamp we have overwritten System.Timestamp AS time
.
Lastly, we want to filter for log results that actually have a responseTime
defined, which WHERE (responseTime > 0)
will do for us.
Resulting in this query:
![16_construct_query](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/3984ec6b29-1722602896/16-construct-query-1600x.png)
Before we save this as done, we can test the query by uploading test data:
![17_upload_test_data](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/d8c6802e09-1722602896/17-upload-test-data-1600x.png)
If you run the test, you should get some positive results:
![18a_test](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/7dfb25012a-1722602897/18a-test-1600x.png)
We can now save our query, and then start the Stream Analytics job (this might take a couple of moments). Once, the running sign is up, refresh your Web App url. Then you should see an event:
![20_event](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/b72659de5f-1722602897/20-event-1600x.png)
That single event we witnessed before created a pino
dataset in Power BI. Go ahead and create a couple more data points by refreshing your web page.
Visualize the data
Now, we leave https://portal.azure.com/ and return to https://powerbi.microsoft.com/ . Expand tour pinoPowerBi
workspace. If it is not visible, select it under Workspaces
:
![21_pbi_expand](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/92bd23f12f-1722602897/21-pbi-expand-1600x.png)
Now, under DATASETS
, click on the pino
, and select a line graph.
![22_pick_visualization](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/99af2e91b3-1722602897/22-pick-visualization-1600x.png)
To populate our graph, we drag time
under Axis
. And, we drag responsetimes
under Values
:
![23_pick_axis](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/7c13acb392-1722602897/23-pick-axis-1600x.png)
To view our graph simply maximise it:
![24_graph](https://www.nearform.com/media/pages/insights/azure-real-time-insights-and-visualisations/430e989ecf-1722602897/24-graph-1600x.png)
To keep this graph save it to a report.
Caveat Emptor: Time Zones
Pay attention to time zone being displayed. I've noticed 3 time-zones being used in graphs across Azure and Power BI: The time-zone I'm in, the time-zone my company is in, and GMT +0.
Azure Resource Manager templates
We provisioned Event Hubs and Stream Analytics using the Azure Portal point-and-click interface because we wanted a first-time or prototyping user to be up-and-running with Azure as quickly as possible. As your product or usage matures though, you would want to describe your infrastructure in code. This is what Azure Resource Manager templates are for. These templates can be harnessed from the Azure CLI , to deploy, update, or delete your resources in a single action.
Conclusion
Power BI and Azure might not be fully integrated on a UI level yet, but the back end integration works great. Stream Analytics real-time insights combined with Power BIs embedded visualization functionality allows opportunities to present business value metrics to management in custom apps.
Here ends our series on setting up a basic Node.js app using Azure. Next, we will look at using Docker to provision a Node.js app on Azure Web Apps.
[1]: /wp-content/uploads/jekyllsite/blog/2017/07/02b-create-workspace.png Scott Rodgerson
Insight, imagination and expertly engineered solutions to accelerate and sustain progress.
Contact