azure event hubs, stream analytics & power bi (by sam vanhoutte)

INTEGRATION DAY 2015

Sam Vanhoutte

Azure Event Hubs, Stream Analytics & Power BI

Nice to meet youSam VANHOUTTECTO

7 year - BizTalk V-TSP1st year - Integration MVP

[email protected]+32 474 849 993@SamVanhoutte

be.linkedin.com/in/

samvanhoutte/

> 60 Active integration customers

International Focus - HQ in BEFocused on integration solutions

2000 Belgium2004 France2013 Portugal

60 employees > 50 consultants BizTalk certifiede-news + SoMe

2012 & 2013

Partner of the Year

Award FinalistApplication Integration

The Internet of Things

3

The Internet of Everything

M2M communicationSpecial purpose

devicesSmart things

The IoT value chain

Ingestor (broker)

Collection Presentation and action

Event producers Transformation Long-term storage

Event Hubs

Storage adapters

Stream processingCloud gateways

(web APIs)

Field gateways

Applications

Legacy IOT (custom protocols)

Devices

IP-capable devices(Windows/Linux)

Low-power devices (RTOS)

Search and query

Data analytics (Excel)

Web/thick client dashboards

Service bus

Azure DBs

Azure storage

HDInsight

Stream Analytics

Devices to take actionMachine Learning

Demo scenario

6

Event Hubs Stream Analytics Power BI

Ingest

Speeding tickets

Traffic jam detection

Power BI tools

7

• Power BI designer (when no office license)

• PowerQuery >> Get data

• PowerPivot >> Prepare data

• PowerView >> Present data

Event Hubsan Azure Service Bus service

(collection & ingestion)

Azure Service BusAzure Service Bus

Relay Request – Response services

Queue High-Reliability Async MessagingSessions, Scheduled Delivery, etc.

Topic Queues with rules-based pub/sub

Notification Hub Mobile push notification servicesMillions of notification targets

Event Hub partitioned event ingestion

Event Hub – IOT at Scale

Event Sources Cloud Services

Storage & Analytics

Custom Code & 3rd Party Services

Web/Mobile User Interfaces

Integration Services

Event Hub

- Hyper Scale -- Fully Managed -- Interoperable -

- Secure -- Cost Effective -

Introducing Azure Event Hubs

Azure Event Hub

Up to 32 partitions via portal, more on

request

Parti

tions

Event Producers

> 1M Producers> 1GB/sec Aggregate Throughput

Direct

PartitionKeyHash

Throughput Units:• 1 ≤ TUs ≤ Partition Count• TU: 1 MB/s writes, 2 MB/s reads

AMQP 1.0Credit-based flow controlClient-side cursorsOffset by Id or Timestamp

Consumer Group(s)

Receivers

Event Processor Host

IEventProcessor


Producers(sending data to Event

Hubs)

15

Publishers

Event Producers

Direct

PartitionKeyHash

Parti

tions

Very many publishersShort-lived, low throughput: HTTPSLong-lived, high throughput: AMQP

Long lived AMQP connections are billable, HTTPS requests are not; AMQP connection allowance included in tier

Publish to …PartitionId

DirectPartitionKey

PartitionKey selecting PartitionIdPublisher Policy (<eh>/publishers/<name>)

<name> overriding PartitionKey

16

Publishers

Event Producers

Direct

PartitionKeyHash

Parti

tions

Very many publishersShort-lived, low throughput: HTTPSLong-lived, high throughput: AMQP

Long lived AMQP connections are billable, HTTPS requests are not; AMQP connection allowance included in tier

Publish to …PartitionId

DirectPartitionKey

PartitionKey selecting PartitionIdPublisher Policy (<eh>/publishers/<name>)

<name> overriding PartitionKey

Demo:• Create Event Hubs• Send events to Event Hub

17

Securing publishers

➔ SAS Policies are defined in the portal➔ Not linked to publishers and limited in numbers

➔ Create Signatures per device➔ Have a timespan➔ Are unique to a publisher

➔ Revoking or blacklisting publishers➔ Manage Rights needed for this

18

Securing publishers

➔ SAS Policies are defined in the portal➔ Not linked to publishers and limited in numbers

➔ Create Signatures per device➔ Have a timespan➔ Are unique to a publisher

➔ Revoking or blacklisting publishers➔ Manage Rights needed for this

Demo:• Generating SAS Signatures• Using Signatures by publishers• Revoking publishers

Reminder ! Sam, increase your instances


Consumers(reading data to Event

Hubs)

Like subscriptions

Consumer groupsReceivers read from a consumer groups

Consumer Group(s)

Checkpointing per CG

Create receivers per

partitionMaximum 20

CGConsumer

Grp

{Default}

Like subscriptions

ConsumersPartitioned consumer model (not competing consumers!)

Each at their own pace

PartitionEvent Log

ID, Time, [Data]

ID, Time, [Data]

ID, Time, [Data]

ID, Time, [Data]

ID, Time, [Data]

Time

ID

22

EventProcessorHostOut of the box Lease management

Nuget package

Managed cursors by

client

Uses storage for state and

CPIEventProcess

or

23

EventProcessorHostOut of the box Lease management

Nuget package

Managed cursors by

client

Uses storage for state and

CPIEventProcess

or

Demo:• Implementing EventProcessorHost• Dashboarding demo

24

Securing consumers

➔ Using SAS key / values or ACS with SBAZTool➔ Full SAS support for consumer groups will

come


Pricing & Tips + tricks

26

Pricing

Basic: Up to 100 connections, no extensionStandard: 1000 connections incl.

Price (US Dollars)

Throughput Unit Hour (Basic) 0.015 TU per hour

Throughput Unit Hour (Standard) 0.03 TU per hour

Ingress Events 0.028 per 1,000,000 events

Cost Brokered Connections (1k-100k) 0.00004 connection/hour

Cost Brokered Connections (100k-500k) 0.00003 connection/hour

Cost Brokered Connections (500k+) 0.00002 connection/hour

Storage Overage >TUs*84GB local-redundant Azure storage charge-through

27

Tips & tricks

Multiple hubs per namespace

Reuse senders & factories

Use consumer groups

Azure Stream Analytics complex event

processing.

Introduction(set up a basic ASA job)

Scenarios

Smart grid CRM alerting sales with customer scenario

Data and identity protection services

Real-time fraud detection

Click-stream analysis Real-time financial portfolio alerts

Connected car scenarioReal-time financial sales tracking

30

Real time analytics

Millions events / sec

Continuous stream of

data CorrelateFast time to

value

High availability SQL Syntax

Easy test & debug in

portal PowerBI !

.999% SQL

31

End to end Architecture overview

Data Source

Collect Process ConsumeDeliver

Event Inputs- Event Hub- Azure Blob

Transform- Temporal joins- Filter- Aggregates- Projections- Windows- Etc.

Enrich

Correlate

Outputs- SQL Azure- Azure Blobs- Event Hub- Table storage- PowerBI

☁

BI Dashboards

Predictive Analytics

AzureStorage

• Temporal Semantics

• Guaranteed delivery

• Guaranteed up time

Azure Stream Analytics

Reference Data- Azure Blob

32


Data Source




Enrich

Correlate

Outputs- SQL Azure- Azure Blobs- Event Hub

☁

BI Dashboards


AzureStorage






Demo:• Set up Stream Analytics• The first job

Azure Stream Analytics complex event

processing.

Query Syntax(SQL like a pro)

Functions & supported types

Aggregate functionsCount, Min, Max, Avg, Sum

Scalar functionsCast

Date and time: Datename, Datepart, Day, Month, Year, Datediff, Dateadd

String: Len, Concat, Charindex, Substring, Patindex

Types

Type Descriptionbigint Integers in the range -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807).

float Floating point numbers in the range - 1.79E+308 to -2.23E-308, 0, and 2.23E-308 to 1.79E+308.

nvarchar(max) Text values, comprised of Unicode characters. Note: A value other than max is not supported.

datetime Defines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock and relative to UTC (time zone offset 0).

35

Tumbling windowsGroup events, based on timestamping

Tumbling window

Aggregate per time interval

Hopping window

Schedule overlapping windows

Sliding window

Windows constant re-evaluated

36


Data Source




Enrich

Correlate


☁

BI Dashboards


AzureStorage






Important:• For PowerBI you need

organizational account

37


Data Source




Enrich

Correlate


☁

BI Dashboards


AzureStorage






Demo:• Traffic management (speed tickets)• Traffic management (average speed)• Traffic management (suspected cars)

38

Tips & tricks

Use Timestamp by

Chose your windowing logic

Consumer groups

Don’t delete blobs while job

runs

azure event hubs, stream analytics & power bi (by sam vanhoutte)

Technology