MAXIME LEMAITRE – 9/7/2015

Microsoft Azure Storage… on the road to a scalable architecture …

Agenda

• Introduction• What is a Storage Account ?• Storage Services• Quotas & Pricing• Working with Azure Storage• Demo• Conclusion• Question

Standard Storage Accounts

Blob storage stores file data. A blob can be any type of text or binary data, such as a document, media file, or application installer.

Table storage stores structured datasets. Table storage is a NoSQL key-attribute data store, which allows for rapid development and fast access to large quantities of data.

Queue storage provides reliable messaging for workflow processing and for communication between components of cloud services.

File storage (Preview) offers shared storage for legacy applications using the standard SMB 2.1 protocol. Azure virtual machines and cloud services can share file data across application components via mounted shares, and on-premise applications can access file data in a share via the File service REST API.

A standard storage account gives you access to Blob storage, Table storage, Queue storage, and File storage.

4

How to create a Storage Account ?(in less than 5 minutes)

Note : To use this new account, you just need the AccountName and Access Keys

5

Storage QuotasRESOURCE DEFAULT LIMITTB per storage account 500 TBMax size of a single blob container, table, or queue 500 TBMax number of blob containers, blobs, shares, tables, queues, entities, or messages per storage account

Only limit is the storage account capacity

Max size of a file share 5 TB (Unlimited number of files)Max 8 KB IOPS per persistent disk (Basic/Standard) 3001/5001Total Request Rate (assuming 1KB object size) per storage account

Up to 20,000 IOPS, entities per second, or messages per second

Target Throughput for Single Blob Up to 60 MB per second, or up to 500 requests per second

Target Throughput for Single Queue (1 KB messages) Up to 2000 messages per secondTarget Throughput for Single Table Partition (1 KB entities) Up to 2000 entities per secondTarget Throughput for Single File Share (Preview) Up to 60 MB per secondMax ingress2 per storage account (Europe) 5 Gbps if GRS3 enabled, 10 Gbps for LRSMax egress2 per storage account (Europe) 10 Gbps if GRS3 enabled, 15 Gbps for LRS

6

Pricinghttp://azure.microsoft.com/en-us/pricing/details/storage/

Additional €0.0027 per 100,000 transactions for all Standard storage types

Page Blobs & Disks

Block Blobs

Tables & Queues

7

• Build apps with first class client library support for .NET, Java, Android, C++, and Node.js.

• Data in Azure Storage is also accessible via REST API, which can be called by any language that makes HTTP/HTTPS requests

Clients

Also available in Node.js, Java, C++, Python, Xamarin, Ruby, …

8

• Azure Blob storage is a service for storing large amounts of unstructured data, such as text or binary data, that can be accessed from anywhere in the world via HTTP or HTTPS. Common uses include :– Serving images or documents directly to a browser– Storing files for distributed access– Streaming video and audio– Performing secure backup and DR– Storing data for analysis by

an on-premises or Azure-hosted service

• Concepts– Container : A container provides a grouping of a set of blobs. All blobs must be in a container.An

account can contain an unlimited number of containers. A container can store an unlimited number of blobs.

– Blob : A file of any type and size. There are two types of blobs that can be stored in Azure Storage: block and page blobs. Most files are block blobs. A single block blob can be up to 200 GB in size. This tutorial uses block blobs. Page blobs, another blob type, can be up to 1 TB in size, and are more efficient when ranges of bytes in a file are modified frequently. For more information about blobs, see Understanding Block Blobs and Page Blobs.

– URL format: http://<storage account>.blob.core.windows.net/<container>/<blob

Blob Storage

9

• A NoSQL datastore for large amounts of structured data. Accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for storing structured, non-relational data. Common uses include :– Storing TBs of structured data for web– Storing datasets that don't require complex joins,

foreign keys, or stored procedures and can be denormalized for fast access

– Quickly querying data using a clustered index– Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries

Concepts– Table : A table is a collection of entities. Tables don't enforce a schema on entities, which means a

single table can contain entities that have different sets of properties. The number of tables that a storage account can contain is limited only by the storage account capacity limit.

– Entity: An entity is a set of properties, similar to a database row. An entity can be up to 1MB in size.– Properties: A property is a name-value pair. Each entity can include up to 252 properties to store

data. Each entity also has 3 system properties that specify a partition key, a row key, and a timestamp. Entities with the same partition key can be queried more quickly, and inserted/updated in atomic operations. An entity's row key is its unique identifier within a partition.

– URL format: http://<storage account>.blob.core.windows.net/<table>

Table Storage

10

• Azure Queue storage is a service for storing large numbers of messages that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account. Common uses include :– Creating a backlog of work

to process asynchronously– Passing messages from

an Azure Web role to an Azure Worker role

Concepts– Queue: A queue contains a set of messages. All messages must be in a queue.– Message: A message, in any format, of up to 64KB.– URL format: http://<storage account>.blob.core.windows.net/<queue>

Queue Storage

11

• Offers shared storage for applications using the standard SMB 2.1 protocol. Azure virtual machines and cloud services can share file data across application components via mounted shares, and on-premises applications can access file data via the File storage API. Common uses include :– Migrating on-premises applications that rely on file shares to run on Azure virtual machines or cloud

services, without expensive rewrites– Storing diagnostic data such as logs,

metrics, and crash dumps in a shared location

– Storing tools and utilities needed for developing or administering Azure VM or cloud services

– Storing shared application settings, for example in configuration files

Concepts– Share: A File storage share is an SMB 2.1 file share in Azure. All directories and files must be created

in a parent share. An account can contain an unlimited number of shares, and a share can store an unlimited number of files, up to the capacity limits of the storage account.

– Directory: An optional hierarchy of directories.– File: A file in the share. A file may be up to 1 TB in size.– URL format: https://<storage account>.file.core.windows.net/<share>/<directory/directories>/<file>

Files Storage

12

Partitions affect load balancing and scalability for each of the storage services.

• Blobs: The partition key for a blob is container name + blob name. This means that each blob has its own partition. Blobs can therefore be distributed across many servers in order to scale out access to them. While blobs can be logically grouped in blob containers, there are no partitioning implications from this grouping.

• Messages: The partition key for a message is the queue name, so all messages in a queue are grouped into a single partition and are served by a single server. Different queues may be processed by different servers to balance the load for however many queues a storage account may have.

• Entities: The partition key for an entity is table name + partition key, where the partition key is the value of the required user-defined PartitionKey property for the entity.

All entities with the same partition key value are grouped into the same partition and are stored on the same partition server. This is an important point to understand in designing your application.

Partitions in Azure Storage

13

• Provides a local environment that emulates the Azure Blob, Queue, and Table services for development purposes (File Shares is not supported)

• Uses a local MS SQL Server instance (LocalDB) and the local file system to emulate the Azure storage services

• a single fixed account and a well-known authentication key

• Service endpoints

Working with Storage emulatorhttps://azure.microsoft.com/en-us/documentation/articles/storage-use-emulator/

14

Usefull toolshttp://storagetools.azurewebsites.net/

Visual Studio + Azure SDK Azure Storage Explorer

15

Example #1a better front-end process to optimize images

Images (PNG/JPG/GIF/SVG)

Block Blob

CDN endpoint

Gulp process (Imagemin + upload to Azure)

16

Example #2Optimazer

Azure BlobStorage optimized

images

Worker RoleScan web pages

and optimize images

Azure TableStore pages &

images results/stats

Azure Web SiteWeb Api +

Angular

SchedulerAutomatically add

page to scanAzure Queue

Pending scans, optimizations …

Demo

Questions

References

• https://azure.microsoft.com/en-us/documentation/articles/storage-introduction/• http://azure.microsoft.com/en-us/documentation/services/storage/• https://azure.microsoft.com/en-us/documentation/articles/storage-introduction/• https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-tables/ • https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-queues/ • https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-files/ • https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/

About Us• Betclic Everest Group, one of the world leaders in online

gaming, has a unique portfolio comprising various complementary international brands: Betclic, Everest Poker/Casino, Bet-at-home, Expekt, Imperial Casino, Monte-Carlo Casino…

• Through our brands, Betclic Everest Group places expertise, technological know-how and security at the heart of our strategy to deliver an on-line gaming offer attuned to the passion of our players. We want our brands to be easy to use for every gamer around the world. We’re building our company to make that happen.

• Active in 100 countries with more than 12 million customers worldwide, the Group is committed to promoting secure and responsible gaming and is a member of several international professional associations including the EGBA (European Gaming and Betting Association) and the ESSA (European Sports Security Association).

We want our Sports betting, Poker, Horse racing and Casino & Games brands to be easy to use for every gamer around the world. Code with us to make that happen.

Look at all the challenges we offer HERE

Check our Employer Page

Follow us on LinkedIn

WE’RE HIRING !