exploring azure cloud storage
TRANSCRIPT
![Page 1: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/1.jpg)
K.Mohamed FaizalLead Consultant @ NCS (P) [email protected]://faizal-comeacross.blogspot.com
AzureUG.SG 26th Jan 2011.
Exploring Windows Azure Cloud Storage
![Page 2: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/2.jpg)
About Me
14
![Page 3: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/3.jpg)
Agenda
Cloud Storage Account OverviewTablesBlobsQueuesStorage Viewer / Explorer Tools
![Page 4: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/4.jpg)
Cloud Storage
The types of Azure Storage• Blobs – A file system• Tables – Hierarchical Data Tables• Queues – Asynchronous Messaging• Drives – NTFS file systems (Blob)Each item in storage is stored in
triplicate.
Blobs QueuesTables
![Page 5: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/5.jpg)
Make a guess!
What is the maximum size of data, a storage account can hold?a) 1 GBb) 100 GBc) 1 TBd) 100 TB
http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx
Answer: d) 100 TB
![Page 6: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/6.jpg)
Windows Azure and SQL Azure
Azure Storage SQL Azure
Vision Highly scalable, highly available store in the Cloud
Scalable, highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational? No Yes – but with some limitations
Analogy File System RDBMS – as it is
Maximum Amount of data in a single “database”
100TB 50GB (up to Oct 2010)
Price per GB per month
$ 0.15 $ 9.99
![Page 7: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/7.jpg)
Creating a Storage Account
![Page 8: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/8.jpg)
With client libraryAutomatically referenced in a cloud projectSupplies a nice .NET API on top of the REST API
You will need the following:Account Name - movieconversionAccount Shared Key – NOcqFlqUwJPLlm…Endpoint - blob.core.windows.net
-or-UseDevelopmentStorage=true
![Page 9: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/9.jpg)
?Table Storage
![Page 10: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/10.jpg)
Windows Azure TablesProvides Structured Storage
Massively Scalable TablesBillions of entities (rows) and TBs of dataCan use thousands of servers as traffic grows
Highly Available & DurableData is replicated several times
Familiar and Easy to use APIADO.NET Data Services – .NET 3.5 SP1
.NET classes and LINQREST – with any platform or language10
![Page 11: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/11.jpg)
TablesIs not relationalCan Not-
Create foreign key relationships between tables.Perform server side joins between tables.Create custom indexes on the tables.No server side Count(), for example.
ADO.NET Data ServiceCreates and consumes data via REST-based API'sThe Table Service REST API implements the WCF Data Services API.
![Page 12: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/12.jpg)
Table Structure
Account: MovieData
Star WarsStar TrekFan Boys
Table Name: Movies
Brian H. PrinceJason ArgonautBill Gates
Table Name: Customers
Account
Table
Entity Tables store entities.
![Page 13: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/13.jpg)
Table Storage ConceptsEntitiesTablesAccounts
moviesonline
Users
Movies
Email =…Name = …
Email =…Name = …
Genre =…Title = …
Genre =…Title = …
13
![Page 14: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/14.jpg)
Table Data ModelTable
A storage account can create many tablesTable name is scoped by accountSet of entities (i.e. rows)
EntitySet of properties (columns)Required properties
PartitionKey, RowKey and Timestamp
14
![Page 15: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/15.jpg)
Required Entity PropertiesPartitionKey & RowKey
Uniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
15
![Page 16: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/16.jpg)
Required Properties
All entities must have the following properties:
TimestampPartitionKeyRowKey
PartitionKey + RowKey = “primary key”
![Page 17: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/17.jpg)
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast & Furious … 2009
Action The Bourne Ultimatum
… 2007
… … … …
Animation
Open Season 2 … 2009
Animation
The Ant Bully … 2006
PartitionKey(Category)
RowKey(Title)
Timestamp
ReleaseDate
Comedy Office Space … 1999
… … … …
SciFi X-Men Origins: Wolverine
… 2009
… … … …
War Defiance … 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast & Furious … 2009
Action The Bourne Ultimatum
… 2007
… … … …
Animation
Open Season 2 … 2009
Animation
The Ant Bully … 2006
… … … …
Comedy Office Space … 1999
… … … …
SciFi X-Men Origins: Wolverine
… 2009
… … … …
War Defiance … 2008
Partitions and Partition Ranges
Server BTable = Movies
[Comedy- Western)
Server ATable = Movies[Action - Comedy)
18
Server ATable = Movies
![Page 18: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/18.jpg)
Table OperationsTable
CreateQueryDelete
EntitiesInsertUpdate
Merge – Partial UpdateReplace – Update entire entity
DeleteQueryEntity Group Transaction (new)
![Page 19: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/19.jpg)
Creating a Table
Tables can be created and destroyed in code.Wire up your storage in the on_start method.Use a CloudTableClient to access the table.
![Page 20: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/20.jpg)
Entity: Inherit the Required FieldsBy inheriting from TableServiceEntity we can easily store our entity in an Azure Table.
![Page 21: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/21.jpg)
Create a Context Class
WCF Data Services needs a context class. This represents your connection to the table.
![Page 22: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/22.jpg)
Adding an Entity
Other operations are just like normal ADO.NET Data Services,such as delete, update, etc.
![Page 23: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/23.jpg)
Deleting an Entity
![Page 24: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/24.jpg)
Key Selection: Things to Consider> Scalability
> Distribute load as much as possible> PartitionKey is critical for scalability> Hot partitions can be load balanced
> Query Efficiency & Speed> Avoid frequent large scans> Parallelize queries
25
![Page 25: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/25.jpg)
Movie Listing – Solution 1> Why do I need multiple PartitionKeys?
> Account name as Partition Key > Movie title as RowKey since movie names
need to be sorted> Category as a separate property
> Does this scale?
26
PartitionKey(Account name)
RowKey(Title)
Category …
moviesonline 12 Rounds Action …
moviesonline A Bug’s Life
Animation …
100,000,000 more rows
… … …
moviesonline Office Space
Comedy …
moviesonline Platoon War …
50,000,000 more rows
… … …
moviesonline WALL-E Animation …
![Page 26: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/26.jpg)
Server A
Movie Listing – Solution 1
> Single partition - Entire table served by one server
> All requests served by that single server> Does not scale
PartitionKey(Account name)
RowKey(Title)
Category …
moviesonline 12 Rounds Action …
moviesonline A Bug’s Life
Animation …
100,000,000 more rows
… … …
moviesonline Office Space
Comedy …
moviesonline Platoon War …
50,000,000 more rows
… … …
moviesonline WALL-E Animation …
ClientClient Request
Request
Request
Request
27
![Page 27: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/27.jpg)
Movie Listing – Solution 2
PartitionKey(Category)
RowKey (Title)
Action Fast & Furious
… 10000 more Action movies
Action The Bourne Ultimatum
… 100000 more Action & Animation movies
Animation Open Season 2
… 100000 more Animation movies
Animation The Ant Bully
Comedy Office Space
… 1000000 more Comedy & SciFi movies
SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War Defiance
> All movies partitioned by category> Allows system to load balance hot partitions> Load distributed> Better than single partition
ClientClient Request
Server A
Request
Request
Request
28
Server B
Request
Request
Request
Request
PartitionKey(Category)
RowKey (Title)
Action Fast & Furious
… 10000 more Action movies
Action The Bourne Ultimatum
… 100000 more Action & Animation movies
Animation Open Season 2
… 100000 more Animation movies
Animation The Ant Bully
Comedy Office Space
… 1000000 more Comedy & SciFi movies
SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War Defiance
![Page 28: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/28.jpg)
Key Selection: Case Study 2
> Log every transaction into a table for diagnostics> Scale Write Intensive Scenario> Logs can be retrieved for a given time
range
29
![Page 29: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/29.jpg)
Logging - Solution 1> Timestamp as Partition Key
> Looks like an obvious choice> It is not a single partition as time moves
forward > Append only> Requests to single partition range> Load balancing does not help> Server may throttle
PartitionKey(Timestamp)
Properties
2009-11-15 02:00:01
…
2009-11-15 02:00:11
…
100000 more rows …
2009-11-17 05:40:01
…
2009-11-17 05:40:01
…
80000 more rows …
2009-11-17 12:30:00
…
2009-11-17 12:30:01
…
ApplicationsClientReques
t
Server A
Request
2009-11-17 12:30:01
…
Request
2009-11-17 12:30:02
…
Request
2009-11-17 12:30:03
…
Server B
30
![Page 30: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/30.jpg)
Server A
Server B
PartitionKey(ID_Timestamp)
Properties
01_2009-10-12 05:10:00
…
… …
100000 more rows …
09_2009-11-15 12:31:00
…
… …
20000000 more rows …
10_2009-10-05 05:10:10
…
5000000 more rows …
… …
900000 more rows …
19_2009-11-17 12:20:02
…
ApplicationsClientReques
tReques
tReques
tReques
t
Logging Solution 2 - Distribute "Append Only”
> Prefix timestamp such that load is distributed> Id of the node logging> Hash into N buckets
> Write load is now distributed > Better throughput> To query logs in time range
> Parallelize it across prefix values
15_2009-11-17 12:30:01
…
09_2009-11-17 12:30:22
…
19_2009-11-17 12:30:10
…
01_2009-11-17 12:30:01
…
31
![Page 31: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/31.jpg)
?Table Storage Demo
demo
![Page 32: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/32.jpg)
?Overview Blob Storage
![Page 33: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/33.jpg)
What is a BLOBBinary Large OBject
![Page 34: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/34.jpg)
BLOBs in Azure & SizeEach BLOB can be up to 1TB in sizeYou can have as many containers and BLOBs as you want.Containers can be created/destroyed on the fly
Account: MovieConversion
- Job1.mpg- Job2.mpg- Header.png
Container: Originals
- Job 9.mpg- Index.docx- Job8.pdf
Container: Completed
http://<account>.blob.core.windows.net/<container>/<blobname>http://cohowinery.blob.core.windows.net/images/pic01.jpg
blobs
![Page 35: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/35.jpg)
Blob Storage Concepts
BlobContainerAccount
sally
images
PIC01.JPG
PIC02.JPG
movies MOV1.AVI
![Page 36: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/36.jpg)
Windows Azure Storage Services
Blobs – Provide a simple interface for storing named files along with metadata for the file
New Type of BlobPage Blob
New Blob OperationsUpdate BlobCopy BlobSnapshot BlobLease Blob
New Ways of Accessing and Serving Blob ContentCustom Domain NamesRoot Blob ContainerContent Delivery Network accessShared Access Signatures
Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)
New Features
![Page 37: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/37.jpg)
?Overview Queue Storage
![Page 38: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/38.jpg)
What is a queue?
Queues are used to store messagesThey are FIFO, and one wayA queue name must be lower case name, and URL friendly
MessageQueueAccount
order processing
customer ID order ID http://…
customer ID order ID http://…
cohowinery
![Page 39: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/39.jpg)
Account, Queues and MessagesAn account can create many queues
Queue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size <= 8 KBTo store larger data, store data in blob/entity storage, and the blob/entity name in the messageMessage now has dequeue count 40
![Page 40: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/40.jpg)
Azure Boot Camp
?Q
& A
Ask your questions.
?Q
& A
Ask your questions.
![Page 41: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/41.jpg)
What Next
Our next UG Meeting on 30th March 2011Microsoft Office Level 21, Singapore
![Page 42: Exploring azure cloud storage](https://reader033.vdocuments.site/reader033/viewer/2022052411/5562f671d8b42a6f598b4722/html5/thumbnails/42.jpg)
Storage Viewer / Explorer Tools
Azure Storage Explorerhttp://azurestorageexplorer.codeplex.com/Myazurestorage.comhttps://myazurestorage.com/Cloud Xplorerhttp://clumsyleaf.com/products/cloudxplorerCerebrata Cloud Storage Studiohttp://www.cerebrata.com/products/cloudstoragestudio/