cloud iaas storage managementcloudcomputing.ieee.org/images/files/education/study... · 2016. 3....

Cloud IaaS Storage Management

Cloud Strategy Partners, LLC

Sponsored by: IEEE Educational Activities and IEEE Cloud Computing

Course Presenter’s Biography

IEEE eLearning Library Cloud IaaS Storage Management Transcript pg. 2 / 24

This IEEE Cloud Computing tutorial has been developed by Cloud Strategy Partners, LLC. Cloud Strategy Partners, LLC is an expert consultancy firm that specializes in Technology and Strategy relating to Cloud Computing.

Course Summary


In this tutorial, we will review why Storage management is one the most demanded and responsible tasks in managing enterprise IT infrastructure. We will discuss the components of Storage management including operations such as backup, replication, and archiving that can be implemented locally, over network or in cloud. We will also review cloud storage as one of the infrastructure services benefits from the cloud based economy of scale. Finally, we will review cloud storage types including block, object, bucket and blob storage.

Transcript


Outline Our Outline for this lesson is

• Storage Management basics/services Backup, Replication, Archiving –Local, Network, Cloud

• Cloud storage scale out models and storage types Cloud storage scaling and consistency Ephemeral Block Storage Persistent Object/Blob Storage. Persistent Block Storage Cloud Storage Services Amazon storage services: EBS, S3, Glacier, and supporting services Microsoft Azure Storage

• Storage Management in OpenStack: Swift, Cinder • Summary and take away

Storage Management Services Storage Management is a very wide topic covering many aspects. Some of the most important areas in this large topic are: Backup Archiving Replication Mirroring Encryption and access control Data destruction. We will consider many of these in this Lesson.

Storage Management Terms: Backup, Replication, Archive First let’s consider the Storage Management Terms of Backup, Replication, and Archive. Each of these topics protects the data owner from losing their data in the event of a software, hardware, or even environmentally related issue. Note the different definitions of Backup, Archive, and Disaster Recovery. Having processes for each of these is important in protecting your data. For example, you might have a backup of your data, but in the event of a Disaster, you no longer have the computer or storage systems themselves to utilize the data on. So a coordinated strategy using each of these concepts is important in protecting ones data. In developing such a storage management strategy, one needs to develop requirements and objectives for that in quantitative terms. Here you will see several of these terms and their meanings defined. RPO is the Recovery Point Objective RTO is the Repair Time Objective MTBF is the Mean Time Between Failures MTTR is the Mean Time To Repair PIT is the Target Point-in-Time replica and continuous replica

Transcript


On the subject of Replica, one should also consider the Consistency of the Replica, making sure that intermediate files (such as files staged on disks or in cache) either are –or are not –part of the replica. Defining these requirements will be key in guiding one to designing solutions and processes for storage management.

Backup and Archive - Differences Backup and Archive have different purposes: recovery versus long-term preservation and retention. This is related to – but orthogonal –to Disaster Recovery. A Disaster Recovery strategy might include an active “Mirror” system or active “Mirror site” which stays synchronized with the Operational Site –and is ready to snap into action. It might include configured but “standby” systems ready to accept a backup from the original site, in the event that site stops functioning. Backups are Stores/copies all data: active and inactive data in their original structure. Normal systems operation and data continue to be used. Backup applications are optimized for large scale recovery: written in large objects to dedicated hardware; include also off-site copy Archives are: Archive stores inactive or older data for extended period of time. Archive media are optimized for long term storage. Archive typically stores and provides access to individual objects or files such as files, email, databases together with metadata.

Backup Methods and Backup Targets How Backups are Made: Hot backup is made while applications are up and running, and users accessing their data. Cold backup (also called offline backup) requires applications to be shutdown during the backup process. Backups go to a variety of media:

Transcript


Tape is a traditional and still used backup media Disks are replacing tapes due to decreased cost and increased volume Cloud backup is widely offered by large CSP and smaller specialized cloud backup providers (due to provider side economy of scale)

Generic Backup Architecture All Backup shares a particular architecture The data originates in the application running in the application server. Alternatively, there may be a dedicated part of the application cluster or a special client which connects to the application cluster or server, whose job it is do perform the backup. In any case, the data comes from the application. While Tracking Information goes to the Backup Server, the Backup Data is persisted on a storage node and then to the backup device. We see several common Backup topology options Direct attached LAN based SAN based backup Cloud based backup

Backup and Restore Operations - Backup This slide illustrates the Backup and Restore Operations for a Backup operation Follow the steps of the Backup Server, the Storage Node, the Backup Device the Backup Client The Storage Node, and the Storage Device Note how the Backup Catalog is used

Backup and Restore Operations - Restore This same topology slide illustrates the Backup and Restore Operations for a Restore operation Follow the steps of the Backup Server, the Storage Node, the Backup Device

Transcript


the Backup Client The Storage Node, and the Storage Device Note how the Backup Catalog is used

Data Deduplication for Backup Data Duplication is a real problem for backup. While changing data may represent a large quantity of storage, typically enterprises, labs, and other IT shops have even more data which is largely static, or might only change occasionally, Repeated Backups will store this so-called Duplicated Data every time a backup is done, over and over, thereby taking up backup resources (media and capacity) and also time (and labor). So solving the Data deduplication problem is quite important. Data Deduplication is the process of identifying and eliminating redundant data Data deduplication methods which have arisen include: • File level deduplication (also called single instance storage) detects and removes redundant copies of identical files • Subfile deduplication detects redundant parts of files As explained in the slide, each of these has positive and negative characteristics

Data Archiving Solutions Architecture Now we will look at Data Archiving Solutions Organisations often require retention of their data for the purposes of business processes provenance, historical value, according to law requirements Email archiving Activity log, business activity File archiving There are many Cloud based solutions Cloud storage as archiving media Full cloud and hybrid cloud/on-premises numerous cloud archiving applications

Transcript


Backup Best Practices There are some important Backup Best Practices One well known rule is the 3-2-1 rule: 3 copies, 2 media, 1 offsite 3 copies are made 2 copies guards against media failure. Its best to have these off-site geographically distributed. 1 is kept on-site for convenient use. This leaves at least one off-site A Typical configuration includes Production storage, primarily backup (on-site), secondary backup (off-site or in cloud) with a WAN accelerator Cloud backup and archiving Uses Storage as a Service vs Backup as a Service One Reminder is to Special attention to remote office backup solutions.

Storage Replication Storage Replication is a key technology used most often as a part of a Disaster Recovery implementation Storage replication service is a managed service in which stored or archived data is duplicated in real time, usually over a Storage Area Network (SAN) Storage replication service provides an extra measure of redundancy to protect data and improve storage/data availability in case the main storage backup system fails There are several Uses for Replicas • Alternative source for backup • Fast recovery • VM migration (hypervisor based: VM snapshot and VM cloning) • Testing platform • Data migration Immediate access to the replicated data minimizes downtime and its associated costs The service, if properly implemented, can streamline disaster recovery processes by generating duplicate copies of all backed-up files on a continuous basis. It can also speed up and simplify recovery from a natural or human-caused disaster such as a fire, flood, hurricane, virus, or worm.

Transcript


Local Replication Technologies Replication can be Local, or Remote. First Local Replication Technologies will be considered. Host based usually use a capability called Local Volume Manager or LVM. LVM based replication replicates one logical volume into two different physical volumes The Benefit of LVM and Local Replication is simplicity LVM has Limitations: it causes additional load for CPU and that replica can only be accessed from the original host Another approach is to use the Copy on First Write (CoFW) principle which creates snapshot – called “snap” for short -of original filesystem gradually as data are written to the original filesystem. In Storage array based replication the Storage array environment does data replication

Remote Replication Technologies We looked at Local Data Replication, now to look at Remote Replication Technologies There are two types of Remote Replication Technologies Synchronous and Asynchronous remote replication In Synchronous, the Write operation includes both writing to the source and copying to the target Either these writes occur Synchronously or Asynchronously As to Topology, there are several options Host based remote replication Storage array based remote replication Network based remote application Three site Replication (cascade/multihop). These are illustrated in the Slide

Continuous Data Protection (CDP) One special type of continuous backup or real-time backup is known as Continuous Data Protection (CDP) As the Illustration on the Slide shows, there is typically a CDP Appliance All the writes to the production volume (source) are also sent to the CDP appliance This is a convenient way to implement a real-time Backup CDP is tricky, for the bandwidth and performance concerns detailed in the Slide.

Virtual Machines (VM) Replication In an environment which ubiquitously uses Virtual Machine (VM) capabilities, such as a Private Cloud, a special type of Backup combined with Disaster Recovery has emerged That is Virtual Machines (VM) Replication

Transcript


Here techniques use VM snapshot and VM cloning In conjunction with Data Replication this allows one to almost instantly re-start a running application in a new place It allows what is called Live migration and VM images replication Hypervisor-to-hypervisor VM migration And Array-to-array migration sequence These scenarios are illustrated in the Slide

Cloud Storage: Scalability, Replication, Consistency Now let us consider directly Cloud working storage. As clouds get larger, some of the main challenges in implementing Cloud Storage is Scalability, Replication, and Consistency In order to understand these issues, one must consider several aspects of Cloud Architecture which are completely different from the usual IT considerations First one must understand the Replication and Scale Concepts in Cloud Next Software vs Hardware Strategies to achieve scale must be examined Scalability and Replication Requirements must be studied Finally we explore High Availability (HA) and Consistency expectations in Clouds Then we can understand how Cloud Storage is implemented, and more importantly why those approaches are taken

Economy of Scale, For Storage and In General One of the overriding considerations of implementing cloud, and Cloud Storage in particular, is that Clouds are constructed with scale in mind, not only scale, but economy of scale Clouds concentrate lots of resources into as dense as possible a physical setup, to get the highest efficiencies and utilization of shared resources. This includes not only IT resources but Physical Plant Efficiencies in Building, Power, and Cooling Also, several different network connections are brought into the same facility, providing a huge Network Variety to the subscribers of the cloud In constructing the cloud (and building up Cloud Storage), some important guiding architectural principles have been used. One of these is to utilize Granular (Small) Building Blocks. This way Failure has limited “Blast Radius”. Eg, when one uses very large servers or storage systems, a failure takes out a lot of capacity. A smaller system failing, takes with it a smaller capacity,

Transcript


Similarly, smaller building blocks ae easier to arrange and re-arrange, and to add, to respond to use patterns. Smaller building blocks Enable Efficient Use of Resources, because you don’t have large capacities which go unused Finally, Automation in clouds is pervasive. The more automated the more efficient a Cloud is. Automation is important because: People to Machines Ratio Also cannot Scale Geometric Automation Becomes a Requirement To Reduce Labor Requirement To Provision Services To Handle Failures Finally, large purchasing gives on large purchasing power, from power and network to IT equipment,

Simplicity Scales – for Storage, for Everything! Another important design blueprint in constructing clouds and cloud storage, is that “Simplicity Scales”. Generally speaking complex small systems because geometrically more complex if one tries to make them large systems. When one tries to make them massive systems they become completely intractable. Consider some complex architectural design patterns which are very tempting to utilize in Cloud. One idea which often arises is to try to utilize distributed state, to keep different parts of the cloud aligned perfectly with other parts. This applies to storage designs in particular. We all know that distributed state is complicated, to manage, it is even more complicated to manage across massive scale systems. Large scale systems have exhibit problem scenarios which sometimes are the result of complex interactions of different parts of the large system and are impossible to replicate (to fix them). Finally, large systems introduce unique issues which add to the difficulty of making things work at scale. For example the sheer physical size of a cloud, measuring the farthest different from one “side” of the cloud to the other, might be many meters. Some designs call for clouds which are either split, or interworking, across WAN links, where they may be hundreds of kilometers apart. In these systems we see problems caused by Speed of Light over large areas –Extremely subtle problems in Time Synchronization and Network Latencies, or problems in Harmonics, with tables and data structures Bouncing, Flapping, Amplifying –it is Impossible to predict system feedback/instability problems.

Transcript


So many adopt a philosophy of using Small Loosely Coupled Building Blocks to Avoid this type of behavior. Instead, we Use Tried and True “Legos” as components. We Avoid Distributed State totally. And we Scale “like the Internet” –using pessimistic connective algorithms (with retry to compensate), using local traffic techniques like routing between switched LAN segments instead of trying to make giant switched LAN segments do everything.

Software vs. Hardware Strategies for Scaled, Distributed Storage Typical IT scenarios have utilized specialized hardware and dedicated systems to implement scaled, distributed storage. These have worked well in the traditional datacenter, but not as well for cloud. Consider RAID for storage – it Does not lend itself to Distribution across independent data centers It Works as long as Host Server does not fail Consider SAN or NAS for Storage -Configuring SAN or NAS servers that can replicate and be redundant across independent data centers solves one concern Now we have introduced an additional point of failure (the SAN or NAS servers themselves) which must in themselves be made redundant and fail-over by automation In general, the Challenge with Hardware Approach is it attempts to be 100% transparent to Applications What if this wasn’t a strict requirement?

Cloud Storage Use Cases and Requirements Now to understand what types of storage models are actually needed by Cloud. To start, an Operating System must be able to boot and run on a VM. This implies that some storage must be available to that OS, and it will expect a root with strict POSIX filesystem behavior to boot along with the OS. This Data is LOCAL to VM; it is called Ephemeral Instance Storage. This Data is “Ephemeral” because it is temporary to that instantiation (that boot duration). This type of storage is Easy to Implement and Easy to Use as it looks like a regular local disk as far as the booted OS is concerned Continuing, that OS wants to utilize storage which is going to be persistent and stick around, between OS boots, and can be used by several different applications running on the cloud at once. So there are all kinds of additional storage types that are of interest to applications.

Transcript


First will be additional mounted drives, or block storage capability. Next will be Object storage, which look like raw drive partitions, perfect for letting applications lay down structure on top of. Finally will be databases of all kinds. What is common is that all of these data types need to be persistent, that is, hold the data in a durable fashion accessible to many different OS’s and Applications running on the cloud. While less important to the application itself, but inherent to the fact that these are running on a Cloud, the actual implementation of each of these models is that they need to be distributed. Utilizing one storage location does not provide the durability need nor the scalability. What then is compromised, depending on the storage model, is the consistency of the various distributed storage elements. The precise details of the consistency depend on the on the implementation of each back end

Scale-out and High Availability (HA) Drives Need to use Replication and Variable Consistency It is then easy to understand that one must not trade-off the storage reliability (durability), and we do not want to trade off scale-out and high availability, so we then can achieve these goals through ‘tricks and compromises’ with Replication and Consistency Replication Fulfils HA Requirements For Availability: store everything in multiple places; use caches/CDN for geographical distribution For Reliability: utilize automation driven, anti-corruption processes What emerges are Difficulties in Implementation: we find challenges in distributed atomic operations, timing, network latencies: “Inconsistent” data behavior emerges We must Understand and Use Consistency Variability to Enable more use of Replication and Scale-out

Examples of Cloud Consistency Effects Let us understand Examples of Cloud Consistency Effects in Storage Microsoft’s Azure, provide only strongly consistent storage services to their applications. They are unable to provide these interfaces across geographically dispersed endpoints

Transcript


Amazon Simple Storage Service (S3), offer only weak consistency but geographically dispersed availability. Amazon’s SimpleDB, for example, provides both eventually consistent reads and consistent reads, with the latter experiencing a higher read latency and reduction in read throughput So each service, as they have been made to work, indeed provides for behaviors which the application developer needs to be aware of.

Formalizing Storage Consistency to Understand the Cloud Offerings Now we should move towards Formalizing Storage Consistency to Understand the Cloud Offerings As mentioned, what is quite different is that application developers need to understand details of consistency behavior for the specific storage capability they are using, in order to make their applications work correctly at scale A wide variety of consistency models are available in Cloud Offerings. We Categorize them into 6 Consistency Guarantees.These consistency guarantees are based on a simple model in which clients perform read and write operations to a data store. The data is replicated among a set of servers, but the details of the replication protocol are hidden from clients.

Six Consistency Guarantees Strong consistency is particularly easy to understand. It guarantees that a read operation returns the value that was last written for a given object. If write operations can modify or extend portions of a data object, such as appending data to a log, then the read returns the result of applying all writes to that object. In other words, a read observes the effects of all previously completed writes. Eventual consistency is the weakest of the guarantees, meaning that it allows the greatest set of possible return values. For whole-object writes, an eventually consistent read can return any value for a data object that was written in the past. More generally, such a read can return results from a replica that has received an arbitrary subset of the writes to the data object being read. By requesting a consistent prefix, a reader is guaranteed to observe an ordered sequence of writes starting with the first write to a data object. For example, the read may be answered by

Transcript


a replica that receives writes in order from a master replica but has not yet received an unbounded number of recent writes. In other words, the reader sees a version of the data store that existed at the master at some time in the past. This is similar to the “snapshot isolation” consistency offered by many database management systems. Bounded staleness ensures that read results are not too out-of-date. Typically, staleness is defined by a time period T, say 5 minutes. The storage system guarantees that a read operation will return any values written more than T minutes ago or more recently written values. Alternative, some systems have defined staleness in terms of the number of missing writes or even the amount of inaccuracy in a data value. I find that time-bounded staleness is the most natural concept for application developers. Monotonic Reads is a property that applies to a sequence of read operations that are performed by a given storage system client. As such, it is often called a “session guarantee.” With monotonic reads, a client can read arbitrarily stale data, as with eventual consistency, but is guaranteed to observe a data store that is increasingly up-to-date over time. In particular, if the client issues a read operation and then later issues another read to the same object(s), the second read will return the same value(s) or the results of later writes. Read My Writes is a property that also applies to a sequence of operations performed by a single client. It guarantees that the effects of all writes that were performed by the client are visible to the client’s subsequent reads. If a client writes a new value for a data object and then reads this object, the read will return the value that was last written by the client (or some other value that was later written by a different client). (Note: In other papers, this has been called “Read Your Writes,” but I have chosen to rename it to more accurately describe the guarantee from the client’s viewpoint.) These last four read guarantees are all a form of eventual consistency but stronger than the eventual consistency model that is typically provided in systems like Amazon. None of these four guarantees is stronger than any of the others, meaning that each might result in a read operation returning a different value. In some cases, as will be shown later, applications may want to request multiple of these guarantees. For example, a client could request monotonic reads and read my writes so that it observes a data store that is consistent with its own actions.

Variable Consistency is a Main Reason Why Re-Platforming Server to Cloud Applications is Tricky One can draw four main conclusions from this exercise:

Transcript


All of the six presented consistency guarantees are useful. Observe that each guarantee appears at least once in Table 4. Systems that offer only eventual consistency would fail to meet the needs of all but one of these clients, and systems that offer only strong consistency may underperform in all but two cases. Different clients may want different consistencies even when accessing the same data. Often, systems bind a specific consistency to a particular data set or class of data. For example, it is generally assumed that bank data must be strongly consistent while shopping cart data needs only eventually consistency. Even simple databases may have diverse users with different consistency needs. Clients should be able to choose their desired consistency. The system cannot possibly predict or determine the consistency that is required by a given application or client. The preferred consistency often depends on how the data is being used. Moreover, knowledge of who writes data or when data was last written can sometimes allow clients to perform a relaxed consistency read, and obtain the associated benefits, while reading up-to-date data. What about the cost of eventual consistency? The main argument often expressed against providing eventual consistency is that it increases the burden on application developers. This may be true, but the extra burden need not be excessive. The first step is to define consistency guarantees that developers can understand; observe that the six guarantees presented in Table 1 are each described in a few words. By having the storage system perform write operations in a strict order, application developers can avoid the complication of dealing with update conflicts from concurrent writes. This leaves developers with the job of choosing their desired read consistency. This choice requires a deep understanding of the semantics of their application, but need not alter the basic structure of the program.

Models for Storage in Cloud Computing Cloud Computing includes several storage models. Some of these come out of necessity, for running existing software. There are also a large number of storage models which have emerged on Cloud which are totally new, becoming possible because of the architecture of Cloud, or becoming needed (where there was not a need before) because of the unprecedently large data volumes found on Cloud, which simple didn’t exist before. The first type of storage comes from the need for Root and System Drives for a VM. These are File systems mounted on Block Storage. These need to be delivered along with a created VM and conversely don’t really need to exist after a VM disappears. Therefore this type of storage is called Ephemeral Block Storage (meaning it has the same lifecycle as the VM). The second type is Additional Block Storage on which to mount File systems, not Ephemeral (Persistent). This block storage will stick around in the cloud independent of any VMs. Presumably VMs, once created, will mount these drives and have access to them that way. The structure of block storage must be maintained by the actual storage implementation. In

Transcript


general block storage is implemented by simply attaching to existing SAN or NAS storage devices which always provide a block interface. Or it could be implemented in software by the CloudOS as we will see below. The next type of storage really emerged on the cloud following the model of a tape archive. In the legacy use of tapes, one could have multiple “archives”, one after the other, on the tape. The archive had no structure to it, it was a sequence of bits. One could directly place a disk image onto a tape archive for example. Or a “tar” file (which compares to a modern day ZIP file in some ways) could be placed onto a tape archive (in fact, tar stands for Tape ARchive). Tape archives were “buckets of bits” on a sequential magnetic media. On the cloud, we have this same notion which we call object, or BLOB, or bucket storage. Object storage is implemented on the cloud in clever ways to provide for replication of data for high availability. It is often the lowest cost storage option on a cloud. Finally, many applications need highly structured storage, like a database, for their applications. While many applications use RDBMs systems with very complicated SQL queries, many application use very simple databases, without complex joins or stored procedures, they need some simple way to do column or table based lookup. As we will see later, implementing databases on a distributed architecture yields some specific challenges and to answer these challenges using different trade-offs, there are many different kinds of database and database-like choices.

VM’s and Ephemeral Block Storage The first cloud storage model is Ephemeral Block Storage. When one requests a VM on a Cloud, it comes with one or more drives which the OS can mount “/” and “/usr” or “C:” and “D:” for example These are initialized as configured by the Boot Image They live as long as the VM does When the VM dies, the storage dies too (hence the word “ephemeral”) Sometimes this is called “instance store”.

Ephemeral Block Storage Implementation Now let us take a look at Ephemeral Block Storage Implementation The first and most cost-effective/common technique is to use the existing Compute server w/local drives (common option, AWS uses). Local Drives are often organized in a separate enclosure, as Just a Bunch of Disks (JBOD), or sometimes in a RAID configuration This has some key attributes: Low Latency Low Cost Longer Live Migration No guest H/A Managed by Cloud OS

Transcript


Shared storage NAS or SAN is another option. NAS or SAN drives are usually packaged as a proprietary network attached server with networking, storage, and processing modules This has some different key attributes: Higher Cost, More features Thin provisioning Faster Migration Standby H/A Not managed by Cloud OS

AWS Ephemeral Block Storage Looking more deeply into AWS Ephemeral Block Storage AWS Ephemeral Block Storage provides an OS with everything it needs to boot and run a Root file system The size of Ephemeral Block Storage (called “instance store” in AWS) ranges from 900 MB to up to 48 TB, and varies by instance type. Larger instance types have larger instance stores. When you launch an instance store-backed AMI, each instance store volume available to the instance is automatically mapped. Swap space in Linux can be used when a system requires more memory than it has been physically allocated. Amazon Linux AMIs automatically enable and use swap space from the instance store

Persistent Block Store Now looking more deeply at Persistent Block Storage Cloud Block Storage is a block level storage solution that allows one to mount drives or volumes to the OS on their VM Mount a drive to a Cloud Server to scale storage without paying for more compute capability. A high performance option for databases and high performance applications, usually leveraging solid state drives (SSDs) for speed. Some clouds have A standard speed option using SATA drive for customers who just need additional storage on their Cloud Server. Usually it is Lower cost per GB for standard performance of additional storage, and Higher cost per GB for high performance of additional storage. Interactions with Cloud Block Storage occur programmatically via the Cloud Block Storage API usually via a RESTful (Representational State Transfer) web services interface.

Transcript


AWS Persistent Block Store - EBS Let us look at Amazons implementation of Block Storage Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances in the AWS Cloud. Each Amazon EBS volume is automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability. Amazon EBS provides three volume types: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic.

Block Storage in OpenStack Cinder is a Block Storage service for OpenStack. It's designed to allow the use of either a reference implementation (LVM) to present storage resources to end users that can be consumed by the OpenStack Compute Project (Nova). The short description of Cinder is that it virtualizes pools of block storage devices and provides end users with a self-service API to request and consume those resources wit Architected as the application storage for performance sensitive workloads, Cinder is the project name for the block storage service within OpenStack. Different than the Swift object storage service, Cinder presents persistent block level storage devices for use with OpenStack compute instances. The block storage system manages the creation, attaching and detaching of the block devices to servers. Block storage volumes are fully integrated into OpenStack Compute and the Dashboard allowing cloud users to manage their own storage needs without requiring any knowledge of where their storage is actually deployed or on what type of device.

Persistent Object or Blob Store Persistent Object or Blob Store is different from block or file storage. Again, think about it like what we use to call a “file” or “archive” on a tape. This is a stream of bit between archive markers. You can stream from the tape or to the tape, you can’t seek around in within an archive. Access to Object Storage is via API to a container (or “bucket”) at application-level, rather than via OS at filesystem-level

Transcript


Byte-level interaction is not possible, entire objects are stored or retrieved with a single command interaction can only occur via a single API endpoint Filesystem level utilities (e.g. POSIX utilities) cannot interact directly with Object Storage Object Storage is one (or potentially few in the case of multi-region deployments) giant volume, Metadata typically lives with the object Your application would use something like name-value pairs for the Metadata with the Object In object storage, there is no structure, no directory tree. It Uses a flat structure, storing objects in containers, rather than a nested tree structure Durability levels at scale are extremely high because usually 3 file replicas are made The simplicity of the requirements lends for extremely scalable implementations with low cost drives and pure software implementation, keeping costs low

Object Storage in AWS Let’s examine the characteristics of the Object Storage implementation in Amazon AWS. S3 is the name of Amazon’s Object Store System; a bucket is a container for objects stored in S3. The object named photos/puppy.jpg is stored in the johnsmith bucket, then it is addressable via http://johnsmith.s3.amazonaws.com/photos/puppy.jpg Objects consist of object data and metadata. The data portion is opaque to S3 The metadata is a set of name-value pairs that describe the object. These include some default metadata, such as the date last modified, and standard HTTP metadata, such as Content-Type Keys are the unique identifier for an object within a bucket. Every object in a bucket has exactly one key. Think of S3 as a basic data map between "bucket + key + version" and the object itself With object storage, one has to know what they are doing! Different Regions of AWS behave differently with Object Storage, as the slide details.

Object Storage in Windows Azure Windows Azure also stores binary data -blobs -in containers called Blob Storage Windows Azure provides two different kinds of blobs Block blobs, each of which can contain up to 200 gigabytes of data. As its name suggests, a block blob is subdivided into some number of blocks. If a failure occurs while transferring a block blob, retransmission can resume with the most recent block rather than sending the entire blob again. Block blobs are

Transcript


a quite general approach to storage, and they're the most commonly used blob type today. Page blobs, which can be as large at one terabyte each. Page blobs are designed for random access, and so each one is divided into some number of pages. An application is free to read and write individual pages at random in the blob. In Windows Azure Virtual Machines, for example, VMs you create use page blobs as persistent storage for both OS disks and data disks. Applications can access blob data in several different ways Directly through a RESTful (i.e., HTTP-based) access protocol. Both Windows Azure applications and external applications, including apps running on premises, can use this option. Using the Windows Azure Storage Client library, which provides a more developer-friendly interface on top of the raw RESTful blob access protocol. Once again, both Windows Azure applications and external applications can access blobs using this library. Using Windows Azure drives, an option that lets a Windows Azure application treat a page blob as a local drive with an NTFS file system. To the application, the page blob looks like an ordinary Windows file system accessed using standard file I/O. In fact, reads and writes are sent to the underlying page blob that implements the Windows Azure Drive

Object Storage in Windows Azure The OpenStack Object Store project, known as Swift, offers cloud storage software to store and retrieve lots of data with a simple API. It's built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound. Swift provides redundant, scalable object storage using clusters of standardized servers capable of storing petabytes of data Objects and files are written to multiple disk drives spread throughout servers in the data center, with the OpenStack software responsible for ensuring data replication and integrity across the cluster. Swift has many features for both end users and sysadmins running the system. Versioned writes CORS ACLs Arbitrarily large objects Static website hosting Signed, expiring URLs Custom metadata Bulk operations Multi-range requests

Transcript


Data Storage Supporting Services in AWS Amazon has a wide portfolio of Data Storage service beyond the basic object and block capabilities. Data storage is part of the Data Lifecycle Management workflow that include the following stages Ingest – Filtering/Formatting –Storage –Transform –Archive Supporting AWS storage services following this workflow are: Amazon Import/Export Amazon Storage Gateway Service Amazon Glacier long term storage Amazon DirectConnect Amazon CloudFront

AWS Import/Export To look deeper at AWS Import/Export It is for getting your large data into the cloud! AWS Import/Export accelerates moving large amounts of data into and out of the AWS cloud using portable storage devices for transport. AWS Import/Export transfers your data directly onto and off of storage devices using Amazon’s high-speed internal network and bypassing the Internet. For significant data sets, AWS Import/Export is often faster than Internet transfer and more cost effective than upgrading your connectivity. Common Use Cases for AWS Import/Export Data Cloud Migration Content Distribution Direct Data Interchange Offsite Backup Disaster Recovery The slide details specifications and options for AWS Import/Export.

Amazon Storage Gateway The next Amazon Data service is the AWS Storage Gateway. This is a service connecting an on-premises software appliance with cloud-based storage to provide seamless and secure integration between an organization’s on-premises IT environment and AWS’s storage infrastructure. Please see the illustration in the Slide. AWS Storage Gateway supports industry-standard storage protocols

Transcript


It Provides low-latency performance by maintaining frequently accessed data on-premises while securely storing all of your data encrypted in Amazon Simple Storage Service (Amazon S3) or Amazon Glacier. AWS Storage Gateway supports three configurations: Gateway-Cached Volumes: store primary data in Amazon S3, and retain your frequently accessed data locally. Gateway-Stored Volumes: to ensure low-latency access to your entire data set, you can configure your on-premises data gateway to store your primary data locally, and asynchronously back up point-in-time snapshots of this data to Amazon S3. Gateway-Virtual Tape Library (Gateway-VTL): To support legacy backup applications.

Amazon Glacier The next Amazon feature to consider is Amazon Glacier. It is a low-cost cloud archive storage service that provides secure and durable storage for data archiving and online backup. Multi-tier architecture: Amazon Glacier is optimized for data that is infrequently accessed and for which retrieval times of several hours are suitable. It Costs as little as $0.01 per gigabyte per month, a significant savings compared to on-premises solutions, and no payment upfront Features include Secure Transfer over SSL and data encryption at rest with AES256 symmetric key, access control using AWS Identity and Access Management (IAM) Durable Amazon Glacier is designed to provide average annual durability of 99.999999999% for an archive Flexible to scale to needs

Storage Alternatives: Gluster (RedHat) Gluster is a scale-out network-attached storage file system. It has found applications including cloud computing, streaming media services, and content delivery networks. GlusterFS was developed originally by Gluster, Inc., then by Red Hat, Inc., after their purchase of Gluster in 2011. Think of Gluster as an alternative to Swift and Cinder in OpenStack. GlusterFS aggregates various storage servers over Ethernet or Infiniband interconnect into one large parallel network file system. Gluster stores data as files and folders, and uses tokens to identify the location of a file within the cluster. Tokens, which are stored as extended attributes of a file, are themselves distributed across directories thereby enhancing load balancing while avoiding the need for a

Transcript


dedicated metadata server. When a client accesses a file, Gluster translates the requested file name to a token and access the files directly. Gluster Filesystem enables you to configure the cluster to replicate files across storage devices, thereby high availability to files and data in your storage environment.

Storage Alternatives: Ceph (DreamHost) Ceph Object Storage and/or Ceph Block Device services are modules to be integrated into Cloud Platforms, deploy a Ceph Filesystem or use Ceph for another purpose, all Ceph Storage Cluster deployments begin with setting up each Ceph Node, your network and the Ceph Storage Cluster. A Ceph Storage Cluster requires at least one Ceph Monitor and at least two Ceph OSD Daemons. The Ceph Metadata Server is essential when running Ceph Filesystem clients. Ceph stores a client’s data as objects within storage pools. Using the CRUSH algorithm, Ceph calculates which placement group should contain the object, and further calculates which Ceph OSD Daemon should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and recover dynamically.

Summary and Take Away Storage management is one the most demanded and responsible task in managing enterprise IT infrastructure It includes such operations as Backup, Replication, Archiving that can be implemented locally, over network or in cloud Cloud storage as one of infrastructure services benefits from the cloud based economy of scale Cloud storage types include: block, object, bucket or blob storage The major cloud IaaS providers Amazon Web Services and Microsoft Azure provide variety of storage services OpenStack as one of popular cloud platforms supports block storage, object storage and also special storage for VM images Alternative storage platforms such as Gluster or Ceph optimized for different purposes can be implemented both on the devoted cloud IaaS platform and on the top of Amazon of Azure block storage.

cloud iaas storage managementcloudcomputing.ieee.org/images/files/education/study... · 2016. 3....

Documents