the challenge of cloud storage - iomart

The Challenge of Cloud Storage

“Wouldn’t it make that headache a little less painful by pushing towards lower spend pay-as-you-use models? Perhaps, and it’s here that the power of Cloud Storage is really coming to the fore.”

IntroductionData is big and getting bigger. According to Gartner we are doubling the amount of data we produce every couple of years, in the past two years alone an array of products and services have been produced which take that data, analyse it and create new products and services as a result. What you buy in Tesco; how fast you drive your car; how often the trains are delayed; how often you click ‘Like’ etc. -all this data is being tracked, recorded and ultimately held on a storage platform somewhere.

The growth in data is unprecedented and exponential. It is having an increasing impact on how we all conduct our business and is leading to larger and more complex storage requirements. Developers no longer even consider storage as being a key challenge or factor when designing applications. There was a time when the volume of storage that a product consumed would be critical – remember reading the “Recommended System Specifications?” Now storage and compute is just presumed capable by many in house development teams. Don’t have enough storage on your local device? No problem, we’ll store it in the cloud instead.

Whilst consumers and even developers are giving less heed to storage, the explosion in utilisation is giving cause to pause for many Infrastructure architects. While this is a headache in terms of the numbers – Petabytes (1000⁵) are ten to the penny now so the world is talking about Zettabytes (1000⁷) and even Yottabytes (1000⁸) - it is becoming an even bigger headache in terms of what we do with our data. Managing an organisation’s IT systems has never been simple but today it is more complicated than ever and

so we need to look externally for systems, processes and third parties who can help. Many believe it is now no longer cost effective to invest hundreds of thousands of pounds in solving this problem locally. After all – hardware moves quickly and a three to six year cycle on equipment can be a painful and expensive Cap-Ex decision to make. Wouldn’t it make that headache a little less painful by pushing towards lower spend pay-as-you-use models? Perhaps, and it’s here that the power of Cloud Storage is really coming to the fore.

The concept of cloud hosting is now moving into a phase of more general adoption even by enterprise organisations for their websites, software and their applications so you might not be surprised by the number of organisations now choosing to store their data in ‘the Cloud’. Microsoft is pushing it in its consumer television advertising – and we’ve even seem the often sombre business orientated marketing department of IBM delivering key messages about cloud adoption during the ad breaks of Coronation Street.

The names involved lend their own weight to the growing message that Cloud is here to stay and Cloud Storage falls right into that category. However - storage isn’t something to take lightly – it’s often underestimated just how mission critical it can be. Poorly managed cloud storage can put your entire business at risk and it is critical that you are making the right choices when it comes to the physical equipment running it, or indeed the vendor you have chosen to run it for you. Robust storage requires a highly reliable cloud provider and considerable technology and business planning.

For further details: visit www.iomart.com or email us at: [email protected] or call: +44 (0)800 040 7228

“Cloud storage uses highly virtualised infrastructure and is generally done by a hosting company. You can then retrieve that data when you need it. ”

Cloud Storage – what is it really?Let’s not forget though that Cloud Storage means different things to different people – Dropbox and OneDrive are technically Cloud Storage – but you wouldn’t dream of putting large scale database operations in their clouds. However, places do exist where you would put that sort of data. It’s a case of horses for courses - but for the sake of simplicity for this paper, we’ll stick to defining cloud storage as the process whereby you keep your data in a safe place off-premise.

The concept was largely pioneered by Amazon who led the way with storing data on virtual machines. The AWS method still sees you sending data off to Amazon and then using your own computers to communicate with those machines and access the information you have entrusted to them to protect.

Cloud storage uses highly virtualised infrastructure and is generally done by a hosting company. You can then retrieve that data when you need it.

People expect data to be available everywhere – some of the data you store will be data you need access to every day. Other data stored might not be accessed more than once every few months or even once every year but has to be kept for legal reasons, for example. Confused? It wouldn’t be surprising if you were – the choice and range of options is at times baffling. Fast or Slow, iSCSI or Fibre, Block, Object or File? Solid State or Spinning Disk? What about de-duplication and compression? We’ll try to highlight some of these questions and what you should be looking out for.

Is your data hot or cold?Data in any location whether it’s your laptop, a spare hard drive or a Virtual Machine for your finance systems can be hot or cold Even that doesn’t simplify it enough. You generally won’t have specific hard drives which are hot. We’re talking right down to individual blocks on the disk. Let’s look at an example:

You power on your laptop every day. Around 10% of the standard laptop is taken up by the Operating

System. Of this 10% around 30-40% of the files and folders are utilised during your laptop’s startup routines every single time. These files which are used repeatedly over and over again are deemed as hot data. Your computer can’t live without using them.

On the flip side of that, we have the 40 spreadsheets you used whilst generating annual reports for 2012. You haven’t looked at them and are not likely to look at them again for some time. This, is cold data.

So what? Isn’t it all just one big drive?Technically yes it is. However storage technologies are advancing rapidly allowing us to carve up these drives at the disk layer to better optimise our experiences. Let’s revert back to our earlier example – laptop vendors like Dell, HP and Lenovo are becoming increasingly conscious that storage makes the single biggest difference to the experience of a laptop user. They could, theoretically, ship every laptop with a dedicated Solid State Drive but these are still relatively expensive for models above 256GB. Plus most users want to store hundreds of gigabytes of their home movies, photographs and music. It’s not cost effective.So along comes the Hybrid Hard Drive. A clever mix of traditional spinning disk and solid state memory. These devices are also smart enough to split up hot and cold data on the fly.

What this means is that you’re suddenly dealing with a laptop which boots really fast because all the hot data is on high-performance storage, whilst your movies and music sit on a more cost-effective spinning disk. It’s a very simple example, but one which extrapolates to the Enterprise very easily.

When you speak to any of the big storage vendors, they are all incorporating this type of technology in their latest storage arrays because they know that it helps them meet performance and price points which make sense.

When making your decision on purchasing storage – make sure you are asking if your vendor’s technology supports ‘Tiering’ of data to move hot and cold data as appropriate to make the absolute best use of your spend.


“It’s important when making your decision on storage that you understand the requirements of your business – knowing what type of storage they need is key to that.”

Types of storage – object, block or file?There are different ways of storing data – essentially, it can be broken down into three primary methods: object, block and/or as files, which we’ll look into in more detail in a moment.

It’s important when making your decision on storage that you understand the requirements of your business – knowing what type of storage they need is key to that.

Object StorageIn theory this is the least utilised and least understood storage technology today.

Object storage stores and manages data as individual objects – as opposed to traditional systems which utilise file hierarchy.

Each object stored is a bundle of data which is kept in its entirety along with all its metadata. The object is given an identification and is retrieved by an application by presenting this ID to the storage structure. These objects are stored in a flat structure and might be stored locally or in geographically dispersed places. It has no concept of folders and typically requires a separate management file system to make accessing the data simpler.

Object storage is more scalable than traditional file system storage because it’s a lot simpler. Instead of organising files in a directory hierarchy, object storage stores files in a flat layer of containers and uses unique IDs or keys to retrieve them. This means it needs less metadata than file systems to store and access files, and it reduces the management needed by storing the metadata with the object. This means object storage can be scaled out almost endlessly by adding nodes. So, for instance if I have a large file I might break it up into 50 chunks and then put these chunks onto object storage. It’s my responsibility to put those bits together again. There is a layer above object storage which involves applications and these make sense of the data.

So who uses Object Storage? Well ebay runs on EMC Atmos so it’s an enormous application and uses object storage. If you put an image up on ebay it breaks it

into chunks - the application knows exactly where those chunks are and presents you with the picture. It’s also able to make incredibly effective use of Object Storage because many images are identical. How many pictures do you think ebay needs to store of a generic stock iPhone 5S? Not as many as you would think. As object storage is able to identify files by their meta data it’s easier for software developers to write algorithms which interrogate object storage and effectively deploy “store once, use many times” strategies. The beauty of the EMC Atmos platform is that it does this for you automatically.

OpenStack Swift is another option. It operates similarly where each chunk of data is stored as an object. It also has the ability to then write that data to different locations, for instance different computers within the same data centre or even go geo-diverse to different continents, if necessary.

Backup headaches for object based storage can be greatly simplified by deploying the likes of Openstack Swift or EMC Atmos. Typically what you do is take very cheap computers and relatively cheap storage which you know is not mission critical – say I’ve got 3 of them so I will write 3 pieces of information. If one fails I start creating a 3rd copy somewhere else. If you are familiar with the concepts of RAID on local disk, think of it like that only for entire storage arrays across multiple locations.

Object Storage typically requires you to buy a third party application that can use the object storage. Or you have to involve a developer who will integrate the object storage into your application.

Benefits of Object Storage:• You can choose between the different types of Object Storage based on price, performance and resilience• It is a cheaper storage option because it runs on cheaper commodity components• It provides geocaching so you can access your copy locally• The data is always available when you need it but has a low performance requirement

Object storage is great for:• Storing media, web assets and backups. • Protecting data by deploying multiple nodes and splitting data across those nodes


“One approach, which we prefer to use at iomart - is that you don’t provide locally attached storage at all. Instead you provide SAN Storage. With SAN Storage you are centralising all the storage by presenting a disk to each server whether it be Virtual or Physical from a central location.”

• Delivering ‘dropbox-esque’ solutions to the Enterprise without the worry of expensive File Servers or the complexity of the likes of Distributed File Systems.

Object storage disadvantages:• Needs an application to store the data• Slow - only really useful for relatively cold data• No real deduplication

Block Storage Block storage, in its simplest description is how most Windows operating systems interact with their underlying hard drives.

A block is a chunk of data and when appropriate blocks are combined it creates a file. A block has an address and an application retrieves the block by contacting that address. The application then decides where to place the data and how the blocks are stored, combined and accessed. There is no metadata associated with the block apart from the address – it has no description, no association and no owner. It only takes on a form when it is combined with other blocks.

Block storage has been the most popular type of data storage because it allows the application to extract the best performance. It enables you to make changes to any storage system on the local disk or SAN (Storage Area Network) without impacting users and requiring downtime. This means that you can move, expand or change the storage infrastructure while the application remains online.

Whereas Object Storage is typically utilised to deliver slower data – like archives and backup – block storage is often extremely performance dependent. Typically you will find systems like SQL Server or even large scale Exchange platforms which absolutely require block storage to operate.

A lot of vendors address this in different ways. Traditionally you’ll go onto a virtual machine and have local storage presented to you, for instance you share the local hard drives with different customers. This makes the guarantee of performance of the machines

difficult. You could be sharing a physical machine with someone whose website is infrequently accessed or with someone who’s running a very data intensive application in which case your performance of storage would be incredibly poor.

Some companies are addressing this by going with locally mounted SSDs – Solid State Drives – and using Flash to increase performance but the same issue remains that you are sharing physically attached drives with other customers so whilst the scale of performance on SSD for block storage is much higher, over time it still remains difficult to guarantee quality of service.

The other approach – which we prefer to use at iomart - is that you don’t provide locally attached storage at all. Instead you provide SAN Storage. With SAN Storage you are centralising all the storage by presenting a disk to each server whether it be Virtual or Physical from a central location. The machine doesn’t have a hard drive, it gets storage assigned to it over the network. The beauty of this is that the majority of storage vendors are now incorporating Flash as a caching tier to most Enterprise arrays. So traditional systems are able to make use of Flash without necessarily having to front load the server with lots of expensive SSD’s.The key thing with SAN Storage is it doesn’t really have a disadvantage. The devices themselves are generally specialised at delivering Block Storage, they are highly resilient and most importantly they can also be presented in performance tiers.

Block Storage TieringAs we mentioned earlier in the article when dealing with hot/cold data – you can have the same issue with Block Storage but across entire hard disks. Your SQL hard drive for example will nearly always want to be really fast – but equally you might have a drive filled with backups connected to the same server which doesn’t need to be fast at all. SAN Tiering allows you to do this easily – disk A goes on SSD, disk B goes on 7.2k SAS – at the click of a button.

Some storage vendors such as EMC take this a step further with their Fully Automated Storage Tiering (FAST). Once the disks have been carved and presented to the relevant server the SAN then monitors performance and evaluates if individual drives need to be moved to more highly performing disks. This alleviates the management overhead for you and also provides superb performance benefits for the end user.

Most vendors also allow you to manually define those tiers. Whilst the SAN Controllers are pretty good at determining it for you – you can make decisions based on budget, department priority, or just who sends more cakes to the IT team on a Friday! Typically, slow is cheap but then you can get bottlenecks and so you can move up to another tier. The beauty is you don’t have to change anything on the local machine. Let’s look at another example:


“In terms of the physical infrastructure, more and more vendors are now developing specially adapted storage technologies to help deal with the headache and performance overhead of file shares. EMC.”

You’ve just started within a new company and the first thing on your list of priorities is that the developers are telling you a database application is performing really poorly. They’ve chosen block storage as the delivery mechanism and insist it isn’t their application causing the problem. If your predecessor has invested in block storage with the ability to tier data you’re in luck. You can simply choose to prioritise the IO for that particular disk or Logical Unit Number and, providing you have the capacity within the device, they should see immediate performance improvement. That is, of course, presuming that it isn’t their application at fault!At iomart we selected the EMC VMAX because of its fantastic ability to deliver incredible IO numbers with industry leading Tiering capabilities. We know that if a customer approaches us with similar problems to those highlighted in the above example that we can simply “turn it up to 11” for them and remove any bottlenecks without them having to lift a finger. It’s a very compelling and powerful tool to have in our arsenal of storage solutions.

Benefits of block storage:• Great performance• Improved management of varied storage assets• Granularity• Little or no metadata• Reduced time and costs associated with moving data• Investment and operational savings

Disadvantages of block storage:• Performance comes at a price

File StorageFile storage is easy to set up and use but presents different challenges that often make it difficult to manage. Historically adding new files storage to an environment ended up with too many files in one place and too few in others which reduced performance and utilisation rates. Consolidation of files often led to downtime as IT departments literally had to lift and shift drives from one location to another before carrying out synchronisation and mergers between the old and new file systems. File storage virtualisation however addresses both of these issues by enabling storage file systems to be changed without downtime and by transparently moving files without changing their name or address. These issues can be addressed using highly scalable on premise solutions like isilon or possibly using one of the new generation of Cloud NAS solutions like Panzura, Ctera or Nasuni which deliver files locally while storing the colder data on the Cloud

Enterprise File Sync and Share is now starting to be deployed today. The technology has its roots in consumer-based systems which easily allowed users to access files from different devices.

The software either allowed users to copy files to the cloud storage or let them synchronise files or

folders to the cloud. In addition to the convenience of accessing files from any device over the internet, the systems allowed users to share files with colleagues and friends in a simple convenient way. The ease of use and simplicity meant that users started using these solutions within the corporate environment and although this provided a number of obvious advantages it also introduced a number of serious risks.

Data security was the key problem. Company data could be leaving the organisation to be stored on systems that may be insecure, may contravene corporate data sovereignty rules and users may be unintentionally sharing restricted files with anyone in their social circle. To address these issues a number of Sync and Share solutions have been developed specifically for the corporate market. These solutions still provide the convenience of synchronising and sharing files but have added other features like file versioning, user access controls, directory integration, push distribution and continuous backup. These features allow organisations to strike an appropriate balance between security and agility, allowing users to create and share files that will be available on any device, that are automatically backed up and have powerful collaboration features such as version control, user access control and expiry dates.

In terms of the physical infrastructure, more and more vendors are now developing specially adapted storage technologies to help deal with the headache and performance overhead of file shares. EMC are able to deliver the storage backends through systems like isolon – however the most important factor is how users interact with the storage and we are increasingly seeing this governed by third party software developers who are developing storage agnostic solutions which can work across a wide range of storage vendors. This provides incredible benefits in terms of cost savings as well as keeping on the cutting edge of how users would like to interact with the corporate IT department.

Benefits of File Sync & Share:• Allows files to be accessed anywhere• Allows files to be easily accessed from different devices• Facilitates collaboration

Disadvantages of File Sync & Share• Security can be an issue

Storage VirtualisationIn many ways storage virtualisation is the completion of a full scale 360 on how IT administrators interact and utilise storage. In the early years everyone was absolutely convinced that local storage was effectively ’the answer’ to all issues – particularly performance.We then moved through the formative years where people moved to SAN storage and all of the benefits


which we highlighted earlier in the article around block storage in particular have become prevalent. However, as the Software Defined Data Centre gathers pace and more and more services are virtualised, including Compute and Network, so the focus has spun around towards Storage once more.

Consumers of IT, whether it be at home or in the business, are no longer satisfied with fixed-installation solutions with limited mobility or restrictive upgrade capabilities. To help address this a number of vendors, VMWare chief amongst them, have returned to local disk to address these concerns.

For the purposes of this paper we will focus on VMWare’s VSAN technology, however there are up and coming alternatives such as ScaleIO.

In a nutshell – VMWare are taking all of the local disk capacity in your existing virtualisation clusters and making best use of them. They require a minimum of three nodes however, in today’s virtualisation market this is not a lot. Each node has to be equipped with an enterprise grade SSD which performs nearly all the read and write caching for the system.

Once you are equipped with these tools the VSAN product will take on management of each node – we’re simplifying it here and not doing the system any real justice but again think of this like networked RAID. You can take entire servers offline for maintenance and, providing there is sufficient resilience in place, VSAN will work around it.

It’s a relatively new technology but one which is making waves in the storage market. We’re not sure yet whether it will be appropriate for highly intensive workloads but VMWare’s own bench-testing is impressive and a number of the larger hardware vendors are lining up to deploy VSAN Certified builds of their existing hardware stacks.

Storage Virtualisation really completes the circle for Virtualisation in general – now providing you with the ability to virtualise everything in your data centre from Compute, to Network and now Storage – it cannot be ignored as a method of storage delivery going forward.

The main thing about Storage Virtualisation is you can add more resources in a very cost effective manner. Bear in mind though that SAN provides the same

benefits – more storage or more performance and the ability to scale up and down as you require. You can also mix it – some machines on high performance and some on low - giving you all this excellent value. However as ever, be careful with selecting the correct storage technology for the business case. It’s most definitely not a one-size-fits-all solution.

The choice – public, private or hybrid?The choices are all of the above but also do you need consumer-level file hosting or enterprise level cloud storage? It’s a big decision for most businesses and one which should not be taken lightly.

It is unlikely that a single storage architecture will fully address all the challenges presented by your business systems. So what are the main security and compliance considerations you need to address when deciding how and where to store your data?

SecurityData Security is King. There’s also a take on another popular saying that “nobody got fired for buying the most secure solution!” In a world of hacking, spying and data leaks we’ve seen the reputation of entire organisations crumble because of a failure to correctly secure and encrypt critical data.

Many storage solutions now offer EAS – Encryption At Rest. This ensures that every block of data is safe and secure even in the unlikely scenario that someone were to remove the hard drives from the premises. Many public solutions will offer nowhere near this level of protection so it is usually reserved for the world of private installations where it can be costed appropriately. Remember that encrypted data, whilst more secure, is also more expensive to store because it cannot by its very nature take advantage of things like data de-duplication. Whilst focusing on the storage side of things is important – you also need to be confident particularly when dealing with PCI DSS or ISO compliance that management access to the data is secure. It’s all well and good making sure that the disks themselves are encrypted but if you make it easy for someone to just lift the keys and access the management portal you’re asking for trouble. Ensuring that management access to any of your storage technologies is locked down either via VPN or at the very least by a restricted IP list is an absolute must.

It’s also this worry which rules out many public solutions which have completely public facing portals for managing, deleting and creating new services. There have been several examples recently where a failure to secure these management portals has led to the downfall in some cases of entire businesses.

Your local security policies should also extend to your internal staff – in short, don’t allow the office junior


“public, private or hybrid?The choices are all of the above but also do you need consumer-level file hosting or enterprise level cloud storage? It’s a big decision for most businesses and one which should not be taken lightly.”

systems administrator free range to make potentially critical decisions on your storage security. Audit your access lists regularly and make sure the people with access are responsible and most importantly fully trained. Whilst many sys admins are truly experts in some fields, not many can extend to managing a full storage stack without the odd hiccup along the way.Finally of course, consider your physical security. The preference should always be to house your storage particularly if it’s mission critical in N+1 or Tier 2+ datacentres. Don’t forget to audit those regularly too, ask to visit the site and check that the company you have selected is actually doing what it promised.

Data sovereignty Data sovereignty is the concept that the data you have converted to binary form and stored is subject to the laws of the country to which your business is subject. This requires the business to trust its cloud storage provider. This is a hugely important point and one that should not be taken lightly. Do you know where your cloud provider is storing your data?

Are you sure that they are storing it in a data centre in the country that you want it to be stored in or are they using a third party data centre located outside your geographical boundaries – this can happen. This is where the cloud provider has to be open about where their servers are hosted and where they need to provide the strictest of SLAs which you can check thoroughly.

Increasingly more and more sectors are interested in where data is stored. PCI DSS processes make it mandatory to know and understand this information. There are also law enforcement considerations to make – would your customers be happy with their data being handed over to a US law enforcement agency if prompted to do so via something like the PATRIOT act for example?

If you have multiple geographical sites in different territories, or you are storing multiple copies of your data in multiple locations where is your data to be sovereign? Essentially your data is subject to the laws of the country hosting the server which holds it as well as the country in which your particular part of the business is situated.

The laws and requirement for this vary from country to country, but many countries have very specific and constraining laws for data sovereignty that can have substantial impact on logical and physical cloud and storage architectures. It is critical that companies create a data privacy and sovereignty governance framework. This must meet the requirements that consumer data does not leave the country of origin, personnel outside that country (provider) do not have access to any aspect of the data, and all operations (provider) must be performed by in-country residence staff.

Managed hosting service providers can help with these issues – particularly around guarantees on where data exists (or does not exist as the case may be) however once again please take steps to audit and verify this information wherever possible.

Data Destruction & PrivacyIncreasingly organisations are being pressed to be able and capable to destroy data on specific users or systems. Google for example has recently been the subject of law making by the EU to allow users to be ‘forgotten.’

This presents an enormous challenge for storage administrators. Particularly if you have chosen to house your data within a public storage location like Amazon’s S3. If you ever require certification of storage destruction this becomes practically impossible within the Public environment. However if your organisation has opted for the Private model it should be a simpler exercise as you have full control over the entire array right down to the last disk.

There are certainly significant considerations to make here – if your organisation holds or maintains any sort of data on individuals then now would be the time to start making decisions based on your ability to remove data of an individual and if necessary prove how this was completed.

When it comes to data destruction services of storage a wide array of options are available but in most major cities around the world localised disk destruction firms will be able to format, degausse and physically destroy disks on site to the same standards demanded by the US Department of Defence. The challenge is and always will be getting access to these disks – think carefully where you store your data if you think this could be an issue for you.

SummaryCloud Storage by its very description means that you will have to rely on a third party hosting provider, so ensuring you have the right partner is absolutely paramount. Data security, uptime guarantees, the ability to scale and the reliability and speed of your provider’s network all come into play. While this can be a nerve-racking thought it can also be the opportunity your business needs to overcome some of its organisational difficulties. It all comes down to making sure you choose the right storage to suit your differing business requirements and if you get it right you can certainly operate a lot smarter and faster than you might have been able to before.

Resourceshttp://uk.emc.com/collateral/software/15-min-guide/h2977-virtualization.pdf


Images for illustrative purposes only. © iomart 2014. Lister Pavilion, Kelvin Campus, West of Scotland Science Park, Glasgow, G20 0SP

“Cloud Storage by its very description means that you will have to rely on a third party hosting provider, so ensuring you have the right partner is absolutely paramount.”

the challenge of cloud storage - iomart

Documents