PowerPoint Presentation

Manila An Update from Liberty

October 29, 2015OpenStack Summit - TokyoAkshai Parthasarathy, NetAppSean Cohen, Red HatThomas Bechtold, SUSE

Welcome to this session at the end of the day on Thursday at OpenStack Summit - Tokyo. We hope you had a great Summit and want to thank you for being here with us this late on Thursday. This session is Manila An Update from Liberty.

Sean Cohen () - @thomasbechtold@gmail.com I've created the frame slide for mitaka, what else can we highlight besides replication? (such as support to use default network by multi tenant drivers / extend share with HDFS_native driver etc)Tom B () - Here's a list of possible summit topics: https://etherpad.openstack.org/p/manila-mitaka-summit-topicsAkshai Parthasarathy () - thx. I'll look into it.Tom B () - Given that we have a lot of feature slides, should we skip this one?Sean Cohen () - Agreed, moving it to backup slidesAkshai Parthasarathy

Technical Marketing Engineer, NetApp@akshaiSean Cohen

Principal Product Manager, Red Hat@SeanCohen_RHThomas Bechtold

OpenStack Cloud Engineer, SUSE

Today's Presenters

I am Akshai Parthasrathy, Technical Markeing Engineer for all things Cloud Computing and OpenStack at NetApp. Here with me are Sean Cohen, Principal Product Manager from Red Hat and Tom Bechtold, OpenStack Cloud Engineer from SUSE.

AgendaWhat is Manila?Why use Manila?Use CasesSaharaContainersLiberty UpdatesDistributions IntegrationSUSE DemoRedHat DemoUpcoming in MitakaShare Replication DemoQ+A

Lets take a look at the agenda for this talk. Some of you may recognize Manila as a city in the Phillipines or relate it to those manila folders in your filing cabinet. We will introduce Manila to you, in the context of OpenStack. Next, well cover why you want to use Manila - the advantage or value. There are many use cases for Manila, and well talk about some of them shortly. Well then jump into the main topic of this session - the work that was done for Liberty. Sean and Tom will take you through Manila integration into RHEL OSP and SUSE Cloud along with two demos. We then talk about features in Mitaka and Ill close-off with a third demo with a state-of-the-art feature called Manila replicas.

Akshai Parthasarathy () - sean: where are the sahara slides?Manila: The OpenStack Shared File Service ProgramBringing self-service, shared file servicesto the cloud

Very simply, Manila is for file shares what Cinder is for block storage.Manila dispenses, in a self service open REST API, shared file systems out to tenants of a cloud. Using Manila, we can get a 1GB NFS share and specify the network range that should have access to the share. Or, we can provision a 1TB CIFS share, do authentication with Active Directory and ensure specific tenant networks have access the share. So were dealing with shared file-systems in Manila. Therefore, in Manila, unlike Cinder, we have a networking component. We may want to export an NFS share only to that particular Neutron or Nova network for a tenant. So, there is a little bit more magic behind the scenes to make sure that the storage thats behind that filesystem can access that network and do it securely.

Manila HistoryBeginningsJunoIncubated ProjectCommunity InceptionPuppet SupportShare ServersTempest IntegrationKiloDriver ModesDevStack Plug-inStorage PoolsDefault Share TypeManage/UnmanageManila UI

The first time people started hearing about Manila was in the OpenStack Atlanta Summit, back in May 2014. At that time, the program was overflowing with interest and we were extremely motivated to continue working in the 6-month cadence. We introduced the Manila capability in Juno, and submitted it for consideration as a core service in the Kilo. Manila went through cycles of Continuous Development and Rigorous Testing and we persisted in putting out a significant number of new feature releases and bug fixes in Juno and Kilo.

Manila Today - Production ReadyNumber of Drivers: 14Blue Prints Completed: 51Major Blue Prints:Share InstancesREST API MicroversionsExperimental APIsExtend & ShrinkConsistency Groups (CGs)Share Migrationsetc.

So, lets get to where were at today. Most importantly, Manila is production ready. We have a total of 14 storage drivers for Manila. There were 51 total features (or blueprints) completed. We went down slightly on this number from Kilo, but that was because we spent more time on bug fixes this time around. We had a total of 184 completed bug fixes. The trends are on the right. We have rolled out a number of new features for you to take advantage of. They include share instances, REST API Microversions, Experimental APIs, and others. Tom and Sean will lead you through these soon.

Sean Cohen () - This is great, exactly what I had in mind... and down the lines of our session abstract, showcasing how Manila is exploding with new features, use cases, and contributersManila Deployment Options and BenefitsSingle Storage Virtual Machine (SVM) / Multi SVMdriver_handles_share_servers=False/TrueBenefit

Network PluginsStandalone Network PluginNova Network PluginNeutron Network PluginBenefit

Today, a Manila share driver may be configured in one of two modes. We can either use Manila to manage the lifecycle of share servers on its own or use Manila to merely provide storage resources on a pre-configured share server. This mode is defined using the boolean option driver_handles_share_servers in the Manila configuration file. It provides flexibility to deploy Manila shares the way you would like and is available with NetApp Clustered Data ONTAP today. The Manila architecture also three concrete network plugins. This allows operators to choose from a variety of options for how network resources are assigned to their tenants networked storage. The three plugins are Standalone Network Plugin for pre-configured networks, Nova Network plugin for Nova Networks, and most importantly, Neutron Network plugin for Neutron networks. Each of these plugins support a variety of segmentation options.

Contributions/New Entrants

Welcome and Thank You:CloudBase SolutionsFujitsuScalityNECNTTLetv Cloud Computingand others

Source:http://stackalytics.com/?module=manila&metric=loc&release=liberty

Code ContributionsReviews

Heres the breakdown of contributions for Manila, as of Liberty. NetApp, Red Hat and Suse are major contributors to the program and we welcome new contributors from the community. It should also be pointed out that NetApp has been a pioneer in the Manila project and a leader through all its releases.

For the Liberty release, we would like to thank the following members of the community: CloudBase, Fujitsu, Scality, NEC, NTT, and Letv Cloud Computing.

Why Use Manila?Shared File ServicesSpendingTotal CapacityIDC: Age of Cloud File ServicesWhy not?

Youve had a look at what Manila is. Let us take a look now at why you want to be using Manila.

There is an explosion of data today. Estimates for spending on file-based storage solutions say that it will reach north of $34.6 billion in 2016. The diversity of applications often depend on the performance, scalability and simplicity in management. OpenStack, as the leading open open source IaaS capability, with Manila as a service, is a production-ready option for deploying infrastructure with file-based services.

So the question really becomes why not use Manila with OpenStack?

Use CasesStandalone Shared File Services ManagementEnterprise ApplicationsDevOpsHybrid CloudSharesAutomationand Integrationwith Manila APIHeatDatabaseas aServiceSaharaContainersetc.

There are numerous use cases for Manila, the file share services project. These include: Standalone File Services Management, Enterprise Applications, DevOps, Sahara, Containers, Database as a Service,, Automation and Integration with Manila API, Heat, Hybrid Cloud Shares, and much much more. Lets dive into some of these now.

Standalone Share File Services ManagementReplace Legacy SystemsImprove SLAIntegrate with Automation Frameworks

The first use case well cover is Standalone Share File Services Management. One of the things we hear as we talk to NetApp customers is that a lot of them have one guy in a back office that has written a set of Perl scripts to dispense shares. Hell go in and run a Perl script to dispense a CIFS share when a new request comes in. There are probably 1 or 2 guys in a company that know how the script works and if they decide to take another job or you switch to a different technology youve got to re-invent that entire infrastructure. Not to mention that they break the consumer interface for that. So, we see a lot of interest in Manila for replacing those home-grown legacy systems with an open/standard API that is production ready. Manila provides the same level of self-service to create a share, delete a share, take snapshots and take other actions - in a completely vendor agnostic framework.

Enterprise ApplicationsLower Cost, Heterogeneous InfrastructureApplication Porting: Move Workloads to OpenStack

Another use case we want to address is the movement of existing enterprise applications to OpenStack. One of the common things we hear from customers Im using a virtualization technology that works great and has a lot of good functionality but is really expensive. A big reason to move to O/S is just cost. IT budgets arent growing a huge amount and you want to do more with less and O/S provides a pretty compelling value proposition to make that a reality. Manila ensures that workloads that are built assuming the existence of a shared file system can move over to O/S and get all the cost benefits that O/S or KVM might avail for us. Another reason is that, I dont have to re-write those apps to use an object store, for example. Perhaps over the next 5 years or 10 years I will rewrite those apps to a Swift interface or S3 interface. But frankly, there are a lot of apps that work today with file-based services and people dont want to fix it if it aint broke. Manila provides a way to move those apps to the O/S infrastructure and leverage the cost benefits.

DevOps/Continuous IntegrationPluggable InfrastructureStorage Service CatalogAnalyticsContainersSnapshots & ClonesConsistency Groups

The #1 use case for Manila is DevOps. You can have clones of Manila in a snap and this speeds up the lifecycle heavily in a fast-paced DevOps environment. OpenStack is all about being a pluggable infrastructure. We can take any type of backend and expose it using share types. The use of the storage service catalog allows you to separate the needs of your file-share workloads. You can have your IT archive into a different share type using the Manila API you dont need to have SSDs being used for your archives. We can provide you the storage you need when you need it, but also make sure that it is stored on the right back-end. For analytics, you have a HDFS driver since the Kilo release. You can scale out your analytics workloads using Manila. You can even have Manila file shares mapped to containers. Sean will cover both Analytics with Sahara and Containers later in this talk. A new feature we introduced in Liberty is Consistency Groups or CGs. With CGs, you can take application-and-database snapshots that are consistent and taken at the same point in time. This is a great feature to have in multi-tiered applications. If you bring it all together - Manila File Services Management, Enterprise Applications, DevOps, Continuous Integration, Analytics, and Containers, we can rock with OpenStack Manila. Let me now pass it over to Tom and Sean to lead us through important features implemented in Liberty.

New in Liberty

thanks Akshai. Im going to talk about new features together with Seana lot happened during the Liberty cycle!for example oversubscription, cosistency groups, driver hooks and morwe also adapted and integrated more into other components. for example we use a tempest plugin, microversions and diskimage builder elementsso in general lot of useful features for users, administrators and driver developers

Share Instances

Needed for features like:

share migrationsreplication

new concept in Libertyneeded for share migration and replicationonly visible for administrators. users don't recognize this changethe main goal of share instances is to decouple share UUIDs which are visible to the user from the UUIDs for share_instances which are visible to drivers.that way the driver can create/delete share_instances or switch the connection between share_instances without changing what the user is seeing.

REST API Microversionshttp://docs.openstack.org/developer/manila/devref/api_microversion_dev.htmlhttp://docs.openstack.org/developer/manila/devref/api_microversion_history.html

Example:

client 1 with server 1 => version 2.3client 1 with server 2 => version 2.3client 2 with server 1 => version 2.5client 2 with server 2 => version 2.10

implemented in Nova during Kilo cycleadapted by Manila and other projects will adapt the concept, toomake it possible to evolve the API incrementally - even backwards incompatible changeshow does it work:client sends version it supportsserver uses that API then and returns the expected return values

Experimental APIsExpected to change at any timecan be removed without deprecation periodUsage: needs to set header X-OpenStack-Manila-API-Experimental: truehttp://docs.openstack.org/developer/manila/devref/experimental_apis.htmlhttp://developer.openstack.org/api-ref-share-v2.html

@api_version(min_version='2.1', max_version='2.9')def show(self, req, id): .... stuff ....@api_version(min_version="2.4", experimental=True)def my_api_method(self, req, id): .... stuff ....

gives the advantage that developers can try out things and APIs can evolveusers can play with new APIs nowslides with the blue test tube are talking about experimental APIsplan for M release is to remove all extensions and add them to the current API

Extend & Shrink

http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html

shrink only supported via CLInot all drivers support shrinkingextend share with HDFS_native driver is planned for Mitaka

Consistency Groups (CGs)grouping different shares together for the purpose of application data protection (focus of snapshots for disaster recovery)example use case: database data and log files are on different shareswhen doing snapshots for both shares, the data on the shares must be consistentsupported actionscreate CG with mutliple sharescreate snapshot of CGcreate CG from snapshot

With CGs you can group together different shares into a consistency group. This is needed for data protection and data consistency.example: database data and application data or logfiles are on different shares. doing a snapshot of both shares needs to be done at the same point in time when the data is consistent. CGs are doing that.So you can now: group shares into a CG, snapshot a CG and create a new CG from a snapshot for data recovery

new capability: consistency_group_support. Possible values:None - No support for CGshost - shares in a CG must be on pool(s) on the same host that also match the CG share typepool - shares in a CG must live in the same pool as the CGUnlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.API experimental

External CI For All Drivers

And others ...

we have noe external CI for all out drivershuge effort by all the vendors - so thanks a lot to all drivers vendors for the hard worktempest tests are now running for all the drivers for every changesetDuring Mitaka cycle, minimum CI requirements

Removed drivers:Hitachi HDS SOP (Scale out platform)

OversubscriptionTunable for setting provisioned capacity and a subscription ratio added

Addresses 'infinite' and 'unknown' drivers reported capacity that may lead to oversubscription

thin_provisoning support needed

you can now oversubscribe your available capacity. for that the backend needs to support thin provisioningmax_over_subscription_ratiodefault is 20 (provisioned capacity can be 20 times the total physical capacity)

LAST SLIDE TOM

Share MigrationsShare Migration allows a share to be migrated from one host pool to another hostpool through the "manila migrate " command, as well as allowing to perform migration between different backends.Basic implementationThe fallback approach for migration is rsync. Slow, inefficientVendors can utilize API for optimized migration

API experimentalexample use case: maintenance for one backend, so move the share to another backendshould work okay with any backend that doesn't have share serversshare retype not yet implemented

Availability ZonesThe availability zones support which was inherited from Cinder was reworked this cycle:Added public API extensionAllow to preserve AZ if creating a share from a snapshot and set AZ in Share API or Share Manager.AZs will benefit share replication and also give end users control of the locality of their data w.r.t. consumers of the data.

Sahara IntegrationUse casesStores binaries for job templates - NFS is ideal for this caseInput and output data sources - Manila-provisioned HDFS and NFS offers more optionsMount NFS share APIBinaries and data I/O from an NFS share pathNew development in Sahara this cycleMount shares at cluster creation or auto-mount when a share is used for EDPManila-provisioned HDFSData sources and data processing on Sahara-external clustersTesting and process verification of extant Manila features this cycleComing soon - NFS Hadoop driver (run jobs on your NFS shares)

Manila, at its core, provides basic provisioning and management of file shares to users of an OpenStack cloud. The Sahara project provides a framework to expose Big Data services such as Spark and Hadoop. Together these two projects create a solution that is greater than the sum of its parts.

Natural synergy and popular demand led these three teams to develop a joint solution to expose Manila file shares within the Sahara construct to solve real Big Data challenges

Sahara Integration - Current ImplementationsData sources on Manila-provisioned HDFSAPI to mount NFS shares to clusters (job binaries and data sources)

Images by Weiting Chen (Intel)

Share mount API does require a network copy into local HDFS for data sources.

Manila-provisioned HDFS: insecure cluster integration only at this time.

** Make reference to the Tuesday demo showing a Sahara data processing job running with binaries, data sets, and results hosted in Manila file shares mounted on a Sahara cluster.

Manila + Containers Ceph ExampleSimply mount --bind share intocontainer namespace NFS re-export from hostmount and export fs on host private host/guest net avoid network hop from NFS service VMHost mounts CephFSBind Manila share/volume into container Further integration requires integration with both Nova and Manila to manage the attach/detach process

HOSTMMRADOS CLUSTERCONTAINERMANILANATIVE CEPHCEPH.KONOVA

The current target is to use the new VSOCK zero-configuration sockets, that requires no configuration on the guest--so we can continue to treat it as a black box--and only a simple network id assignment on the host.

Distributions Integration

Let us now talk about the upcoming features in Mitaka

Manila in SUSE OpenStack CloudThe Manila service is tech preview in SUSE Cloud 5Fully supported in SUSE OpenStack Cloud 6Crowbar deployment tool integrationController HANetApp driverCustom driver possible

SUSE OpenStack Cloud 5 has Manila as tech previewVersion 6, expected in January has Manila fully supportedIm going to show a demo video of the deployment integration in the current Cloud 6 beta versionCrowbar used as deployment toolChef BarclampsHA integration

Manila in RHEL OpenStack Platform 7The Manila service is tech preview in RHEL OpenStack Platform 7RHEL OpenStack director deployment tool Offer integration withGlusterFS nativeGluster NFS NetApp driversManila Certification program in RHEL OpenStack Platform 8Introducing NFS-Ganesha and Gluster Automated Volume Management (based on Heketi)

Using GlusterFS via NFS-Ganesha, the Manila file shares are abstracted from underlying hardware and can be grown, shrunk, or migrated across physical systems as necessary. Storage servers can be added or removed from the system dynamically with data rebalanced across the trusted pool of servers, where the data is always online this addresses the File Share Elasticity required to provide a Scale-out / Scale-down NAS on demand. It also delivers File Share Availability as GlusterFS enables you to replicate the whole storage system between different data centers and across geographic locations.

Upcoming in Mitaka

Let us now talk about the upcoming features in Mitaka

Upcoming in MitakaMount AutomationRolling UpdatesExport Location MetadataManila QoSCapability ListsInteraction Between New FeaturesShare Migration Loose-endsRemove All ExtensionsArchitectural Directions For New 1st-party DriversShare Replication

Note: dont deep dive into each one. Choose 1 or 2 and briefly cover others.

So here is a list of upcoming features we have for you in mid 2016. Mount Automation: there has been a lot of interest to have Manila shares automatically mounted on instances and significant amount of work is already in place. Were going to complete it for Mitaka. Rolling upgrades to spin up new API services without any disruption is also scheduled for the next release. Ill leave this up there for a few seconds. If you have questions about any of these, feel free to see me after this session.

Akshai Parthasarathy () - sean - over to you. please change and highlight accordingly.Non Disruptive OperationsHigh AvailabilityAvailability ZonesFailures within an AZHigh Availability SolutionClustered StorageFailure of an AZHigh Availability SolutionShare ReplicationManila State of the Art: Share Replication

As with any service in an enterprise or the Cloud, were always looking for non-disruptive operations. The way we provide this is through high availability or HA. In OpenStack we have the concept of Availability Zones. We already have the technology to provide HA within an AZ today. This is achieved through clustering of storage controllers, including Clustered Data ONTAP. Ok, so you can tolerate a malfunctioning storage unit or loss of network connectivity to a storage controller using clustering magic. But, what if you have the entire AZ going down, say due to loss of power? You want to have failover and failback from one data-center or Availability Zone. This is a really hot state-of-the-art feature (that just got finalized a week ago for Mitaka) and Ill wallk you through a demo.

Demo #3: Share Replication

Let me show you state of the art for Manila - Share Replication.

GET INVOLVED WITH MANILA!

Manila Resourceshttps://github.com/openstack/manila https://github.com/openstack/python-manilaclient https://github.com/openstack/manila-ui https://github.com/openstack/manila-image-elements https://wiki.openstack.org/wiki/Manila https://launchpad.net/manila

#openstack-manila on IRC (Freenode)Weekly meetings @ Thursday, 15:00 UTC

NetApp: http://netapp.github.io Red Hat: https://www.redhat.com/en/insights/openstack Suse: https://www.suse.com/products/suse-cloud/

Get involved with Manila today. We always welcome new members to the community. This is all the links for Manila and NetApp, Red Hat and Suse. The NetApp website is netapp.github.io, and the Red Hat and Suse links are here as well. Please visit them to learn more.

MANILA RELATED SESSIONS

MANILA GENERAL SESSIONSManila and Sahara: Crossing the Desert to the Big Data Oasis: Tuesday, Oct 27 12:05pmOpenStack Manila Hands-on Lab Session: Tuesday, Oct 27 2:00pmThe State of Ceph, Manila, and Containers in OpenStack: Wednesday, Oct 28 4:40pm

UPCOMING MANILA SESSIONManila contributors meetup: Friday, Oct 30, 9:00am

Here are the Manila related sessions that you can refer to on Youtube. There is also a meetup tomorrow please attend.

Q&A

October 29, 2015OpenStack Summit - Tokyo

With that, I would like to call this talk to a close and open up the floor for any questions you may have. Please provide any feedback you may have through the OpenStack Summit App. Thank you for staying with us till the end of this Tokyo Summit.