1. Inkscope a ceph GUI initiative Alain DECHORGNAT Orange Labs Ghislain CHEVALIER Orange Labs September 18, 2014 Orange Labs

2. 2 Agenda Ceph at Orange Labs Inkscope project Demo Parallel with Calamari Next steps Q&A CephDay Paris September18,2014 3. 3 Since 6 years, research on innovative concepts to enrich the cloud storage functionalities Data growth control Storage costs reduction Bottleneck avoidance Data durability insurance Ease of data accessibility (ATAWADAC) Development of an internal cloud storage solution Since June 2013 CEPH is identified as a real challenger Acquire expertise on Ceph Study how the innovative concepts can be implemented into Ceph (erasure codes, multi-tiering, multi-siting, store-edge, indexation) Be an actor in CEPH community Be a CEPH tamer CEPH at Orange Labs source : Crypto Kraken - Menvielle/Martin CephDay Paris September18,2014 4. 4 Federation Management of heterogeneous storage services Storage location regarding costs and performance Storage contracts content = f(data, contract, metadata) Smart data placement Storage orchestration (best storage infrastructure) Data robustness (replication/erasure code, sharding) Multi-tiering Right Data, Right Tier Implementation in Ceph Crushmap management Dynamic tiering Multi-siting Right Data, Right Site Implementation in Ceph Rados Gateway Federation Ceph Monitoring and Administration Inkscope initiative R&D Works CephDay Paris September18,2014 5. 5 Ceph visualization and operation through command lines No open-source Ceph GUI available in late 2013 Breizh Meetup helps us to launch the project Our choices: Open source Web app Use of Ceph rest API Modularity, simplicity Contributors: A.Dechorgnat, Ph.Raipin (Orange labs), A.Sall (trainee) E. Mourgaya (Arkea) Why Inkscope ? CephDay Paris September18,2014 6. 6 cluster inkscopeViz Web client inkscopeCtrl Server part Provides an advanced REST API inkscopeProbe Collects system and ceph infos Feeds a mongoDB database inkscopeMonitor (not developed) Monitoring of Ceph metrics stored in db Feeds monitoring tools like Nagios Inkscope modules Supervision framework (Nagios)inkscopeProbe inkscopeCtrl inkscopeMonitor inkscopeViz inkscope server ceph APIs inkscope client CephDay Paris September18,2014 7. 7 Inkscope demo CephDay Paris September18,2014 8. 8 Dashboard Ceph cluster status at a glance OSD visualization and details Free and used capacity Functional or not Pools details Used capacity Parameters PG states Object storage (S3/swift) Users management Buckets management Crushmap visualization Inkscope functionalities summary CephDay Paris September18,2014 9. 9 Tiering Wring the neck of the crushmap to fulfill tiering requirements Work in progress on Ceph From a basic topologythrough a hw-based topology to an application-based topology CephDay Paris September18,2014 lets see it in inkscope 10. 10 Inktank dev open source since May 2014 mailing-list : ceph-calamari presented as diagnostic tool 4 modules calamari-web cthulhu salt-master rest-api Calamari is in the wild CephDay Paris September18,2014 11. 11 Parallel Calamari/Inkscope : functionalities Calamari Inkscope Visualization Dashboard Yes Yes OSD Yes Yes Host Yes Yes PG Yes Yes Crushmap No Yes Object No Yes Graphs Cluster Yes No Pools IOPS Yes No Hosts Yes No Management Console No Yes OSD Yes (operation only) Partial (stop, in/out) Pools Yes (limited parameters) Yes Cluster settings (flags) Yes No S3/swift users No Yes S3/swift buckets No Yes S3/swift zones/regions No No Monitoring Alerts No No Logs Yes No CephDay Paris September18,2014 12. 12 Parallel Calamari/Inkscope : technologies CephDay Paris September18,2014 Client HTML5 HTML5/AngularJs Javascript libraries: AngularJS IndexedDb BackboneJS Javascript libraries: AngularJs addons (ngTable, dialog, ui Bootstrap) Visualization with D3.js, NVD3, Rickshaw Compass, Sass CSS (needs gem and Ruby), Bootstrap 3 CSS Bootstrap CSS and also Grunt, NodeJs, make to build the client Server - Apache with Django (python scripts) Apache with Flask (python scripts) - packages: django, flask, python-dev, libcairo, liqpq5, psutil, gevent, sqlalchemy, pytz, alembic, zwig, zeromq, m2crypto Python packages: flask, pymongo, simples3, python- dev, psutil PosgreSql MongoDB Carbon, Graphite, Diamond Shellinabox (optional) Saltstack - Supervisor - Vagrant and all their dependancies Calamari Inkscope 13. 13 Planned in Inkscope Extend or replace Ceph Rest API request by command lines powered by Salt Improve S3/swift features : zones and regions management Improve probes operations Implement objects visualization Plug the monitoring module to feed Nagios/Shinken Other ideas Simulation : impact calculation in case of crushmap update (storage capacity, bandwidth,..) Fusion with Calamari? Contact taken with Calamari team Next steps for inkscope CephDay Paris September18,2014 14. 14 https://github.com/inkscope/inkscope alain.dechorgnat@orange.com ghislain.chevalier@orange.com Q&A CephDay Paris September18,2014