the challenges and opportunities of open source storage
TRANSCRIPT
Jian Zhang, [email protected], Senior Software Engineer
Jack Zhang, [email protected] , Senior Enterprise Architect
April, 2016
2
Agenda
• A brief introduction
• Ceph in PRC
• The stability challenges and optimizations
• The performance challenges and optimization
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
3
Agenda
• A brief introduction
• Ceph in PRC
• The stability challenges and optimizations
• The performance challenges and optimization
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
4
A brief introduction
• Intel Cloud Storage Engineering team
• Deliver optimized open source storage solutions on Intel platforms
• What did we do on Ceph
• Ceph performance analysis, development, and optimization
• Tools and BKM
• CeTune; Virtual storage management (VSM); COSBench – Cloud Object Storage Benchmark tools
• Performance tunings and BKMs
• Customer engagements
• Working with 35+ customers to help them enabling Ceph based storage solutions
5
Agenda
• A brief introduction
• Ceph in PRC
• The stability challenges and optimizations
• The performance challenges and optimization
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
6
Ceph @ PRC
Ceph is much hotter in PRC with the growth of Openstack
Redevelopment based on the upstream code
More and more companies move to OpenSourcestorage solutions
Intel/Redhat held two Ceph days at Beijing and Shanghai
350+ attendees from 250+ companies
Self media-reports and re-clippings
More and more PRC code contributors
ZTE, XSKY, H3C, LETV, UnitedStack, AliYun, Ebay..
7
Ceph @ PRC
Active & spontaneous local community – ceph.org.cn
Grow Ceph ecosystem
– Three meetups @ Beijing and Shanghai
– On average ~100 attendees
– More cities – Zhengzhou Next
– Technical problem discussion
– 1094 users for BBS, 1025 for wechat group, 2265 wechat public account subscribes
Ceph localization
Ceph documents localization –http://docs.ceph.org.cn/
Ceph local mirrors
Material from http://ceph.org.cn
8
Who is using Ceph?
Telcom
With System scale and data growth, performance requirement is on the increase, and become new burden and bottleneck for wholeIT infrastructure
– High Storage cost (30%-50% of Total IT cost)
– Limited scalability of traditional Disk Array makes difficulties in operation
Searchable Examples: ChinaUnicom
CSP/IPDC
Non-TAB IPDC customers are moving away from SAN/NAS solution to open source scale out storage solution such as Ceph. The service work functionally and performance is not up to par. Tier 2 CSPs are building their storage service with Ceph.
Searchable Examples: LeTV, Ctrip, PLCloud
OEM/ODM
OEM/ODM building Ceph based storage solutions
Searchable Examples: H3C, Quanta Cloud Technology
Enterprise & Research Institutes
Building Ceph based storage products
Searchable Examples: http://www.slideshare.net/inktank_ceph
9
The Feedbacks
Customers & Developers & End users from CSP, IPDC, Telcom, Enterprise, and Research Institutes
• Ceph is awesome!
• Performance is acceptable
• Never Lost Data
• It’s awesome we can deliver three type of services in one single platform
Beyond Performance
• Can you deliver a training to us on the architecture/code?
• Is there any tool to deploy/test/manager a Ceph cluster?
• Hey, can you help, our Ceph cluster met some performance problems.
• The cluster is complaining slow requests…
• Why my OSD up and down again and again?
27%
16%
17%
26%
9%6%
Top Current Ceph Problems
Performance Documentation Stability
code complexity Ecosystem other
….With Ceph, Perfect World is able to build high performance, highly scalable and reliable software defined storage solution that to providing KVM private cloud service to many high load critical applications. In the deployment process, Ceph proves software defined storage is capable of delivering high reliability as well as high performance.
Zhang XiaoyiVP
Perfect World
10
Agenda
• A brief introduction
• Ceph in PRC
• The stability challenges and optimizations
• The performance challenges and optimization
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
11
The Stability problems --Motivation
Ceph performance is good when the cluster is stable, but there are some stability problems
OSD Flapping
Slow Requests
Sometimes it’s more efficient to handle it with third-party software
12
The Stability problems --OSD Flapping
Reported from customer1:3 nodes, each nodes has 1 monitor and 9 ODS
Cluster network and public network on separate network
Reproduce Steps: 1. Cutoff node2 cluster network
2. OSD becomes un stable, start to flapping, finally into stable status after 40mins (OSD on node 1&3 is up, while on node 2 is down).
Log:2015-08-26 16:00:14.672658 7f8f38e56700 -1 osd.17 1769 heartbeat_check: no reply from osd.1 since back 2015-08-26 15:59:53.848924 front 2015-08-26 16:00:11.058129 (cutoff 2015-08-26 15:59:54.672641)…..2015-08-26 16:02:33.721984 7f8f30120700 0 log_channel(cluster) log [WRN] : map e1807 wrongly marked me down
MON
OSD
MON
OSD
MON
OSD
Public Network
Cluster Network
13
The Stability problems --Optimizations: OSD Flapping
Smart Failure Detection
mon: support min_down_reporter conuted by host (#6709)
– Nowadays we use the "osd_min_down_reporters“ as threshold to mark an OSD down. We would like to extend the semantic to allow it counted by any crush level(host , rack, etc) thus user could require failure reports from at least two nodes to mark an OSD down, which should prevent an isolated host make trouble to the cluster.
– In general this PR will prevent OSDs in an isolated node marking other health OSD down.
osd/OSD.cc: shutdown after flapping certain times (#6708)
– Shutdown the OSD after certain times of reboot(flapping), thus speed up the convergence of cluster.
Do not try to boot up if we have no available peers(#6645)
– We should be isolated in this case, do not boot
Ignore report from an OSD in down state(WIP)
Better failure report
– Will aggregate the failure message like so admin could identify the connective issue easily
– cannot reach _% of peers outside of my $crushlevel $foo [on front|back]
– cannot reach _% of hosts in $crushlevel $foo [on front|back]
14
The Stability problems --Slow Request
Slow Requests: A phenomenon usually come along with performance issues If a ceph-osd daemon is slow to respond to a request, it will generate log messages complaining about
requests that are taking too long. The warning threshold defaults to 30 seconds, and is configurable via the OSD op complaint time option.
When this happens, the cluster log will receive messages. Optracker will record the request initiated time, and report slow requests.
– 2015-08-26 16:13:42.928244 7f8f38e56700 0 log_channel(cluster) log [WRN] : 16 slow requests, 16 included below; oldest blocked for > 30.961164 secs– 2015-08-26 16:13:42.928264 7f8f38e56700 0 log_channel(cluster) log [WRN] : slow request 30.894584 seconds old, received at 2015-08-26 16:13:12.033564: osd_op(client.394232.0:2123827
rbd_data.85642ae8944a.000000000000085c [set-alloc-hint object_size 4194304 write_size 4194304,write 3670016~524288] 7.eced69f0 RETRY=6 ack+ondisk+retry+write+known_if_redirectede1916) currently reached_pg
Need a better mechanism to detect what caused the slow requests — beyond Optracker’s scope.
Possible causes Possible solutions Opportunties
• A bad drive (check dmesg
output)
• A bug in the kernel file system
bug (check dmesg output)
• An overloaded cluster (check
system load, iostat, etc.)
• A bug in the ceph-osd daemon.
• Remove VMs Cloud Solutions
from Ceph Hosts
• Upgrade Kernel
• Upgrade Ceph
• Restart OSDs
• Better Error Handling • Smart Failure Report/detection
15
The Stability problems --optimizations How to detect the failure in a fixed (or short) time
End users are unlikely to wait and would like to involve to fix the problems ASAP.
Time out based solution is hard to control
How shorten the cluster state convergence time
Better nested failure handling
Smart device failure detection
– e.g., smart info, perfcounter of an OSD instance?
How to expose more message about failure types (hints) from the log
How to trouble shooting the problems?
16
Agenda
• A brief introduction
• Ceph in PRC
• The stability challenges and optimizations
• The performance challenges and optimization
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
17
The performance problems – Ceph on All Flash Array--Motivation
Storage Providers are Struggling to achieve the required high performance
There is a growing trend for cloud provider to adopt SSD
– CSP who wants to build EBS alike service for their OpenStack based public/private cloud
Strong demands to run enterprise applications
OLTP workloads running on Ceph
high performance multi-purpose Ceph cluster is the key advantages
Performance is still an important factor
SSD price continue to decrease
18
The performance problems – Ceph on All Flash Array-- Configuration
2x10Gb NIC
Test Environment5x Client Node
• Intel® Xeon™ processor E5-2699 v3 @ 2.3GHz, 64GB mem
• 10Gb NIC
5x Storage Node• Intel Xeon™ processor E5-
2699 v3 @ 2.3 GHz • 128GB Memory• 1x 1T HDD for OS• 1x Intel® DC P3700 800G
SSD for Journal (U.2)• 4x 1.6TB Intel® SSD DC
S3510 as data drive• 2 OSD instances one each
S3510 SSD
CEPH1
MON
OSD1 OSD8…
FIO FIO
CLIENT 1
1x10Gb NIC
Note: Refer to backup for detailed test configuration for hardware, Ceph and testing scripts
FIO FIO
CLIENT 2
FIO FIO
CLIENT 3
FIO FIO
CLIENT 4
FIO FIO
CLIENT 5
CEPH2
OSD1 OSD8…
CEPH3
OSD1 OSD8…
CEPH4
OSD1 OSD8…
CEPH5
OSD1 OSD8…
19
The performance problems – Ceph on All Flash Array--Observations & Analysis
Leveldb became the bottleneck
Single thread leveldb pushed one core to 100% utilization
Omap overhead
Among the 3800+ threads, average ~47 threads are running, ~ 10 pipe threads and ~9 OSD op threads are running, most of OSD op threads are sleep (top –H)
Osd op threads are waiting for throttle of filestore to be released
Disable omap operation can speedup release of filestore throttle, which makes more OSD op thread in running state, average ~ 105 threads are running.
Throughput improved 63%
High CPU consumption
70% CPU utilization of Two high-end Xeon E5 v3 processors (36 cores) with 4 S3510s
Perf showed that most of CPU intensive functions are malloc, free and other system calls
*Bypass omap: Ingore object_map->set_keys in FileStore::_omap_setkeys, for tests only
20
The performance problems – Ceph on All Flash Array--Tuning and optimization efforts
• Up to 16x performance improvement for 4K random read, peak throughput 1.08M IOPS
• Up to 7.6x performance improvement for 4K random write, 140K IOPS
4K Random Read Tunings 4K Random Write Tunings
Default Single OSD Single OSD
Tuning-1 2 OSD instances per SSD 2 OSD instances per SSD
Tuning-2 Tuning1 + debug=0 Tuning2+Debug 0
Tuning-3 Tuning2 + jemalloctuning3+ op_tracker off, tuning fdcache
Tuning-4 Tuning3 + read_ahead_size=16 Tuning4+jemalloc
Tuning-5 Tuning4 + osd_op_thread=32 Tuning4 + Rocksdb to store omap
Tuning-6 Tuning5 + rbd_op_thread=4 N/A
-
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
Default Tuning-1 Tuning-2 Tuning-3 Tuning-4 Tuning-5 Tuning-6
No
rma
lize
d
4K random Read/Write Tunings
4K Random Read 4K random write
21
The performance problems – Ceph on All Flash Array--Tuning and optimization efforts
1.08M IOPS for 4K random read, 144K IOPS for 4K random write with tunings and optimizations
1
2
4
8
16
32
64
128
0 2 0 0 0 0 0 4 0 0 0 0 0 6 0 0 0 0 0 8 0 0 0 0 0 1 0 0 0 0 0 0 1 2 0 0 0 0 0 1 4 0 0 0 0 0
LATE
NC
Y(M
S)
IOPS
RANDOM READ PERFORMANCERBD # SCALE TEST
4K Rand.R 8K Rand.R 16K Rand.R 64K Rand.R63K 64k Random Read
IOPS @ 40ms
300K 16k Random Read IOPS @ 10 ms
1.08M 4k Random Read IOPS @ 3.4ms500K 8k Random
Read IOPS @ 8.8ms
0
2
4
6
8
10
0 2 0 0 0 0 4 0 0 0 0 6 0 0 0 0 8 0 0 0 0 1 0 0 0 0 0 1 2 0 0 0 0 1 4 0 0 0 0 1 6 0 0 0 0
LATE
NC
Y(M
S)
IOPS
RANDOM WRITE PERFORMANCERBD # SCALE TEST
4K Rand.W 8K Rand.w 16K Rand.W 64K Rand.W
23K 64k Random Write IOPS @ 2.6ms
88K 16kRandom Write IOPS @ 2.7ms
132K 8k Random Write IOPS @ 4.1ms
144K 4kRandom Write IOPS @ 4.3ms
Excellent random read performance and Acceptable random write performance
The performance problems – Ceph on All Flash Array--Ceph*: SSD Cluster vs. HDD Cluster
• Both journal on PCI Express*/NVM Express* SSD
• 4K random write, need ~ 58x HDD Cluster (~ 2320 HDDs) to get same performance
• 4K random read, need ~ 175x HDD Cluster (~ 7024 HDDs) to get the same performance
ALL SSD Ceph helps provide excellent TCO (both Capx and Opex), not only performance but also space, Power, Fail rate, etc.
Client Node• 5 nodes with Intel® Xeon® processor E5-2699 v3 @ 2.30GHz,
64GB memory• OS : Ubuntu* Trusty
Storage Node• 5 nodes with Intel Xeon processor E5-2699 v3 @ 2.30GHz,
128GB memory • Ceph* Version : 9.2.0, OS : Ubuntu Trusty• 1 x P3700 SSDs for Journal per nodeCluster difference: SSD cluster : 4xS3510 1.6TB for OSD per nodeHDD cluster : 10 x STAT 7200RPM HDDs as OSD per node
22
0
50
100
150
200
4K Rand.W 4K Rand.R
No
rma
lize
d
Performance Comparison
HDD SSD
~ 58.2
~175.6
23
Agenda
• A brief introduction
• Ceph in PRC
• The performance challenges and optimization
• The stability challenges and optimizations
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
24
Opportunities – Ceph 3D NAND and 3D Xpoint™ Technology3D NAND and 3D Xpoint™ technology is emerging
Can we solve today’s performance problems with Intel® 3D Xpoint™ technology?
– Persistent Memory based KV database
– Persistent Memory based BlueStore backend
– Client side cache – Hyper-converge solutions
How to build Cost-Effective Ceph Storage Solutions with 3D NAND Solid State Drives?
25
NAND Flash vs 3D XPoint™ Technology for Ceph tomorrow
Enable higher capacity OSDs at lower price
3D MLC and TLC NAND
Higher performance, opening up new use cases, DRAM extended, Key/value…
3D XPoint™ Technology
Accelerating Solid State Storage in Computing Platforms
3D NAND2D NAND
32 Tiers
CAPACITY Enables high-density flash devices
COST Achieves lower cost per gigabyte than 2D NAND at maturity
CONFIDENCE 3D architecture increases performance and endurance
26
Intel Continues to Drive Technology
3D Xpoint™ TECHNOLOGYA new class of non-volatile memory Media
1Technology claims are based on comparisons of latency, density and write cycling metrics amongst memory technologies recorded on published specifications of in-market memory products against internal Intel specifications
1000Xfaster
THAN NAND1
1000Xendurance
OF NAND1
10Xdenser
THAN DRAM1
Nand-like densities and dram-like speeds
27
NAND SSDLatency: ~100,000X
Size of Data: ~1,000X
Latency: 1XSize of Data: 1X
SRAM
Latency: ~10 MillionXSize of Data: ~10,000X
HDD
Latency: ~10XSize of Data: ~100X
DRAM3D XPoint ™
Memory MediaLatency: ~100X
Size of Data: ~1,000X
STORAGE
MEMORYMEMORYTechnology claims are based on comparisons of latency, density and write cycling metrics amongst memory technologies recorded on published specifications of in-market memory products against internal Intel specifications.
3D Xpoint™ TECHNOLOGYBreaks the Memory Storage Barrier
28
Intel® Optane™ (prototype) vs Intel® SSD DC P3700 Series at QD=1
Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Server Configuration: 2x Intel® Xeon® E5 2690 v3 NVM Express* (NVMe) NAND based SSD: Intel P3700 800 GB, 3D Xpoint based SSD: Optane NVMe OS: Red Hat* 7.1
29
Storage Hierarchy Tomorrow
Hot
3D XPoint™ DIMMs
NVM Express* (NVMe) 3D XPoint™ SSDs
Warm
NVMe 3D NAND SSDs
Cold
NVMe 3D NAND SSDs
SATA or SAS HDDs
~6GB/s per channel~250 nanosecond latency
PCI Express* (PCIe*) 3.0 x4 link, ~3.2 GB/s<10 microsecond latency
SATA* 6GbpsMinutes offline
DRAM: 10GB/s per channel, ~100 nanosecond latency
PCIe 3.0 x4, x2 link<100 microsecond latency
Comparisons between memory technologies based on in-market product specifications and internal Intel specifications.
Server side and/or AFABusiness ProcessingHigh Performance/In-Memory AnalyticsScientificCloud Web/Search/Graph
Big Data Analytics (Hadoop*)Object Store / Active-ArchiveSwift, lambert, HDFS, Ceph*
Low cost archive
30
31
3D Xpoint™ opportunities – KV database
A new Key-Value DB on Persistent Memory
Alternative of LevelDB or RocksDB in Ceph to speedup metadata operations
Implemented with Intel NVM Library
– Bw-Tree as internal index
– a mapping table that virtualizes both the location and the size of pages
– Delta update to maximize CPU cache hit ratio
– libpmemobj transaction to support KV transaction
– Use libpmemobj library to manage disk space
BlueStore
Persistent-Memory KV
PMEMDevice PMEMDevice
Metadata
Libpmemlib
Libpmemobj
DAX Enabled File System
mm
ap
Load
/sto
re
File
FileFile
AP
I
AP
I
Data
Libpmemblk
32
3D Xpoint™ opportunities – Bluestore backend
Three usages for PMEM device
• Backend of bluestore: raw PMEM block device or file of dax-enabled FS
• Backend of rocksdb: raw PMEM block device or file of dax-enabled FS
• Backend of rocksdb’s WAL: raw PMEM block device or file of DAX-enabled FS
Two methods for accessing PMEM devices
• libpmemblk
• mmap + libpmemlib
BlueStore
Rocksdb
BlueFS
PMEMDevice PMEMDevice PMEMDevice
Metadata
Libpmemlib
Libpmemblk
DAX Enabled File System
mm
apLo
ad/s
tore
mm
apLo
ad/s
tore
File
FileFile
AP
I
AP
I
Data
3D Xpoint™ opportunities – Client side cache Overview
• Client Side cache: caching on compute node
• Local read cache and distributed write cache
• Independent cache layer between RBD and Rados
• Extensible Framework
• Pluggable design/cache policies
• General caching interfaces: Memcached like API
• Data Services
• Deduplication, Compression when flushing to HDD
• Value add feature designed for 3D Xpoint™ device
• Log-structure object store for write cache
OSD OSD OSD OSD OSD OSD OSD OSD OSD OSD OSD OSD
Compute Node
ReadCache
WriteCache
VM1 VM2 VMn
Compute Node
ReadCache
WriteCache
VM1 VM2 VMn
meta meta
33
PM PM PM PM PM PM PM PM
dedup/compression
Caching tier
HDD Capacity tier
34
3D NAND - Ceph cost effective solution
Enterprise class, highly reliable, feature rich, and cost effective AFA solution
NVMe SSD is today’s SSD, and 3D NAND or TLC SSD is today’s HDD
– NVMe as Journal, high capacity SATA SSD or 3D NAND SSD as data store
– Provide high performance, high capacity, a more cost effective solution
– 1M 4K Random Read IOPS delivered by 5 Ceph nodes
– Cost effective: 1000 HDD Ceph nodes (10K HDDs) to deliver same throughput
– High capacity: 100TB in 5 nodes
W/ special software optimization on filestore and bluestore backend
Ceph Node
S3510 1.6TB
S3510 1.6TB
S3510 1.6TB
S3510 1.6TB
P3700M.2 800GB
Ceph Node
P3520 4TB
P3520 4TB
P3520 4TB
P3520 4TB
P3700 & 3D Xpoint™ SSDs
3D NAND
P3520 4TB
3D Xpoint™
35
Agenda
• A brief introduction
• Ceph in PRC
• The performance challenges and optimization
• The stability challenges and optimizations
• The opportunities – Ceph with future 3D NAND and 3D XPoint™ Technology
• Summary
36
Summary
Ceph becomes more popular w/ growth of OpenStack in China
Challenge still exists w/ stability and performance
Intel is working with community to improve Ceph performance
37
Acknowledgements
This is a joint team work.
Thanks for the contributions of Chendi Xue, Jianpeng Ma, Xinxin Shu and Yuan Zhou.
Thanks for the contributions of ceph.org.cn from Hang Geng @XiangCloud.
38
Legal Notices and Disclaimers
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.
No computer system can be absolutely secure.
Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.
Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K.
The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.
Intel, Xeon and the Intel logo are trademarks of Intel Corporation in the United States and other countries.
*Other names and brands may be claimed as the property of others.
© 2016 Intel Corporation.
Backup
39
NVM Library
40
Source: Typical NVDIMM Software Architecture, http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf
41
[global]debug paxos = 0/0debug journal = 0/0debug mds_balancer = 0/0debug mds = 0/0mon_pg_warn_max_per_osd =
10000debug lockdep = 0/0debug auth = 0/0debug mds_log = 0/0debug mon = 0/0debug perfcounter = 0/0debug monc = 0/0debug rbd = 0/0debug throttle = 0/0debug mds_migrator = 0/0debug client = 0/0debug rgw = 0/0debug finisher = 0/0debug journaler = 0/0debug ms = 0/0debug hadoop = 0/0debug mds_locker = 0/0debug tp = 0/0debug context = 0/0debug osd = 0/0debug bluestore = 0/0debug objclass = 0/0debug objecter = 0/0
osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k,delaylogosd_mkfs_type = xfsfilestore_queue_max_ops = 5000osd_client_message_size_cap = 0objecter_infilght_op_bytes = 1048576000ms_dispatch_throttle_bytes = 1048576000osd_mkfs_options_xfs = -f -i size=2048filestore_wbthrottle_enable = Truefilestore_fd_cache_shards = 64objecter_inflight_ops = 1024000filestore_queue_committing_max_bytes = 1048576000osd_op_num_threads_per_shard = 2filestore_queue_max_bytes = 10485760000osd_op_threads = 32osd_op_num_shards = 16filestore_max_sync_interval = 10filestore_op_threads = 16osd_pg_object_context_cache_count = 10240journal_queue_max_ops = 3000journal_queue_max_bytes = 10485760000journal_max_write_entries = 1000filestore_queue_committing_max_ops = 5000journal_max_write_bytes = 1048576000osd_enable_op_tracker = Falsefilestore_fd_cache_size = 10240osd_client_message_cap = 0
Ceph All Flash Tunings debug log = 0debug filer = 0/0debug mds_log_expire = 0/0debug crush = 0/0debug optracker = 0/0debug rados = 0/0debug heartbeatmap = 0/0debug buffer = 0/0debug asok = 0/0debug objectcacher = 0/0debug filestore = 0/0debug timer = 0/0mutex_perf_counter = Truerbd_cache = Falsems_crc_header = Falsems_crc_data = Falseosd_pool_default_pgp_num = 32768osd_pool_default_size = 2rbd_op_threads = 4cephx require signatures = Falsecephx sign messages = Falseosd_pool_default_pg_num = 32768throttler_perf_counter = Falseauth_service_required = noneauth_cluster_required = noneauth_client_required = none