Download - Jvm goes big_data_sfjava
![Page 1: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/1.jpg)
JVM goes BigData
srisatish.ambati AT gmail.comDataStax/OpenJDK4/12/2011@srisatish
![Page 2: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/2.jpg)
Motivation
• A compendium of recent jvm scale issues while working with big data.
• This talk will not have details on big data.
• Thanks Sasa!
![Page 3: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/3.jpg)
Trail Ahead
synchronizedNonblocking Hashmap A state transition viewCollectionsSerializationUUIDGarbage Collection The free parameters! Generations, Promotion, Fragmentation OffheapQuestions & asynchronous IO
![Page 4: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/4.jpg)
tools of trade
• What the JVM is doing:– dtrace, hprof, introscope, jconsole, visualvm, yourkit,
gchisto, zvision
• Invasive JVM observation tools:– bci, jvmti, jvmdi/pi agents, logging
• What the OS is doing:– dtrace, oprofile, vtune, perf
• What the network/disk is doing:– ganglia, iostat, lsof, nagios, netstat, tcpdump
![Page 5: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/5.jpg)
![Page 6: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/6.jpg)
synchronized
under the hood– Fast path for nocontention thin lock– Bias threads to lock or bulk revoke bias– Store free biasing
![Page 7: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/7.jpg)
JMM: happensbefore, causalityPartial ordervolatilePiggybackingFutureTaskBlockingQueuejsr133
![Page 8: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/8.jpg)
* Java Concurrency in Practice, Brian Goetz
![Page 9: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/9.jpg)
java.util.concurrent also holds locks!
![Page 10: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/10.jpg)
Tomcat under concurrent load!
![Page 11: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/11.jpg)
Nonblocking collections: Amdahl's > Moore's!
State, Actions – key/value pairs!get, put, delete, _resize
ByteArray to hold DataConcurrent writes: using CAS
No locks, no volatileMuch faster than locking under heavy load
Directly reach main data array in 1 step
Resize as neededCopy Array to a larger Array on demand. Post updates
![Page 12: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/12.jpg)
Death & Taxes: Java Overheads!
• Cost of an 8char String?
• Cost of 100entry TreeMap<Double,Double> ?
8bhdr
12bfields
4bptr
4bpad
8bhdr
4blen
16bdata
A: 56 bytes, or a 7x blowup
48bTreeMap
40bTreeMap$Entry
16bDouble
16bDouble
A: 7248 bytes or a ~5x blowup
![Page 13: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/13.jpg)
yourkit: memory profile
![Page 14: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/14.jpg)
Which collection: Mozart or Bach?
Concurrency: Nonblocking HashMap Google Collections
Overheads Watch out for perelement costs! Primitives can be hard to manage!
Sparse collections Average collection size in enterprise is ~3
![Page 15: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/15.jpg)
java.io.Serializable is S.L..O.…WTrue to platform Use “transient” ObjectSerialField[] Avro Google Protocol Buffers, Externalizable + byte[] Roll your own
serializable
![Page 16: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/16.jpg)
ser+deser smaller is better
https://github.com/eishay/jvmserializers.git
![Page 17: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/17.jpg)
avro
• Schema– No per datum overheads– Optional code gen
• Types are runtime• Untagged data• No manuallyassigned field IdsCons:• Schema mismatches• Runtime only checks
![Page 18: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/18.jpg)
googleprotobuffer
• Define message format in .proto file
• All data in key/value pairs• Generate sources• .builder for each class
with getter/setter
![Page 19: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/19.jpg)
thrift
• Type, Transport, Protocol, Version, Processors
• Separation of structure from protocol & transport
• TCompactProtocol, etc– tag/data, compression
• TSocket, TfileTransport, etc• colocated clients & servers
![Page 20: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/20.jpg)
UUIDjava.util.UUID is slow
● dominated by sha_transform costs● Leachsalz (128bit)
Turns out that default PRNG (via SecureRandom)Uses /dev/urandom for seed initialization Djava.security.egd=file:/dev/urandom
● PRNG without file is atleast 20%40% better.Use TimeUUIDs where possible – much faster
Alternatives: JUG – java.uuid.generator, com.eaio.uuid
~10x faster
http://github.com/cowtowncoder/javauuidgenerator
http://jug.safehaus.org/
http://johannburkard.de/blog/programming/java/JavaUUIDgeneratorscompared.htm
![Page 21: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/21.jpg)
/**
* Returns a {@code String} object representing this {@code UUID}.
*
* <p> The UUID string representation is as described by this BNF:
* <blockquote><pre>
* {@code
* UUID = <time_low> "-" <time_mid> "-"
* <time_high_and_version> "-"
* <variant_and_sequence> "-"
* <node>
* time_low = 4*<hexOctet>
* time_mid = 2*<hexOctet>
* time_high_and_version = 2*<hexOctet>
* variant_and_sequence = 2*<hexOctet>
* node = 6*<hexOctet>
* hexOctet = <hexDigit><hexDigit>
* hexDigit =
* "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
* | "a" | "b" | "c" | "d" | "e" | "f"
* | "A" | "B" | "C" | "D" | "E" | "F"
* }</pre></blockquote>
*
* @return A string representation of this {@code UUID}
*/
public String toString() {
return (digits(mostSigBits >> 32, 8) + "-" +
digits(mostSigBits >> 16, 4) + "-" +
digits(mostSigBits, 4) + "-" +
digits(leastSigBits >> 48, 4) + "-" +
digits(leastSigBits, 12));
}
Leachsalz UUID
![Page 22: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/22.jpg)
PerfTop: 1485 irqs/sec kernel:18.6% exact: 0.0% [1000Hz cycles], (all, 8 CPUs)
samples pcnt function DSO _______ _____ ________________________________________________________________
1882.00 26.3% intel_idle [kernel.kallsyms] 1678.00 23.5% os::javaTimeMillis() libjvm.so 382.00 5.3% SpinPause libjvm.so 335.00 4.7% Timer::ImplTimerCallbackProc() libvcllx.so 291.00 4.1% gettimeofday /lib/libc2.12.1.so 268.00 3.7% hpet_next_event [kernel.kallsyms] 254.00 3.6% ParallelTaskTerminator::offer_termination(TerminatorTerminator*) libjvm.so PerfTop: 1656 irqs/sec kernel:59.5% exact: 0.0% [1000Hz cycles], (all, 8 CPUs)
samples pcnt function DSO _______ _____ ________________________________________________________________ 6980.00 38.5% sha_transform [kernel.kallsyms] 2119.00 11.7% intel_idle [kernel.kallsyms] 1382.00 7.6% mix_pool_bytes_extract [kernel.kallsyms] 437.00 2.4% i8042_interrupt [kernel.kallsyms] 416.00 2.3% hpet_next_event [kernel.kallsyms] 390.00 2.2% extract_buf [kernel.kallsyms] 376.00 2.1% ThreadInVMfromNative::~ThreadInVMfromNative() libjvm.so 321.00 1.8% T.3542 libjvm.so 298.00 1.6% __ticket_spin_lock [kernel.kallsyms] 296.00 1.6% Timer::ImplTimerCallbackProc() libvcllx.so 255.00 1.4% Unsafe_GetInt libjvm.so
![Page 23: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/23.jpg)
summary
TimebasedUUIDs vs. UUIDsuse ~4 times less kernel time on creation!No SHA library calls!optimized toString()Much faster than standard java.util.UUID Better Instructions per clocks as well. If on EC2: Watch out for noncacheable file access to /dev/urandom!
![Page 24: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/24.jpg)
String theory of Java!
byte[] vs. char[]If ver > jdk16u21 try XX:+UseCompressedStringsAppend performance (gc) differs: Strings vs. StringBufferscom.google.common.base.Joiner
• Join text for cheap, • skipNulls or useForNulls()
com.google.common.base.Splitter
![Page 25: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/25.jpg)
“Null References: A billion dollar mistake” C.A.R Hoare
“I call it my billiondollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.” qconlondon, '09
![Page 26: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/26.jpg)
Best Practices:Garbage Collection
![Page 27: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/27.jpg)
verbose:gc
GC Logs are cheap even in production
Xloggc:gc.log
XX:+PrintGCDetails
XX:+PrintGCTimeStamps XX:+PrintTenuringDistribution
A bit expensive/obscure ones: XX:PrintFLSStatistics=2 XX:CMSStatistics=1
XX:CMSInitiationStatistics XX:+PrintFLSCensus
![Page 28: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/28.jpg)
Three free parameters
Allocation Rate: your workload!Size: defines runway! Live Set, memoryPause times: Stoppages!
![Page 29: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/29.jpg)
Four free parameters
Allocation Rate: your application load!Size: defines runway! Live Set, system memoryPause times: Stoppages!
(fourth: Overheads of GC – Space & CPU.)
![Page 30: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/30.jpg)
Part I: Sizingto be Xmx == Xms or not?Young generation:
Use Xmn for predictable performance
edensurvivor spaces
new Object()survivor ratio
jvm allocates
TenuringThreshold
promotion
old gen
![Page 31: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/31.jpg)
Part II: Pick a collector!
Serial GC – Serial new + Serial OldParallel GC (default) Parallel Scavenge + Serial OldUseParallelOldGC : Parallel Scavenge + Parallel OldUseConcurrentMarkSweep: ParNew, CMS Old, Serial OldG1/Experimental
![Page 32: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/32.jpg)
Reading GC logs – a topic/tool
Full GC is STWInitial Mark, Rescan/WeakRef/Remark are STWLook for promotion failuresLook for concurrent mode failures
![Page 33: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/33.jpg)
... 995.330: [CMSconcurrentmark: 0.952/1.102 secs] [Times: user=3.69 sys=0.54, real=1.10 secs] 995.330: [CMSconcurrentprecleanstart]995.618: [CMSconcurrentpreclean: 0.279/0.287 secs] [Times: user=0.90 sys=0.20, real=0.29 secs] 995.618: [CMSconcurrentabortableprecleanstart]995.695: [GC 995.695: [ParNew (promotion failed)Desired survivor size 41943040 bytes, new threshold 1 (max 1) age 1: 29826872 bytes, 29826872 total: 720596K>703760K(737280K), 0.4710410 secs]996.166: [CMS996.317: [CMSconcurrentabortablepreclean: 0.218/0.699 secs] [Times: user=1.39 sys=0.10, real=0.70 secs] (concurrent mode failure): 4100132K>784070K(5341184K), 4.7478300 secs] 4780154K>784070K(6078464K), [CMS Perm : 17033K>17014K(28400K)], 5.2191410 secs] [Times: user=5.70 sys=0.01, real=5.22 secs]...
![Page 34: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/34.jpg)
Tuning CMS
Don’t promote too often! Frequent promotion causes fragmentation
(avoid never tenure) TenuringThreshold
Size the generations Min GC times are a function of Live Set
Old Gen should host steady state comfortably
Avoid CMS Initiating heuristic XX:+UseCMSInitiationOccupanyOnly
Use Concurrent for System.gc() XX:+ExplicitGCInvokesConcurrent
![Page 35: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/35.jpg)
GC Threads
Parallelize on multicores XX:ParallelGCThreads=4
(default: derived from # of cpus on system)
*8 + (n5)/8
XX:ParallelCMSThreads=4
(default: derived from # of parallelgcthreads)
Strategy A:
Tune min gcs & let appl data in eden
![Page 36: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/36.jpg)
Did someone ask about defaults? if (FLAG_IS_DEFAULT(ParallelGCThreads)) { assert(ParallelGCThreads == 0, "Default ParallelGCThreads is not 0"); // For very large machines, there are diminishing returns // for large numbers of worker threads. Instead of // hogging the whole system, use a fraction of the workers for every // processor after the first 8. For example, on a 72 cpu machine // and a chosen fraction of 5/8 // use 8 + (72 8) * (5/8) == 48 worker threads. unsigned int ncpus = (unsigned int) os::active_processor_count(); return (ncpus <= switch_pt) ? ncpus : (switch_pt + ((ncpus switch_pt) * num) / den); } else { return ParallelGCThreads; }
![Page 37: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/37.jpg)
Fragmentation
Performance degrades over timeInducing “Full GC” makes problem go awayFree memory that cannot be used
Round off errorsReduce occurrenceUse a compacting collectorPromote less oftenUse uniform sized objects
![Page 38: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/38.jpg)
Not enough large contiguous space for promotion
Small objects still can fit in the holes!Compaction – stop the world.Unsolved on Oracle/Sun Hotspot Azul Systems Pauseless JVM.
![Page 39: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/39.jpg)
JRockit Mission Control
![Page 40: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/40.jpg)
Example
Application suddenly transitions to backtoback full gcs.
Cannot use free mem – too many holes!
![Page 41: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/41.jpg)
Tools
• GCHisto• jconsole• VisualVM/VisualGC• Logs• Thread dumps• yourkit memory profile, snapshots
![Page 42: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/42.jpg)
GCSpy
![Page 43: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/43.jpg)
Gone 0xff the heap !!
ByteBuffer.allocateDirect(16 * 1024 * 1024)Also can be mapped memory of a file regionStore longlived objects outside jvm Managed by native i/o ops.JNA: dynamically load & call native libraries
without compile time decl like JNIWorks for limited use cases in the lab. Ex: Terracotta, Hbase, Cassandra
![Page 44: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/44.jpg)
Gone 0xff the heap ?Issues to consider:No clear api to deallocate from this region
● See jbellis patch to JNA179 for FreeableBufferObject cleanup relegated to finalization Single finalizer thread, Bug ID: 4469299Behind WeakReference processing in jdk16u21
Workaround:XX:MaxDirectMemorySize=<size> Manually Trigger System.gc() to avoid “leak”
![Page 45: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/45.jpg)
Virtually there!
Ballooning driver for Memory: Disable it!Time (TSC) issue! It's relative!Scheduling when # of threads > # of vcpus.. Tickless _nohz kernelGC Thread starvation = STW pauseslarge ec2 instances are not all equal..DirectPathIO & vtd, rvi – Watch out for Sockets!Tools: Performance counters still not virtualized!
![Page 46: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/46.jpg)
summary
• JVM is still the most popular platform for deployment for the new languages!
• JVM heartburn around scale!– Serialization– UUID– Object overhead– Garbage Collection– Hypervisor
![Page 47: Jvm goes big_data_sfjava](https://reader033.vdocuments.site/reader033/viewer/2022061218/54b7a3b24a795993718b478b/html5/thumbnails/47.jpg)
References
Chris Wimmer, Chris Wimmer, http://wikis.sun.com/display/HotSpotInternals/Synchronizationhttp://wikis.sun.com/display/HotSpotInternals/SynchronizationRussel & Detlefs Russel & Detlefs http://www.oracle.com/technetwork/java/biasedlockingoopsla2006wp149958.pdfhttp://www.oracle.com/technetwork/java/biasedlockingoopsla2006wp149958.pdfGoogle Protocol Buffers Google Protocol Buffers http://code.google.com/p/protobufhttp://code.google.com/p/protobufThrift Thrift http://incubator.apache.org/thrift/static/thrift20070401.pdfhttp://incubator.apache.org/thrift/static/thrift20070401.pdfLeachSalz Variant of UUID LeachSalz Variant of UUID http://www.upnp.org/resources/draftleachuuidsguids00.txthttp://www.upnp.org/resources/draftleachuuidsguids00.txtHans Boehm, Hans Boehm, http://www.hpl.hp.com/personal/Hans_Boehm/gc/complexity.htmlhttp://www.hpl.hp.com/personal/Hans_Boehm/gc/complexity.htmlBrian Goetz, JSR133 Brian Goetz, JSR133 http://www.ibm.com/developerworks/java/library/jjtp03304/http://www.ibm.com/developerworks/java/library/jjtp03304/GCSpy GCSpy http://www.cs.kent.ac.uk/projects/gc/gcspy/http://www.cs.kent.ac.uk/projects/gc/gcspy/Understanding GC logs Understanding GC logs http://blogs.sun.com/poonam/entry/understanding_cms_gc_logshttp://blogs.sun.com/poonam/entry/understanding_cms_gc_logs
Cliff Click's http://sourceforge.net/projects/highscalelib/Cliff Click's http://sourceforge.net/projects/highscalelib/