scalecamp 2009 - last.fm vs xbox
DESCRIPTION
A brief talk I gave at Scale Camp, hosted by The Guardian.http://www.scalecamp.org.ukTRANSCRIPT
Scaling war stories
• Last.fm vs Xbox360
• Quick and hacky profiling
• Discuss
Wednesday, 26 May 2010
What happens when scale is forced on you?• 24 Million Xbox360 users
• Estimated ~5 times peak traffic increase
• No control over source of client
• External partner, with their own deadlines
• Must be reliable at launch. Err? (The PR problem)
Wednesday, 26 May 2010
A dose of reality
• You won’t have the resources you’d like
• People (skills, availability,
• Budget (new servers would be nice)
• Time (more of this would be nice too)
Wednesday, 26 May 2010
B-b-but...
• It’s not an ideal world
• I’m not a systems developer (I spend half my time on the front end)
• Quite hacky. Open to suggestions, alternatives, your experiences
Wednesday, 26 May 2010
HTTP profiling
• Had a development Xbox
• Stuck a proxy box in the middle
• Profiled traffic with Wireshark
Wednesday, 26 May 2010
Wireshark
Wednesday, 26 May 2010
Making a list...
• API calls made
• In order, as uses cases
• Bad uses that can be factored out
• What breaks when call X is disbabled
• Proxy routed to a dev version of API
Wednesday, 26 May 2010
Xbox Beta
• Launched to 1,000 people (At midnight on a Friday). Then to a further 9,000 people
• Gave a nice real usage sample
• Great, the most common calls are... completely pointless.
• frequent != expensive (necessarily)
Wednesday, 26 May 2010
Custom Logging gen-time
LogFormat "%h %l %u %t \"%r\" %>s %b \"% \{Referer}i\" \"%{User-Agent}i\" %{gentime}n"
<?phpapache_note("gen-time", $x);
Wednesday, 26 May 2010
Abusing bash
awk 'BEGIN \ {FS="\t"} {a[$2]++;b[$2]=b[$2]+$1} \END \{for (i in a) printf("%.2f %s\n", b[i]/a[i]/1000000, i)} '
Text
Wednesday, 26 May 2010
Stats
Count Total Average
71638 trackgetInfo 15789 0.2253941 artistgetImages 19647 0.3615150 radiogetPlaylist 6962 0.46 7308 librarygetArtists 2402 0.33 5020 usergetRecentStations 1674 0.33 4979 adsgetvideos 1810 0.36 4205 radiotune 923 0.22 3155 tracklove 0 0.00 1507 artistgetinfo 440 0.29 1258 usergetRecommendedArtists 1457 1.16 1135 usergetInfo 171 0.15 1130 geogetTopArtists 575 0.51 1128 radiogamerstations 298 0.26 1102 taggettopartists 1488 1.35 1021 trackban 0 0.00 1006 usergetLovedTracks 271 0.27 340 libraryaddArtist 0 0.00 206 authgetMobileSession 38 0.19 136 usersignUp 32 0.24 123 userterms 16 0.13
Wednesday, 26 May 2010
zomg, graphs!
Wednesday, 26 May 2010
Wednesday, 26 May 2010
Wednesday, 26 May 2010
Wednesday, 26 May 2010
Profiling with Xdebug• Profile PHP, stack traces, memory usage, etc
• http://xdebug.org
extension=xdebug.soxdebug.max_nesting_level=500xdebug.default_enable=0xdebug.profiler_enable_trigger=1xdebug.profiler_append=1xdebug.trace_output_name=%R.cache.out
Wednesday, 26 May 2010
Making sense of it
Wednesday, 26 May 2010
Contingencies
• From invisible, to user affecting
• Disable DB writes, speed up and prevent blocking on failure
• Suppress badly architected Flash Player (and less evil taste-o-meter)
• Kill % of method calls at perlbal
Wednesday, 26 May 2010
Wednesday, 26 May 2010
Monitoring porn
• Launch control with: 3 projectors, 5 x 24” monitors, a dozen dev/sysops with laptops
• A moose, a cow and a fail train
Wednesday, 26 May 2010