google developer day 2007 - nestoria: from mash-up to business
DESCRIPTION
A case study of Nestoria. Discuss the difference between a simple mash-up and a business that runs on mash-ups. Also touch on some challenges we have faced.TRANSCRIPT
this space left intentionally blank
why perl is awesome andall other languages suck
Mike [email protected]
from mash-up to business
Mike Astle - Lokku Ltd.Google Developer Day31 May 2007 - London
i like questions
Milestones
• Start dev April 2006• Launch London June 2006• Launch UK Sept 2006• Launch Spain May 2007
Numbers
• In the UK, have 850,000 listingsrefreshed continually
• Boils down to about 500,000 properties• 100,000+ users per month• 4 programmers• No dedicated operations staff
Houses on a map…so what?
this is harder than it looks
The Plan
• Talk about some of the technicaldecisions we made at Nestoria
• Go over a few challenges that we havefaced
• Hopefully there are lessons to learnfrom our experiences
first, you design
Things to worry about
• Architecture• Services vs. APIs• How to scale?• i18n• Future products
then you choose a platform
MS vs. Everything Else
Java?
That Leaves LAMP
PHP?
Ruby?
LAMR?
Python?
Perl
Perl
Utility Belt == CPAN
Apache
MySQL
GNU/Linux(Debian)
now choose adevelopment environment
Source Control
• SVN…duh• Does anybody use CVS anymore?• Love the trac
Coders
• A few veterans• As many interns / junior coders as fit in
the room
Highly Collaborative(but not XP or Agile)
Emphasis
• Collective decision-making• Individual responsibility for deadlines• Many small success rather than few
large projects
Why write tests?
• Find bugs, of course• Increase confidence to make changes
to unfamiliar systems or libraries• Promote good design / expose bad
design
Design
• Informal process (no UML, minimaldesign patterns)
• Concentrate on the things that will reallyhurt - e.g. i18n
• Not everything has to be an object
Documentation
• POD for interfaces• twiki for systems• Schedule regular refreshers
Outsource
• If you can spec it faster than you canwrite it, consider farming it out
• Many, many site full of hungry coders• Have had good results with
scriptlance.com but there are lots ofothers out there
how to deploy?
Virtualization?
• Sounds like voodoo to me• Extra layer to worry about• Easy enough to do everything in user
space
Wigwam
• 12 subdomains• Live data• 22 services• Many playpens per box• Releases several times a week
big ISP vs. small ISP
• Bigger - more consistent• Smaller - more personal• Bigger - discrete choices• Smaller - continuous advice
• We went small
Hardware
• Know your downtime limits• Be reasonable!• Load balancing - expensive and
complex - we do it with DNS through avendor
• Have a failure plan
TheHawkeyePrinciple
but is anybody using it?
Logging
• Need your own, and it needs to beflexible
• Use the free tools as a sanity check
Reporting
• An important area of expertise, not anannoying afterthought
• Glossary• Every mean needs a variance• A/B Testing (a.k.a Bucket Testing) is
hard to do right
Also…
• Build tools for ad hoc queries, they>will< get used
• Keep an audit trail of changes• Effective tools for drilling down on
sessions allows non-techs to helpdebugging
problems we have solved
http://www.nestoria.co.uk/covent-garden/flat/buy/bedrooms-2/maxprice-
1000000/minprice-250000/bathrooms-2
• Alderly Edge => Alderley Edge• Bayswater London => Bayswater• Berkhampsted => Berkhamsted• Birminham => Birmingham• Bletchingly => Bletchingley• Broxbourne Station => Broxbourne• Cambuslank => Cambuslang• Canary Warf => Canary Wharf• Claphamjuction => Clapham Junction• Covent Garden London => Covent
Garden
Geocoding
• Transform address into lat/longcoordinates
• 1,000+ calls a minute• Very high data variance (few cache hits)• Royal Mail data for UK• Teleatlas data for Spain
• Albert Villas, Gilbert Mews, LEIGHTON BUZZARD,• Bedfordshire, LU7 1NF• Lake Lock Drive, Wakefield West Yorkshire• Church Lane, London• C ELMINGTON ROAD, LONDON• Fountain Road (Flat 2), Edgbaston• London SE18, UK• Lee, London, UK• Free Trade Wharf, 340 The Highway,
Wapping/Limehouse London• Stunning 3 Storey Contemporary Barn Conversion In
The Popular Village Of Utkinton• A.4-1 GREAT NORTHERN TOWER,WATSON
STREET,CITY CENTRE,• MANCHESTER GREATER MANCHESTER• 15, ASTLEY HOUSE, LONDON
(((^[BEGLMNS][1-9]\d?) | (^W[2-9] ) | ( ^( A[BL] | B[ABDHLNRST] |C[ABFHMORTVW] | D[ADEGHLNTY] | E[HNX] | F[KY] | G[LUY] | H[ADGPRSUX] |I[GMPV] | JE | K[ATWY] | L[ADELNSU] | M[EKL] | N[EGNPRW] | O[LX] |P[AEHLOR] | R[GHM] | S[AEGKL-PRSTWY] | T[ADFNQRSW] | UB | W[ADFNRSV] |YO | ZE ) \d\d?) | (^W1[A-HJKSTUW0-9]) | (( (^WC[1-2]) | (^EC[1-4]) |(^SW1) ) [ABEHMNPRVWXY] ) ) (\s*)? ([0-9][ABD-HJLNP-UW-Z]{2}))
what next?
Future Projects
• Expose more data via API (e.g.historical aggregate data)
• Implicit personalisation (the systemlearns from users)
• More international expansion• Systemic testing to improve user
experience
like what you see?
http://www.lokku.com/jobs
thanks!
[email protected]://www.lokku.com
http://www.nestoria.co.ukhttp://www.nestoria.es
http://www.lokku.com/lpw06_geocoding/http://www.lokku.com/lpw06_wigwam/