foursquare com scala lift
TRANSCRIPT
foursquare.com & scala/lift
Harry Heymann 1/11/2010
Background
foursquare.com is a "mobile social network"Users "check in" at locations to inform their friends of where they are, to participate in a game, and use the sytem to gather information on what's going on around themYou guys checked in here right? :-)
Background
foursquare.com originally developed in PHP (running on Apache, using MySQL as a datastore)Written by someone who wasn't an engineer. Poor code quality.A rewrite was clearly required to move forward, but the choice of language and toolkit was wide open anything was possibleI chose to use Scala & The Lift Web Framework
Why Scala/Lift
I had a strong background in Java so the switch to Scala was easyI like type and compiled code, so something like Python/Django or Ruby/Rails was never seriously consideredJava/Wicket would have been an obvious choice at this point, but experimenting with it for a little bit left me cold. I found it to be not as typesafe as one would want (too many things were just Objects) and the lack of first level functions clearly hindered the expressiveness of the toolkit
Why Scala/Lift
Scala felt natural. I'd been programming in Java using a very functional/immutable data structures way for a while and Scala just made that easier and more naturalI was very very impressed with the ease of writing AJAX using Lift. Just inline the function to run when a control is clicked in the client and the framework takes care of the rest. No HTTP to worry aboutI wanted to do something a little fun/different. It's the sort of decision that can be made at a startup without the institutional pressures of a larger company.
The Rewrite
Started (by my self) on day one of the job.Sessions shared between Lift/PHP code using the database for sessionid -> userid mapping. Never really worked 100% perfectly. Scala code serving live pages in production by day 3.90% complete rewrite took 90 days.Last 10% (which required changes on the iPhone client, thus drastically slowing things down) took an additional 60 days. Completed TODAY with release of version 1.5 of client.
The Rewrite
Lift doesn't play very well with MySQL. We switched to PostgreSQL, and had a disastrous downtime filled weekend fixing various bugs (mostly on the PHP side of things) that cropped up because of differences. If you're starting a Lift project go with PostgreSQL from day 1
A quick AJAX example
"todo_checkbox" -> ajaxCheckbox( userStatus.equals(Full(TipUserStatus.todo)), (b: Boolean) => { val currentUser = User.currentUser.open_! if (b) markTodo(currentUser) else clearStatus(currentUser) val className = if (b) "tip_checked" else "tip_todo_unchecked" SetElemById("tip_todo"+id, Str(className), "className")}, ("id", "tip_todo_checkbox"+id)))
lazy loading in 6 lines
<lift:Util.lazyLoad> <lift:Settings.twitter> <!-- markup for twitter settings) </lift:Settings.twitter></lift:Util.lazyLoad>
lazy loading in 6 lines
def lazyLoad(xhtml: NodeSeq) = { val id = randomString(6) val (name, exp) = ajaxInvoke(() => { SetHtml(id, xhtml) })
<div id={id}> Loading...<img src="/img/ajax_spinner.gif" height="32" width="32" alt="wait"/> {Script(OnLoad(exp.cmd))} </div> }
A quick look at our API Implementation
def dispatch: LiftRules.DispatchPF = { case req@Req(List("api", "v1", "addtip"), _, PostRequest) => () => wrap(req, requireAuth(addTip)) case req@Req(List("api", "v1", "addvenue"), _, PostRequest) => () => wrap(req, requireAuth(addVenue)) case req@Req(List("api", "v1", "cities"), _, GetRequest) => () => wrap(req, cities) // ... }
A quick look at our API Implementation
def wrap(req: Req, f: (Req, Box[User]) => Elem) = { def getResponse() = { MasterAuthenticator.authenticate(req) match { case Full(authResult) => (f(req, Full(authResult.user)), 200) case Empty => (f(req, Empty), 200) } } req.path.suffix match { case "" | "xml" => { val (xml, code) = getResponse XmlCodeResponse(Utility.trim(xml), code) } case "json" => { val (xml, code) = getResponse JsonCodeResponse(xmlToJson(xml), code) } case _ => XmlCodeResponse(<error>Invalid Suffix</error>, 501) }}
A quick look at our API Implementation
def xmlToJson(xml: Elem): JsExp = { Xml.toJson(xml) map { case JField("id", JString(s)) => JField("id", JInt(s.toInt)) case JField("geolat", JString(s)) => JField("geolat", JDouble(s.toDouble)) case JField("geolong", JString(s)) => JField("geolong", JDouble(s.toDouble)) }
JsRaw(Printer.compact(render(json3)))}
A quick look at our API Implementation
def cities(req: Req, currentUser: Box[User]) = { val geolat = getAttr(req, "geolat", asDouble) val geolong = getAttr(req, "geolong", asDouble) val cities = (geolat, geolong) match { case (Full(lat), Full(long)) => City.activeCities(currentUser, Geolocation.sortOn(lat, long)) case _ => City.activeCities(currentUser, City.alphaSort) } <cities>{cities.flatMap(GenXml(_))}</cities> }}
A quick look at our API Implementation
def apply(city: City): Elem = { <city> <id>{city.id.is}</id> <name>{city.displayName.is}</name> <shortname>{city.shortName.is}</shortname> <timezone>{city.tz.is}</timezone> <geolat>{city.geolat}</geolat> <geolong>{city.geolong}</geolong> </city> }
Upsides
Virtually every page on the site is better, more interactive, and prettier than it was before.Writing AJAX style code actually easier in Lift that writing traditional forms (which isn't to say that handling traditional forms is hard). All web programming should work like this.New code is smaller (fewer lines), easier to understand, has fewer bugs, and more features. "Four stars to @foursquare - 1st site in a while I have taken a good look at that didn't have a single security issue (that I could find)" - @rasmus
Downsides (with apologies to dpp)
Lift has stateful servers. Ultimately this will lead to challenges in production, though it hasn't been too bad so far.Mapper (the Lift ORM layer) is still fairly immature, and comes with a fair number of quirksThe Lift SiteMap is heavier weight than we need. A simpler version for sites that don't need Menu functionality would be helpful.As we scale up, we're likely to hit strange/unexpected issues that no one has seen before. A more commonly used environment might come with more easily accessible global knowledge. But, the Lift community has been VERY helpful thusfar. Small can still be helpful.
Some stats
~14,000 lines of scala. Includes the website, the mobile website, & the Rest API (including an OAuth server implementation).~6,000 lines of markup
A pretty small amount of code for a fully functional web startup.
The Future
We still using a single PostgreSQL instance as our only datastore, this model will no get us too much further.Mapper <-> JSON functionality has been implemented in Lift and so now we need to decide where/how to store that JSON (Memcached, Casandra, Goat Rodeo?).We'll be forging some new ground in the scala world here, hopefully it will work out well. Ultimately though the CS concepts here are independent of the language being used.
