building things fast - and getting approval

Building Things Fastand getting approval

Simon Willison, An Event Apart Chicago 2009

Web development four years ago

sucked

This talk

• Individual techniques and tools

• From one developer to a group

• From a group to an organisation

• Because the tools have never been better

• Because the sooner you get your stuff in front of users the sooner you’ll realise

how wrong you were

• Because it’s cheaper

• Because it’s (a lot) more fun

Why?

1 hour kick off meeting x 6 people 6 man-hours

4 hours prototying x 1 developer 4 man-hours

• Interactive development

• Client-side prototyping

• Web frameworks

• Web APIs

• Open source

Rapid development trends

Interactivedevelopment

Have a good idea if your code will work

before you evensave it to a file

JavaScript FirebugCSS “edit styles” bookmarklet

Python python, ipythonRuby irbPHP phpsh (by Facebook)Java BeanShellPerl Devel::REPL

Search for “language REPL” or “language shell”

http://xkcd.com/303/



Client-side prototyping

Server-side = deployment headaches

The sameorigin policy

http://en.wikipedia.org/wiki/Same_origin_policy



Evil website

Your bank

iframe

JavaScript

JSON-PJSON with Padding

http://twitter.com/statuses/user_timeline/simonw.json?callback=foo

foo([ { "favorited": false, "in_reply_to_status_id": 4420410018, "in_reply_to_screen_name": "andrewgodwin", "in_reply_to_user_id": 8749512, "source": "web", "text": "@andrewgodwin CMS that do full revision tracking are pretty useful - I've had success with MediaWiki as a CMS backend in the past", "created_at": "Sun Sep 27 17:31:24 +0000 2009", "truncated": false, "id": 4421168167, "user": { "id": 12497, "verified": false, "profile_sidebar_fill_color": "FFFFFF", "profile_text_color": "000000", "followers_count": 5664, "protected": false,



LEARN JQUERY NOW!

What does jQuery code look like? Here's thequick and dirty:

Getting Started With jQuery

How jQuery Works

Developer Resources

Mailing List

is a new kind of JavaScript Library.

jQuery is a fast and concise JavaScript Library that simplifies HTML

document traversing, event handling, animating, and Ajax interactions for

rapid web development. jQuery is designed to change the way that you

write JavaScript.

PRODUCTION (19KB, Minified and Gzipped)

DEVELOPMENT (120KB, Uncompressed Code)

Current Release: v.1.3.2

GRAB THE LATEST VERSION!

CHOOSE YOUR COMPRESSION LEVEL:

jQuery Plugins UI Blog About Donate

Download Documentation Tutorials Bug Tracker Discussion

Lightweight Footprint CSS3 Compliant Cross-browser

WHO'S USING JQUERY?

JQUERY RESOURCES

jQuery.getJSON( “http://twitter.com/statuses/user_timeline/simonw.json?callback=?”, function(data) { alert(data);});

http://twitter.com/statuses/user_timeline/simonw.json?callback=?




http://j.mp/afeedapart



<div id="feed"> <ul> <li class="tweet group" id="template"> <a href="http://twitter.com/simonw" title="simonw" style="background-image: url('http://a1.twimg.com/profile_images/408476120/django-cowboy_normal.jpg');" class="profile-image"></a> <div class="tweet-text"> <span>Here come the tweets...</span> <div class="posted">posted <span>0 minutes ago</span> by <a href="http://twitter.com/simonw"> simonw</a> </div> </div> </li> </ul></div>

http://twitter.com/simonw


http://a1.twimg.com/profile_images/408476120/django-cowboy_normal.jpg'








var search_url = 'http://search.twitter.com/search.json?q=%23aea&callback=?';var last_tweet_id = null;jQuery(function($) { var template = $('#template').clone(); template.attr('id', ''); $('#activity-indicator').hide(); function refresh() { var url = search_url; if (last_tweet_id) { url += '&since_id=' + last_tweet_id; } $('#activity-indicator').show(); jQuery.getJSON(url, function(data) { $('#template').remove(); $('#activity-indicator').hide(); last_tweet_id = data.max_id; $('#feed li').removeClass('new-tweet'); data.results.reverse();

http://search.twitter.com/search.json?q=%23aea&callback=?'




jQuery.getJSON(url, function(data) { $('#template').remove(); $('#activity-indicator').hide(); last_tweet_id = data.max_id; $('#feed li').removeClass('new-tweet'); data.results.reverse(); $.each(data.results, function() { var tweet = template.clone(); tweet.find( '.tweet-text span:first' ).html(this.text); tweet.find('a:first').css( 'background-image', 'url('+this.profile_image_url+')' ); // ... tweet.hide(); $('#feed ul').prepend(tweet); tweet.slideDown(); }); });

jQuery, JSON-P and 45 lines of JavaScript

HOME DOCUMENTATION MY PROJECTSsimonwillison Sign Out

YDN Yahoo! Query Language Documentation Console

Recent Queries

get my profile data

get my friends

get all my friends profiles

get my friends nicknames

get my last added friend

get my friends sorted by nickname

get 10 flickr "cat" photos

get a flickr photo by photo ID

get the flickr image url by photo ID

get san francisco geo data

get san francisco woeid

get geo details about san francisco

find sushi restaurants in san francisco

find sushi restaurants in san francisco with an average rating of 4.5

find events in north beach

find all upcoming events in north beach sorted by start date

find the next upcoming event in north beach

get rss feed from yahoo news top stories

search the web using the title of the top yahoo news story

extract the headlines from yahoo finance using xpath

aggregate and filter multiple rss feeds

Example Queries

Show Community Tables

socialsocial.connections

social.connections.updates

social.contacts

social.presence

social.profile

social.profile.image

social.profile.status

social.updates

flickr

geo

local

maps

meme

music

mybloglog

Data Tables (94)Whats this?

Your YQL Statement

show tables

XML JSON TEST Permalink

The REST query

http://query.yahooapis.com/v1/public/yql?q=show%20tables&format=xml

How do I use this? COPY URLFlash

TREE VIEW

<?xml version="1.0" encoding="UTF-8"?><query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="94" yahoo:created="2009-09-27T06:03:36Z" yahoo:lang="en-US" yahoo:updated="2009-09-27T06:03:36Z" yahoo:uri="http://query.yahooapis.com/v1/yql?q=show+tables"> <diagnostics> <publiclyCallable>true</publiclyCallable> <user-time>1</user-time> <service-time>0</service-time> <build-version>3130</build-version> </diagnostics> <results> <table>atom</table> <table>csv</table> <table>feed</table> <table>flickr.photos.exif</table> <table>flickr.photos.info</table> <table>flickr.photos.interestingness</table> <table>flickr.photos.recent</table> <table>flickr.photos.search</table> <table>flickr.photos.sizes</table> <table>flickr.photosets.photos</table> <table>flickr.places</table> <table>flickr.places.info</table> <table>geo.placemaker</table> <table>geo.places</table> <table>geo.places.ancestors</table> <table>geo.places.belongtos</table> <table>geo.places.children</table> <table>geo.places.neighbors</table> <table>geo.places.parent</table> <table>geo.places.siblings</table> <table>geo.placetypes</table>

FORMATTED VIEW

Version 1.0 | Copyright © 2009 Yahoo! Inc. All rights reserved. Copyright | Privacy Policy | Forum

Building a toolboxYQL

But...

Beware the Phantom WebOr JavaScript will destroy us all

Server-side prototyping

Modern web frameworks

• Ruby on Rails, Symfony, CakePHP...

• “Full-stack” - everything you need

• DRY; eliminate boiler-plate

• Model layer, separate templates

“Web development on journalism deadlines”

The MPexpenses scandal

2005 FOI request

• Prototype that became a product

• One week from concept to launch

• Design, ops, QA involvement

• Load tested “live” on EC2

• Database models / ORM

• Automatic admin interface

• Display / validate / redisplay forms

• Templates, and template inheritance

• Well designed URLs!

Django features

Simon Willison's WeblogNew entry / Comments

Home › Entries

Select entry to change

Welcome, simon. Documentation / Change password / Log out

Add entry

2002 2003 2004 2005 2006 2007 20082009

Go

Title Created Comments open

Django ponies: Proposals for Django 1.2 Sept. 28, 2009, 11:32 p.m.

Hack Day tools for non-developers July 28, 2009, 2:23 p.m.

Teaching users to be secure is a shared responsibility July 16, 2009, 8:04 p.m.

Facebook Usernames and OpenID June 13, 2009, 5:01 p.m.

djng - a Django powered microframework May 19, 2009, 12:13 a.m.

rev=canonical bookmarklet and designing shorter URLs April 11, 2009, 5:37 p.m.

List of SxSW 2009 panels with "social" in the title March 14, 2009, 11:02 p.m.

A few notes on the Guardian Open Platform March 10, 2009, 2:28 p.m.

Pragmatism, purity and JSON content types Feb. 6, 2009, 10:19 a.m.

Rate limiting with memcached Jan. 7, 2009, 10:27 p.m.

DjangoCon and PyCon UK Sept. 15, 2008, 3:20 p.m.

Announcing dmigrations Sept. 3, 2008, 7:17 p.m.

Back to full-time employment Aug. 22, 2008, 5:26 p.m.

The point of "Open" in OpenID June 24, 2008, 8:12 a.m.

Debugging Django May 22, 2008, 12:35 a.m.

jQuery style chaining with the Django ORM May 1, 2008, 12:31 p.m.

wikinear.com, OAuth and Fire Eagle March 22, 2008, 2:34 p.m.

Django People: OpenID and microformats Jan. 24, 2008, 2:02 a.m.

Django People Jan. 23, 2008, 2 a.m.

Yahoo!, Flickr, OpenID and Identity Projection Jan. 7, 2008, 11:33 p.m.

Comet works, and it's easier than you think Dec. 5, 2007, 4:22 p.m.

Figuring out OpenSocial Nov. 2, 2007, 10:29 a.m.

Questioning Steve Ballmer Oct. 1, 2007, 11:57 p.m.

Filter

By createdAny dateTodayPast 7 daysThis monthThis year

By comments openAllYesNo

Title:

Slug:

Body:

Created:

Django ponies: Proposals for Django 1.2

ponies

<p>I've decided to step up my involvement in Django development in the run-up to Django 1.2, so I'm currently going through several years worth of accumulated pony requests figuring out which ones are worth advocating for. I'm also ensuring I have the code to back them up - my innocent <a href="http://code.djangoproject.com/wiki/AutoEscaping">AutoEscaping proposal</a> a few years ago resulted in an enormous amount of work by Malcolm and I don't think he'd appreciate a repeat performance.</p>

<p>I'm not a big fan of branches when it comes to exploratory development - they're fine for doing the final implementation once an approach has been agreed, but I don't think they are a very effective way of discussing proposals. I'd much rather see working code in a separate application - that way I can try it out with an existing project without needing to switch to a new Django branch. Keeping code out of a branch also means people can start using it for real development work, making the API much easier to evaluate. Most of my proposals here have accompanying applications on GitHub.</p>

<p>I've recently got in to the habit of including an "examples" directory with each of my experimental applications. This is a full Django project (with settings.py, urls.py and manage.py files) which serves two purposes. Firstly, it allows developers to run the application's unit tests without needing to install it in to their own pre-configured project, simply by changing in to the examples directory and running <samp>./manage.py test</samp>. Secondly, it gives me somewhere to put demonstration code that can be viewed in a browser using the runserver command - a further way of making the code easier to evaluate. <a href="http://github.com/simonw/django-safeform">django-safeform</a> is a good example of this pattern.</p>

<p>Here's my current list of ponies, in rough order of priority.</p>

<h4>Signing and signed cookies</h4>

<p>Signing strings to ensure they have not yet been tampered with is a crucial technique in web application security. As with all cryptography, it's also surprisingly difficult to do correctly. <a href="http://vnhacker.blogspot.com/2009/09/flickrs-api-signature-forgery.html">A vulnerability in the signing implementation</a> used to protect the Flickr API was revealed just today.</p>

<p>One of the many uses of signed strings is to implement signed cookies. Signed cookies are fantastically powerful - they allow you to send cookies safe in the knowledge that your user will not be able to alter them without you knowing. This dramatically reduces the need for sessions - most web apps use sessions for security rather than for storing large amounts of data, so moving that "logged in user ID" value to a signed cookie eliminates the need for session storage entirely, saving a round-trip to persistent storage on every request.</p>

<p>This has particularly useful implications for scaling - you can push your shared secret out to all of your front end web servers and scale horizontally, with no need for shared session storage just to handle simple authentication and "You are logged in as X" messages.</p>

Date: 2009-09-28 Today |

Time: 23:32:04 Now |

Comments open

Simon Willison's WeblogNew entry / Comments

Home › Entries › Django ponies: Proposals for Django 1.2

Change entry

Welcome, simon. Documentation / Change password / Log out

History View on site

Rapid development

in a group

/dev/fort

e-mail

password

or OpenIDwhat is this?

desiredusername

Register nowAlready a member? » LOG IN

By registering with Wildlife Near You, you areagreeing to our terms and conditions of use.

» Go for it

This is alpha.wildlifenearyou.com - please don't link to or Twitter this yet, we're still fixing up some loose ends

Welcome back!

Wildlife Near You is a site for you to share yourpassion for animals. Search for a place you'vebeen, add in which animals you spotted, and addany photos you have of them!

REGISTER LOGIN FEEDBACK

Try tigers, llamas near brighton, new zealand or your favourite animal

LOOK FOR » Search

General informationAddress: Regent's Park, London, England, NW1 4RY, United Kingdom (map)

Phone: 020 7722 3333

VISIT THEIR OFFICIAL WEBSITE

Animals you might see

Abyssinian Ground-hornbillAfrican Brush-tailed Porcupine African Penguin African Wild Dog

Alpaca American Black VultureBactrian camel Bearded PigBlack-and-white-casqued Hornbill

Black-cheeked LovebirdSee all species that have been spotted here.

Have you been here?Tell us which animals you saw.

Average rating

Based on ratings of 3 trips by 3 spotters.

Have you been here? Why not addyour trip?

Have we got our facts right?Suggest changes to our information

Photos


London Zoo


LOOK FOR search

It was great!

simon went to Les Géants du Cielin August 2009

SightingsGriffon VultureGyps fulvus

Barn OwlTyto alba

Eurasian Eagle OwlBubo bubo

Trip rating

Photos

CommentsYou need to log in to post comments.


My trip to Les Géants du Ciel in August2009


LOOK FOR search

owls near chicago

Search

Alternatively, search for “owls” everywhere, or everything near “chicago”.

owls near Chicago, IL, USA

Show results on a map

National Zoo593 miles away, in Washington, United States. Visit their official

website.

Oregon Zoo1757 miles away, in Portland, United States. Visit their official website.

Fowlmere RSPB Bird Sanctuary3939 miles away, in Fowlmere, United Kingdom. Visit their official

website.

London Zoo3949 miles away, in London, United Kingdom. Visit their official

Matching species

Spectacled OwlPulsatrix perspicillata

Burrowing OwlAthene cunicularia

Tawny OwlStrix aluco

Barn OwlTyto alba

Snowy OwlBubo scandiacus

More species matching “owls”…


Search: “owls” near “chicago”


Pair programming

The KJ method

Great for learningNot so great for shipping

Making your organisation

more hackable

Innovation

Hack days

Every Wednesdayis Hack Day

Dev Labs

RSS feeds

Screen scraping(scrAPIs)

(You don’t have to use regular expressions any more)

JavaScript YQL

Python BeautifulSoup, html5lib

Ruby hpricot + many others

PHP PHP Simple HTML DOM Parser

With YQL

select * from data.html.cssselect where url="www.yahoo.com" and

css="#news a"

Example query: http://j.mp/IhF1b

http://developer.yahoo.net/blog/archives/2009/04/yql_execute.html

http://www.yahoo.com

http://www.yahoo.com

http://j.mp/IhF1b

http://j.mp/IhF1b



Open source

Open source is an ecosystem

Open source

• It’s (mostly) not about the ongoing price

• Free to evaluate

• No lockin

• No “we’ve paid for it, we have to use it”

• Awesome community and support*

* Depending on both the project and your own attitude

Should you release your own code?

An Open Source project is for life, not just for Christmas

simonw Dashboard Inbox 28 Account Settings Log Out

Search GitHub…Explore GitHub Gist Blog Help

simonw / bugle_project Description: Group collaboration tools for hackers in forts. editHomepage: Click to edit editPublic Clone URL: git://github.com/simonw/bugle_project.git Your Clone URL: [email protected]:simonw/bugle_project.git

Added BSD license file

simonw (author)June 08, 2009

commit 741fff120b583c78d6860e28f9c51db618f48862tree bd0268213a3db68168e987dac00aa1e01ff05522parent 30528491762acf27fae07a36084e7e3189475ff8

history

bugle_project /

name age message

.gitignore May 31, 2009 local_manage.py is now supported to make the py... [simonw]

LICENSE.txt June 08, 2009 Added BSD license file [simonw]

README.txt June 08, 2009 Clarified dependencies [simonw]

__init__.py May 30, 2009 Created project [simonw]

bugle/ June 08, 2009 New shared _blasts.html template, re-implemente... [simonw]

ext/ May 30, 2009 Added ridiculously simple registration system, ... [simonw]

manage.py May 31, 2009 local_manage.py is now supported to make the py... [simonw]

registration/ June 08, 2009 Added reserved usernames [simonw]

screenshot.png June 08, 2009 Tweaked readme, added screenshot [simonw]

settings.py June 08, 2009 Added reserved usernames [simonw]

Source Commits Network (2) Fork Queue Issues (0) Downloads (0) Wiki (1) Graphs Admin

master all branches all tags

228

GitHub: open source for developers who are scared of commitment

Calls To Action!

• Use interactive development

• Build client-side prototypes with JSON-P

• Learn a modern web framework

• Hire a fort

• Organise hack days

Further reading

• http://simonwillison.net/

• http://news.ycombinator.com/

• http://programming.reddit.com/

http://simonwillison.net

http://simonwillison.net

http://news.ycombinator.com

http://news.ycombinator.com

http://programming.reddit.com

http://programming.reddit.com

building things fast - and getting approval

Technology

open source

lasttweetid

united kingdom

united states

feed li

feed ul

23aeaamp

txt june 08