How we built Caibangzi.com

Robin Lu (caibangzi.com)

IntroductionPersonal Investment Management. Tool + Community

Founders: We are engineers!

Technology:

Ruby On Rails

MySQL，Linux，Lighttpd ...

TopicsProblems we are facing

Architecture evolution of caibangzi

Maintenance

Q&A

Peaks!

High Peak

Heavy Computation

Visiting Peak, Computation Peak and Cache expiration Peak are in the same time. :...(

Evolution - StartLaunched at March, 2007

Hardware: 1 x 1U server

1xAMD Opteron 250, 2G Memory, 250G SATA, RAID 1

Bandwidth

Shared

less than 3Mbps

Evolution - StartApache 2 + mod_proxy_balancer

mongrel x 8

caibangzi

rails 1.1.6

MySQL 5

memcached

All in one box

First ChallengeMay, 2007

Extremely time consuming actions blocked the site.

Performance tuning

Optimize Rails cache

Second Challenge

August, 2007

We were running out of bandwidth

Solution:

Money

upgrade to 5Mbps

Server upgrading

High growth rate after August, 2007

Hardware upgrading in Sept. 2007

still 1 x 1U

2 x AMD 2214 2.2GHz

4G RAM

Until Sept. 2007Apache 2 + mod_proxy_balancer

mongrel x 20

caibangzi

rails 1.1.6

MySQL 5

memcached

Still in one box

Something happened in Oct. 2007

Stock Index reached 6000

Daily pv reached 1 million

Bigger challenge came

Several Changes We Made

Upgrade bandwidth to 10Mbps

Separate APP server and DB server

Load Balancer

Distributed Computation

2 BoxesWeb Server

event mongrel x 30

caibangzi

rails

MySQL 5

memcachedAPP

Tianyuanjia

Load balancer

Run with 2 mongrels

behindbalancer

Lighttpd 1.4

Load balancerSlow actions block the others with balancer:

by request

by traffic

round-robin

Apache

HAproxy

Nginx

Note: Nginx fixed it with ‘fair’ balancer later

Caibangzi Choice of Balancer

Lighttpd 1.5 beta

proxy-core.balancer = "sqf"

http://redmine.lighttpd.net/issues/show/1362

For Rails application, choice of balancer is sometimes more important than choice of web server

More on Web Server

disable keep-alive

serve static files with web server instead of mongrel

tune configurations according to your web server documentations

Distributed solution

BackgrounDRB

based on Distributed Ruby (DRB)

use ruby thread

high overhead during context switching

very slow when concurrent connections higher than 10

Distributed solution

FastBackground

our own distributed computation solution

based on Eventmachine

simple, but fast and works for us

Rails upgrading

Early of 2008, we upgraded to Rails 2.0

expired several plugins

refactory to be more RESTful

cookie session

multi view => wap.caibangzi.com

Open API

API Based on OAuth

integrate caibangzi into your own sites

Other APPs support several F8 clones

xiaonei.com, hainei.com, UCHome

Maintenance

Munin

easy to use and extend

Monit

Crontab

With all that....we handled:

pv/day 1Mrails req/day 1.8M

peak rails req/sec 160total req/day 5Mpeak req/sec 430

It’s the fact. Not the capacity.Our CPU usage never goes higher

than 30%

Q&A