hidden gears of your application - sergej kurakinserver side near real-time job start load...
TRANSCRIPT
![Page 1: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/1.jpg)
Hidden gears of your application
Sergej Kurakin
![Page 2: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/2.jpg)
Problem
● Need for quick response● Need for many updates● Need for different jobs done● Need for task to be done as different user on
server side● Near real-time job start● Load distribution
![Page 3: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/3.jpg)
![Page 4: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/4.jpg)
Job Queue
● You put job to queue● Worker takes the job and makes it done
![Page 5: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/5.jpg)
![Page 6: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/6.jpg)
Job Queue using Crons
● Many different implementations● Perfect for small scale● Available on many systems/servers● Crons are limited to running once per minute● Harder to distribute load
![Page 7: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/7.jpg)
![Page 8: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/8.jpg)
Gearman Job Server
● Job Queue● http://gearman.org/● C/C++● Multi-language● Scalable and Fault Tolerant● Huge message size (up to 4 gig)
![Page 9: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/9.jpg)
Gearman Stack
![Page 10: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/10.jpg)
![Page 11: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/11.jpg)
Gearman Job Types
Normal Job● Run Job● Return Result
Background Job● Run Job in
Background● No Return of Result
![Page 12: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/12.jpg)
Gearman Parallel Tasks
![Page 13: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/13.jpg)
Gearman Supported Languages
● C● Perl● NodeJS● PHP● Python
● Java● C#/.NET● Ruby● Go● Lisp
![Page 14: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/14.jpg)
Job Priority
● Low● Normal● High
![Page 15: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/15.jpg)
Gearman Worker Example<?php// Reverse Worker Code$worker = new GearmanWorker();$worker->addServer();$worker->addFunction("reverse", function ($job) { return strrev($job->workload());});while ($worker->work());
![Page 16: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/16.jpg)
Gearman Client Example<?php// Reverse Client Code$client = new GearmanClient();$client->addServer();print $client->do("reverse", "Hello World!");
![Page 17: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/17.jpg)
Gearman Client Example<?php// Reverse Client Code$client = new GearmanClient();$client->addServer();$client->doBackground("reverse", "Hello World!");
![Page 18: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/18.jpg)
Running Worker in Background
● CLI● screen / tmux● supervisord - http://supervisord.org/● daemontools - http://cr.yp.to/daemontools.html● daemon - http://en.wikipedia.org/wiki/Daemon_
(computing)
![Page 19: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/19.jpg)
High Availability
![Page 20: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/20.jpg)
Persisted Job Queue
● libmysqlclient● libdrizzle● libmemcached● libsqlite3● Postgresql
![Page 21: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/21.jpg)
![Page 22: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/22.jpg)
Security
● No security at all in Gearman● Protect by yourself: 127.0.0.1, Firewalls● Ask your Sysadmins for help: Firewalls,
Secure Tunnels● Do some crazy APIs
![Page 23: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/23.jpg)
Running Example
![Page 24: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/24.jpg)
Usage Examples
● Image Resize● Search Index updates● Email sending● External API calls● Small tasks that can be done in background
![Page 25: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/25.jpg)
Alternatives
● RabbitMQ - http://www.rabbitmq.com/● Redis - http://redis.io/● Amazon SQS - http://aws.amazon.com/sqs/● IronMQ - http://www.iron.io/mq● Beanstalkd - http://kr.github.io/beanstalkd/● JFGI
![Page 26: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/26.jpg)
Links
● http://gearman.org/● http://gearman.org/manual/● http://gearman.org/getting-started/● http://php.net/manual/en/book.gearman.php● https://github.com/sergej-kurakin/gearman-
demo
![Page 27: Hidden gears of your application - Sergej Kurakinserver side Near real-time job start Load distribution Job Queue You put job to queue Worker takes the job and makes it done Job Queue](https://reader033.vdocuments.site/reader033/viewer/2022050515/5f9edfd729bd1862eb088817/html5/thumbnails/27.jpg)
Questions?
Sergej Kurakin
Work Email: [email protected] Email: [email protected]://www.linkedin.com/in/sergejkurakin
Special thanks to authors of all pictures used in this presentation.