slideshare - magento imagine - do you queue

Download Slideshare - Magento Imagine - Do You Queue

Post on 17-May-2015

1.064 views

Category:

Technology

6 download

Embed Size (px)

TRANSCRIPT

Do you queue?

Do you queue? Characteristics of scalabilityKevin SchroederTechnology EvangelistZend Technologies All rights reserved. Zend Technologies, Inc. All rights reserved. Zend Technologies, Inc.About me Past: Programming/Sys Admin Current: Technology Evangelist/Author/Composer @kpschrade

All rights reserved. Zend Technologies, Inc.I blog at eschrade.comFollow us!(good things will happen to you)

Zend Technologies

http://twitter.com/zend

http://twitter.com/kpschrade (me!)

All rights reserved. Zend Technologies, Inc.Twtpoll resultsCould your PHP apps benefit from being able to process data or execute asynchronously?

All rights reserved. Zend Technologies, Inc.Why would you want to queue?PerformanceExecute logic apart from the main request (asynchronicity)ScalabilityThe ability to handle non-immediate logic as resources are available All rights reserved. Zend Technologies, Inc.Typical anatomy of a PHP Application| 6PresentationApplication ControlDatabase AccessBusiness LogicPresentationApplication ControlBusiness LogicPresentationBad for scala-bility! All rights reserved. Zend Technologies, Inc.6| 7 | 8-Feb-11PresentationDatabase AccessBusiness LogicApplication ControlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpGood for Scalability All rights reserved. Zend Technologies, Inc.7What helps make software scalable?Defined tasksLoose couplingResource discovery All rights reserved. Zend Technologies, Inc.

The Golden Rule of ScalabilityIt can probably wait All rights reserved. Zend Technologies, Inc.Asynchronous execution usesPre-caching dataData analysisData processingPre-calculating (preparing data for the next request)

Data is out of date once it leaves the web serverImmediacy is seldom necessary

All rights reserved. Zend Technologies, Inc.

CharacteristicsA sledgehammer can hit a machineScalability and High Availability are yin and yangA site that cant keep running is not scalableA site that cant scale will fail (if it gets really popular)Machines can be added and removed at willNot cloudy necessarilyNo single point of failureData exists in at least two, preferably at least three, places All rights reserved. Zend Technologies, Inc.ConsiderationsWaste disk spaceControl usage (dont let users do anything they want)Pre-calculate as much as possibleCalculate and cache/storeKeep data processing off the front end serversDont just cacheDont let it substitute for thoughtCache hit rates can be meaningless if you have hundreds of cache hits for a request All rights reserved. Zend Technologies, Inc.ConsiderationsBuild a deployment mechanism with NO hardcoded values like directory or resource locationsMake as much as possible configurable/discoverableDecouple/PartitionDont tie everything (relationships and such) into the databaseUse queues/messagingStomp interfaces are really good for PHP Can also use Java BridgeZend_Queue has several interfacesTry to use stateless interfaces(polling is more scalable than idle connections) All rights reserved. Zend Technologies, Inc.OptionsUse Cron /w PHP CLI (not the best idea)People mostly use cron simply due to availabilityIts an available option, not necessarily the best option.Use GearmanUse home-grown (dont do this)Use pcntl_fork() (NEVER do this)Use Zend Server Job Queue All rights reserved. Zend Technologies, Inc.Your only real optionsGearman*Zend Server Job QueueFreeLightweightOpen Source(mostly) language agnosticDistributed queuingReady to goIntegrates with Event MonitoringIntegrates with Code TracingRuns over a widely known protocolLoad distribution can be accomplished outside of the queue* I am not an expert on Gearman. Corrections will be taken in the spirit that they are given.Very cloud friendlyFor obvious reasons, I will focus on Zend Server All rights reserved. Zend Technologies, Inc.

Linux(rpm/web repositories)

IBM i(PTF)

Windows(MSI)

What the heck is Zend Server?ApplicationPerformanceAccelerationOptimizationCachingReliability &ManagementMonitoringRoot-CauseConfigurationScale-OutClusteringJob QueueDownloadsBusiness-gradePHPHot FixesSupportJava BridgeZend FrameworkPHPZend ServerZend Server16 All rights reserved. Zend Technologies, Inc. We released Zend Server and Zend Server Community Edition in April this year Products written from scratch based on our very extensive experience with Platform technologies Full integrated stack, native installer, ZF and Studio integration, software updates, all new UI, Both editions have been very well received by users (love performance boost, ease of use, deployment) Great fit to our partners were working with Varien/Magento, KnowledgeTree, MCS, Next step will round up web app server offering to support high availability and scalability more on this in the next few months

16Job Queue Architecture Elastic BackendProsScale the backend as necessaryDefault (easy) mechanismConsGetting the job status requires using a DBUsers!Web Server /w JQWeb Server /w JQWeb Server /w JQWeb ServerWeb ServerWeb ServerLoad Balancer All rights reserved. Zend Technologies, Inc.Job Queue Architecture Elastic FrontendProsEasy to communicate with the Job Queue server handling the jobConsRequires you to build your own interface (or just use mine!)Users!Web ServerWeb ServerWeb ServerWeb Server /w JQWeb Server /w JQWeb Server /w JQLoad Balancer All rights reserved. Zend Technologies, Inc.Kevins favorite way to implement itCreate a task-handling controllerCreate an abstract task classUnderstands the Job QueueSelf containedIf Elastic Backend: connects to localhostIf Elastic Frontend: connects to load balancer (my preferred), load balanced JQ servers manage themselvesExecute the task, have it serialize itself and send it to send to the task handler All rights reserved. Zend Technologies, Inc.Classes involved in the demoManagerHandles connecting to the queue and passing results back and forthJobAbstractAbstract class that a job would be based off ofResponseThe response from the manager when a job is queued. Contains the server name and job numberGetRemoteLinksScans a given web page and reports back a list of all the links on the page All rights reserved. Zend Technologies, Inc.Execution FlowCreate job and set dataExecute jobJob passes itself to the queue managerManager serializes jobManager uses HTTP call through a load balancer to queue the jobThe queue on the other end returns the job id and server nameJob ID and server name is passed to the clientClient polls the manager to get a completed jobWhen the job is returned pass the serialized version of the executed job All rights reserved. Zend Technologies, Inc.Demo All rights reserved. Zend Technologies, Inc.Neato, but how in the world does this pertain to Magento? All rights reserved. Zend Technologies, Inc.How?Long running front-end requests kill front-end responsiveness which kills the user experience40% will wait no more than 4 seconds for a website (Forrester)Calculating cache data on-the-fly can lead to bad user experiences (abruptly slow page load times) and consistency problemsLong running page requests can push up against max_execution_timeFor example, handling payments from a slow CC APISimply setting it to run longer is a band-aid on a broken bone

All rights reserved. Zend Technologies, Inc.Solution?Build Zend Server integration extensionsZendserver_JobqueueZendserver_MonitorUtilize the event system to intercept actionsAsync_Payment (for payments, in this case)Profit! (scalability AND performance, actually) All rights reserved. Zend Technologies, Inc.Building a jobInstall the Zend Server Magento extensionCreate a class that extends Zendserver_Jobqueue_JobAbstractCall $job->execute(); All rights reserved. Zend Technologies, Inc.Payment as an exampleHooks an observer into controller_action_predispatch_checkout_onepage_saveOrder, core_block_abstract_to_html_afterChecks if a defined template is being renderedInjects JavaScript that changes the review.save() methodInjects a new URL for submitting the transaction which creates a jobPasses POSTed data to a jobJob executes by submitting the payment URL on behalf of the browser, storing the resultBrowser pings the server to see if the job has completed executing yet All rights reserved. Zend Technologies, Inc.Scaling the solutionWebserverWebserverWebserverDatabase or Session ClusteringLoad BalancerJob Queue/ WebserverJob Queue/ WebserverJob Queue/ Webserver* Duplicate deployments on ALL machines

All rights reserved. Zend Technologies, Inc.DownloadsZend Serverhttp://www.zend.com/serverZendserver_Jobqueuehttps://github.com/kschroeder/Magento-ZendServer-JobQueueZendserver_Monitorhttps://github.com/kschroeder/Magento-ZendServer-MonitorAsync_Paymenthttps://github.com/kschroeder/Magento-Async-PaymentJob Queue library & demohttps://github.com/kschroeder/ZendServer-JobQueue-Job-API

http://bit.ly/hWJBYw All rights reserved. Zend Technologies, Inc.Magento/Zend Case Study - http://bit.ly/horpFF

Follow us!

Zend Technologies

http://twitter.com/zend

http://twitter.com/kpschrade (me!)

All rights reserved. Zend Technologies, Inc.

Get this information and all the examples at eschrade.com

All rights reserved. Zend Technologies, Inc.

Recommended

View more >