Our way to a Performant, Self Healing, Highly Available

Magento Stack

Homework

• Which features are shipped by default with Magento?• Which features are really needed for our Shop?• How does they scale? • How does the default implementation working with high load scenarios like tv-ads?• How does the default implementation is cacheable?• Which features are not shipped per default we need to develop on our own?• Which knowledge we have in house to deliver all the required features?

First Step

Making the core lightweight and more scaleable.

First Step

Making the core lightweight and more scaleable.

First Step

Making the core lightweight and more scaleable.

Making the core lightweight and more scaleable.

Removed not required core modules and also the ones which brings poor performance.

• AdminNotification• Authorizenet• Backup• Captcha• Compiler• ConfigurableSwatches• Connect• Downloadable• Google*• Newsletter• Oauth• PageCache• .... and even more

Making the core lightweight and more scaleable.

Added great community modules

• Zookal_Mock• Ecomdev_LayoutCompiler• Inchoo_PHP7 (Directly in the affected core files)

Making the core lightweight and more scaleable.

Varien_Db_Select

• Introduced SQL_USE_CACHE AND SQL_NO_CACHE support• Introduced FORCE INDEX• Introduced IGNORE INDEX• Introduced USE INDEX

Making the core lightweight and more scaleable.

Magento Indexer

• Rewrite all indexer to use SQL_NO_CACHE• use CREATE TEMPOARY TABLE to avoid growing transaction log• Implemented Table rotation to reclaim table space• all index events are processed asyncronius via rabbitmq (except stock after checkout)• Array_map(‘intval‘, $entityId) to avoid mixed data types in queries• process full reindex in chunks to reduce possibility of slave replication lag• Implemented ORDER BY NULL for indexers to avoid tmp_tables_on_disc

Making the core lightweight and more scaleable.

indexer for url rewrites

• Mysql implementation based on EcomDev_UrlRewrite with some parts as UDF in C• add new unsigned integer column rewrite_hash• Crc32 over store_id and request path• add index over rewrite_hash and first 10 characters of request_path

Making the core lightweight and more scaleable.

Redis Cache Backend

• removed Credis libary and directly speak over pecl_redis• add the possibility to make cache tags persistent (max memory policy)• implemented master / slave connections

Making the core lightweight and more scaleable.

Magento Admin

• all SQL Queries in the admin interface by default use SQL_NO_CACHE• Long running tasks are handled by rabbitmq via worker

Making the core lightweight and more scaleable.

Block caching / layout.xml

• Introduced caching of html and also of tags and object id (required for varnish tagging)• Cache tags are also applied to parent block • removed symlink check• make layout handles for category and product configurable if needed

Making the core lightweight and more scaleable.

Block caching for products

• created a common product block • template and caching configurable over layout.xml• holepunching possible for price block• accepts product id or full product data

• fetches data from collection with LIMIT 1 and also caches collection response• implemented Observer AddStockStatusToCollection within product load• product urls are unique and can be directly joined

• use column target_path and store_id with CONCAT

Making the core lightweight and more scaleable.

Navigation Block

• Removed Usage of Varien_Data_Tree which is horrible slow• $this->getCategoryCollection()->getItemById($categoryId)• Cache query result for $this->getCategoryCollection()

Making the core lightweight and more scaleable.

Cache Locking

• add cache locking during config generationa• add block cache locking confgurable via layout.xml

Making the core lightweight and more scaleable.

Mage_Core_Controller_Varien_Front::dispatch

• added fastcgi_finish_request (php-fpm only)

Infrastructure

Infrastructure

• Stackformation by AOE Media• all nodes are based on gentoo compiled for their use case

• Provisioned with Packer and Managed by Chef• All Services HA• Galera Cluster• Cache Replication• Varnish Cache with Tagging and Segmentation• mostly spot instances

• different instances for night day and peak• Scaling handled by Lamda and a C Daemon which watches for termination notification• Fast upscaling / slow downscaling• Downscaling happends only if instances are reaching the next payment iteration• all instance related metrics calculated down to one cpu core (scaling over spot price)

• Elasticsearch for catalog and search• Google Analytics API for feeding dynamic weighting for products