migrating one of the most popular ecommerce platforms to mongodb
DESCRIPTION
Is it worthwhile to migrate a heavy SQL application to MongoDB? In this talk we show our insights and positive outcomes on the migration of a popular open source eCommerce platform (Magento) to MongoDB. The talk covers interesting migration challenges and techniques both from a data and software point of view.TRANSCRIPT
Migrating One of the Most Popular eCommerce Platforms to MongoDBMongoDB Munich 2013, October 14th
Aron Spohr, fashion4home GmbH, Berlin
Will it blend?
What is Magento?
● Open eCommerce platform● Serves all primary features of a Webshop● Written in PHP● Works on top of MySQL out of the box● Extensible architecture● Runs on over 180,000 sites● eBay owned since 2011
The Shopping Cart
● Quote● Items● Addresses
● one Model for each● one MySQL-Table for each
Data Structure
quote_id discount_amount grand_total
560 100.00 299.00
561 0.00 1028.40
item_id quote_id product_id qty price
100 560 1257 1 39.90
101 560 1349 2 140.10
address_id quote_id city street type
388 560 Munich Hauptstr. 33a shipping
389 560 Berlin Zahlstrasse 12 billing
390 561 Hamburg Geheimweg 3 shipping, billing
quote
quote_item
quote_address
as a developer in// Loading a quote from database$quote = Mage::getModel(‘sales/quote’)->load(560);
// Loading a filtered collection of quote items from database$items = $quote->getItemCollection();$items->addFieldToFilter(‘product_id’, 1257);$items->load();
SELECT * FROM sales_quote WHERE quote_id=560;
SELECT * FROM sales_quote_item WHERE quote_id=560 AND product_id=1257;
every Model has ● a Resource Model to load/save one record from/to DB● a Collection Model to load multiple records from DB
Persistence in
Model Resource Model
DBCollection Model
Model Model Model
App
licat
ion
function load($object, $id) {
$stmt = “SELECT * FROM “ . $this->getTableName() . ” WHERE “ . $this->getIdFieldname() . ”=$id”;
$data = $sqlAdapter->fetchRow( $stmt );
$object->setData( $data );
}
Resource Model basics of
function load() {
$stmt = “SELECT * FROM “ . $this->getTableName() . ” WHERE “ . $this->renderFilters();
foreach($sqlAdapter->fetchRows( $stmt ) as $row) {$object = new Object();$object->setData($data);$this->addItem($object);
}}
Collection Model basics of
Why should we change that?
● You don’t have to● It always depends on your application
Reasons we had:● Have more/other scalability options● Make it easier to work with raw data● Be more flexible with your data schema● Learn more about the software you are using
Let’s get started
● not change the way Magento works● a very simple, self-explainable data schema● make it easy to migrate old data● not lose any existing features
Data Structure - mongoDB{
quote_id: 560, discount_amount: 100.00, grand_total: 299,
item: [{item_id: 100, product_id: 1257, qty: 1, price: 39.9},{item_id: 101, product_id: 1349, qty: 2, price: 140.10}
],address: [
{address_id: 388, city: ‘Munich’, street: ‘Hauptstr. 33a’, ...},{address_id: 389, city: ‘Berlin’, street: ‘Zahlstrasse 12’, ...}
]
}
SELECT * FROM quote_item;
db.quote.find( {}, { ‘item’: 1 } );
● { ‘item’:[ {item_id: 100, product_id: 1257, qty: 1 },{item_id: 101, product_id: 1349, qty: 2 } ] }
● { ‘item’:[ {item_id: 102, product_id: 4421, qty: 1 } ] }● { ‘item’:[ {item_id: 103, product_id: 2301, qty: 1 },
{item_id: 104, product_id: 5511, qty: 1 } ] } ● ...
Do we still have a table?
a Tool to simplify our work with mongoDB...
Loading a collection of thingsloadDataCollection(‘/quote/item’, array());
● { item_id: 100, product_id: 1257, qty: 1 }● { item_id: 101, product_id: 1349, qty: 2 }● { item_id: 102, product_id: 4421, qty: 1 }● ...
db.quote.find( {}, { ‘item’: 1 } );
● { ‘item’:[ {item_id: 100, product_id: 1257, qty: 1},{item_id: 101, product_id: 1349, qty: 2} ] }
● { ‘item’:[ {item_id: 102, product_id: 4421, qty: 1} ] }● ...
The path to our data
● Tells us where to find the data● Is very similar to a table name
/item/quoteName of Collection
Name of Array in document
We rewire all Table names in
quote
quote_item
quote_address
/quote
/quote/item
/quote/address
$rows = $newAdapter->loadDataCollection($this->getTableName(),$this->renderFilters() );
foreach($rows as $row) {$object = new Object();$object->setData($data);$this->addItem($object);
}
The new Collection Model
loadDataCollection(‘/quote/item’, array(product_id => 1257));
● { item_id: 100, product_id: 1257, qty: 1 }● { item_id: 342, product_id: 1257, qty: 2 }● ...
db.quote.find( { ‘item.product_id’: 1257 }, { ‘item’: 1 } );
● { ‘item’:[ {item_id: 100, product_id: 1257, qty: 1} ] }● { ‘item’:[ {item_id: 342, product_id: 1257, qty: 2} ] }● ...
Loading a collection of things (filtered)
as a developer in
// loading a filtered collection of quote items from database$items = Mage::getModel(‘sales/quote_item’)->getCollection();
$items->addFieldToFilter(‘quote_id’, 560); $items->addFieldToFilter(‘product_id’, 1257);
$items->load();
Loading a collection of things (filtered)
loadDataCollection( ‘/quote/item’, array( ‘quote_id’ => 560, ‘product_id’ => 1257) );
db.quote.find( { ‘item.quote_id’: 560, ‘item.product_id’: 1257 }, { ‘item’: 1 } );
- no result
- no result
● This is not a relational database anymore● Quote Items may not have a quote_id in our schema
Loading a collection of things (filtered)
loadDataCollection( ‘/quote{quote_id:560}/item’, array(product_id=> 1257));
● { item_id: 100, product_id: 1257, qty: 1 }
db.quote.find( {‘quote_id’: 560, ‘item.product_id’: 1257}, { ‘item’: 1 } );
● { ‘item’:[ {item_id: 100, product_id: 1257, qty: 1} ] }
On-the-fly Tablename completion
/quote{quote_id:$quote_id}/item
getTablename()
/quote{quote_id:560}/item
completePath(filters, properties)
as a developer in
// load a single item from db, change qty, save it$item = Mage::getModel(‘sales/quote_item’)->load(101);$item->setQty(2);$item->save();
// add a product to cart$item = Mage::getModel(‘sales/quote_item’);$item->setQuoteId(560)->setProductId(1566)->setQty(1);$item->save();
Loading a single record
loadData( ‘/quote{quote_id:560}/item’, ‘item_id’, 100);findOne({ ‘quote_id’: 560, ‘item.item_id’: 100}, {‘item.$’: 1});
loadData( ‘/quote/item’, ‘item_id’, 100);loadData( ‘/quote{quote_id:$quote_id}/item’, ‘item_id’, 100);findOne({ ‘item.item_id’: 100}, {‘item.$’: 1});
Result for all three{ item_id: 100, product_id: 1257, qty: 1 }
Saving a single record
InsertingsaveData( ‘/quote{quote_id:560}/item’,
array(‘item_id’ => 732, ‘product_id’ => 1257, ‘qty’ => 1));db.quote.update( { quote_id: 560 }, { $push : {‘item’ =>
{ item_id: 732, product_id: 1257, qty: 1 }} });
UpdatingsaveData( ‘/quote/item’, array(‘item_id’ => 732, ‘qty’ => 2));saveData( ‘/quote{quote_id:$quote_id}/item’, ...);db.quote.update( { item.item_id: 732 },
{ $set : { item.$.qty: 2 } } );
function load($object, $id) {
$data = $adapter->loadData($this->getTablename(),$this->getIdFieldname(),$id);
$object->setData( $data );
}
The new resource model
function save($object) {
$id = $this->getNewId();$data = $adapter->saveData(
$this->getTablename(),$this->getIdFieldname(),$id,$object->getData());
$object->setId($id);}
The new resource model
create a simple application to● load using the old resource model● save using the new resource model
Migration of old data in
Model
Old Resource Model DB
App
licat
ion
New Resource Model
Thanks a lot
Firefly Glow Cube
Dining Table Campagna