create a scalable mobile media site with drupal and mongodb
Post on 10-May-2015
2.304 Views
Preview:
TRANSCRIPT
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Mobile Media Site using Drupal & MongoDB
Presented By:Yash Badiani, Big Data Practice Lead, CIGNEX DatamaticsGaurav Khambhala, Technical Lead, CIGNEX Datamatics30th July, 2013
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
About Acquia
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Acquia – Enterprise Guide to Drupal
3
Co-founded by the Drupal project’ creator in 2007, Acquia has helped more than 2,000 clients manage
growth and scale of their digital experiences.
Acquia provides wide range of Drupal products and services, including:
Acquia Cloud Enterprise GardensAcquia Network
Drupal Training Professional Services
Drupal Commons
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
About CIGNEX Datamatics
4
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
What Does CIGNEX Datamatics Do?
5
Since 2000, CIGNEX Datamatics has implemented over 400 Open Source enterprise
solutions addressing business requirements related to Portals, Content & Big Data
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Where We Can Help You
6
SOLUTIONS
Managed Cloud Services - Develop, Deploy, Manage
Annual Product Subscription: Liferay, Alfresco, Magento, Hadoop, Selenium
Extended Development Center – Center of Excellence
UI Development Integration Customization Migration Testing Training Support (24*7)
User eXperience Platform
Portals Liferay, Magento,Drupal, Adobe CQ
• Intranet • Extranet• Social
Collaboration
• Mobile Portals
• E-Commerce
Enterprise Content Management
Content Alfresco, Drupal, Magento, Adobe CQ, Moodle, Ephesoft
• WCM• DM• RM• DAM
• E-Commerce• E-learning• ERP• Imaging Solutions
SERVICES
Making Data WorkBig Data Hadoop, MongoDB,
Hbase, Neo4j, Solr
• Analytics• Mobile• Social• Web• Real-time
• DW - BI
• Log Processing and Analysis
• Enterprise Search
Velocity
Complexity
Volume
Variety
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Part of Datamatics (DGSL)
• Mission– Experts in improving
Enterprise productivity through Process Engineering & Information Management Solutions
• Key Highlights– Founded in 1975
– Publicly listed in India
– Annual consolidated revenue of US$100 Million
– Fortune 500 clients
– 4,400+ employees across 22 offices in 9 countries
Strategic Alliances
7
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
About the presenters
• Yash Badiani is the Big Data Practice Lead at CIGNEX Datamatics and focuses on Big Data Technologies including MongoDB & Hadoop. He has worked extensively on large Data warehousing & Business Intelligence projects with tools such as Business Objects, Microsoft SQL Server, Microstrategy, IBM Cognos.
• Gaurav Khambhala works at CIGNEX Datamatics as Technical Lead. He is
the senior member of the PHP Practice at CIGNEX Datamatics and is involved on various technology initiatives like Big Data where he focuses on integration of PHP with NoSQL sources like MongoDB. He has a wide industry experience in software development & management in Open Source technologies such as Drupal, Moodle & Wordpress.
8
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Agenda
• The Mobile Media Use Case• Requirements and Challenges• Solution : Mobile Media site using Drupal & MongoDB• Why Drupal and MongoDB?• Demo and Solution Features• Benefits • Summary
9
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
The Mobile Explosion!
10
By 2015, at least 60% of information workers will
interact with their content applications via a mobile
device
Employees work on proposals and presentation on
mobile devices while travelling
People use digital assets (videos, images) longer on
Tablets and Mobiles compared to desktops
Based on a report by a leading IT advisory firm
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Mobile Media Use Case
• Mobile Media site includes the following features:– Store a variety of Images & associated metadata– Massively Scalable to store billions of images– Access through Mobile– Create /Edit Albums– Add Images to the Albums– Add / Edit Metadata of Images– Search Images / Albums by date, metadata, albums, etc– Social Media features – Likes, comments
11
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Requirements of Mobile Media sites
12
• Fast performance• Large user base• Concurrent CRUD• Access through various channels
• Millions of digital assets• Variety of content• Complexity of data
• Rich UI features• Social features• Mobile access• Fast search
• Elastic scaling• Cost effectiveness• Centralized storage• Ease of Maintenance
• HIGH availability• Automatic failover• User management
Velocity Volume
User experience
Scalability
Security & Availability
• Easy integration• Shorter dev cycle• Faster deployment• Ease of schema design
Flexibility & Agility
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Standard Three Layered Data Architecture
13
File System
Metadata in RDBMS
Search
Standard Three Layered Storage
Application layer
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Limitations of RDBMS
• Support limited to terabytes– No support for petabytes to
exabytes
• Manage only structured data– No support for semi-structured and
unstructured data
• RDBMS don't scale inherently– Scale up/Scale out (Load Balancing
& Replication)
• Hard to shard / partition– Large data files
• Both read / write throughput not possible
– Transactional / Analytical databases
• Specialized hardware - expensive
14
RDBMS can’t manage all dimensions of data with speed & at lower cost.
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
NoSQL is the right solution
• They are schema less• Designed to support huge data volumes
– Facebook 135 billion messages/month; Twitter 7TB data/day
• Scalable replication and distribution mechanism– Thousands of machines distributed around the world
• Massive write performance with asynchronous inserts and updates• Designed to give high query performance• Runs on commodity hardware• Most NoSQL databases are Open Source
15
Not SQLOnly
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
NoSQL – Data Models
16
Column FamiliesUsage: Read/Write Intensive
Popular databases: Hbase, Cassandra
Document StoreUsage: Working with Occasionally
changing/consistent dataPopular databases: CouchDB,
MongoDB
Graph Database
Usage: Spatial Data storagePopular databases: Neo4j, Bigdata
Key Value / Tulip StoreUsage: Briskly changing data and
high availabilityPopular databases: Riak, Redis,
Azure Table storage
NoSQL Databases
• 4 broad data models• 120+ variants available in the market
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Requirements of Mobile Media sites - Recap
17
• Fast performance• Large user base• Concurrent CRUD• Access through various channels
• Millions of digital assets• Variety of content• Complexity of data
• Rich UI features• Social features• Mobile access• Fast search
• Elastic scaling• Cost effectiveness• Centralized storage• Ease of maintenance
• HIGH availability• Automatic failover• User management
Velocity Volume
User experience
Scalability
Security & Availability
• Easy integration• Shorter dev cycle• Faster deployment• Ease of schema design
Flexibility & Agility
Mobile Media Site
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Drupal with MongoDB Solution
18
Themes
CoreModules
Nodes TaxonomyUserRoles
Forms &Menu
CustomModules
Workflow ForumsComments& Ratings
Tagging
PHP
Web Services
3rd party & Internal
Applications
MongoDBDriver
MongosRouting Process
Replica Set
MongoDB MongoDB
Replica Set
MongoDB MongoDB
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Why Drupal?
19
PluggableArchitecture
Data Abstraction Layer
Easy to Upgrade
Secure
Active CommunityWidely Adopted
Scalable
3rd Party Tools Integration
User Management& Permissions
HTML5 & CSS Support
Drupal Open Source CMS
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Websites using Drupal
20
Website: Whitehouse.gov Website: Data.gov.uk
Website: mtv.co.uk
Website: research.yahoo.com
Website: pdx.eduWebsite:
EndPoverty2015.org
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Why MongoDB?
21
Agile and Scalable
Full Index Support
Document Oriented Storage
ReplicationQuerying
Atomic Updates
Data Processing and aggregation
High Availability
Open Source NoSQL Solution
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Customers using MongoDB
• Centralized data management platform
• 2 billion+ documents
• 20 TB of photo metadata
• TV episodes and series
• Risk solutions auditing data
22
Source: http://www.10gen.com/customers
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Demo
23
• Media site on mobile simulator• Like & comment on an image on mobile simulator• Mobile site on web browser• Verify ‘Like’ & comment of the same image on web browser• Search images & access control
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Solution Features
Architecture and Design
24
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Architecture
25
User MetadataInd
exes Albums
ImageMetadata GridFS
Form API
Drupal API
CustomModule
Browser/MobileTheme
MongoDB PHP Driver
Menu API
UserMobileDevice
ImageMetadata
GridFS
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Add Album
Flow
26
Add Image
View Album
View Individual Images
Like Image
Comment Image
Add Tags to Images
View Counter
Search ImagesBy Tags
User Metadata Albums GridFS Image Metadata
DBRef
DBRef DBRef
DBRef
MongoDB Collections
Use
r A
ctio
ns
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
27
Schema Design
User Metadata GridFS
Albums Image Metadata
• User ID• DBRef (Album)• Tags• Thumbnail• Likes• View Counter• Comments• Permission
• FS.Files• FS.Chunks
• User ID• Tags• Title
• Make• Model• Date Time• Aperture• Exposure• DBRef (GridFS)
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
28
Schema DesignImage Metadata
Albums
User Metadata
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
29
MongoDB Monitoring Service (MMS)
• DB Storage
• Cursors
• Replica Sets
• Network Connections
• Non Mapped Virtual Memory
• Opcounters
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Benefits
Drupal MongoDB
Most advanced content management solutions
Scalability – billions of content items, millions of users
Highly customized websites Performance – FAST writes through sharding, reads through indexes
Most search friendly CMS Data safety through replication
Less coding, high on automation Centralized single system for data storage
Powered by 7000+ plugins and extensions
Monitoring through MMS
Active community, real time assistance Enterprise support through 10gen
30
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Summary & Key Takeaways
• MongoDB provides the RIGHT fit for CMS applications with flexibility, scale & speed
• Drupal’s advanced & automated CMS features and tight integration with MongoDB makes it the right choice for building agile websites
• Both Drupal & MongoDB are feature rich and being Open Source, provide significant cost benefits
31
CIGNEX Datamatics Confidential www.cignex.com
CIGNEX Datamatics Confidential
Thank you. Questions?
CIGNEX Datamatics makes Open Source work for you!
32
Yash BadianiBig Data Practice Lead
yash.badiani@cignex.com
Gaurav KhambhalaTechnical Lead
gaurav.khambhala@cignex.com
top related