webinar: how penton uses mongodb as an analytics platform within their drupal digital platform
DESCRIPTION
NorthPoint Digital worked with the Penton and MongoDB teams to deliver a MongoDB based solution, Govalytics, to serve city and county governments. We will review the design decisions made and steps taken to implement and integrate into the existing digital platform. In the session, we will review: How Govalytics fits into Penton's entire digital platform? What were the business drivers for choosing MongoDB (with Product Owner testimony) and why it was so successful? How NorthPoint Digital implemented a complete, highly interactive UX solution powered by MongoDB as part of an integrated solution and not just as a database Roadmap for the future – how the solution was designed to be independently scalableTRANSCRIPT
Penton is a professional information services company. We engage and inform millions of professionals every day, helping them decide how to grow and solve their most critical business challenges. As a leading provider of workflow solutions and insightful content, marketing services and networking via in-person and online events, we help businesses do business.
We focus our industry expertise on five core growth markets: agriculture, transportation, natural products/food, public infrastructure, and industrial manufacturing and design.
Penton has many SaaS business products within its digital platform, and continues to add to their portfolio each year.
Govalytics:Public Infrastructure GroupAtlanta, GA
Who We Are:
Ben WalkerDirector of Brand Management, GovalyticsPenton: Public Infrastructure Group
• Responsible for product development and marketing functions related to new SaaS/DaaS products
• MID, MBA, PMP and Lean Six Sigma Black Belt product developer
• Experience in marketing, with skill sets in market research, analytics, branding, market segmentation and strategy
• Background in the design field, including disciplines such as product design, web design, UX, graphic design and furniture design
Who We Are:
About NorthPoint Digital:
• Privately held and technology agnostic• Headquartered in New York City with offices in Boston and
Philadelphia• Deliver scalable content and community solutions and services:
Strategy
/ Customer Research/ Digital Strategy/ Mobile Strategy/ Multi-Channel/ Analytics
Experience Design
/ Persona Development/ Information Architecture/ Content Strategy/ User Experience + Design
- Mobile/Tablet Applications
- Responsive Design
Technology
/ Platform Strategy + Implementation/ Architecture + Solution Design/ CMS Implementation/ 3rd Party Integrations/ Analytics/ Search/ Mobile Development
Some of NorthPoint Digital’s Clients
Below are a few of the companies we have helped succeed in the digital arena.
• 20+ years of experience across a diverse set of technologies
• Led projects for media companies, research foundations, insurance companies, hospitals, and think tanks
• Focus on scalable, cross-platform solutions
• Background in music production, UX and service design
Who We Are:
Ian MarianoProject ManagerNorthPoint Digital
Overview
“With close ties to local governments and
extensive experience in the public infrastructure space we know that winning government work can be extremely challenging. Government work represents
$3 trillion annually and we wanted to find an easier way to identify opportunities within governments well before the bidding process. Govalytics provides marketers with the critical intelligence and workflow tools needed to navigate the complicated government sector
and win more business.”
- Gregg Herring, Vice President and Public Infrastructure Market Leader
Overview
Origin & Market Pain:• Through extensive experience in the public infrastructure space, know the pain
points for marketers• Have close ties to both local governments and marketers to them• Marketers are constantly looking for more efficient ways to find opportunities
within governments, but struggle to find the right kind of work within the right kind of governments
• Our group has white-labeled a RFP service to help our customers, but quickly came to realize more forward looking analytics was needed
What is Govalytics?• A sales and marketing tool that informs the user as to which governments they
should be establishing relationships with well before the issuance of a RFP• Budget analytics• Capital project data• Government official contact information• An answer to, “Which governments should I be marketing to and why?”
Govalytics Main Areas
1.Budgetary Data Analytics – Tracks the data analytics for each local government
• Total Expenditures, Capital Assets, Long-Term Debt and Budgets
– Current year and 5 years back, down to the department level for accepted and actual budgets providing:• Historical Trending • Projections
2.Top Government Officials– Contact information for city officials – Allows users to track who is in office and changes in
leadership
3.Capital Project Data– Project information for any current or future
project– Projected yearly expenditures for future years
within the projects
8 Government Department Functions
• Public Safety• Public Works• Utilities• Planning and Development• General Government/Judicial• Education• Health and Human Services• Parks/Recreation/Culture
Government 1:Dept. of SafetyDept. of Parks & Rec
Government 2:Fire DepartmentDept. of Fun Times
Both Normalized:Public SafetyParks/Recreation/Culture
New York-Newark-Jersey City, NY-NJ-PALos Angeles-Long Beach-Anaheim, CAChicago-Naperville-Elgin, IL-IN-WIDallas-Fort Worth-Arlington, TXHouston-The Woodlands-Sugar Land, TXPhiladelphia-Camden-Wilmington, PA-NJ-DE-MDWashington-Arlington-Alexandria, DC-VA-MD-WVMiami-Fort Lauderdale-West Palm Beach, FLAtlanta-Sandy Springs-Roswell, GABoston-Cambridge-Newton, MA-NHSan Francisco-Oakland-Hayward, CARiverside-San Bernardino-Ontario, CAPhoenix-Mesa-Scottsdale, AZDetroit-Warren-Dearborn, MISeattle-Tacoma-Bellevue, WAMinneapolis-St. Paul-Bloomington, MN-WISan Diego-Carlsbad, CATampa-St. Petersburg-Clearwater, FLSt. Louis, MO-ILBaltimore-Columbia-Towson, MDDenver-Aurora-Lakewood, COCharlotte-Concord-Gastonia, NC-SCPortland-Vancouver-Hillsboro, OR-WASan Antonio-New Braunfels, TXOrlando-Kissimmee-Stanford, FLGovalytics will offer data for the top 50 MSAs in Q1 2014.
Coverage: Top 25 Metropolitan Statistical Areas (MSAs)
Govalytics needed:
• Something scalable• We were launching with the top 25 MSAs, but would quickly be adding the
next 75
• Wanted to be able to add features without breaking the model• We were continuing to add more features and functionality
• Capital Projects• ToDos• Alerts
• We wanted to add more capabilities in the future• State data• More advanced searching and statistical analysis
• Ability to aggregate data
What We Needed: Business Drivers
Wireframes
Implementing an Integrated Solution
Implementation: Challenges
The business needs for Govalytics needed to address the convergence of:
• Building an engaging, interactive and analytics tool for the intended audience
• Delivering accurate analysis in a reliable, timely and secure manner
• Simplicity of maintenance, extensibility and content management
Implementation: Key Features
Some of the key features that Govalytics needed to support included:
• A highly interactive UX
• The ability to track the financial data analytics for each local government and offer five years of data providing historical trending and show projections
• Include government contact information from the mayor to the fire chief -- initially nearly 25,000 individuals needed to be in the database
• Include information for any current or future government capital project with projected yearly expenditures for future years within the projects
Implementation: Database
Several database technologies were evaluated according to the needs of the application and Penton operations:
• SQL including MS SQL and Oracle
• Oracle not an option (overall cost and corporate shift away from Oracle)
• Traditional SQL not flexible enough for evolving needs
• NoSQL as a solution
• Cassandra (key/value tabular hybrid)
• CouchDB (JSON document based)
• MongoDB (BSON document based)
Implementation: MongoDB
MongoDB was chosen as the database for several reasons that “fit” the application solution:
• The “document-like” structure of government data
• An application workload focused mainly on the retrieval of data
• No fixed schema - flexibility and extensibility in data representation
• Operational simplicity for scaling
• Built-in features that support the data analytics need such as map reduce and aggregation
• Product Maturity
Implementation: Architecture
Implementation: Architecture
• UX• ASP.NET MVC, Angular JS, HighCharts JS• Consumer of the PHP API• Proxy for Solr Searching and Filtering
• API: PHP / Drupal• Abstract away access to MongoDB• Government data import• General Administration
• Solr – Searching and Filtering the Data• DataImportHandler used to index• High performance index of the Government data• Future: Universal Search
• MongoDB• Government and other document-esque storage• Analytics (Forecasting, Aggregation, Map-Reduce)
Implementation: MongoDB Flexible Schema
• Governments (Cities, Counties, Towns)• Initial representation of a government
• Financial Data• Contact Data
• Extended later for• Key Statistics• Capital Projects• Government Scoring / Grading
• User Preferences• Profile preferences including default search• Saved searches• Started with Government and Function “favorites” • Added To-Dos, Notes, Alerts and Notifications
Implementation: MongoDB Flexible Schema
{ _id: 'ga-atlanta-city', name: 'Atlanta City', population: [], city_officials: [], total_expenditures: [], …}
Implementation: MongoDB Flexible Schema
{ _id: 'ga-atlanta-city', name: 'Atlanta City', population: [], city_officials: [], total_expenditures: [], … capital_projects: [], …}
Implementation: MongoDB Map-Reduce
Traversing the Government data to distill useful information. A simple example:
db.governments.mapReduce( function() { for (var i = 0; i < this.population.length; i++) { emit(this.population[i].year, this.population[i].value); } }, function(k, v) { return Math.min.apply(Math, v); }, { out: "calculated_population_min” });
Implementation: MongoDB Map-Reduce
Additional uses of Map-Reduce in Govalytics include:
• Calculation of minimums, maximums, averages and benchmarking data across
• National
• MSAs
• State
• Population Brackets
Implementation: MongoDB Aggregation
Used to perform additional minimum, maximum and rollup calculations. Example:
var k = 'statistic.key’;
var r = db.governments.aggregate({ $group: { _id: 0, min: { $min: k }, max: { $max: k } }});
Implementation: MongoDB Functions and Forecasting
• Extended MongoDB with additional functions
• db.system.js for storing reusability
• db.loadServerScripts() to make them available
• We created Govalyitcs-specific functions for:
• Forecasting data (best-fit curve projection)
• Normalizing imported data
• Processing the importing data
• Flattening data for subsequent search
Roadmap
Roadmap
Govalytics will add in the near future:
• Revenue/Expenditure analysis charts• Whitepapers section• Article/content archive• Universal search• State-based data• Resources compilation• Olark instant messaging with sales/help desk• Linkedin contact info• APIs for various users
* A product is never “finished,” but in a constant state of improvement and positioning itself in the market to better aid its users.
Screenshots
How Penton Won Local Government Business
Thank You!
Ian MarianoNorthPoint Digital www.northps.com
@northps
Ben WalkerPenton | Govalytics
www.govalytics.com @Govalytics