december 2015, volume 3, number 6 business 2015...

Executive Dashboard Running Smart Big Data Story Hour Craig Stedman: Hadoop’s Uncertain Identity

december 2015, Volume 3, Number 6

Business InformationINsIght oN maNagINg aNd usINg data

o o

More Than Thingsthe Internet of things could simply be about getting stuff online. but companies that put Iot data to use will find that there’s a lot more work—and rewards— involved in connecting devices.

plUS: manufacturing leads the Way on Iot

2015O N L I N E

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour

Iot opeNs up NeW aNalytIcs FroNtIer

INdustrIal World ahead oN Iot

craIg stedmaN: hadoop’s uNcertaIN IdeNtIty

2 business information • december 2015

EvERyTHIng EvolvES ovER time. Lately, it seems a lot of things have arrived at a place called the Internet. Thanks to advances in sensor technology, just about everything can be connected to the Internet, allowing data from all kinds of devices to be collected and analyzed. This phenomenon is known as the Internet of Things, or IoT, a phrase credited to British technologist Kevin Ashton from a presentation he gave in 1999.

But after more than 16 years, we’re still hearing about the IoT’s potential. “The IoT is a concept that is, in itself, transformational,” reads a recent Gartner report. Al-though many algorithms and tools can be applied to the IoT, the report continues, it presents new challenges that require features typically not found in traditional analyt-ics. With that in mind, mass use of IoT applications could well be a few more years away. In this issue of Business Information, we take a look at IoT users and vendors that are getting a jump on this much-hyped technology.

In the cover story, executive editor Craig Stedman presents compelling evidence of real-world IoT applications and the benefits they’re having. It takes some doing. “We knew there was inherent value in [IoT] data,” said Christopher Dell, senior director of product development and management at Intelligent Mecha-tronic Systems, or IMS. “We just didn’t know how to

unlock that value.” But they did. So why aren’t more businesses buying into the IoT?

Like IMS, organizations looking to collect and analyze IoT data, writes Stedman, often find they first need to beef up their IT architectures with big data management technologies and advanced analytics tools.

Those investments don’t have to be overwhelming. Some manufacturers may have to upgrade old equipment for new machines that accommodate the IoT, explains executive editor David Essex in another feature, but it may be possible to add connectivity to existing equip-ment in some cases. “It’s shocking ... how many very large industries can benefit from [the IoT] and haven’t thought about it,” said FreeWire Technologies CEO Ar-cady Sosinov.

Also in this issue, the Meeting Room feature describes how British Gas is updating its energy-use meters and dumping customization in favor of standardization. Our look at emerging technologies and concepts—What’s the Buzz?—helps illustrate the meaning of data storytelling. And Hindsight reveals potential successors to some of Hadoop’s core components. n

Is your company considering incorporating the IoT into its planning strategy? Write to me at [email protected].

EDIToR’S noTE | roN karjIaN

Connecting All Things great and Small

mailto:[email protected]

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour





TREnD SpoTTER | executIVe dashboard

35%

31%

29%

25%

24%

24%

19%

17%

14%

11%

10%

source: tdWI’s EmErging TEchnologiEs for BusinEss inTElligEncE, AnAlyTics, And dATA WArEhousing; 216 bI aNd data proFessIoNals selected all applIcable optIoNs

Clouds Dispersethe cloud is becoming more accepted by It pros in data management and is used for a variety of purposes.

What they plan to or already use the cloud for:

data warehouse

analytics platform

sandbox for analytics

platform as a service

data integration

software as a service for analytics

hadoop

data security to protect cloud data

data quality

Internet of things connectivity

Infrastructure as a service for analytics

source: tdWI’s EmErging TEchnologiEs for BusinEss inTElligEncE, AnAlyTics, And dATA WArEhousing, WhIch reported results From a surVey oF 403 bI aNd data proFessIoNals

35% use the cloud in some way for data

management or analytics

35% are thinking about it

17% say they would never use the cloud,

compared with 25% in 2013

13% are not sure

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour





TREnD SpoTTER | VerbatIm

“ It’s a replacement. spark is much faster than hadoop is. and from a productivity standpoint, you don’t have to do [analytical] modeling in a separate tool.”hakaN joNssoN, data scientist for the lifelog product team at sony mobile communications

“ spark is going to replace a large part of what we use mapreduce for now. and, over time, if spark continues to expand its func-tionality, it could completely replace mapreduce.”joe hsy, director of cloud services platforms and tools for cisco’s Webex unit

“ spark doesn’t really store anything. processing in spark is replacing mapreduce and yarN, but the storage layer is going to be hadoop for a long time.”srIdhar alla, big data architect at mass media company comcast

“ If you want to be revolutionary, you can say hadoop is dead. but hadoop isn’t dead.”charlIe crocker, business analytics program lead at software vendor autodesk

Team player or Future Rival?Business information asked attendees at strata + hadoop World 2015 in New york city if they believe the spark processing engine is a complement to hadoop or an alternative to the big data framework and components such as the yarN resource manager and mapreduce programming environment.

“ there are four or five performance-challenging [surveillance] patterns in our portfolio ... target- ed for spark. long term ... the jury is still out.”brett shrIVer, senior director of market regulation technology at the Financial Industry regulatory authority, or FINra

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour




RAREly DoES CHAngE occur on so large a scale as the for-midable challenge British Gas is confronting. The U.K. energy provider needs to replace 15 million energy-use meters with new, government-mandated “smart” meters by 2020.

To handle this massive undertaking, the entire logis-tics process had to become a lot more nimble. So British Gas pared down its SAP supply-chain infrastructure to the essentials. The company, based in Staines, England, is using big data analytics to fine-tune the process.

When phil Crannage was appointed core systems di-rector at British Gas two-and-a-half years ago, one of his first assignments was to organize the meter-replacement project. “This project was only one of maybe 100 we had going on at the time,” said Crannage, who is responsible for the organization’s SAP system. “It was just quite an important one for me because it was a talisman for the way we were delivering change for the business.”

He wanted to prove that change can happen quickly and doesn’t have to cost a fortune as long as the project’s outcome remains the primary focus. As Crannage put it, “Let’s give the business what they want to achieve, not [necessarily] what they asked for.”

Abandoning some of the old methods of doing busi-ness was critical to the project. Users were comfortable


Running Smart

nAME: phil crannage

TITlE: core systems director

oRgAnIzATIon: british gas

HEADqUARTERS: staines, england

TREnD SpoTTER | meetINg room

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






working with customizations in SAP modules for billing, scheduling, warehouse management, human resources and customer relationship management, or CRM. So Crannage set about convincing users that the IT depart-ment had strong ideas for a more streamlined logistics process. He asked stakeholders to imagine what a world-class logistics operation would look like: “ ‘If I can give you that, would you change the way that you work to meet how I’m going to deliver it?’ And they said, ‘Of course we would.’ ”

A SIMplIFIED AppRoACHCrannage then turned his attention to enlisting the ser-vices of an outside integrator. He and his project team drew up a request for quotes that included a list of de-sired outcomes for the entire project. Tata Consultancy Services Ltd. (TCS) was selected because the company demonstrated it could use standard SAP features to achieve those goals.

“We contractually bound TCS to limit the amount of customization,” Crannage said, so a governance process was established, whereby a steering committee would decide which customization requests to accept or reject. It helped that British Gas supply-chain general manager Glyn Williams supported the approach. “He said ‘no’ to things that I would say ‘yes’ to,” Crannage noted. “He really got the concept of trying to stay as standard as possible.”

Heavily customized materials records, for example, would now be standardized and updated to handle smart

meters. “There are some custom fields in there, but they’re vastly reduced in number,” Crannage said. Data capture was shifted from early in the logistics process to later, when it was most helpful in provisioning meters. The old CRM user interface was re-architected for HTML 5 and cloud-friendly RESTful APIs. “That enables us to make quite significant process changes without making any back-end changes,” Crannage explained.

Throughout the newly streamlined logistics process, a Hadoop cluster captures data; business users analyze it in QlikView with help from data scientists put in place by Crannage. “We have a lot of people looking at the data,” he said. “Fifteen million installations is not a small number, so we’ve got to be as efficient at delivering this as possible. Where can we cut a few seconds off a job, or where [can] we save some money on materials?”

A major challenge was training more than 1,500 smart-energy engineers, 150 scheduling agents, 60 warehouse and logistics workers and 8,000 CRM users to solve problems on their own. The solution was to

Read more profiles of business and IT professionals.

“ lET’S gIvE THE BUSInESS WHAT THEy WAnT To ACHIEvE, noT [nECESSARIly] WHAT THEy ASkED FoR.” —phil Crannage

http://searchcontentmanagement.techtarget.com/essentialguide/Change-Agents-Leaders-in-business-applications

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






colocate the business and IT teams, which encouraged seamless collaboration “to the point where you didn’t really know there was a difference between the orga-nizations,” said Ash Jokhoo, director of programs and engagement at Centrica, the holding company of Brit-ish Gas. “That really helped people clear their minds to work in a new way and embrace SAP not in the way that we’ve used SAP before, but in the way SAP has already designed the system.”

In phil They TrustCrannage insists the new logistics process isn’t as com-plicated as it may appear. Smart meters are distributed to local British Gas engineers, daily job orders are transmit-ted to them and the replaced meters are returned. “We have to have stock locations all around the country,” he said. “The engineers go to the local post office to collect meters.” British Gas averaged about 3,000 swap-outs a day during the summer.

The next phase will focus on increasing the project’s capacity to handle additional meters and engineers as well as larger volumes of transactions, which calls for more error handling and exception management. Mean-

while, British Gas uses Hadoop analytics reports to help improve energy efficiency for customers that have the smart meters. The new metering system consists of smart meters for recording gas and electricity usage, a commu-nications hub and an in-home smart energy monitor.

Crannage’s many years of program management expe-rience and proven people skills coincide with his philos-ophy on the important role IT and big data analytics can play in transforming a business. “Trust is a big factor,” he asserted. “You can’t just say it’s so. You have to have a compelling case. And once you’ve got that compelling case, it’s actually quite easy to bring people along. [But] they still need to see progress. They need to believe in you.” —DAvID ESSEx

“ FIFTEEn MIllIon InSTAllA- TIonS IS noT A SMAll nUM-BER, So WE’vE goT To BE AS EFFICIEnT [AS poSSIBlE].” —phil Crannage

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour





TREnD SpoTTER | What’s the buzz?

AS oRgAnIzATIonS increasingly use business intelligence and analytics tools to drive more effective decision mak-ing, IT teams are faced with the challenge of explaining their findings in ways business executives can easily understand. That has led to the rise of data storytelling: crafting a narrative based on analytics information, usu-ally with the aid of charts and other data visualizations.

THE BUzzProponents say storytelling engages business managers better than simply handing them unvarnished data to decipher. Pamela Peele, chief analytics officer at health-care system UPMC’s insurance services division, calls a former journalist hired to tell data stories one of her most important team members. BI vendors are touting the storytelling features of their products, education outfit TDWI has added a course on the topic to its con-ferences, and colleges such as Boston University and Georgetown offer workshops and adult education classes in storytelling.

THE REAlITyA search for data storyteller on the jobs site Indeed yielded just two listings, and both were job descriptions for data analysts with storytelling skills. Tapping such

professionals is one way to go, but data scientists adept at crafting data stories and communicating with busi-ness users can be a rare breed. Also, it’s possible to go overboard with visualizations and create graphics that, as consultant Claudia Imhoff put it, “are lovely to look at but unintelligible.” That scenario isn’t likely to produce a story with a happy ending. —CRAIg STEDMAn

Big Data Story Hour

IllustratIoN: chrIs seero


HoME

DATA MAnAgEMEnT | craIg stedmaN

IoT opEnS Up nEW AnAlyTICS FRonTIERdata flowing from the Internet of things creates opportunities to analyze equipment performance and track the activities of drivers and users of wearable devices. but the It needs are nearly as big as the Iot itself.

Every day, Intelligent MechatronicSystems Inc. collects 1.6 billion data points from hundreds of thousands of automobiles in the U.S. and Canada.

The cars are equipped with devices that track driving distance, acceleration, fuel use and other information on how the vehicles are being operated—data that In-telligent Mechatronic Systems, or IMS, uses to support usage-based insurance programs and fleet and traffic management initiatives. Until the middle of this year, the data was stored in a MariaDB relational database, but the open source software imposed rigid limits on how the data could be structured, which complicated efforts to analyze it.

“We knew there was inherent value in the data,” said Christopher Dell, senior director of product develop-ment and management at IMS. “We just didn’t know how to unlock that value.”

So in August, after a yearlong project, IMS added an Apache Cassandra NoSQL database along with data in-tegration and analytics tools from Pentaho. Now, data



flows through the MariaDB system into Cassandra, giving the Waterloo, Ontario, company’s data scientists more flexibility in formatting the information. That setup lets the analytics team perform finer-grained anal-ysis of customer driving behavior in search of patterns and trends that could help insurers fine-tune their us-age-based policies and rates.

In addition, Dell said the new technologies should allow IMS to better handle future data growth, which is expected to be generated by two developments. The com-pany hopes a newer smartphone app for collecting vehi-cle data will make its subscriber base “skyrocket.” There’s also an ongoing move to combine driving information with other types of data from the Internet of Things—or IoT—such as weather records and telematics captured by so-called smart home systems.

Like IMS, organizations looking to collect and ana-lyze IoT data often find they first need to beef up their IT architectures. That applies to companies on both the consumer and enterprise sides of the IoT fence: The challenges of pulling in and processing large amounts of data from onboard diagnostics gear, industrial sen-sors, fitness trackers, phones and other devices know no business boundaries. Upgrades typically include big data management technologies such as Hadoop, the Spark processing engine and NoSQL databases, plus advanced analytics tools that can support machine learning and other algorithm-driven applications. And in many cases, a mix of technologies is needed to meet all IoT analytics needs.

Big Data ExpansionCisco Systems Inc.’s WebEx unit installed a Hadoop cluster three years ago to store data from mobile devices and PCs that connect to its Web and video conferencing services. After initially developing some standalone an-alytics applications for individual departments, WebEx in early 2014 adopted a unified strategy for tracking usage, analyzing performance and diagnosing technical problems at its end and on customer networks. But this year, Cisco—based in San Jose, Calif.—had to expand the Hadoop system and augment it with additional big data tools. The expansion was prompted by new types of analytics as well as an increasing data load that amounts to multiple terabytes daily, with the total collected ap-proaching a petabyte.

Joe Hsy, director of WebEx’s cloud services platform and tools, said his team added 30 nodes to the Cloud-era-based cluster in the fall, boosting the total to more than 100 nodes. And last summer, WebEx began using the Apache Kafka message-queuing technology to more

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour




CHRISTopHER DEll, senior

director of product development

and management at Intelligent

Mechatronic Systems, deployed

a noSql database and new

analytics tools to better handle

IoT data.



quickly feed telemetry data into the Hadoop cluster and other systems to support real-time performance monitor-ing and alerting.

In addition, the conferencing unit is starting to use Spark to filter streams of incoming data into subsets

for analysis and to power a prototype machine learn-ing application aimed at improving its ability to detect fraudulent phone calls. The infrastructure also includes Cassandra, used to store event logs for diagnostics, and Oracle databases that hold meeting-history data. At

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour




Data privacy a Big Thing on IoTWHEn THE lIFElog architecture development and ana-

lytics team at sony mobile communications set up a

program for capturing and analyzing data from wear-

able devices linked to the activity tracking app, its

biggest challenge wasn’t technical in nature. It was

getting clearance from the company’s legal depart-

ment on the proposed data collection practices and

privacy protections.

jonas sellergren, who manages the team, said at

strata + hadoop World 2015 in New york city that it

took some doing to balance “this opportunity and

the risks a company exposes itself to with potentially

sensitive data.” and there’s a lot of data involved: cur-

rently 5 terabytes per month, which sony processes

and analyzes in a spark system running in the ama-

zon Web services cloud.

to help ease the legal concerns, hakan jonsson, a

data scientist on the lifelog team, said they adopted

an “obvious-data-usage principle” specifying they

should only collect data that can benefit the app’s

users. In one case, he said, “we decided not to collect

a piece of data that was useful to us [because] we

couldn’t find a use for it from the user point of view.”

data privacy on the Internet of things is a partic-

ularly big deal in consumer applications. during a

strata session at the conference, charles givre, a data

scientist at consultancy booz allen hamilton, detailed

his use of a connected-car device sold by automatic

labs Inc. If the data collected isn’t protected properly,

he said, “you can really start to get a scary level of in-

sight into someone’s daily life.”

In an interview before the conference, automatic

labs’ vice president of software engineering, rob Fer-

guson, emphasized that privacy is a top priority, even

inside his company. “everyone on my staff has train-

ing and clear guidelines that they’re not supposed to

look up information without a driver’s consent,” he

said. n

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






the front end, WebEx primarily relies on Platfora’s Ha-doop-based analytics tools, plus Tableau and Excel for business users.

The growing number of technology options enables organizations like WebEx to do more with IoT data than they could previously. But managing Hadoop, Spark and related open source tools can be a challenge, Hsy said, pointing to bugs and administration shortcomings partly due to the fast pace of development and updates on those technologies and commercial versions of them.

Looking for a balance between new features and sta-bility, WebEx typically stays one release behind the latest version of the technologies it’s using. “We’re an opera-tions team, so we can’t be cutting edge,” Hsy said. But he added that keeping an eye on development plans and big data trends is crucial in such a fast-moving environment. “That’s a big part of what we have to do as a team—not only keep the systems running but look to the future to what’s coming next.”

IoT analytics is still the province of early adopters. According to a TDWI survey conducted in May, only 16% of the 303 respondents said their organizations were analyzing IoT data. Another 33% said possible initiatives were being considered [see “Not Now, Maybe Later”]. But in its fourth annual “hype cycle” report on the IoT, published in July, consulting and market research firm Gartner predicted that most IoT technologies and pro-cesses are still five to 10 years away from mainstream adoption. There’s a good reason for that. Gartner analysts wrote that IoT deployments pose “formidable” challenges

not now, Maybe laterthe percentage of organizations actively analyzing Iot data remains in the low double digits, although one-third of survey respondents said the topic is on the table in their companies.

source: tdWI’s emErging TEchnologiEs for BusinEss inTElligEncE, AnAlyTics And dATA WArEhousing; based oN respoNses From 303 It aNd busINess proFessIoNals; IllustratIoN: photoraIdz/Istock

16% yes

33% No, but we’re thinking about it

33% No, and

we have no plans

18% don’t know

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour





and typically require investments in a “daunting array” of technologies, plus new data management and analytics skills.

parsing petabytes of DataFor many companies, the volumes of data that the IoT can deliver take them into an entirely new processing realm. “We’re collecting data at fairly extraordinary rates, at least in my world,” said John Dyck, global director of software business development at Rockwell Automation Inc.

Over the past three years, the Milwaukee-based com-pany has built a new architecture based on Microsoft’s Azure cloud platform to capture data on equipment and plant operations from manufacturing control sys-tems at customer sites. According to Dyck, more than 100 companies in the oil and gas industry and various manufacturing verticals are currently using the cloud setup; Rockwell manages and analyzes the data for about two-thirds of them, while the others handle these tasks themselves.

The data Rockwell stores in the cloud architecture already measures “in the single-digit petabytes overall,” Dyck said. The amount generated by individual plant-floor devices is easy to handle. But collecting information from a thousand pieces of equipment in a factory multi-ple times per second—and repeating that process across multiple facilities and different customers—is “a whole different story,” he noted. “It took us a while to address that and work through it.”

Eventually, the company created a schedule akin to a shipping manifest for managing the collection process; it also built in buffering and store-and-forward mech-anisms that help ensure data is gathered “even when the network pipeline gets narrow or goes offline,” Dyck explained.

And after initially relying solely on Microsoft’s rela-tional Azure SQL Database, Rockwell last year added HDInsight—the vendor’s Hadoop distribution—as a sec-ond-stage repository to boost its ability to handle all the incoming data.

The industrial automation company also is tapping the Hadoop system, along with Microsoft’s Azure Machine Learning analytics technology, to help move be-yond conventional performance monitoring and develop predictive models that can expose possible equipment failures before they occur. Rockwell is running pilot proj-ects on that with a half-dozen customers, Dyck said. In the past, he added, predictive maintenance “was almost a pipe dream” because most manufacturers left the data


joHn DyCk, global director of

software business development

at Rockwell Automation, said

access to more data allows

data scientists to build

algorithms that make more

accurate predictions.

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






generated by different devices in isolated databases. But now, with access to larger pools of data, Rockwell’s data scientists are building automated algorithms that, Dyck said, “are making predictions we can be confident about.”

Such algorithms require a mix of current and

historical data, and getting the historical data out of all those isolated manufacturing databases requires a robust architecture. “If it takes you six months to ingest three years of history, that’s kind of a deal-breaker,” acknowl-edged Alex Bates, co-founder and CTO at software ven-dor Mtell. In March, the company announced a platform

Skills Shortage Makes IoT Hirings pricey IT MAy BE the Internet of things, but it takes people to

make Iot data management and analytics projects

work. the problem is there aren’t enough workers

with skills in hadoop and other big data technologies,

at least at salary levels that won’t bust budgets.

joe hsy, director of the cloud services platform and

tools at cisco systems Inc.’s Webex conferencing unit,

said the competition in the job market is intense on

both the processing and analytics sides. Webex uses

hadoop, spark, cassandra and other in-demand tech-

nologies to collect and analyze data from the mobile

devices and pcs of people using its Web and video

conferencing services. to help ensure that his team

can use those technologies effectively, hsy said he is

working to expand the skills of existing employees,

while “aggressively hiring” new workers when he

finds them.

the skills shortage was one of the reasons

automatic labs Inc. adopted databricks’ cloud-based

spark platform after the san Francisco company

initially began doing its own development with the

base apache spark open source software. “It’s very

difficult to find infrastructure engineers in san Fran-

cisco,” said rob Ferguson, vice president of software

engineering at automatic labs. “and the cost is just

tremendous—probably $200,000 a year per person.”

It’s the same at rockwell automation Inc., which

used microsoft’s azure cloud technologies to build

a platform for storing and analyzing data captured

from manufacturing equipment at companies that

use its industrial automation tools. “We don’t want to

be building infrastructure that’s plumbing,” said john

dyck, rockwell’s global director of software business

development. “We want to focus on manufacturing

functionality and analytics because that’s our value-

add.” n

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






combining a predictive maintenance application with a data repository based on the Hadoop distribution from MapR Technologies.

Of course, not all the data that’s there for the taking on the IoT is necessarily golden. Like IMS, Automatic Labs Inc. sells devices that plug into the onboard di-

agnostics port in cars and collect data to track vehicle performance metrics. The devices send back everything that’s made available through the port, amounting to mil-lions of rows of data per day, according to Rob Ferguson, the San Francisco company’s vice president of software engineering.

The data goes into an Amazon Simple Storage Service, or Amazon S3, repository for processing and analysis on a pair of cloud-based clusters running Databricks’ version of Spark. But not all of the data is used. “Right now, we collect more data than we know what to do with,” Fergu-son said.

He added that Automatic Labs wants to hold onto as much of the information as possible, partly to enable more advanced analytics applications for new usage-

How you DidIntelligent mechatronic systems uses the data it captures from automobiles to send graphical reports on vehicle and driving metrics to the smartphones of drivers after they complete trips.

source: INtellIgeNt mechatroNIc systems

oF CoURSE, noT All THE DATA THAT’S THERE FoR THE TAkIng on THE InTERnET oF THIngS IS nECESSARIly golDEn.

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






based insurance and fleet management services it’s launching. But to keep the size of the company’s Amazon S3 stores from getting out of hand, Ferguson’s team filters out some data “in cases where we’re getting more noise than signal.” For example, he said voltage data has been skewed by the growing use of hybrid cars such as the Toyota Prius.

As for IMS, the billions of data points it’s collecting from cars don’t add up to a huge amount of bytes. Dell said the company has a total of 75 terabytes in its sys-tems. Initially, however, IMS spread the data across separate environments for the different insurers it works with. Pulling all that information together into the Cas-sandra database and making it consistent without affect-ing the data’s integrity “was more of a challenge than we expected upfront,” he noted.

Now, the company is using the centralized repository to push forward on new analytics initiatives. In addition to the trip reports it sends to the smartphones of drivers [see page 15, “How You Did”], IMS released in Septem-ber a set of analytics tools and dashboards that allow in-surers to track their usage-based insurance programs and identify driving behaviors that could trigger changes to insurance policies.

Dell’s team also launched a 1.0 version of the analyt-ics tools for internal business users in operations and finance. “We kept it fairly locked down on what they

could do,” he said, adding that he wanted to make sure the upgraded architecture could handle the additional workload. He’s planning a second release in early 2016 that will give internal users more freedom to run queries against the Cassandra system. And with the new database in place and tested out, Dell said he’s now confident that IMS “won’t have to pull data into another data store to do analytics.”

For many other organizations trying to tap into IoT data, though, the challenges of combining various big data platforms and analytics tools into an architecture that can stand up to the flood of information are just beginning. n

CRAIg STEDMAn is an executive editor in TechTarget’s Business Applications and Architecture Media Group. Email him at [email protected].

RoB FERgUSon, vice president

of software engineering at

Automatic labs, works with

his team to keep data that isn’t

useful out of the company’s

Amazon Simple Storage Service

repository.



HoME

InDUSTRIAl WoRlD AHEAD on IoTmanufacturers have a jump start connecting “things,” and names like ge and rockwell automation provide a blueprint for tackling the challenges of collecting and analyzing mounds of sensor data.

IMplEMEnTATIon | daVId essex

For decades, the most advancedmanufacturers have been outfitting their plants and warehouses with the kinds of networked sensors now being hyped as the Internet of Things.

But those systems were rarely networked via the Inter-net Protocol—the minimum definition of the Internet of Things, or IoT— much less monetized through big data analytics, and sensor data was often discarded.

Companies wanting to invest in IoT technologies can build on that foundation of sensors. But before they do, manufacturers must first decide which of the three main applications of the industrial IoT—supply chains, facto-ries or products—to focus on. Each one requires “very different thought processes,” according to Bill McBeath, an analyst at ChainLink Research.

Supply-chain deployments are showing up frequently in the trucking industry, where logistics platform pro-viders such as Descartes have long tapped telematics and GPS to optimize routes. “We see more and more people taking advantage of the sensors that are already in the trucks,” McBeath said.

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






Data from a truck’s GPS and sensors, which can detect hard braking and acceleration, are analyzed for fleet as-set management, route optimization and driver perfor-mance. “Some [trucks] are starting to have other kinds of sensors that have more to do with the cargo, like tem-perature sensors,” he continued.

This collected data can sometimes be processed in real time. Cloud platform provider GT Nexus has part-nered with supply-chain analytics vendor TransVoyant to combine GPS data with information about the cargo on ships and trucks, thus providing shippers with more precise details on the location and status of individual shipments. Also tapped are data streams that can affect shipping and delivery routes. Data about weather, street traffic and even information from Twitter feeds can, for example, indicate crowd patterns surrounding an event such as a ballgame or concert. Armed with all this addi-tional data, shipping and delivery companies can better serve their customers by providing more accurate esti-mates of arrival times, McBeath noted.

Another vendor, Fleet Advantage, normalizes data collected from trucks so clients can offer their customers new leasing options. “They compensate for the terrain, the time of year and the heaviness of the load so they can rank [trucks] and … say which are performing better,” McBeath said.

Some trucking companies are making even more so-phisticated advances with sensor technology. Mining giant Rio Tinto, from its Australian operations center in Perth, controls self-driving trucks at its Pilbara mines

about 900 miles away. “The whole [operation] can run like a gigantic factory with a bunch of robots,” McBeath explained.

A Rio Tinto competitor, mining equipment maker Joy Global, employs sensor technology in a remote-con-trolled extraction device that can be sent deep into mine-shafts to do dangerous work normally handled by miners. “That particular machine has something like 7,000 sen-sors on it,” McBeath noted.

Meanwhile, tech vendors are devising less disruptive ways to add sensors, he said. Bluvision, for instance, makes peel-and-stick sensors for temperature and vibra-tion that can go on electric motors. A small box plugs into a power outlet, reads data from sensors and trans-mits it over Wi-Fi.

Wheel Away power WorriesAnother potential catalyst for IoT implementation is the concept of product as a service, or selling the product’s

BIll MCBEATH, an analyst

at Chainlink Research, pointed

to the trucking industry as one

market that is taking advan-

tage of and experimenting

with sensors.

(Continued on page 20)



home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour




FreeWire Charges Ahead

MAnUFACTURERS ARE REAlIzIng that smarter analytics

of bigger data from more sensors can extend the life

of their expensive equipment. FreeWire technologies

Inc., based in san leandro, calif., takes that concept a

step further by making its large, long-lasting battery

packs mobile and outfitting them with wireless Inter-

net to sell “charging as a service.”

FreeWire’s primary market, electric vehicles, has

an infrastructure problem: not enough charging sta-

tions. but what if it’s possible to make charging sta-

tions mobile by recycling second-life batteries? that’s

what FreeWire’s mobi charger does, with the help

of an on-site technician who takes orders and pay-

ments through a smartphone app. In another market,

FreeWire’s mobi gen replaces diesel generators that

power the bright lights used for nighttime highway

projects and other construction sites.

sensors play a big role in testing and matching the

battery modules that go into each power pack. “each

module is connected to a machine that is able to basi-

cally simulate real-world uses,” sosinov said. “In that

simulation, it tests a lot of things in the module—ev-

erything from heat to resistance internally to voltage

to current to the weather conditions, et cetera.” a

central processing unit, or cpu, consolidates sensor

data and sends it to a sQl database in the amazon

Web services cloud.

sosinov’s experience has taught him a few things

about making Iot projects practical. he advises

against trying to build too much in-house. “there’s a

lot of low-hanging fruit out there, where you can go to

current technology and, frankly, all you have to do is

network it to create this really powerful platform,” he

said. FreeWire will soon replace its home-grown cel-

lular modems and cpu with Iot gateways from euro-

tech, which works with cellular carriers to ensure Iot

data gets first priority.

In addition, sensors typically can be used off-the-

shelf. “they usually communicate via standard pro-

tocols like [controller area network],” sosinov ex-

plained. “all you have to do is read that protocol.” n

The Mobi gen (left) and the Mobi Charger (right) are mobile charging stations that can power personal devices and electric vehicles, among other things.

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






services rather than its features. McBeath called this trend “more revolutionary than evolutionary. People are trying to figure out what changes they should make to the product itself and what kinds of sensors to put in. There are opportunities for new kinds of data and new kinds of service that didn’t exist before.”

FreeWire Technologies Inc., based in San Leandro, Calif., offers two products—or services—that would not have been possible before the advent of the IoT’s rela-tively inexpensive sensors and cloud computing. The company’s Mobi Gen mobile power generator and Mobi Charger mobile electric-vehicle charging station rely on an Internet connection to operate. “There’s data going up to the cloud, and then there’s data and commands going back down to the Mobi,” said CEO Arcady Sosinov.

That data stream travels to two locations: a dashboard app used by FreeWire and its customers to monitor Mobi Gen and, say, turn off power to items connected to it, and the E-car Operation Center, where Siemens software serves as an interface between electric meters and utility companies. “Utilities can now see Mobi as a distributed asset,” Sosinov said.

The Mobis reach the Web through a 4G cellular con-nection, but they also have mesh networking, so if one device goes out of range, then another one can output the data to the cloud. “We output a stream of data every five seconds from each device, so over the course of a month, it’s probably 100 MB per month per device,” Sosi-nov explained, adding, “It’s shocking to me how many

[IoT] things are not actually expensive, and how many very large industries can benefit from it and haven’t thought about it.”

Upgrades to the Shop FloorCompanies such as General Electric that are both users and purveyors of IoT tools are at the forefront of the technology. Likewise, Bosch, Cisco Systems, Intel and Siemens are getting into the IoT business and at the same time are using the technology to improve their own pro-duction processes. Industrial and building automation mainstays Johnson Controls and Rockwell Automation Inc. also have significant deployments underway.

Rockwell, based in Milwaukee, has completed about 80% of the planned IoT upgrade of its global plants, ac-cording to John Nesi, the company’s vice president of market development and head of its Connected Enter-prise initiative. EtherNet/IP, an open network standard for interoperability between corporate IT and industrial applications, underpins Rockwell’s effort. Integrating


ARCADy SoSInov, CEo of

FreeWire Technologies, uses

IoT technology to operate two

of his company’s products,

the Mobi gen and the Mobi

Charger.

(Continued from page 18)

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






Rockwell’s SAP ERP system with its own manufacturing execution system, FactoryTalk, is also a major part of the upgrade.

Rockwell’s IT and operations departments collabo-rated on developing a common methodology and process for normalizing data, Nesi said. He recommends that companies considering an IoT project integrate a small team of IT pros into operations so IT expands beyond its traditional role of compiling reports.

To accommodate the IoT, some manufacturers may have to swap out decades-old equipment for new ma-chines that have current connectivity built in. But it may well be cheaper to simply add controllers to the process. “The control system is typically less than 10% of the costs of the machine, 20% at most,” Nesi said.

Since the Internet can be a relatively inexpensive way to connect people, computers and sensors, one of the IoT’s potential strengths could be in divining all this collected data. But even if manufacturers have already installed sensors and the IP gateways to link them to enterprise applications, they’re still having trouble jus-tifying the monetary costs associated with extracting intelligence from all that connectivity without first es-tablishing a tangible use, according to Maribel Lopez, founder of Lopez Research.

“You’ve got all that, and it still hasn’t gotten you any-where,” Lopez said. “You still have to figure out, ‘Am I going to buy Hortonworks Hadoop or Apache Spark?’ [And] where does the data go?” There might also have to be an API to bring data back into the software that runs

the manufacturing process.The data choices are generally divided into two cate-

gories, according to Lopez. Platforms and vendors such as Axeda, Bright Wolf and LogMeIn focus on collecting data, while others such a GE’s Predix and the industry cloud jointly introduced by Siemens and SAP supply the analytics. Predix, for example, “figures out if something’s going to go wrong,” Lopez said.

Structured data problems also can be easily handled by Hadoop-style products. “I think the big data guys,” she added, “can take what they generically have and apply it to the problem with a combination of marketing and


Rockwell Automation Factory- Talk software aggregates data from IoT sensors, allowing managers to assess the plant’s output and quality.

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






A Dose of Realism for IoT

MARIBEl lopEz, FoUnDER of lopez research, thinks

manufacturing may have a head start on Internet of

things, or Iot, applications, but “it’s a lot of talk and

not a lot of action,” she said. “First of all, the phrase

ioT is meaningless because it doesn’t talk about any-

body doing anything that’s useful. just connecting

your stuff is not enough.”

For the Iot to be meaningful, several things must

happen, she reasoned. First, “things” must be con-

nected securely—hardly a no-brainer. It can be hard

to get wireless connectivity into manufacturing fa-

cilities that are laden with concrete walls and heavy

iron pipes and machinery. because of that potential

interference, the manufacturing process “may, in

fact, need to be a wired Ip network,” lopez sugges-

ted.

provisions might have to be made for error-free

data transmission if the application requires it. the

data must get into workers’ hands, perhaps in mobile

apps on ruggedized tablets. “that’s not necessarily

cheap hardware,” she said. “once you collect that

data, you have to process it somewhere. do you haul

it all the way back to the cloud, or do you process it

locally?”

then there’s the debate

about whether the data

should be real-time. lopez

explained, “sending me an

alert every five seconds

that says the manufacturing plant is 25 degrees is

not useful. saying that the vibration is out of range

is interesting yet not sufficient. saying the vibration

is out of range and if it continues for the next two

hours, it’s going to shut down the plant—that’s more

interesting.”

lopez has noticed a distinct shift in favor of local

data management and analytics over the past year.

“you’re going to want to do a lot more of that locally

than you would have thought,” she said. the tera-

bytes of data coming from a turbine every day, for ex-

ample, can’t realistically be sent over a cellular data

network to the cloud then back for analysis.

“you need to figure out how you’re going to do

some local processing of that data,” lopez explained,

“and you need to figure out some rules around what

matters and what doesn’t when it comes to getting

the analytics back. think about right time versus

real time.” n

lopez

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour






maybe a few additional tool sets.”Connectivity alone won’t cut it. “Nobody gets budget

funding based on that,” Lopez emphasized. “The one thing that you need to be able to do with [the] IoT is pick a very specifically scoped use.” And it helps to get approval for infrastructure that can accommodate other uses later.

Safety FirstSecurity is another important consideration. Too many manufacturers rely on “security by obscurity,” according to Nesi, gambling that no one will bother to crack into their rusty legacy systems. What they need is a security system that’s multipronged and involves multiple ven-dors. Nesi advised companies to be skeptical of any pro-vider that claims it can handle all IoT security issues. The biggest culprits can be ERP vendors; their systems are usually limited to supply-chain transactions, Nesi said.

As for the numerous IoT standards in the works, Nesi said, “they’re constantly evolving, and that work is never done … The [network] stack is very deep and complex. There are standards at all these different levels,” from how devices talk to each other to security methods such as digital signatures. But the lack of approved standards isn’t a reason to delay implementing IoT apps.

“Don’t wait,” Nesi warned. “One of the challenges for a 100-year-old manufacturing company is how to now act like a startup. But I think that’s what they need to do: Fail fast and do lots of little experiments and learn as they

go, because it takes a long time to transform what you do. And if you wait, you’re just postponing the learning process.”

The IoT is forcing manufacturers to rethink their busi-ness models, according to Lopez. She sees IoT costs as analogous to the way cloud software became financially attractive because it could be paid for out of operational expenses, which are generally easier to justify than capi-tal expenses.

Lopez also sees similarities in the way the cloud shifted maintenance responsibilities to the provider. “I think this is a similar business model in terms of hard-ware,” she stated. Instead of the hardware, IoT providers will use software and sensor networks to sell uptime. “They’ll make sure that the asset is never out of service and that they’re getting paid to do that,” Lopez said. n

DAvID ESSEx is an executive editor in TechTarget’s Business Applications and Architecture Media Group. Email him at [email protected].

(Continued from page 21)

joHn nESI, Rockwell Automa-

tion’s vice president of market

development, said a lack of

approved standards shouldn’t

cause companies to delay im-

plementing IoT applications.




HoME

pooR MApREDUCE. Until late 2013, it was a critical cog in all Hadoop systems, serving as both the cluster resource manager and the primary programming and processing environment for the open source big data framework. But then things started to change.

The Apache Software Foundation’s Hadoop 2 release added a new technology called YARN that usurped the resource management role and opened up Hadoop to applications other than MapReduce batch jobs. A still-growing gaggle of vendors rolled out SQL-on-Ha-doop tools that let users write analytical queries against Hadoop data in standard SQL instead of MapReduce. And the Spark processing engine burst onto the scene, with proponents claiming it can run batch jobs up to 100 times faster than MapReduce, while supporting

higher-level programming in popular languages such as Java and Python.

With all those forces arrayed against it, MapReduce has been, er, reduced in stature—like an old steam en-gine being forced to give way to sleeker diesel locomo-tives. That sense was palpable at the Strata + Hadoop World 2015 conference in New York City, where various attendees talked about trying to get away from Map- Reduce—in the words of one speaker, “as soon as possi-ble and as much as possible.”

The no-more MapReduce sentiment reached its apex at a session on MapReduce Geospatial, an open source toolkit for use in processing satellite images and other large sets of raster data. It turned out that the develop-ers had just switched the technology from MapReduce to Spark. The result was faster performance and a 25% reduction in the code base, according to speaker Ryan Smith, an analytics manager at satellite imaging com-pany DigitalGlobe. After the session, Smith acknowl-edged that it’s probably time to come up with a new name for the toolkit.

And it isn’t just MapReduce. The Hadoop Distributed File System, or HDFS—the other core component of Ha-doop’s first incarnation—also finds itself a bit on the run these days. At the Strata conference, Cloudera, the lead-ing Hadoop distribution vendor, announced a columnar

hadoop’s uncertain IdentityNew technologies are augmenting—and in some cases replacing—the core components of hadoop. Welcome to the new, not-so-settled reality of the big data era.

HInDSIgHT craIg stedmaN

home

edItor’s Note

executIVe dashboard

VerbatIm

ruNNINg smart

bIg data story hour





HInDSIgHT | craIg stedmaN

data store called Kudu as a potential alternative to HDFS for applications involving real-time analytics on stream-ing data. Hortonworks, another Hadoop vendor, intro-duced a separate piece of software for managing the flow of data between different systems, with no requirement that HDFS be part of the picture.

Neither MapReduce nor HDFS is going away anytime soon. There are too many applications built on top of them for that to happen, and plenty of Hadoop users do remain committed to the pairing, at least for some of their big data processing needs. But it’s entirely likely that new Hadoop systems will be deployed without either of the two technologies that once comprised Hadoop.

Will they still really be Hadoop clusters? That’s kind of an existential question. But Hadoop’s evolution—or iden-tity crisis, perhaps—is indicative of the unsettled data management environment ushered in by the big data era. We’re living in a polyglot world with a variety of technol-ogy choices for different data processing and analytics needs. Relational software is still included, of course, but so are Hadoop, Spark, NoSQL databases and a vast and ever-expanding ecosystem of other big data tools. And Hadoop’s position at the center of that ecosystem isn’t guaranteed forever—except, maybe, in name only. n

Business Information is a SearchDataManagement.com e-publication.

Ron karjian, Managing Editor

Moriah Sargent, Associate Managing Editor

Scott Wallask, News Director

David Essex, Executive Editor

lauren Horwitz, Executive Editor

jan Stafford, Executive Editor

Craig Stedman, Executive Editor

linda koury, Director of Online Design

Martha Moore, Production Editor

Doug olender, Publisher, [email protected]

Annie Matthews, Director of Sales, [email protected]

TechTarget, 275 Grove Street, Newton, MA 02466 www.techtarget.com

©2015 TechTarget Inc. No part of this publication may be transmitted or repro-duced in any form or by any means without written permission from the publish-er. TechTarget reprints are available through The YGS Group.

About TechTarget: TechTarget publishes media for information technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and process-es crucial to your job. Our live and virtual events give you direct access to inde-pendent expert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts.

COVER: zONADEARTE/ISTOCK

Stay connected with us.

linkedInFacebook

Read more columns by Business Infor-mation editors.

http://searchdatamanagement.techtarget.com/


http://reprints.ygsgroup.com/m/techtarget

http://www.linkedin.com/company/3002580?trk=tyah&trkInfo=tas%3Asearchdataman%2Cidx%3A1-1-1

https://www.facebook.com/businessinformationmagazine

http://searchdatamanagement.techtarget.com/essentialguide/Trend-watch-Data-management-and-business-intelligence-technologies

december 2015, volume 3, number 6 business 2015...

Documents