certified for data engineering [big data] · 2020-02-04 · big data management] to azure made it a...

4
Goals Create new digital processes and revenue streams based on digital assets and Internet of Things (IoT) sensors on energy infrastructure and maritime vessels Compress development cycles and scale digital services up and down as needed to satisfy fluctuating customer demand while controlling costs Ensure that data from any source can be trusted, verified, and compliant with the ISO 8000-8 international data quality standard Solution Connect an Azure-based data platform, on- premises data centers, a Cloudera Hadoop ecosystem, and IoT devices with Informatica Data Engineering Integration (DEI, formerly known as Big Data Management or BDM) Automate the deployment and management of Apache Hadoop clusters with Databricks Unified Analytics Platform, which integrates with Informatica DEI Verify data from databases, APIs, email, flat files, and event hubs with Informatica Data Engineering Quality [formerly known as Big Data Quality] before processing Results Enables the creation of new digital services to increase energy efficiency, reduce emissions, improve grid stability, and accelerate ship inspections up to 30x Accelerates developer and data engineering productivity and reduces costs by scaling clusters up and down automatically Allows DNV GL to treat data as an asset, a prerequisite for success in a data-driven future, while remaining compliant with international standards Informatica Data Engineering Integration [formerly known as Big Data Management or BDM] allowed us to digitalize our maritime surveys, reducing the inspection process for a fleet of vessels from 30 days to one day.” Jørgen Stang Data Scientist DNV GL Certified for Data Engineering [Big Data]: DNV GL Integrates Digital Assets, Accelerating Maritime Certifications

Upload: others

Post on 22-May-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Certified for Data Engineering [Big Data] · 2020-02-04 · Big Data Management] to Azure made it a perfect fit for adding data management capabilities to our Veracity data platform,”

Goals Create new digital processes and revenue streams based on digital assets and Internet of Things (IoT) sensors on energy infrastructure and maritime vessels

Compress development cycles and scale digital services up and down as needed to satisfy fluctuating customer demand while controlling costs

Ensure that data from any source can be trusted, verified, and compliant with the ISO 8000-8 international data quality standard

Solution Connect an Azure-based data platform, on-premises data centers, a Cloudera Hadoop ecosystem, and IoT devices with Informatica Data Engineering Integration (DEI, formerly known as Big Data Management or BDM)

Automate the deployment and management of Apache Hadoop clusters with Databricks Unified Analytics Platform, which integrates with Informatica DEI

Verify data from databases, APIs, email, flat files, and event hubs with Informatica Data Engineering Quality [formerly known as Big Data Quality] before processing

Results Enables the creation of new digital services to increase energy efficiency, reduce emissions, improve grid stability, and accelerate ship inspections up to 30x

Accelerates developer and data engineering productivity and reduces costs by scaling clusters up and down automatically

Allows DNV GL to treat data as an asset, a prerequisite for success in a data-driven future, while remaining compliant with international standards

Informatica Data Engineering Integration [formerly known as Big Data Management or BDM] allowed us to digitalize our maritime surveys, reducing the inspection process for a fleet of vessels from 30 days to one day.”

Jørgen StangData ScientistDNV GL

Certified for Data Engineering [Big Data]: DNV GL Integrates Digital Assets, Accelerating Maritime Certifications

Page 2: Certified for Data Engineering [Big Data] · 2020-02-04 · Big Data Management] to Azure made it a perfect fit for adding data management capabilities to our Veracity data platform,”

Creating a safe and sustainable future is at the heart of what DNV GL offers. Whether it is delivering a new service to discover structural problems for wind turbines, implementing a hull planned maintenance system, or certifying maritime vessels, DNV GL prides itself on “building the invisible infrastructure of trust.”

Norway-based DNV GL is an international accredited registrar and classification society created in 1864. It provides risk assessment, quality assurance, and management system certification services for a range of industries including renewable energy, oil and gas, electrical, food and beverage, automotive, aerospace, and healthcare. DNV GL is also one of the world’s leading ship classification societies, providing rules and standards for how to build and operate maritime vessels.

DNV GL’s business is based on data, which it uses to improve energy efficiency, optimize productivity, conduct risk assessments, predict failures, forecast performance, and reduce operational costs for its customers. It even built its own data platform called Veracity to provide an adaptive data ecosystem that securely connects key players in the maritime, oil and gas, and energy industries to drive business innovation and digital transformation. Built on Azure for scalability, Veracity connects asset owners and operators with domain experts and data scientists, enabling the exchange of datasets, applications, APIs, and insights. Customers benefit from predictive maintenance and improved logistics.

Although big data can bring great rewards, bad data carries substantial risk. DNV GL is well positioned to create new revenue streams with digital transformation, leveraging data from Internet of Things (IoT) sensors that are becoming standard on new energy infrastructure and data-smart maritime vessels. By using this data to automate traditionally manual processes such as ship inspections and certifications, DNV GL can accelerate customer service and substantially reduce costs.

To take advantage of new digital opportunities, DNV GL had to shorten its cycles for development and data engineering. It also needed the flexibility to scale data-driven services up and down as needed to satisfy fluctuating customer demand, while controlling costs. Finally, it had to ensure that data from any source can be trusted, verified, and compliant with data quality requirements specified in the ISO 8000-8 international standard, which DNV GL helped to create.

“Powerful data management can provide many benefits to our consumers, regulators, and utilities,” says Jørgen Stang, Data Scientist at DNV GL. “When we first started working with big data, all processing and analysis was done in our on-premises data centers. But now we have a hybrid environment that also includes Cloudera Hadoop clusters and Azure cloud. Our customers use many different IoT sensor systems, and we needed to pool all this data together without spending too much time in isolated systems.”

Business Requirements:

• Manage data engineering and big data workloads both on-premises and in the cloud

• Provide a consistent visual development interface as technologies change

• Constantly integrate maritime data from more sources, including IoT

About DNV GLDNV GL is a provider of risk management and quality assurance services, headquartered in Høvik, Norway. It is also a global leader in certifying management systems of companies across industries. The company has 14,500 employees and 350 offices operating in more than 100 countries.

Page 3: Certified for Data Engineering [Big Data] · 2020-02-04 · Big Data Management] to Azure made it a perfect fit for adding data management capabilities to our Veracity data platform,”

Data is fueling our digital transformation,

so we need to find good ways to work with

data to create new services. Informatica

is helping us improve energy efficiency,

reduce emissions, save money, and

improve productivity.”

Jørgen StangData ScientistDNV GL

Embracing a new digital realityAn Informatica PowerCenter user for many years, DNV GL selected Informatica Data Engineering Integration (DEI, formerly known as Big Data Management) to connect its Azure-based customer data platform, on-premises data centers, Hadoop ecosystem, and IoT devices from ships and electrical infrastructure. It also chose to automate the deployment and management of its Hadoop clusters with Databricks Unified Analytics Platform, which integrates with Informatica DEI to create high-volume data engineering pipelines at scale.

“We looked at Informatica and a few other vendors and did a proof of concept,” says Stang. “Informatica Data Engineering Integration [formerly known as Big Data Management] was the best fit for us. It offered pre-built connectors to Azure and Hadoop HDFS, and also made it easy to connect to on-premises data sources.”

Before data from databases, APIs, email, flat files, and event hubs is processed, automated quality rules are applied with Informatica Data Engineering Quality.

“Data quality is a big part of what we’ve always done, it’s just that today we get much of the data from IoT sensors instead of a ship’s log books,” says Stang. “But we don’t just assume that data from IoT sensors on maritime vessels is correct. Instead, we put it through rigorous quality checks with Informatica Data Engineering Quality [formerly known as Big Data Quality].”

“The ease with which we were able to connect Informatica Data Engineering Integration [formerly known as Big Data Management] to Azure made it a perfect fit for adding data management capabilities to our Veracity data platform,” says Stang.

Big insights with trusted, real-time dataWith Informatica Data Engineering Integration, DNV GL can use data to generate insights and create new digital services to increase energy efficiency, reduce emissions, and optimize maintenance schedules for electrical infrastructure and maritime vessels.

For example, by collecting grid frequency data for the Scandinavian energy market over the last 10 years, analyzing power producers’ logs, maintenance records, and downtime events, and correlating weather conditions available as open data, DNV GL can better predict imbalances in power supplies. This data-driven monitoring service has the potential to improve grid stability, preventing large blackouts.

“Informatica Data Engineering Integration [formerly known as Big Data Management] allowed us to digitalize our maritime surveys, reducing the inspection process for a fleet of vessels from 30 days to one day,” says Stang. “We’re also realizing a huge savings in employee travel costs for those inspections.”

Page 4: Certified for Data Engineering [Big Data] · 2020-02-04 · Big Data Management] to Azure made it a perfect fit for adding data management capabilities to our Veracity data platform,”

Informatica 2100 Seaport Blvd., Redwood City, CA 94063, USA Phone: 650.385.5000, USA Toll-free: 1.800.653.3871 www.informatica.com | Facebook | Twitter | LinkedIn

© Copyright Informatica LLC 2019. Informatica, the Informatica logo, and PowerCenter are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at https://www.informatica.com/trademarks.html. Other company and product names may be trade names or trademarks of their respective owners. The information in this documentation is subject to change without notice and provided “AS IS” without warranty of any kind, express or implied.

IN05_3785_1019

Inside The Solution:

• Informatica Data Engineering Integration (formerly known as Big Data Management)

• Informatica Data Engineering Quality (formerly known as Big Data Quality)

Improving time to market from months to weeksWith the ability to scale its Hadoop clusters up and down automatically, DNV GL is accelerating developer and data engineering productivity, leading to shorter development cycles for new digital services.

“The ability to scale our Hadoop clusters automatically is extremely valuable,” says Stang. “With Informatica Data Engineering Integration [formerly known as Big Data Management] and Databricks, we are reducing the time from when someone gets a great idea until we can actually show the business a solution. Before, we needed months to actually implement a new data-driven project. Now we can have the first version running in just a few weeks. Then, we can run short iterations using Agile development techniques and show value to the business very quickly.”

A data engineering foundation for digital transformationWith a modern data management toolset in place, DNV GL can treat data as an asset, a prerequisite for success in its data-driven future, while remaining compliant with the international standards it helped establish.

“Data is fueling our digital transformation, so we need to find good ways to work with data to create new services,” says Stang. “Informatica helps us improve energy efficiency, reduce emissions, save money, and improve productivity. We are now working through all of our services to see how we can use digital tools to improve them.”