Microsoft IT Showcase

How Microsoft uses process mining to accelerate digital transformationAt Microsoft Core Services Engineering and Operations (CSEO), we’re using process mining to accelerate digital transformation within our organization. With process mining we’re examining our business processes using source digital footprints (system of records) to visualize those processes and uncover challenges and bottlenecks. Process mining enables us to dig deeply into our transactional logs and extract meaningful, transformative insights that we can use to accelerate digital transformation and streamline business processes.

Exploring process miningProcess mining is an emerging data science field within business process management that uses an organization’s transactional digital footprints to examine their business processes and discover process challenges. Using the metaphor of an x-ray, process mining is an x-ray of business processes as they are exposed through data stored in a variety of digital systems. Process mining intelligent algorithms use this data to monitor process performance of business processes. Process mining is applicable to any business processes that are mostly (or entirely) repetitive and transactional in nature. The only requirement is having data in the form of transaction steps, and date and time stamps that can be used as a basis for process mining. Most companies that have an IT system supporting their business already have this data.According to Business Wire, the global process analytics market is expected to grow by more than 50 percent year-over-year in the next 5 years, taking from a 180 million-dollar industry in 2018 to an almost 1.5 billion-dollar industry by 2023.The challenge for any large organization is to find a single application that can provide end-to-end visibility over these processes. Process mining provides the unique capability of joining the data together and illustrating the connected nature of business processes. The primary goal of process mining is to analyze different applicable data sources and perspectives, and then aggregate the data into information that business leaders can use when making business decisions. This enables business leaders to identify issues and bottlenecks that aren’t obvious, and then act to remedy them.

Driving digital transformation with process miningDigital transformation at Microsoft is about reimagining how we bring together people, data, and processes to create value for our customers while maintaining a competitive advantage in a digital-first world. Process mining is accelerating digital transformation by providing tools to optimize operations using intelligent processes. This accelerates our business responsiveness, improves service levels, and reduces costs. We want to coordinate people and assets more efficiently, respond to issues in real time, and even solve them preemptively.

Page 2 |  How Microsoft uses process mining to accelerate digital transformation

Figure 1. Driving digital transformation with process mining

Specifically, process mining provides value at various stages of the journey into digital transformation: Before digital transformation. Process mining can identify and prioritize potential digital

transformation scenarios that will result in cost savings and simulate the scenarios to provide approximate results.

During digital transformation. You can use process mining to validate steps in your digital transformation. Process mining can test and confirm design principles by simulating potential business scenarios to ensure validity and efficiency.

After digital transformation. Process mining can provide real-time process monitoring and enable intelligent decision making regarding ongoing process improvements.

Process mining enables us to dig deeply into our big data and extract meaningful, transformative insights that we can then use to accelerate our digital transformation and make our business better.

Using process mining at Microsoft Global TradeAt Microsoft, our challenges are much the same as other organizations. We want more visibility into how our business operates so we can make it better and achieve more. We’re continuing to discover how to use process mining to find our business challenges and move forward with improving and refining the relevant processes. As an example, consider how we’ve implemented process mining at Microsoft Global Trade.Microsoft Global Trade is part of Microsoft’s export shipment process. Global Trade is responsible for picking, packing, and shipping our hardware products to customers and distribution points around the globe. At Global Trade, we had several questions that we needed answers to including: Are the export shipments from our distribution center flowing on time? If not, how many shipments are pending for complete documentation? At what stage are these shipments pending? What is the list of delivery orders that are pending? Which countries/regions are they going to? How are these orders being shipped? Who is the carrier/broker? Can we get an alert if a shipment is going to be delayed at any point in the process?We recognized that by putting process mining to work for Global Trade, we could improve our Global Trade business processes.

microsoft.com/itshowcase January 2019

Page 3 |  How Microsoft uses process mining to accelerate digital transformation

Creating direction for process improvementOne of the primary concerns that we had at Global Trade was the time it took between a sales order being placed and the product being delivered to the customer. The end-to-end process involved a lot of moving parts and different parties. We knew there was room for improvement, but we weren’t sure how to identify exactly where those opportunities were. Our first step was to establish business goals based on the primary issues that we had. Our two primary areas of concern were: Frequent delays of shipments due to a variety of factors, including customs processing,

documentation discrepancies, and inability to consistently identify exactly how or why the shipment was delayed.

Lack of a single, end-to-end view of order tracking from the moment the order was placed to the moment it was signed for by the customer.

Process mining requires process dataOur next step was to identify and examine all of the transactional and logging data sources that are part of the Global Trade process. We were using a supply chain management system comprised of various shipping carriers, a sales system, and an invoice and payment system. Recognizing that the results of process mining are only as accurate as the data used, we worked to connect the various datasets to create a meaningful and clean data source. We ran a data quality exercise, and then began synthesizing the data from the Global Trade process.The Global Trade process begins with the sales order initiated at our production facilities and ends with the delivery of products to the customer. A flowchart of the physical process is illustrated in Figure 2. The three major components to this process are SAP, our enterprise resource planning solution (represented by blue boxes), our logistics and shipping partners (represented by green boxes), our inventory management systems (represented by purple boxes) and our invoice tracking and customs systems (represented by orange boxes).

Figure 2. The Global Trade process

Examining Global Trade with process miningThe process mining workflow for Global Trade involved four distinct stages:

1. Gather digital footprints.2. Apply mining techniques to digital footprints.3. Visualize business processes in real-time.4. Infuse with AI and machine learning.

Gathering digital footprintsWe collected the underlying process data from all of the applicable data sources and assembled it into digital footprints that defined the important qualities of each process step, including the delivery document number, event type, and a date/time stamp defining when the event occurred.

microsoft.com/itshowcase January 2019

Page 4 |  How Microsoft uses process mining to accelerate digital transformation

Delivery Doocument nbr Event Type Event DatetimeStamp8611874448 Delivery Created 12/29/2017 9:448611874448 Shipped 1/2/2018 12:578611874448 Goods Issue 1/2/2018 7:158611874448 Proforma Created 1/2/2018 15:208611874448 CD Created 1/2/2018 15:208611892309 Shipped 1/2/2018 14:018611892309 Goods Issue 1/2/2018 16:298611892309 Delivery Created 12/27/2017 11:148611892309 Proforma Created 1/3/2018 0:328611892309 CD Created 1/3/2018 0:328611895649 Delivery Created 12/28/2017 13:068611895649 Proforma Created 1/3/2018 14:178611895649 CD Created 1/3/2018 14:178611909945 Delivery Created 1/2/2018 10:068611909945 Proforma Created 1/3/2018 14:498611909945 CD Created 1/3/2018 14:498611911655 Delivery Created 1/2/2018 10:53

Figure 3. Digital footprints from the Global Trade process

Applying mining techniques to digital footprintsUsing the digital footprints as our data source, we mined and synthesized the data using process mining algorithms taken from the open source R library. The output from the process mining techniques provided analysis of the Global Trade process in visual form and revealed important details about process order, timing, and other details not immediately apparent in the underlying digital footprints.

Figure 4. Process mining output view of the Global Trade process

The insights from process mining analysis provided a different perspective on the Global Trade process. The data it presented created the opportunity for discussions about how and why our real process differed from the ideal process. It showed us where the process deviated from the ideal,

microsoft.com/itshowcase January 2019

Page 5 |  How Microsoft uses process mining to accelerate digital transformation

allowed us to identify bottlenecks in the process, and provided a measurement of real process performance.

Visualizing business processes in real timeIn our next step, we took the process as it was exposed by process mining techniques and created a visualization for the Global Trade process as it was happening. We used real-time monitoring to immediately begin taking corrective actions in the process as issues occurred. As a result, we were able to resolve problems while the process was ongoing and prevent those problems from causing issues elsewhere in the process.Visualization was the key to a complete understanding of the real-time Global Trade process. We used JavaScript APIs for Microsoft Visio to embed Visio diagrams into Microsoft SharePoint Online pages for access by anyone who needed to examine the process. Figure 5 represents a real-time visualization of the Global Trade process, with box colors identifying process status as defined by business rules extracted from process mining insights.

Figure 5. The Global Trade process visualized in real time

Infusing with AI and machine learningAt Global Trade, we’re currently looking at ways to predict and prescribe future process scenarios using AI and machine learning. For example, we’ve created a bot to perform interactive question and answer handling through a chat interface. The bot uses Microsoft Cognitive Services natural-language processing to answer natural language questions using data synthesized from processing mining activities. It can respond to a question such as, “How long will it take to complete order number 8611620108?” with a real-time, data-driven response in natural language.

Figure 6. The complete process mining journey for Global Trade

How to use process mining in an enterprise scenarioYou can apply process mining to your enterprise using data analytics tools from Microsoft Azure. In stage 1 of the process, capture enterprise business process data and transform it into digital footprints using AppInsights. Then store the digital footprint data in Azure Data Lake for each node of the business process, including date/time stamp and event type. In stage 2, use process mining techniques to synthesize the data for insights analysis or real-time process performance monitoring. For stage 3, use the existing insights and data extracted by process mining to create intelligent tools,

microsoft.com/itshowcase January 2019

Page 6 |  How Microsoft uses process mining to accelerate digital transformation

and enable informed business decisions with Microsoft Azure Machine Learning Studio, Microsoft Cognitive Services, and the Azure Bot Service.

Figure 7. The process mining architecture model

How to further integrate process mining with Azure technologiesReal-time availability of digital footprints is the most important element needed for process mining. Most traditional, on-premises applications don’t have methods to obtain logs for each node of a business process. Retrieving log data on a real-time basis to monitor business process performance is difficult, but the latest version of Application Insights has the capability to perform real-time streaming of digital data directly from the source.

Application InsightsApplication Insights has an API for custom events and metrics that is configurable simply by inserting a few lines of code in a legacy application. When this code is in place, we can get the digital logs of activity with date and time stamp for all activity happening in our business processes. We store these logs in Azure Data Lake, which removes the complexities of ingesting and storing logs while making it faster to get up and running with batch, streaming, and interactive analytics. Azure Data Lake works with existing IT data investments for simplified data management and governance. It integrates seamlessly with operational stores and data warehouses, so we can extend current data applications. You can also consider Azure Data Lake for multi-tenant application architecture ecosystems to centrally manage enterprise digital logs for various transactional applications.

Power BIWe can view the process mining analysis output in multiple ways depending on the user experience we’re trying to achieve. We can render the output in Microsoft Power BI for static visualization of business process flow with real-time refresh. We also use the Visio API for Power BI to create interactive process diagrams with custom handlers for click or tap events within the drawing. We can overlay the diagram with process data to track and monitor process performance in real time.

Addressing challenges in process miningThroughout the Global Trade process mining journey, we’ve encountered challenges to applying the process mining solution. Some of the challenges have been applicable only to Global Trade, but many will be encountered by any organization that implements process mining.

microsoft.com/itshowcase January 2019

Page 7 |  How Microsoft uses process mining to accelerate digital transformation

Data quality is critical. Our results are only as good as our data. We found that we had to examine our data quality up front to ensure that we were getting accurate results from process mining. We established several practices—including regular data consistency checks, creating policies for enabling appropriate logs, and ensuring that data sources were connected appropriately—to allow for relational data flow across the entire process mining dataset.

Look for and address whitespace. We found several places in our processes where we had whitespace in the form of no data or not enough data being recorded by our process systems. For example, if we didn’t receive any information from a shipping company between when a package was picked up and when it was dropped off, that was whitespace in the continuum of our data stream. We identified whitespace in our processes and suggested fixes for the issue wherever possible.

Don’t settle for hard-coded business rules. We found that some delays in processes were due to hard-coded business rules that were either no longer applicable, had become irrelevant, or used outdated values. For example, if our shipping department had 12 hours to pack and ship a product from the time it was ordered, they typically operated to meet that target but not necessarily exceed it. We’ve started creating and using dynamic rulesets that account for what’s happening within our processes and continuously adjust targets and metrics within the process.

BenefitsThroughout our journey with process mining, we’ve realized several benefits—specifically within the context of Global Trade, and for other teams as well: Reduced process time. This affects many aspects of our processes, and was arguably our

biggest improvement. Once we were able to identify the issues and solve them, we found that our business was simply more efficient. We reduced cycle and lead times across our processes and reduced the overall time it takes our products to arrive to our customers.

Increased transparency. Process mining opened up our processes to all parts of our organization. Anyone who wanted to see what our end-to-end process looked like could pull it up on a dashboard and examine the results for any products that were in the process.

Immediate process interaction at all levels. If one of our executives sees that a high-level process is reporting below average performance, that executive can drill down into the process and identify the specific orders or processes that are causing poor performance. 

ConclusionProcess mining has created a new level of insight into and control over our Global Trade business processes at Microsoft. We’re finding new and innovative ways to visualize and monitor our workflows and pinpoint room for improvement. We’re beginning to use process mining to create self-healing business processes that can provide performance feedback to the business process itself, and utilize machine learning to adjust parameters to improve process performance. By embracing and using process mining, we have the ability to dig deeply into our big data and extract meaningful, transformative insights that we can use to accelerate our digital transformation and make our business better.

For more informationMicrosoft IT Showcasemicrosoft.com/itshowcaseIT Showcase Data and AI contentTransforming business through the power of dataMetrics that matter – how we track our digital transformationAzure Data Lake connects supply chain data for advanced analytics

microsoft.com/itshowcase January 2019

Page 8 |  How Microsoft uses process mining to accelerate digital transformation

Bringing digital transformation to the supply chain with Azure IoT Suite

© 2018 Microsoft Corporation. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

microsoft.com/itshowcase January 2019