the ai engineering journey
TRANSCRIPT
© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This presentation, including all supporting materials, is proprietary to Gartner, Inc. and/or its affiliates and is for the sole internal use of the intended recipients. Because this presentation may contain information that is confidential, proprietary or otherwise legally protected, it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates.
The AI Engineering JourneyErick Brethenoux, GartnerApril 15, 2021
2 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
About Erick
Gartner VP Research & AI Research Agenda lead – Aug 2017 IBM Director of Machine Learning & Decision Management Strategy SPSS VP of Corporate Development Lazard VP, Technology Investments, M&A and VC Gartner, Research Director of Advanced Technologies New Science, Research Director Advanced Technologies and Applications French Embassy, Scientific Mission University of Delaware, PhD Cognitive Science
Years of experience: 3.5 years at Gartner, 30 years industry experience
3 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
Key Issue Takeaway:Through 2023, at least 50% of IT leaders will struggle to move their AI predictive projects from proof of concept to production maturity.
4 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
Google circa 2014… models in context - still true!
Source: “Machine Learning: The High-Interest Credit Card of Technical Debt” – NIPS 2014 Workshop
5 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
In two years organizations have (barely) crossed the 50/50 threshold
Proportion of prototypes that make it to productionPercentage of respondents
53%Average proportion of projects that make it from prototypes to production (fully deployed)
2018 2020
47%
© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This presentation, including all supporting materials, is proprietary to Gartner, Inc. and/or its affiliates and is for the sole internal use of the intended recipients. Because this presentation may contain information that is confidential, proprietary or otherwise legally protected, it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates.
MLOps to ModelOps
7 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
KPIs validation
start here
Business Understanding
Data Sources Discovery
Data Wrangling
AnalysisModel Validation
New Data Acquisition
Model Release
End-points identification
Parameters testing
Integration testing
Instantiation validation
re/tuningchallenging
explainingvisualizing
complying
Model activation
Model deployment
Application integration
Model behavior tracking
Production audit procedure
Management & Governance
Establishing a sustainable, repeatable, measurable and continuous operationalization process.
Operationalization cycle / Activation
Development cycle
Operationalization cycle / Release Business drift
Data drift
Mathematical drift
Performance drift
Ethical drift
MLOps Discipline
9 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
Demystifying XOps – DataOps, MLOps, ModelOps, Platform Ops for AI
10 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
Invariant AI Engineering Themes and Governance Principles
• Security• Ethics• Autonomy• Decisioning• Change Management
11 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
Invariant AI Engineering Themes and Governance Principles
• Security• Ethics• Autonomy• Decisioning• Change Management
12 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
AI Security
Manipulating AI at Runtime:• Pertubations or ill-intended inputs (data, behavior) • Query attacks to determine model logic• Changes to Model, Operational rules,
External or Insider Attacker
Training Set Dev Set Test Set
Dataset Used for Model Dev Data After Deployment
Assess &Adjust
DevelopD
eplo
y
Production Data
Run
Training Data Poisoning• Change model outcomes
TrainModel
Model Theft at any time • Subsequent Malicious use/ Replacement• Steal intellectual property • Inversion attack
13 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
AI Ethics
AI Ethics Guidelines
Fair
Interpretable & Transparent
Human-centric & Socially
Beneficial
Accountable
Secure & Safe
15 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
Recommendations
Minimize the technical debt of deployed ML models by monitoring and revalidating their business value by adopting a foundational ModelOps practice – first stage of the AI Engineering discipline.Ensure the integrity, transparency and sustainability of AI models through a systematic process, and enduring governance principles; based on fairness, trustworthiness, reliability and accountability.Identify projects in which a fully data-driven, ML-only approach is unviable and inefficient while exploring the expressive power of Composite AI to solve complex business problems.Extend the skills of AI experts, to cover multiple AI techniques, both from a model development and operationalization perspectives.
16 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.
ModelOps
Composite AI
Generative AI
Tech
nolo
gy C
ompl
exity
Operational Complexity
The AI Engineering Evolution
Adaptive Systems
AI engineering is a discipline focused on the governance and life cycle management of a wide range of operationalized AI and decision models, providing an uninterrupted flow between the development (human or machine-based), and operationalization of AI models (leveraging one or multiple AI techniques).
(Stage 1)
(Stage 2)
(Stage 3)
© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This presentation, including all supporting materials, is proprietary to Gartner, Inc. and/or its affiliates and is for the sole internal use of the intended recipients. Because this presentation may contain information that is confidential, proprietary or otherwise legally protected, it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates.
Thank you