Scribd
Upload
88 views

Arize U - Intro To ML Observability

The document discusses ML observability and the need for tools that help teams monitor machine learning models in production environments. It notes common challenges that plague ML teams like data and feature drift, data leakage, and model performance degradation. It also provides an overview of the key pillars of ML observability including performance monitoring, drift detection, data quality management, and explainability.

Uploaded by

emma adeoye
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
88 views

Arize U - Intro To ML Observability

The document discusses ML observability and the need for tools that help teams monitor machine learning models in production environments. It notes common challenges that plague ML teams like data and feature drift, data leakage, and model performance degradation. It also provides an overview of the key pillars of ML observability including performance monitoring, drift detection, data quality management, and explainability.

Uploaded by

emma adeoye
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

ML Observability

A Critical Piece in Making Models Work in the Real World

Amber Roberts | ML Engineer @ Arize AI | amber@arize.com


Sign up for your Arize account!

https://bit.ly/3FWMOW6

Join our community Slack for Arize


updates, events and platform
support!
Curriculum

● Intro to ML Observability
● Unit 1: Performance Tracing
● Unit 2: Drift Detection
● Unit 3: Data Quality Management
● Unit 4: Feature Importance & Explainability
● Unit 5: Fairness & Bias Tracing
● Unit 6: Unstructured Data & Embeddings
Taking a Model from Research to Production is Hard

Models fail silently this makes troubleshooting painful

Data Feature different Feature Regression in


changes in production changes a cohort

Model in
production
Rewrite as Scripts Data Performance Troubleshooting
Python/ Java broken leakage drifts scripts
84% 51% 54%

Takes >1 week to Want deeper Say business execs


detect & fix a model capabilities to can’t quantify AI ROI
issue monitor & RCA drift

● > 1000 companies surveyed


Every Company is an AI Company

10+ models live 100+ models live 1000+ models live

In production for business critical objectives:

R&D Optimize production Targeted sales Enhanced user experience

Forecasting Maintenance Marketing Customer service


Time spent on ML the last 5 years

CHALLENGES
Teams spend on Time to resolution take Business impacted from Model velocity/training
average less than 1% weeks/days poor model decisions slowed

of time on production Alerts on model issues Real world data is different Fairness & bias issues erode
coming from customers than training brand trust
Time spent on ML the last 5 years

COMMON PROBLEMS PLAGUE ALL TEAMS

Model & data drift Model interpretability Model Readiness

Performance degradation Data quality issues Fairness & bias issues


ML Observability

Reference
ML Observability

ML Əb-ˌzər-və-ˈbi-lə-tē \ noun

Software that helps teams automatically


monitor AI, understand how to fix it when it’s
broken, and improve data & models.
ML Observability vs ML Monitoring
ML Monitoring is just the tip of the
iceberg when it comes to tackling
these challenges in production

ML monitoring is necessary but


doesn’t troubleshoot your problems
in production.

ML Observability goes deeper. It


helps you get to the root cause
and resolve the model failure
mode.

11
© All Rights Reserved
4 Pillars of ML Observability
Performance: Surfacing worst performing slices Drift: Data Distribution Changes over lifetime of model

Data Quality: Ensure high quality inputs & outputs Explainability: Attribute why a certain outcome was made
Sign up for your Arize account!

https://bit.ly/3FWMOW6

Join our community Slack for Arize


updates, events and platform
support!

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy