Production Failures Become Agent Evals

And so Weave helps users do this AI loop that we like to talk about, where unlike traditional software development, it's a much more experimental workflow because at the core of it, you have a model that's stochastic. It's non-deterministic. You don't know exactly, if you give it an input, what the output will be. So this loop is that You put something in production. You put your app in production. And then you monitor what users do with it, what the inputs are that users give and what the outputs are. And you try to find either failure modes or scenarios that you'd like your agent to do better in. You classify all those into groups. And then you form them into what we call offline evaluations. So that's like a big data set of scenarios or cases that we want our application to be good at. And then you can run your evaluation offline while you test your system. You can improve your prompts, you can improve your harness, you can improve the system as a whole against that evaluation and get a score for it. And when you get a better score, you can ship back to production. And now you've sort of completed that loop and you can look for new cases that you'd like to optimize.