How I lead complex technical products through uncertainty.

Repairing a model evaluation

When we compared candidate chat models as possible replacements for the production model, the first evaluation produced pass-or-fail decisions that did not align with its own scores. In the review I led, we found that the rubric was incomplete and unweighted, definitions had shifted during scoring, and evaluators could see model identities.

I designed a controlled rerun: a fixed weighted rubric agreed in advance, consistent scoring ownership, prompt optimization for each candidate, and blinded model identities. I also built the blinding mechanism and prompt-override support into our internal tooling and repaired simulator issues that were blocking the evaluation.

Planning colleagues created and scored the conversations. AI colleagues selected candidates, benchmarked their behavior, and optimized prompts. My role was to repair the decision process and connect the teams needed to run it.

Turning production errors into an operating loop

After launch, our daily reports presented raw error data without the user impact, investigation status, or app-version context needed to make decisions. Noise and real defects were difficult to separate, and a completed engineering task could be mistaken for a production fix.

I designed and implemented the initial AI-assisted triage loop. AI prepared a plain-language summary, possible impact, and suggested action; people retained authority over status, priority, and sprint work. Muted noise returned when it spiked, and a fix remained open until production evidence showed that the issue had stopped on the corrected app version. Regressions returned to the decision queue.

The result was a durable operating change rather than a claimed crash-rate reduction: engineering and planning reviewed one bounded queue, release follow-through depended on production evidence, and the system stayed in daily use after I handed operations to teammates.

Operating across product and engineering

The work connected launch readiness, model evaluation, production operations, and distributed-team leadership. I stayed close enough to the technology to build the first version of an operating system while keeping consequential decisions with people.

At Woven, I led within a large automotive software program. At GAZAI, I worked at startup scale: making uncertainty explicit, improving how the team made technical decisions, and connecting those decisions to delivery.