Senior Product Manager, AI Agents Testing

Job Description

Role

Today, the admins who configure and manage these agents — CX managers, bot builders, operations leads — lack the tools to confidently test agent behavior before going live, measure quality in production, or experiment with changes safely. You'll own the end-to-end product strategy for our Testing & Observability suite — the layer that lets admins simulate conversations against their real knowledge and procedures, score agent quality across accuracy, tone, and policy adherence, run A/B experiments on agent behavior, and catch regressions before they reach end users. This is a strategic opportunity that directly determines whether enterprises can trust and scale agentic AI in their customer service operations.

Key Responsibilities

• Own product strategy and roadmap for AI agent testing — simulation, quality scoring, experimentation, regression detection, and conversatio...