Senior AI QA Architect for Conversational Platform

Responsibilities

• Design and own the end-to-end QA strategy for the Conversational Banking Platform, covering functional, regression, performance, security, and AI-specific evaluation.
• Build and maintain golden datasets, eval suites, and LLM-as-judge frameworks to validate conversational quality across intents, languages, and tenants.
• Define the tenant onboarding QA gate, the certification checklist every new business unit must pass before going live.
• Establish regression strategies for prompt changes, model upgrades, retrieval index updates, and guardrail policy changes.
• Use Langfuse traces to drive evaluation: mine production failures, convert them into test cases, and close the loop with engineering.
• Test NeMo Guardrails configurations against jailbreaks, prompt injection, off-topic drift, and false-positive over-blocking.
• Validate governance and compliance behaviors: data residency, PII handling, regul...