Role: AI QA Automation Lead

Function: Quality Assurance

Location: Bangalore/ Mumbai

Type: Full-time

Compensation: Competitive compensation package with equity participation

Industry: Artificial Intelligence, SAAS, Fintech

About Company

A groundbreaking AI initiative under India's largest private sector company. The company builds AI solutions for every Indian, harnessing green energy to power gigawatt-scale infrastructure.

Partnerships with tech giants like Meta and Google drive innovation at unprecedented scale. You'll work in a fast-paced, innovation-driven environment where engineering excellence meets real impact.

Position Overview

You'll lead the quality strategy and automation frameworks for cutting-edge AI systems serving millions of users across India. You'll own and implement automation-first frameworks that ensure reliability and performance of AI platforms. This role puts you at the forefront of testing agentic workflows, LLM-based applications, and conversational AI systems at gigawatt scale.

Role & Responsibilities

• You define and execute comprehensive quality strategy for LLM-based applications and AI systems
• You design and implement automation-first testing frameworks using AI-specific testing tools
• You lead QA automation initiatives for agentic workflows and conversational AI systems
• You build and maintain CI/CD pipelines with AI model validation and automated testing systems
• You mentor and guide QA engineering teams on AI testing frameworks and methodologies
• You collaborate with ML engineers to test model performance, accuracy, and reliability metrics
• You establish quality metrics and monitoring systems for production AI applications

Must Have Criteria

• 10-20 years of experience in QA automation for large-scale distributed platforms
• Hands-on experience with AI testing frameworks, Python automation using PyTest, and JavaScript/typescript automation using Playwright
• Expert-level proficiency in Python test automation with PyTest framework for AI systems
• Strong experience with JavaScript automation using Playwright for web-based AI applications
• Proven leadership experience managing QA automation teams of 5+ engineers
• Deep understanding of AI model testing, validation metrics, and performance benchmarking
• Experience implementing CI/CD pipelines with automated AI model evaluation and validation

Nice to Have

• Experience with LLM evaluation frameworks and custom AI testing metrics
• Background in testing machine learning pipelines and model deployment systems
• Experience with A/B testing frameworks for AI model performance comparison
• Knowledge of performance testing for AI inference systems and model serving
• Open source contributions to AI testing frameworks or ML evaluation tools

What We Offer

• Opportunity to shape AI testing practices at unprecedented scale
• Work with cutting-edge AI technologies and global tech partnerships
• Competitive compensation package with equity participation
• Access to world-class infrastructure and green energy powered systems
• Career growth in India's most ambitious AI initiative