Role: AI QA Automation Lead

Function: Quality Assurance

Location: Navi Mumbai, Maharashtra

Type: Full-time

Compensation: Competitive compensation package with equity participation

Industry: Artificial Intelligence, Technology

About Company

A groundbreaking AI initiative building the intelligence layer for India at massive scale. Backed by partnerships with global tech leaders like Meta and Google.

The company creates AI solutions designed for real adoption across languages, contexts, and daily needs. Engineered from day one for 100M+ users early with 1B-ready constraints on latency, cost, reliability, and safety.

Position Overview

You'll lead the quality strategy and automation frameworks for cutting-edge AI systems serving millions of users across India. You'll own and implement automation-first frameworks that ensure reliability and performance of AI platforms powering agentic workflows and LLM-based applications. This role puts you at the forefront of defining AI testing practices at unprecedented scale.

Role & Responsibilities

• Define and execute comprehensive quality strategy for LLM-based applications and AI systems
• Design and implement automation-first testing frameworks using AI-specific testing tools
• Lead QA automation initiatives for agentic workflows and conversational AI systems
• Build and maintain CI/CD pipelines with AI model validation and automated testing systems
• Mentor and guide QA engineering teams on AI testing frameworks and methodologies
• Collaborate with ML engineers to test model performance, accuracy, and reliability metrics
• Establish quality metrics and monitoring systems for production AI applications

Must Have Criteria

• 10-20 years of experience in QA automation for large-scale distributed platforms
• Hands-on experience with AI testing frameworks, Python automation using PyTest, and JavaScript automation using Playwright
• Expert-level proficiency in Python test automation with PyTest framework for AI systems
• Strong experience with JavaScript automation using Playwright for web-based AI applications
• Proven leadership experience managing QA automation teams of 5+ engineers
• Deep understanding of AI model testing, validation metrics, and performance benchmarking
• Experience implementing CI/CD pipelines with automated AI model evaluation and validation

Nice to Have

• Experience with LLM evaluation frameworks and custom AI testing metrics
• Background in testing machine learning pipelines and model deployment systems
• Experience with A/B testing frameworks for AI model performance comparison
• Knowledge of performance testing for AI inference systems and model serving
• Open source contributions to AI testing frameworks or ML evaluation tools

What We Offer

• Opportunity to shape AI testing practices at unprecedented scale
• Work with cutting-edge AI technologies and global tech partnerships
• Competitive compensation package with equity participation
• Access to world-class infrastructure and green energy powered systems
• Career growth in India's most ambitious AI initiative