QA Automation Engineer – Agentic AI & RAG Systems

Job Title: QA Automation Engineer – Agentic AI & RAG Systems

Location: Bangalore

Experience: 3–5 Years

Technology Stack: Python, AWS (Step Functions, Bedrock, SageMaker), OpenSearch

Role Overview

We are seeking a highly skilled QA Automation Engineer to drive quality assurance for our Agentic AI orchestration platform. Unlike traditional QA roles, this position focuses on validating non-deterministic AI systems, ensuring the reliability, safety, and accuracy of LLM-driven agents and Retrieval-Augmented Generation (RAG) pipelines.

The ideal candidate will play a critical role in developing testing frameworks for AI reasoning validation, hallucination detection, and workflow orchestration. You will also contribute to building a Human-in-the-Loop (HITL) validation framework using Amazon SageMaker Ground Truth to ensure high-accuracy outputs for financial reporting processes such as ETF AUM calculations.

Key Responsibilities

AI Logic & Response Validation

1. Design and implement automated test frameworks to detect LLM hallucinations and evaluate the reasoning quality of AWS Bedrock agents.
2. Validate AI-generated responses for accuracy, consistency, and grounding in source data.

RAG System Testing

1. Benchmark and validate retrieval accuracy and semantic relevance of data retrieved from Amazon OpenSearch.
2. Ensure RAG pipelines correctly combine retrieved data with LLM-generated responses.

Human-in-the-Loop (HITL) Framework

1. Configure and manage Amazon SageMaker Ground Truth labeling jobs.
2. Define confidence thresholds and workflows that route low-confidence AI outputs to human reviewers for validation.

Workflow & Orchestration Testing

1. Validate complex AWS Step Functions workflows, ensuring proper state transitions, exception handling, and system reliability during agent tool calls.
2. Test integrations across AWS services such as Lambda, API Gateway, and S3.

AI Safety & Adversarial Testing

1. Conduct red-team testing by designing adversarial or “jailbreak” prompts.
2. Validate Bedrock Guardrails for protection against PII exposure, restricted content, and unsafe outputs.

Quality Metrics & Reporting

1. Develop and maintain quality dashboards tracking:
2. Test pass rates
3. Hallucination rates
4. Retrieval accuracy
5. Human intervention frequency
6. Provide regular QA reports to engineering and management teams.

Required Skills & Qualifications

1. Automation & Testing
2. Strong expertise in Python-based automation testing.
3. Hands-on experience with Pytest or similar testing frameworks.
4. Experience testing APIs and backend systems.
5. AWS Ecosystem
6. Practical experience with AWS Lambda, S3, API Gateway, IAM policies, and Step Functions.
7. Human-in-the-Loop Systems
8. Hands-on experience with Amazon SageMaker Ground Truth or other data labeling and verification platforms.
9. Vector Search & RAG Concepts
10. Basic understanding of vector databases such as OpenSearch.
11. Experience validating semantic search and retrieval pipelines.
12. Analytical Thinking
13. Ability to design and maintain Golden Datasets for regression testing of LLM prompts and outputs.

Preferred Qualifications

1. Experience with LLM evaluation frameworks and prompt evaluation methodologies.
2. Familiarity with agent orchestration frameworks such as LangGraph or Strands.
3. Background in financial services, data extraction pipelines, or ETF-related reporting systems.