Apply now »

Sr. Test Engineer - Automation Testing Job

Date:  Nov 22, 2026
Job Requisition Id:  67168
Location: 

Indore, MP, IN Indore, MP, IN

YASH Technologies is a leading technology integrator specializing in helping clients reimagine operating models, enhance competitiveness, optimize costs, foster exceptional stakeholder experiences, and drive business transformation.

 

At YASH, we’re a cluster of the brightest stars working with cutting-edge technologies. Our purpose is anchored in a single truth – bringing real positive changes in an increasingly virtual world and it drives us beyond generational gaps and disruptions of the future.

 

We are looking forward to hire Automation Testing Professionals in the following areas :

 

Role Overview

We are seeking an experienced Senior QA Engineer with strong expertise in AI-based systems, LLM applications, and intelligent agent testing. This role will be responsible for defining and implementing quality strategies for AI-powered platforms, including multi-agent systems, conversational AI, and LLM-driven workflows.The ideal candidate should have hands-on experience validating AI outputs, testing prompt-based systems, evaluating explainability, and ensuring Responsible AI compliance.

 

Key Responsibilities

AI & Agent Testing

Design and execute test strategies for AI/LLM-based applications and multi-agent systems.Validate agent behaviors, reasoning flows, tool usage, and A2A (Agent-to-Agent) interactions.Test follow-up question handling, performance, and contextual continuity.Perform evaluation of prompting techniques and prompt robustness.Validate explainability outputs and reasoning traceability.

Model & Output ValidationConduct qualitative and quantitative evaluation of LLM outputs.Develop evaluation datasets and benchmark test cases.Implement automated scoring mechanisms where applicable.Identify hallucinations, bias, unsafe outputs, and performance inconsistencies.

Responsible AI & Compliance

Ensure AI systems comply with Responsible AI guidelines (fairness, transparency, safety).Perform bias testing, toxicity checks, and adversarial testing.Validate guardrails and content moderation mechanisms.

Automation & Tools

Develop automation frameworks for API, backend, and agent workflows.Work with tools such as:Postman / REST AssuredSelenium / Playwright (PyTest / JUnitAI evaluation frameworks (e.g., LangSmith, Opik, custom evaluation pipelines)Validate streaming responses and real-time agent interactions.

Observability & Monitoring

Validate logging, tracing, and observability of agents.Test telemetry, response time, and performance under load.Ensure proper monitoring of BYOA (Bring Your Own Agent) and out-of-the-box agents.

Collaboration

Work closely with AI engineers, prompt engineers, backend developers, and product teams.Participate in architecture discussions to ensure testability.Provide risk assessments and quality metrics to stakeholders.

 

Required Qualifications

6–10 years of QA experience, with at least 2–3 years in AI/ML or LLM-based systems.Strong understanding of:Large Language Models (LLMs)Prompt engineering conceptsMulti-agent architecturesAPI and backend testingExperience testing conversational AI or agent-based workflows.Proficiency in automation using Python/Java/JavaScript.Experience with CI/CD pipelines and test integration.Strong analytical and debugging skills.

Preferred Qualifications

Experience with LLM evaluation frameworks (e.g., RAG evaluation, vector DB testing).Familiarity with Responsible AI standards and AI governance frameworks.Experience testing streaming APIs.Exposure to cloud platforms (AWS/GCP/Azure).Understanding of embeddings, vector databases, and retrieval systems.

Key Competencies

Strong problem-solving skillsDetail-oriented with a quality-first mindsetAbility to evaluate non-deterministic AI outputsExcellent communication and documentation skillsProactive and ownership-driven

What We Offer

Opportunity to work on cutting-edge AI and agent platformsHigh-impact role in shaping AI quality standardsCollaborative and innovation-driven environment

 

 

At YASH, you are empowered to create a career that will take you to where you want to go while working in an inclusive team environment. We leverage career-oriented skilling models and optimize our collective intelligence aided with technology for continuous learning, unlearning, and relearning at a rapid pace and scale.

 

Our Hyperlearning workplace is grounded upon four principles

  • Flexible work arrangements, Free spirit, and emotional positivity
  • Agile self-determination, trust, transparency, and open collaboration
  • All Support needed for the realization of business goals,
  • Stable employment with a great atmosphere and ethical corporate culture

Apply now »