Senior Test Automation Engineer(AI Testing + python) (0-45 days joiners)

Responsibilities

• Design and execute test plans for agent behavior across reasoning, tool usage, multi-turn flows, and goal completion.
• Create prompt regression suites and synthetic test data for 50+ use cases.
• Build and maintain automated testing frameworks for agent workflows using Python.
• Evaluate agent output using a combination of rubric-based, LLM-based, and scenario-based techniques.
• Test for hallucinations, prompt injection, bias, PII leakage, and response safety.
• Work with observability tools (e.g., LangSmith, Trulens, logging pipelines) to analyze agent execution.
• Collaborate with engineers to iterate on prompt design, tool chaining, and fallback logic.
• Contribute to the development of agent test harnesses and benchmarking pipelines.

️ Skills & Qualifications

Required:

• 3+ years in QA, software testing, or NLP/ML product evaluation.
• Strong Python scripting and test automation experience.
• Experience testing REST APIs, workflows, or backend logic.
• Familiarity with LLMs and basic concepts in prompt engineering.
• Understanding of hallucination, context limits, and RAG-based agents.

Preferred:

• Experience with LangChain, CrewAI, or custom agent frameworks
• Experience with LLM evaluation tools like LangSmith, Promptfoo, Trulens, Ragas
• Exposure to guardrails, moderation APIs, or adversarial testing
• Prior experience in generative AI/LLM-based applications or chatbots

🎯 Bonus Points

• Experience with synthetic data generation
• Familiarity with CI/CD, version control (Git), and Agile QA practices
• Ability to write or grade evaluation prompts using GPT-4 or similar models

📦 What We Offer

• Opportunity to work on cutting-edge GenAI platform used across 50+ enterprise use cases
• Work with cross-functional AI-first teams (MLEs, PEs, PMs)
• Competitive compensation and benefits
• A collaborative, fast-paced environment where quality is a first-class citizen