QA Engineer – AI Automation Validation

Job Description: • Develop comprehensive test strategies: Create and maintain test plans covering functional, performance, and regression testing for AI agents and automation workflows. • Build and maintain automated test frameworks: Develop and maintain automated test scripts/frameworks to ensure bot responses and workflow outputs remain accurate, consistent, and reliable over time. • Build and maintain test assets: Create/curate golden conversation sets, synthetic test data, evaluation rubrics, and regression suites to measure quality over time. • Validate AI agent safety and reliability: Test tool-calling end-to-end (tool selection, parameter mapping, sequencing, retries/timeouts, error handling, and recovery), including negative/edge cases; confirm outputs align to business rules and document risk mitigations. • Test integration/workflow integrity: Validate API contracts, data transformations, and downstream impacts (idempotency, duplicate prevention, partial failures, and latency) across environments. • Manage defects through resolution: Identify, log, prioritize, and track defects; partner with developers to reproduce issues and validate fixes. • Provide release certification: Deliver final quality sign-off prior to production release, ensuring defined acceptance criteria and quality standards are met. • Align on acceptance criteria: Partner with Business Analysts to ensure user stories include clear, testable acceptance criteria and that test coverage maps to requirements. • Document testing and metrics: Maintain test cases, data, results, and quality metrics; communicate outcomes and risks. • Participate in Agile ceremonies: Engage in planning, grooming, standups, and retrospectives to embed quality throughout delivery. • Drive continuous improvement: Identify opportunities to improve automation coverage, testing effectiveness, and quality practices. Requirements: • 3+ years in a QA, test automation, or software testing role with demonstrated experience developing and executing test strategies. • Test automation expertise: Strong hands-on experience building automated test frameworks using tools such as Selenium, Playwright, Cypress, Jest, Pytest, or similar frameworks. • Scripting and programming skills: Proficiency in at least one programming or scripting language (Python, JavaScript, Java, C#) for test automation development. • Agile methodology: Direct experience working in an Agile/Scrum environment with hands-on involvement in sprint cycles and continuous integration. • Defect management tools: Experience using bug tracking and test management tools such as Jira, Azure DevOps, TestRail, or similar platforms. • Bachelors level degree is preferred, but not required. Benefits: • Health insurance • 401(k) matching • Flexible work hours • Paid time off • Remote work options Apply tot his job Apply To this Job