AI Quality Assurance Automation Engineer

Job description

AI Quality Assurance Automation Engineer

Be among the first 25 applicants.

Get AI‑powered advice on this job and more exclusive features.

The way we work and our customer needs are changing! We are going through a huge AI uplift that will be a real game changer in the NZ Market.

Our customers are loving our story and excited about our new service solutions.

As such we are seeking AI Quality Assurance Automation Engineers who want to get on board our journey.

We need someone who is passionate about AI development and wants to push boundaries.

Your software background is key but your passion for AI is a must.

This role will see you join our Digital Engineering team, focusing on delivering innovative AI‑driven software solutions.

We will offer you the opportunity to work on cutting‑edge projects, collaborating with cross‑functional teams to design, develop, and deploy advanced AI models that address complex business challenges.

We need multiple people but we need them now, so please don’t hesitate – this is your chance to put a New Zealand company on the global map!

Responsibilities

• Design, develop, and implement advanced AI‑powered QA automation solutions for testing AI‑based applications, models, and systems; use innovative techniques to identify performance bottlenecks, inconsistencies, and model biases.
  
  
• Build and maintain automated test frameworks for AI systems using a wide variety of automation tools (e.g., Playwright, Appium, pytest, or custom‑built AI testing tools); ensure that test automation strategies are scalable, reusable, and aligned with industry best practices.
  
  
• Perform functional and non‑functional testing of AI models to assess accuracy, bias, scalability, and robustness; conduct regression testing and ensure that AI solutions behave as expected across different scenarios.
  
  
• Stay up to date with the latest AI advancements and research; actively explore and adopt emerging AI technologies (e.g., GPT models, neural networks, reinforcement learning) to continuously improve the quality assurance process.
  
  
• Test and validate AI solutions hosted on both cloud platforms (e.g., AWS, Azure, Google Cloud) and on‑premise AI services; ensure that the AI solutions perform optimally across both environments.
  
  
• Work closely with AI developers, data scientists, and product teams to identify testing requirements, troubleshoot issues, and improve overall AI system quality; provide feedback and insights on AI solution design from a QA perspective.
  
  
• Implement and continuously improve testing methodologies for AI systems; drive automation efficiency and effectiveness, ensuring that testing is not only comprehensive but also aligned with fast‑paced AI development cycles.
  
  
• Demonstrate the place and value propositions for test automation pipelines with AI solutions using PaaS (e.g., Azure DevOps, GitHub Actions).
  
  
• Establish key quality metrics for AI systems, track progress, and report on the effectiveness of AI testing strategies; maintain clear, concise, and actionable documentation and reports on test outcomes, issues found, and resolutions.
  
  
• Communicate abstract concepts and value points to stakeholders in terms that can be understood and discussed.
  
  

Skills & Qualifications

• 4–5 years of experience in test automation, strong focus on AI and machine learning‑based applications and systems.
  
  
• Proficient in programming languages such as Python, C#, Java, or TypeScript; strong proficiency in Python.
  
  
• Hands‑on experience with automation tools like Selenium, Appium, Pytest, or other testing frameworks.
  
  
• Experience working with cloud‑based AI services (e.g., AWS AI/ML, Azure Cognitive Services, Google Cloud AI) as well as on‑premise AI solutions.
  
  
• Knowledge of version control systems (e.g., Git) and CI/CD pipelines for automated testing.
  
  
• Basic understanding of AI/ML concepts, including data preprocessing, model training, evaluation metrics, and model fairness.
  
  
• Strong analytical skills to assess AI system performance, debug issues, and design solutions to improve test coverage and efficiency.
  
  
• Demonstrated ability to stay current with AI industry trends and developments; ability to quickly adapt to new tools, technologies, and methodologies.
  
  
• Excellent teamwork and communication skills; able to work effectively with cross‑functional teams and explain complex technical concepts to both technical and non‑technical stakeholders.
  
  

Preferred Qualifications

• Experience with on‑premise AI services and managing AI deployments in hybrid environments.
  
  
• Familiarity with AI ethics, fairness, and bias testing methodologies.
  
  
• Experience in test‑driven development (TDD) or behavior‑driven development (BDD) in AI projects.
  
  

Why Join Us

• Be part of a dynamic, forward‑thinking team that is passionate about pushing the boundaries of AI technology.
  
  
• Work in an environment that fosters innovation and continuous learning.
  
  
• Opportunity to lead the way in AI QA, making a direct impact on the quality of cutting‑edge AI products.
  
  
• Competitive salary and benefits package.
  
  

If you’re a proactive, tech‑savvy AI QA engineer with a strong automation background and a passion for staying ahead of AI developments, we’d love to hear from you!

Job Details

• Seniority Level: Mid‑Senior level
• Employment Type: Full‑time
• Job Function: Quality Assurance
• Industries: IT Services and IT Consulting
• Location: Auckland, New Zealand

Required Skill Profession

Quality Management & Operations