Software Quality Assurance Engineer – GenAI / LLM

Primary Focus: Software Quality Assurance / Software Testing / Test Automation – GenAI, LLM & Agentic AI Secondary Exposure: Solution Analysis / Technology Solution Design / Enterprise Integration Domain / Project: Global Markets, Capital Markets Banking Technology & Market Risk Technology Role Overview We are looking for a Senior GenAI Quality Engineer / Solution Analyst to design, analyse, test and validate production-grade Generative AI (GenAI), Large Language Model (LLM), RAG and Agentic AI applications within a complex enterprise environment. This is not a traditional manual QA or software testing role . The role combines: • Software Quality Engineering • GenAI / LLM Testing & Evaluation • Agentic AI / AI Agent Testing • UI & API Testing • Test Automation • Solution Analysis • Enterprise Integration Testing • Observability & Troubleshooting You will work across discovery, solution design, development, testing and release , translating business requirements into clear application behaviours and validating end-to-end application quality across user interfaces, APIs, data flows, LLMs, RAG components, AI agents and enterprise integrations. Key Responsibilities GenAI / LLM Quality Engineering • Define and execute end-to-end quality engineering and test strategies covering: • Web / UI workflows • REST APIs • Backend services • Enterprise integrations • GenAI applications • LLM workflows • RAG pipelines • Agentic AI / AI Agent interfaces • Perform GenAI / LLM testing and evaluation covering: • Response quality • Task completion • Grounding • Faithfulness • Relevance • Consistency • Citation accuracy • Hallucination risk • Safe failure behaviour • Test non-deterministic / probabilistic AI systems using: • Evaluation datasets • Repeat testing • Quality thresholds • Acceptance criteria • Regression evaluation • Validate RAG / Retrieval-Augmented Generation solutions, including retrieval quality, grounding and response accuracy. Agentic AI / AI Agent Testing Test end-to-end Agentic AI and AI Agent workflows , including: • Multi-turn conversations • Context handling • Agent planning • Tool selection • Tool calling / function calling • Tool inputs and outputs • State transitions • Memory and state • Human-in-the-loop approvals • Handoffs • Retries • Timeouts • Fallback behaviour • Error recovery • Termination conditions • Partial failures Validate that AI agents behave correctly across both successful and failure scenarios. Software & API Quality Engineering Perform: • Functional Testing • Integration Testing • API Testing • Regression Testing • Exploratory Testing • Negative Testing • Resilience Testing • Basic Performance Testing • End-to-End Testing Design comprehensive REST API tests covering: • API contracts • Authentication • Authorisation • Input validation • Error handling • Idempotency • Rate limits • Downstream system failures Test web application behaviour across browsers and realistic end-user journeys, including: • Loading states • Interrupted sessions • Error messages • Feedback capture • Accessibility fundamentals Test Automation Develop and maintain risk-based test automation that reduces: • Regression testing time • Manual testing effort • Release cycle time • Production risk Use automation frameworks and tools such as: • Playwright • Cypress • Selenium • pytest • REST Assured • Postman • Equivalent UI / API automation frameworks Apply pragmatic automation principles by prioritising stable, high-value and frequently executed test scenarios. GenAI Evaluation & AI Safety Testing Validate LLM and GenAI applications for: • Grounded responses • Hallucinations • Retrieval quality • Citation accuracy • Prompt behaviour • Prompt injection • Unsupported requests • Restricted content handling • Safe failure behaviour • Adversarial scenarios Support AI evaluation / LLM evaluation using appropriate evaluation datasets, quality metrics and repeatable evaluation approaches. Exposure to AI Red Teaming / Adversarial Testing would be advantageous. Observability & Troubleshooting Use application and GenAI observability to identify the source of defects across: • Application • LLM / Model • RAG / Retrieval • Data • API / Integration • Platform Analyse: • Logs • Distributed traces • API requests / responses • Payloads • Network calls • Database records • Agent execution traces Exposure to observability and LLM evaluation tools such as: • Langfuse • LangSmith • OpenTelemetry • Elastic / Elasticsearch • Splunk is advantageous. Solution Analysis & Design The role also acts as a hands-on Solution Analyst for GenAI applications. Responsibilities include: • Partner with product owners, business users, architects, engineers and GenAI specialists during discovery and solution design. • Analyse proposed GenAI use cases and determine whether the requirement should use: • Conventional application logic • Deterministic business rules • Search / retrieval • RAG • Workflow automation • Agentic AI • Human approval • Translate business requirements into: • Functional requirements • End-to-end solution flows • User journeys • Acceptance criteria • Interface behaviour • Decision rules • Non-functional requirements • Map interactions across: • User Interfaces • APIs • LLMs / Models • Prompts • RAG / Retrieval components • Enterprise data sources • AI Agent tools • Downstream enterprise systems • Analyse solution design trade-offs involving: • Quality • Complexity • Cost • Latency • Security • Data access • Maintainability • Operational risk • Identify missing controls, integration assumptions, ownership gaps, failure scenarios and operational risks before development begins. • Support the design of: • Human-in-the-loop approval • Fallback flows • Escalation • Exception handling Solution Documentation Produce practical technical and functional artefacts i…