M28 - Data Engineer

Role Overview We are seeking an experienced Data Engineer to design, build, and deploy a scalable and reliable data lakehouse platform on AWS. The role will focus on developing end-to-end data pipelines, establishing robust data quality frameworks, and delivering production-ready solutions that form the foundation of the organisation's data infrastructure. Key Responsibilities • Design, develop, and maintain scalable ETL/ELT data pipelines using AWS services such as Glue, Step Functions, Lambda, and S3. • Architect and implement data lakehouse solutions using Apache Iceberg, S3 Tables, or similar open table formats. • Implement lakehouse capabilities including schema evolution, partition evolution, and ACID transactions. • Optimise data pipelines and storage solutions for performance, scalability, reliability, and cost efficiency. • Define and implement automated data quality validation frameworks to ensure data accuracy, completeness, and consistency. • Establish data quality metrics, monitoring, and alerting mechanisms across the data platform. • Implement and maintain data governance standards and ensure compliance with relevant policies and requirements. • Develop production-quality code and deploy solutions on AWS cloud infrastructure. • Implement CI/CD practices to support automated, repeatable, and reliable deployments. • Use infrastructure-as-code tools such as Terraform or CloudFormation to provision and manage cloud infrastructure. • Work closely with cross-functional teams to design and deliver data engineering solutions. • Communicate technical designs and concepts clearly to both technical and non-technical stakeholders. • Produce comprehensive technical documentation covering architecture, pipelines, deployment, operations, and support procedures. • Support knowledge transfer and handover of developed solutions to Day 2 operations and support teams. Qualifications & Experience • Degree in Computer Science, Data Engineering, Information Systems, or a related discipline. • 3–5 years of relevant experience in data engineering, ETL/ELT development, or data platform engineering. • Strong hands-on experience with AWS cloud services, particularly Glue, Step Functions, Lambda, and S3. • Strong experience designing and developing scalable data pipelines and data lakehouse architectures. • Hands-on experience with Apache Iceberg, S3 Tables, or similar open table formats. • Good understanding of Apache Iceberg capabilities including schema evolution, partition evolution, ACID transactions, and table optimisation. • Experience implementing automated data quality frameworks, validation rules, monitoring, and data governance controls. • Experience developing and deploying production-grade applications or data solutions in cloud environments. • Experience with CI/CD practices and infrastructure-as-code tools such as Terraform or CloudFormation. • Experience working in a Government Commercial Cloud (GCC) environment is strongly preferred. • Strong analytical, problem-solving, communication, and stakeholder management skills. • AWS certifications such as AWS Certified Data Engineer, AWS Certified Solutions Architect, or equivalent certifications would be an advantage.