Data Engineer - Data Lakehouse Architecture (Public Sector)

• 1-year contract, renewable • Government project • Hybrid work arrangement Role Overview The Data Engineers will design, build and deploy a scalable and reliable data lakehouse platform on AWS. Their responsibilities will include developing end-to-end data pipelines, establishing data quality frameworks and delivering production-ready solutions to support the organisation's data infrastructure. The Senior Consultant will additionally lead complex engineering activities and provide technical guidance and implementation oversight. Key Responsibilities Data Pipelines and Architecture • Design and implement scalable ETL/ELT pipelines using AWS Glue, Step Functions, Lambda and S3. • Build data lakehouse solutions using Apache Iceberg or Amazon S3 Tables, incorporating schema evolution, partition evolution and ACID transactions. • Optimise pipelines for performance, cost and reliability. Data Quality and Governance • Develop and maintain automated data quality validation frameworks. • Establish metrics and monitoring to ensure data accuracy, completeness and consistency. • Implement data governance controls and ensure compliance with applicable standards. Application Development and Deployment • Write production-quality code and deploy solutions on AWS. • Implement CI/CD pipelines and infrastructure as code using tools such as Terraform or CloudFormation to support repeatable and auditable deployments. Collaboration and Handover • Work with cross-functional teams and communicate technical designs clearly to technical and non-technical stakeholders. • Prepare technical documentation and support handover to the Day 2 operations team. Qualifications and Experience • A degree in Computer Science, Data Engineering, Information Systems or a related discipline. • Five to eight years of experience in data engineering, ETL/ELT development or data platform roles. • Strong hands-on experience developing and deploying production solutions using AWS services. • Proficiency in Apache Iceberg or similar open table formats. • Experience in a Government Commercial Cloud (GCC) environment is strongly preferred. • Relevant AWS certifications are an advantage. Skills and Competencies • Strong understanding of data lakehouse architecture, Apache Iceberg features and Amazon S3 Tables optimisation. • Ability to design reliable pipelines and implement automated data quality and governance controls. • Proficiency in production-quality coding, performance optimisation and cloud deployment. • Ability to troubleshoot complex engineering issues and document solutions clearly. /Ref: T24-251