Data Engineer

Job Description · We are looking for an experienced Data Engineer to join the delivery team for the EDW Reporting Modernisation project. · You will be responsible for designing and building the Gold layer Data Mart tables, transformation pipelines, and associated views on the Unified Data Platform (Databricks), working closely with the Tech Lead and Data Modeler to deliver a robust and scalable reporting data layer that caters to Power BI reports. · As the onshore Data Engineer, you will serve as the technical bridge between the Singapore-based design team and the offshore engineering squad, providing day-to-day technical direction, code review, and quality oversight across the full Data Mart build. Key Responsibilities · Design and Development:: o Develop Gold layer Data Mart tables and associated database views in Databricks in accordance with signed-off Source-to-Target Mapping (STTM) documents and design specifications o Implement Silver-to-Gold transformation pipelines using Databricks Workflows, Delta Live Tables (DLT), and PySpark o Create Delta table DDL in Unity Catalog, including schema definition, table properties, and column-level comments o Implement SCD Type 2 merge logic for dimension tables and aggregation logic for fact tables per the grain specification o Configure table partitioning, Z-ORDER clustering, and Liquid Clustering strategies for query performance optimisation o Develop and implement data quality expectations using Delta Live Tables DQ framework per the DQ thresholds defined in the design specification o Schedule and configure Databricks Workflow job DAGs for all pipeline runs, including watermark-based incremental load logic · Testing and Quality Assurance:: o Conduct unit testing for each Data Mart table upon build completion — covering row count validation, null checks, duplicate grain checks, and transformation logic verification o Execute system integration test (SIT) scenarios for all 60 Data Mart tables, including aggregated value checks, referential integrity validation, and SCD integrity checks o Log, investigate, and resolve defects identified during SIT and UAT, performing root cause analysis and documenting fixes o Review and validate unit test outputs from Pune Data Engineers before test results are submitted for Tech Lead review · Offshore Squad Technical Leadership:: o Provide day-to-day technical direction and coding guidance to the 2 offshore-based Data Engineers o Conduct code reviews for all pipeline and DDL code produced by the offshore squad before submission for Tech Lead sign-off o Ensure offshore squad adherence to agreed coding standards, naming conventions, Unity Catalog governance rules, and development best practices o Facilitate knowledge sharing and technical problem-solving with the Pune squad on complex transformation and merge logic · Deployment and Handover:: o Execute Data Mart pipeline and table deployment to the production Databricks environment per the approved deployment runbook o Validate the first production pipeline run and confirm data freshness post-deployment o Contribute to the operational runbook covering pipeline architecture, job schedules, alert thresholds, and common failure scenarios o Support knowledge transfer sessions with the Singtel IT operations team during the handover phase Required Skills and Experience · 5+ years of experience in data engineering with a strong focus on ETL/ELT pipeline development and dimensional data modelling · Hands-on experience with Databricks — including Delta Lake, Delta Live Tables, Databricks Workflows, Unity Catalog, and Databricks SQL · Proficiency in PySpark and Databricks SQL for large-scale data transformation · Strong understanding of dimensional modelling concepts — star schema, SCD Type 2, surrogate key design, fact and dimension table design · Experience implementing data quality frameworks and reconciliation testing · Familiarity with Source-to-Target Mapping (STTM) and translating design specifications into production-grade pipelines · Experience with incremental load patterns — watermark-based, partition-based, or CDC-driven · Proficiency in Git-based version control for collaborative development · Experience with Databricks Unity Catalog access control, table tagging, and lineage tracking · Exposure to Oracle-to-Databricks migration projects or similar platform modernisation programmes · Familiarity with Power BI Semantic Models and how Gold layer table design impacts downstream DAX measure performance · Experience with Databricks Assistant or AI-assisted code generation tools for accelerated pipeline development · Knowledge of CI/CD pipeline setup for Databricks notebook or YAML