Lead Data Engineer (Teradata/Hadoop)

Company Overview OPTIMUM SOLUTIONS (SINGAPORE) PTE LTD is a Singapore-headquartered IT outsourcing firm specializing in software engineering, application development, and IT infrastructure management. Since 1997, it has grown into a leading provider of digital transformation services. Job Summary We are seeking a Senior Data Engineer with expertise in enterprise data platforms and banking domains to lead the design and development of scalable data pipelines across Teradata, Hadoop/Cloudera, and cloud ecosystems while mentoring engineering talent and driving technical excellence. Responsibilities • Lead the architecture and implementation of scalable batch, streaming, and API-based data pipelines to support large-scale data delivery and historical data processing frameworks. • Design and implement Slowly Changing Dimension (SCD) and Change Data Capture (CDC) mechanisms to maintain accurate and up-to-date data products. • Develop semantic-layer data products that enhance data accessibility and usability for business stakeholders. • Optimise complex SQL queries, Spark workloads, partition strategies, and storage frameworks to improve performance for large-volume data processing in regulated enterprise banking environments. • Implement infrastructure-as-code using Terraform to automate provisioning and management of cloud and on-premises resources. • Establish automated Continuous Integration/Continuous Deployment (CI/CD) pipelines and containerized deployment workflows to ensure reliable and repeatable software delivery. • Enforce rigorous testing standards to maintain high-quality, production-ready data engineering solutions. • Lead code reviews and enforce engineering best practices to uphold technical excellence across teams. • Mentor junior engineers to develop their technical skills and promote disciplined delivery in regulated settings. • Collaborate closely with architects, product teams, DevOps, and business stakeholders to align data engineering efforts with organizational goals. Required competencies and certifications • Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related discipline (required education). • At least 10 years of data engineering experience with technical leadership in Teradata and Hadoop/big-data delivery. Preferred competencies and qualifications • Certifications in Teradata, Cloudera, Databricks, Cloud Data Engineering, or Terraform. • Proficiency with core technologies including Teradata, Hadoop/Cloudera, Spark, Hive, Impala, HBase, SQL, Python, Scala/Java, Kafka, Airflow, and Denodo. • Experience with infrastructure and DevOps tools such as Terraform, Kubernetes/OpenShift, Git, and CI/CD automation tools. • Domain expertise in enterprise banking data including Customer, Payments, Risk, Finance, AML, and Regulatory Reporting, with a focus on data quality, reconciliation, lineage, and historical reconstruction. • Demonstrated technical leadership, disciplined delivery in regulated environments, strong communication skills, and accountability for production-ready outcomes.