Databricks Engineer – 12 months contract

Key Responsibilities • Design and build batch and/or streaming data pipelines using Databricks and AWS services. • Develop data processing across Bronze, Silver, and Gold layers using SQL, Python/PySpark, and Delta Lake. • Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources. • Develop data transformations, dimensional or curated datasets, and reusable data-engineering components. • Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring. • Optimise Spark jobs, clusters, partitioning, and storage for performance and cost. • Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover. • Collaborate with platform, application, BI, and data-governance teams. Requirements • 3+ years of experience in data engineering or enterprise data-platform delivery. • Hands-on experience with Databricks, Apache Spark, PySpark, and SQL. • Experience with Delta Lake and layered data architecture such as Bronze/Silver/Gold. • Experience building production data pipelines with proper logging, error handling, and recovery. • Experience with AWS services such as S3, IAM, Glue, Lambda, Kinesis, or related services. • Good understanding of data modelling, data quality, and performance optimisation. • Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming. • Experience with APIs, XML/JSON, event or near-real-time ingestion. • CI/CD and infrastructure/configuration automation using Git-based tools. • Experience with Tableau, BI datasets, or data warehouse integration. • Databricks, AWS, data engineering, AI/ML, or GenAI-related certification(s). • Strong engineering fundamentals and willingness to learn. • Practical problem-solving and attention to data correctness. • Clean implementation, documentation discipline, and reliable delivery. • Ability to work in a fast-paced project environment and collaborate with different technical teams. • Ownership of assigned tasks and openness to technical guidance and feedback.