Data Engineer (Databricks)

Job Summary The Data Engineer will design, develop, and maintain scalable, reliable data pipelines on Databricks and cloud platforms, integrating diverse data sources to support analytics, reporting, and machine learning. Collaborate with cross-functional teams to enhance data platform governance, monitoring, and reliability. Responsibilities • Design, develop, and maintain ETL pipelines for centralized data storage systems such as Delta Lake to ensure efficient data ingestion and transformation • Integrate data from databases, APIs, log files, streaming platforms, and external providers to support diverse analytics needs • Develop data transformation routines to clean, normalize, and aggregate complex or inconsistent datasets for accurate analysis • Apply data processing techniques to handle large-scale and varied data sources effectively • Contribute to the development and enforcement of frameworks and best practices for code development and deployment to ensure quality and consistency • Implement data governance policies aligned with company standards to maintain data integrity and compliance • Collaborate with analytics and product teams to design and operationalize data pipelines that meet business requirements • Work with infrastructure teams to advance cloud-based data platforms leveraging Azure and Databricks technologies • Explore and evaluate new tools and techniques to enhance data platform capabilities on Azure, Databricks, or related platforms • Monitor data pipelines continuously to detect, troubleshoot, and resolve issues promptly, ensuring high availability • Develop monitoring tools, alerts, and automated error-handling mechanisms to maintain pipeline reliability • Analyze business requirements and translate them into data extraction and pipeline development tasks • Participate in requirement grooming and refinement sessions with users to clarify data needs • Optimize pipeline performance and batch scheduling to maximize efficiency and minimize latency • Develop dashboards, reports, scorecards, and data visualizations to support data-driven decision-making • Perform system integration testing (SIT), data validation, and profiling to confirm data accuracy and completeness • Validate completeness and consistency of ETL loads to ensure reliable data delivery • Support user acceptance testing (UAT) and production implementation activities to ensure smooth deployment Required competencies and certifications • Databricks Certified Data Engineer Associate (strongly preferred) • Databricks Certified Data Engineer Professional (strongly preferred) Preferred competencies and qualifications • 3 or more years of experience in data engineering with scalable pipelines • Strong experience designing data solutions including data modelling and distributed computing architectures • Hands-on experience with data processing jobs using PySpark, Spark SQL, and Databricks notebooks/jobs • Experience orchestrating data pipelines with Azure Data Factory (ADF), Airflow, or similar tools • Experience with both real-time and batch data processing • Experience building pipelines on Azure; AWS experience is beneficial • Proficiency in SQL including window functions and performance optimization • Understanding of DevOps tools, Git workflows, and CI/CD pipelines • Familiarity with Scrum methodology and experience working in Scrum teams • Ability to apply Scrum practices in a practical project context • Strong problem-solving and collaborative mindset • Experience with streaming technologies such as Apache Kafka, Apache Flink, or AWS Kinesis • Ability to design and implement real-time data processing pipelines