• Databricks Certification is Mandatory

• Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake).

• Integrate data from databases, APIs, log files, streaming platforms, and external providers

• Develop data transformation routines to clean, normalize, and aggregate data

• Apply data processing techniques to handle complex or inconsistent datasets

• Contribute to frameworks and best practices for code development and deployment

• Implement data governance in alignment with company standards

• Partner with analytics and product leaders to design and operationalize pipelines

• Collaborate with infrastructure leaders to advance cloud-based data platforms

• Explore new tools and techniques leveraging Azure, Databricks, or related platforms

• Monitor data pipelines to detect and resolve issues promptly

• Develop monitoring tools, alerts, and automated error-handling mechanisms

• Analyse business requirements and identify data extraction requirements

• Attend requirement grooming and refinement sessions with users

• Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading

• Optimise performance and batch scheduling

• Develop dashboards, reports, scorecards, and data visualizations

• Perform SIT, data validation, data profiling and confirm data accuracy

• Validate completeness and consistency of ETL Loads

• Support UAT and production implementation

Qualifications

The ideal candidate should possess:

• 3 or more years of experience in data engineering with scalable pipelines

• Strong experience designing data solutions including data modelling and distributed computing architectures

• Hands-on experience with data processing jobs using PySpark, Spark SQL, and Databricks notebooks/jobs

• Experience orchestrating data pipelines with ADF, Airflow, or similar tools

• Experience with both real-time and batch data processing

• Experience building pipelines on Azure, with AWS experience beneficial

• Proficiency in SQL including window functions and performance optimization

• Understanding of DevOps tools, Git workflows, and CI/CD pipelines

• Familiarity with Scrum methodology and experience working in Scrum teams

• Ability to apply Scrum practices in a practical project context

• Strong problem-solving and collaborative mindset

• Experience with streaming technologies such as Apache Kafka, Apache Flink, or AWS Kinesis

• Ability to design and implement real-time data processing pipelines

Certification:

Databricks Certified Data Engineer Associate and Databricks Certified Data Engineer Professional are preferred