Job Specification — Data Engineer x3 (Databricks/PySpark/Azure) - Ireland
Location: Limerick (preferred) OR UK based remote
Pay rate: €400-430 per day
6 month initial contract

Team: Data & Analytics — Supply Chain  Reports to: Engineering Lead, within a SAFe Agile Release Train

1. About the Role
You will join a Data & Analytics team building a large-scale supply chain visibility and decision-intelligence platform. The platform ingests data from multiple enterprise source systems and curates it through a layered (medallion-style) data architecture to power interactive analytics for planners, supply chain leaders, and decision-makers.

The team delivers continuously on a  2-week sprint cadence with regular major release cycles, in a regulated environment where data accuracy and traceability are critical.

Data architecture (medallion-style): L0 (raw) ? L1 (standardized) ? L2 (curated) 2. Role Summary
We are looking for a  Data Engineer to design, build, and maintain the data pipelines that feed the analytics layer. You will own data ingestion, transformation, and quality across the L0 ? L1 ? L2 layers, working in  Databricks/PySpark and  Azure Data Factory, ensuring curated, trustworthy data reaches the downstream applications.

You will work in an Agile squad alongside fellow data engineers  and QA, contributing to a regulated environment where data accuracy and traceability are critical.

3. Key Responsibilities

• Build & maintain data pipelines across the L0 ? L1 ? L2 layers using Databricks (PySpark) and Azure Data Factory (ADF).
• Develop curated data models in the L2 layer that power supply chain KPIs (inventory health, demand/supply waterfalls, network health, plan attainment, risk logic, etc.).
• Integrate new source systems into the platform, mapping raw feeds to standardized and curated schemas.
• Implement data quality and validation controls — automated checks, reconciliation, and traceability from source to dashboard.
• Write and maintain unit tests for your code (rather than relying on manual testing), contributing to automated test runs as part of the pipeline.
• Optimize performance & cost of Spark jobs and data workflows — query tuning, indexing, and partitioning strategies; manage Unity Catalog governance.
• Build and monitor ADF pipelines — orchestrate, schedule, and troubleshoot Azure Data Factory data flows.
• Support releases on a regular major release cadence — participate in SIT/UAT, troubleshoot defects, and support hotfixes.
• Collaborate in sprint ceremonies (planning, refinement, standups, retros) and document key design decisions.
• Maintain CI/CD hygiene — pull requests, database migrations, code quality gates, and artifact management.

4. Required Skills & Experience
Core technical (must-have) Area Requirement Data processing Hands-on experience with  Spark and Delta Lake for data processing and analysis (Databricks / PySpark) Languages Proficiency in  Python (used for SQL-based transformations and pipeline logic) SQL platforms Experience with  SQL Server and/or  Synapse (formerly SQL Data Warehouse) Data modeling Solid understanding of  data modeling principles and concepts Performance tuning Practical experience optimizing data processing, including  indexing and partitioning for query performance Orchestration Comfortable building, monitoring, and managing  Azure Data Factory (ADF) pipelines Testing Experience implementing  unit testing practices for code (not relying solely on manual testing) CI/CD Understanding of  CI/CD processes and their implementation Technical depth Strong technical depth across  databases, testing, and performance tuning Ways of working

• Comfortable in Agile/Scrum (SAFe Release Train experience a plus).
• Experience supporting frequent releases and hotfix cycles.
• Strong collaboration with QA and analytics consumers.

5. Nice-to-Have / Differentiators

• Familiarity with Common Data Layer (CDL) concepts.
• Python-based Spark development and monitoring ADF pipelines at scale.
• neo4j / graph databases and Redis caching.
• Supply chain / manufacturing domain experience.
• Experience with AI-assisted tooling (e.g., GitHub Copilot and similar tools across the SDLC).

6. Tech Stack You'll Work With Layer Technologies Data / ETL Databricks (Spark + Delta Lake, PySpark), Azure Data Factory, Azure Blob, Unity Catalog Databases SQL Server / Synapse, neo4j Graph DB, Redis CI/CD & Quality Jenkins, Bitbucket, Artifactory, Flyway, SonarQube Testing Unit testing, automated test runs in CI Collaboration Jira, Confluence 7. What Success looks like

• First Sprint — Onboarded to the L0?L1?L2 data architecture; environment access set up (Databricks, source control, CI/CD, SQL Server); shadowing a squad through a sprint, first User Story delivered.
• First Release — Independently delivering data-engineering stories within a sprint; writing unit tests for your pipelines; raising clean pull requests that pass code quality gates.
• Success — Owning a data domain end-to-end; contributing to a major release; proposing pipeline or quality improvements.

 


By applying for this role, you consent to SGI contacting you by telephone regarding recruitment services, market updates, and relevant business opportunities

We believe in equal opportunity for all and actively encourage applications from diverse backgrounds, experiences, and perspectives.
Source Group International Ltd is acting as an Employment Business in relation to this vacancy