We're supporting a major financial-systems transformation programme and are looking for a Senior Databricks Engineer to help build and enhance a scalable data-engineering framework used across critical reporting processes. You'll operate as the senior hands-on engineer, shaping reusable libraries, optimising PySpark pipelines, and guiding offshore developers to deliver high-quality, production-ready code.

You'll work across ingestion, validation, transformation, and mapping layers within a Databricks-on-Azure environment, helping to establish consistent engineering patterns and ensuring all deliveries meet high standards of performance, traceability, and maintainability.

What you'd be doing:

• Building and extending reusable Databricks/PySpark libraries (ingestion, validation, transformation, mapping).

• Developing scalable, optimised PySpark pipelines aligned to metadata-driven or hybrid data models.

• Implementing robust validation and control logic suitable for regulated financial environments.

• Leading engineering quality: code reviews, mentoring offshore teams, ensuring standards across readability, testing, and performance.

• Supporting integration paths into downstream reporting systems and managing return-feeds.

• Producing clear documentation, runbooks, and developer guides.

My ideal candidate will have:

• Strong hands-on experience with Databricks and PySpark (optimisation, partitioning, orchestration).

• Solid Azure data-platform experience.

• Proven ability to build reusable frameworks/libraries with clean abstractions.

• Experience guiding or leading offshore engineering teams.

• Comfortable working with evolving requirements while maintaining strong engineering discipline.

• Bonus: exposure to financial or regulatory reporting, Tagetik, or metadata-driven/EAV models.

This is an urgent role, will be remote with once monthly visits to client site and up to £400 per day OUTSIDE IR35.

JBRP1_UKTJ