Senior Databricks Consultant

New Today

We're supporting a major financial-systems transformation programme and are looking for a Senior Databricks Engineer to help build and enhance a scalable data-engineering framework used across critical reporting processes. You'll operate as the senior hands-on engineer, shaping reusable libraries, optimising PySpark pipelines, and guiding offshore developers to deliver high-quality, production-ready code.
You'll work across ingestion, validation, transformation, and mapping layers within a Databricks-on-Azure environment, helping to establish consistent engineering patterns and ensuring all deliveries meet high standards of performance, traceability, and maintainability.
What you'd be doing:
Building and extending reusable Databricks/PySpark libraries (ingestion, validation, transformation, mapping).

Developing scalable, optimised PySpark pipelines aligned to metadata-driven or hybrid data models.

Implementing robust validation and control logic suitable for regulated financial environments.

Leading engineering quality: code reviews , mentoring offshore teams, ensuring standards across readability, testing, and performance.

Supporting integration paths into downstream reporting systems and managing return-feeds.

Producing clear documentation, runbooks, and developer guides.

My ideal candidate will have:
Strong hands-on experience with Databricks and PySpark (optimisation, partitioning, orchestration).

Solid Azure data-platform experience.

Proven ability to build reusable frameworks/libraries with clean abstractions.

Experience guiding or leading offshore engineering teams .

Comfortable working with evolving requirements while maintaining strong engineering discipline.

Bonus: exposure to financial or regulatory reporting , Tagetik , or metadata-driven/EAV models .

This is an urgent role, will be remote with once monthly visits to client site and up to £400 per day OUTSIDE IR35.

TPBN1_UKTJ
Apply
Location:
London
Salary:
£350 - £450 per day
Job Type:
FullTime
Category:
Consulting

We found some similar jobs based on your search