Senior Data Engineer

New Yesterday

Job Description

Senior / Lead Data Engineer – Databricks / Spark (High-Performance Platform)

£850 p/d OUTSIDE IR35 (higher for Lead)

12-month rolling (multi-year programme)

1 day/week – St Paul’s, London


We’re hiring a Senior & Lead Data Engineer to build a Databricks lakehouse platform in a high-performance, business-critical Front office trading environment.


This is a hands-on engineering role focused on building and optimising large-scale distributed data systems.


This is a highly technical team operating at scale, we’re looking for engineers with deep data engineering expertise, strong low-level Spark knowledge, and experience building high-performance systems using modern Databricks and AI-driven platforms.

.

What you’ll do

  • Build and optimise Spark pipelines on Databricks
  • Develop a lakehouse platform (Medallion architecture)
  • Own data modelling, architecture, and pipeline design
  • Work with large-scale data (TB–PB)
  • Drive performance, scalability, and reliability in production


What we’re looking for

  • Strong experience running Spark workloads in production
  • Proven ability to optimise Spark at scale (Tb/PB datasets)
  • Solid Python (Scala beneficial, not essential)
  • Experience with data modelling and lakehouse architecture
  • Ability to debug and improve performance in distributed systems


Important

  • Must have recent, hands-on Spark experience
  • Databricks strongly preferred (not essential if Spark depth is very strong)
  • Experience supporting AI/ML or advanced analytics platforms is a big plus


Nice to have

  • Financial services / trading exposure
  • Experience in performance-critical environments


Not a fit if

  • Primarily BI / reporting focused
  • Spark used only at small scale or outside production
  • No experience with performance optimisation in distributed systems


Stack

Databricks (Azure), Spark, Delta Lake, Python (+ Scala optional)


Bottom line

We’re looking for engineers who can design, build, and optimise Spark-based systems at scale and operate effectively in a performance-critical environment from day one.

Apply
Location:
City Of London
Job Type:
FullTime
Category:
Technology

We found some similar jobs based on your search