Lead PySpark Engineer - Data, SAS, AWS
36 Days Old
Lead Data Engineer - Pyspark / AWS / Python / SAS - Financial Sector
As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising legacy data workflows and supporting large-scale SAS-to-PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment.
Essential Skills
PySpark & Data Engineering
Minimum 5+ years of hands-on PySpark experience.
SAS to Pyspark migration experience
Proven ability to write production-ready PySpark code.
Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDs
Spark Performance & Optimisation
Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning.
Experience troubleshooting and improving distributed data processing pipelines.
Python & Engineering Quality
Strong Python coding skills...
- Location:
- London
- Salary:
- £350 - £380 per day
- Job Type:
- FullTime
- Category:
- Engineering