Lead PySpark Engineer

New Today

Lead PySpark Engineer As a Lead PySpark Engineer , you will design, develop, and optimise complex data processing solutions on AWS. You will work hands-on with PySpark, modernise Legacy data workflows, and support large-scale SAS-to-PySpark migration programmes. This role requires strong engineering discipline, deep data expertise, and the ability to deliver production-ready data pipelines within a financial services environment. Skill Profile: PySpark - P3 (Advanced) AWS - P3 (Advanced) SAS - P1 (Foundational) Key Responsibilities Technical Delivery Design, develop, and fix complex PySpark code for ETL/ELT and data-mart workloads. Convert and refactor SAS code into PySpark using SAS2PY tooling and manual optimisation. Build production-ready PySpark solutions that are scalable, maintainable, and reliable. Modernise and stabilise Legacy data workflows into cloud-native architectures. Ensure accuracy, quality, and reliability across data transformation processes. Cloud & Data Engineering (AWS-Focused) Build and deploy data pipelines using AWS services such as EMR, Glue, S3, Athena . Optimise Spark workloads for performance, partitioning,...
Location:
London
Job Type:
FullTime
Category:
Engineering

We found some similar jobs based on your search