Kafka Data Architect(Streaming And Payment) in Greater London
New Yesterday
Energy Jobline is the largest and fastest growing global Energy Job Board and Energy Hub. We have an audience reach of over 7 million energy professionals, 400,000+ monthly advertised global energy and engineering jobs, and work with the leading energy companies worldwide.
We focus on the Oil & Gas, Renewables, Engineering, Power, and Nuclear markets as well as emerging technologies in EV, Battery, and Fusion. We are committed to ensuring that we offer the most exciting career opportunities from around the world for our jobseekers.
Job Description
We are seeking a Hands-On Data Architect to design, build, and operate a high-scale, event-driven data platform supporting payment and channel operations. This role combines strong data architecture fundamentals, deep streaming expertise, and hands-on engineering in a regulated, high-throughput environment.
You will lead the evolution from legacy data ingestion patterns to a modern AWS-based lakehouse and streaming architecture, handling tens of millions of events per day, while applying domain-driven design (DDD) and data-as-a-product principles.
This is a builder role, not a documentation-only architect position.
Key Responsibilities
Data Products & Architecture
Design and deliver core data products including:
Channel Operations Warehouse (high-performance, ~30 days retention)
Channel Analytics Lake (long-term retention, 7+ years)
Define and expose data APIs and status/statement services with clear SLAs.
Architect an AWS lakehouse using S3, Glue, Athena, Iceberg, with Redshift for BI and operational analytics.
Enable dashboards and reporting using Amazon QuickSight (or equivalent BI tools).
Streaming & Event-Driven Architecture
Design and implement real-time streaming pipelines using:
Kafka (Confluent or AWS MSK)
AWS Kinesis / Kinesis Firehose
EventBridge for AWS- event routing
Define patterns for:
Ordering, replay, retention, and idempotency
At-least-once and exactly-once processing
Dead-letter queues (DLQs) and failure recovery
Implement CDC pipelines from Aurora PostgreSQL into Kafka and the lakehouse.
Event Contracts & Schema Management
Define and govern event contracts using Avro or Protobuf.
Manage schema evolution through Schema Registry, including:
Compatibility rules
Versioning strategies
Backward and forward compatibility
Align domain events with Kafka topics and analytical storage models.
Migration & Modernization
Assess existing “as-is” ingestion mechanisms (APIs, files, SWIFT feeds, Kafka, relational stores).
Design and execute migration waves, cutover strategies, and rollback runbooks.
Ensure minimal disruption during platform transitions.
Governance, Quality & Security
Apply data-as-a-product and data mesh principles:
Clear ownership
Quality SLAs
Access controls
Retention and lineage
Implement security best practices:
Data classification
KMS-based encryption
Tokenization where required
Least-privilege IAM
Immutable audit logging
Observability, Reliability & FinOps
Build observability for streaming and data platforms using:
CloudWatch, Prometheus, Grafana
Track operational KPIs:
Throughput (TPS)
Processing lag
Success/error rates
Cost per million events
Define actionable alerts, dashboards, and operational runbooks.
Design for high availability with multi-AZ / multi-region patterns, meeting defined RPO/RTO targets.
Hands-On Engineering
Write and review production-grade code using:
Python, Scala, SQL
Spark / AWS Glue
AWS Lambda & Step Functions
Build infrastructure using Terraform (IaC).
Implement CI/CD pipelines (GitLab, Jenkins).
Enforce automated testing, performance profiling, and secure coding practices.
Required Skills & Experience
Streaming & Event-Driven Systems
Strong experience with Kafka (Confluent) and/or AWS MSK
Experience with AWS Kinesis / Firehose
Deep understanding of:
Event ordering and replay
Delivery semantics
Outbox and CDC patterns
Practical experience using EventBridge for event routing and filtering
AWS Data Platform
Hands-on experience with:
S3, Glue, Athena
Redshift
Step Functions and Lambda
Familiarity with Iceberg-based lakehouse architectures
Experience building streaming pipelines into S3 and Glue
Payments & Financial Messaging
Experience with payments data and flows
Knowledge of ISO 20022 messages:
PAIN, PACS, CAMT
Understanding of payment lifecycle, reconciliation, and statements
Exposure to API, file-based, and SWIFT-based integration channels
Data Architecture Fundamentals (Must-Have)
Logical data modeling (ER diagrams, normalization up to 3NF/BCNF)
Physical data modeling:
Partitioning strategies
Indexing
SCD types
Strong understanding of:
Transactional vs analytical schemas
Star schema, Data Vault, and 3NF trade-offs
Practical experience with:
CQRS and event sourcing
Event-driven architecture
Domain-driven design (bounded contexts, aggregates, domain events)
If you are interested in applying for this job please press the Apply Button and follow the application process. Energy Jobline wishes you the very best of luck in your next career move.
- Location:
- Greater London
- Job Type:
- FullTime
- Category:
- Data, Architect