AWS Glue Data Engineer

Overview

DeepLight AI is a specialist AI and data consultancy with extensive experience implementing intelligent enterprise systems across multiple industries, with particular depth in financial services and banking.

Responsibilities

• Build scalable ingestion pipelines using AWS Glue, PySpark, and ETL pipeline development.
• Implement AWS Glue jobs for Bronze layer ingestion, applying defined standards and templates.
• Design and execute historical loading mechanisms to bring legacy data into the Lakehouse.
• Optimize Glue job performance (DPU allocation, parallelization, partitioning) according to best practices.
• Collaborate with platform teams to ensure tooling and optimization alignment.
• Automate migration of source tables to Bronze layer, leveraging AI-enabled acceleration.
• Version-control jobs and automate production deployment via Git and Terraform.
• Establish source system connectivity int...