Associate Manager, D&T, Data Engineering

**Job Title: Associate Manager, D&T, Data Engineering**

**Location- Hyderabad**

**Job Content:**

Design and maintain scalable batch and real-time data pipelines on Databricks using PySpark, Structured Streaming, DBT, and Delta Live Tables. Manage cloud infrastructure with Terraform on AWS and deploy workflows using Databricks Asset Bundles with CI/CD best practices. Optimize Spark performance and costs while implementing data governance through Unity Catalog. Collaborate with Data Scientists to produce ML models using MLflow and ensure data quality through automated testing and robust deployment pipelines.

**Your key responsibilities:**

+ Design, build, and maintain scalable batch and real-time data pipelines using PySpark and Structured Streaming on Databricks, including migration of legacy ETL processes to Delta Live Tables (DLT).
+ Develop and manage robust transformation layers using DBT to deliver clean, tested, and well-documented d...