Data Engineer/Lead

Get AI-powered advice on this job and more exclusive features.

• Develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake
• Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
• Developing scalable and re-usable frameworks for ingesting data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
• Working with event-based/streaming technologies to ingest and process data
• Provide technical guidance and support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
• Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best pr...