Databricks Developer (Vancouver)

Responsibilities

• Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
• Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
• Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
• Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
• Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
• Apply best practices in data modeling and data warehousing concepts.
• Perform data validation and quality checks.
• Work with various data formats, including JSON, Parquet, and Avro.
• Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, an...