Data Engineer

Are client is looking for a skilled Data Engineer to join their dynamic team dedicated to building and maintaining our cutting-edge media data lake on Microsoft Azure. This role will focus on developing and supporting data pipelines within a medallion architecture, utilizing Spark and Scala to process and transform large volumes of media data. The successful candidate will be passionate about data, eager to learn, and contribute to a high-performing engineering team.

WHAT YOU WILL DO

• Develop, test, and deploy data ingestion, transformation, and processing pipelines using Azure services (Azure Data Factory, Azure Data Lake Storage).
• Write efficient and maintainable code in Spark and Scala for data manipulation and analysis.
• Contribute to the implementation and maintenance of the medallion architecture (Bronze, Silver, Gold layers).
• Collaborate with senior engineers, architects, and analysts to understand data requirements and implement solutions.
• Monitor data pipelines, troubleshoot issues, and implement optimizations for performance and reliability.
• Implement data quality checks and ensure data integrity across the data • Participate in code reviews and contribute to team best practices.
• Document data pipelines, processes, and technical specifications.

WHAT YOU WILL NEED

• Proficiency in Apache Spark and Scala programming.
• Experience with cloud platforms, preferably Microsoft Azure (Azure Data Factory, ADLS Gen2, Azure Synapse Analytics).
• Understanding of ETL/ELT processes and data warehousing concepts.
• Familiarity with medallion data lake architecture principles.
• Experience with SQL and database technologies.
• Experience with version control systems like Git.
• Familiarity with media or advertising data.
• Good understanding of CI/CD practices.