Data Engineer

Title: Data Engineer - DatabricksLocation: RemoteDuration: Full TimeJob Description:

• Build scalable PySpark/Spark SQL pipelines in Databricks to prepare curated, feature ready datasets for model development and scoring.
• Develop and manage data ingestion and harmonization pipelines.
• Implement transformations, validations, and schema alignment based on model requirements.
• Optimize pipeline performance, scheduling, and orchestration (Jobs, Workflows, Delta Live Tables).
• Collaborate with Solution Architect, data architects, and scientists to ensure accurate feature engineering pipelines.
• Integrate and harmonize data from Snowflake into Databricks, ensuring schema alignment and data consistency.
• Prepare curated, feature-ready datasets that meet Client’s feature definitions and modeling requirements.
• Monitor pipeline health, data freshness, and quality, escalating issues early.
• Follow best practices for versioning, CI/CD, and operational MLOps workflows.
• Work closely with Data Scientists to translate feature logic into efficient, reusable pipelines.
• Implement orchestration and scheduling using Databricks Jobs/Workflows or Delta Live Tables.
• Monitor, optimize, and troubleshoot pipelines for performance, data quality, and SLAs.
• Ensure secure access patterns and compliance with customer data governance requirements.
• Support Dev/QA/Prod deployment of feature pipelines.

Qualifications:

• 7+ years of hands-on experience in data engineering with cloud data platforms.
• Strong experience with Databricks, PySpark, Spark SQL, and Delta Lake.
• Proficiency with Snowflake and cross-platform data integration.