Pipeline Development & Maintenance

• Build and maintain data pipelines for ingesting, transforming, and cleaning healthcare data in the Azure cloud using Databricks, PySpark, and Delta Lake

• Implement pipeline logic from defined specifications, with guidance from senior engineers on architectural decisions

• Build reusable testing frameworks to ensure pipeline reliability

• Monitor pipelines for failures and performance issues, escalating complex problems appropriately

Orchestration & Automation

• Develop and maintain Airflow DAGs with appropriate error handling and retry logic

• Support deployment and configuration of pipelines via Astronomer on Azure using the Astro CLI

• Contribute to improving pipeline reliability and reducing manual intervention

Data Modeling & Storage

• Implement data models for efficient storage and retrieval using Delta Lake, including merge/upsert patterns

• Design and optimize Delta Lake tables for analytic workloads

• Leverage Unity Catalog for data governance, organization, and security

• Apply Change Data Capture (CDC) patterns under the direction of senior team members

• Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and ETL support

Azure Infrastructure

• Work within the Azure ecosystem including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps

• Follow established security and scalability standards when building data infrastructure

• Support infrastructure tasks with guidance from senior engineers on architecture

Data Quality & Monitoring

• Ensure scripts and datasets are well-documented to support enterprise data governance

• Design, implement and continuously improve automated data quality monitoring, reconciliation, and alerting to ensure reliable downstream reporting

• Troubleshoot and resolve pipeline failures, documenting root causes and resolutions

Collaboration & Documentation

• Work closely with data scientists, analysts, and business stakeholders to understand reporting requirements, troubleshoot issues, and improve data assets

• Participate in code reviews, incorporating feedback to improve code quality

• Document pipelines, processes, and implementation decisions clearly and consistently

• Stay current with data engineering technologies, particularly in the healthcare space

• 2–4 years of experience as a Data Engineer or in a closely related data role

• Hands-on experience with Azure cloud services (ADLS, Databricks, or similar)

• Working knowledge of SQL for scripting and data modeling including T-SQL, Spark, and Databricks SQL

• Ability to contribute to technical projects with moderate oversight

• Strong communication skills — comfortable asking questions and providing updates to cross-functional partners

• Familiarity with CI/CD concepts, Git, and version control workflows

• Solid problem-solving skills with a systematic approach to debugging

• Familiarity with healthcare data standards and regulations (HIPAA, HL7, etc.) is a plus

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience)

• Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts.

• Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience.

• Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations.

• Airflow Python Development (Mid): Ability to write and maintain DAGs; understanding of error handling and retry patterns.

• Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage.

• Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps.

• T-SQL (Foundational–Mid): Basic stored procedures, SQL Server querying, and data manipulation.

NICE TO HAVES

• Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers.

• Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, and version control best practices as well as deploying production data pipelines using CI/CD workflows.