Senior Data Engineer (Azure Data Factory/Databricks/PySpark)

Apply now Refer a friend

Job Description

We’re looking for a Senior Data Engineer with strong hands-on expertise in building scalable data pipelines and cloud-native data solutions on Azure. This role focuses on designing and implementing real production systems using Azure Data Factory, Azure Databricks, and modern big data technologies. You will work across distributed data platforms, integrating cloud and on-premises environments, and delivering robust, enterprise-grade data solutions aligned with industry best practices.

Qualifications

Skills & Experience

Strong experience building data pipelines with Azure Data Factory and Azure Databricks
Hands-on expertise in PySpark for distributed data processing
Advanced knowledge of SQL, including complex joins and performance optimization
Experience working with Cosmos DB and Snowflake
Solid programming skills in Python, Scala, or Java
Experience with Kafka or event-driven architectures
Strong understanding of data architecture and distributed systems
Proven ability to deliver production-grade data solutions in complex environments

Technical Requirements

Candidates must demonstrate solid, hands-on knowledge across the following three areas:

1. PySpark (Production-Level Coding)

DataFrame transformations: select, filter, groupBy, agg, withColumn
Window functions: rank, row_number, lag, lead
Joins: inner, left, broadcast joins and usage scenarios
Reading/writing data: Parquet, Delta, CSV
UDFs: syntax, registration, and performance tradeoffs
Ability to produce production-ready code without pseudo-code

2. Azure Databricks (Architecture & Platform Expertise)

Cluster types, auto-scaling, and cluster policies
Notebook orchestration, workflows, and job scheduling
Delta Lake: ACID, schema evolution, time travel, OPTIMIZE, VACUUM
Unity Catalog: governance, lineage, and access control
Integration with Azure Data Factory

3. Hybrid Data Architecture (On-Prem to Cloud Integration)

JDBC/ODBC connectivity from Databricks
Secure credential management with Azure Key Vault and secret scopes
Network architecture: VNet injection, private endpoints, Self-Hosted IR
End-to-end pipeline design from on-prem to cloud
Performance optimization and error handling in hybrid environments

Set Yourself Apart With

Experience designing end-to-end data architectures in Azure ecosystems
Knowledge of Delta Lake features (ACID transactions, schema evolution, time travel)
Experience with Databricks Workflows and orchestration
Understanding of data governance (Unity Catalog)
Experience with hybrid cloud/on-prem integration patterns
Exposure to performance optimization at scale

Additional Information

An inclusive workplace that promotes diversity and collaboration.
Access to ongoing learning and development opportunities.
Competitive compensation and benefits package.
Flexibility to support work-life balance.
Comprehensive health benefits for you and your family.
Generous paid leave and holidays.
Wellness program and employee assistance.

As part of our dedication to an inclusive and diverse workforce, Publicis Sapient is committed to Equal Employment Opportunity without regard for race, color, national origin, ethnicity, gender, protected veteran status, disability, sexual orientation, gender identity, or religion. We are also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, you may contact us at hiring@publicis.sapient.com

Company Description

Publicis Sapient is a digital transformation partner helping established organizations get to their future, digitally enabled state, both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods, fusing strategy, consulting, and customer experience with agile engineering and problem-solving creativity. United by our core values and our purpose of helping people thrive in the brave pursuit of next, our 20,000+ people in 53 offices around the world combine experience across technology, data sciences, consulting, and customer obsession to accelerate our clients’ businesses through designing the products and services their customers truly value.