Lead design, build, and operation of enterprise-scale data platform systems using Spark and Databricks on Azure. Architect cloud-native data pipelines, create reusable data models and governed datasets, drive platform modernization (Delta Lake, metadata-driven designs), mentor engineers, lead incident root-cause analysis, and participate in architecture and governance forums to improve data quality, lineage, and observability.
102261 - Sr. Data EngineerSummary
As a Principal Data Engineer (senior), you will be a technical leader responsible for architecting, building, and operating foundational systems of the enterprise data platform. This role is critical to delivering scalable data processing pipelines, self-service analytics, governance, and advanced analytics capabilities that enable the business to move with speed, agility, and urgency.
You will partner closely with Data Architects and business stakeholders to translate architecture into working systems, modernize platform capabilities, and mentor other engineers. The role provides the opportunity to lead large-scale distributed data systems using Spark and Databricks on Azure, drive platform-wide standards, and influence engineering strategy across the organization.
Responsibilities- Collaborate with Data Architects and business partners to design and evolve enterprise data architecture and platform capabilities.
- Translate architectural strategies into technical execution and influence system design and delivery across teams.
- Serve as a domain expert in data engineering standards, best practices, and patterns across the Data Platform organization.
- Design, code, and optimize complex distributed data processing systems using Spark and Databricks.
- Develop canonical data models, semantic structures, and reusable datasets for reporting and machine learning.
- Drive platform modernization initiatives such as Delta Lake and metadata-driven designs.
- Create reusable frameworks and platform capabilities to accelerate analytics, machine learning, and governed self-service access.
- Provide technical leadership and mentorship to Staff, Senior, and mid-level data engineers to enable high-quality delivery.
- Lead root-cause analysis for major data incidents and implement long-term improvements in data quality, lineage, and observability.
- Participate in architecture reviews, technical design sessions, and platform governance forums.
- Bachelor’s or Master’s degree in Computer Science, Information Systems, or equivalent experience.
- 10+ years of experience in data engineering or a related technical field.
- Expert-level proficiency in SQL, Python, and Spark for large-scale data processing.
- Extensive experience designing and building cloud-native data pipelines, data models, and distributed data systems (Delta Lake, Spark, Unity Catalog, Jobs, Workflows).
- Proven experience with Azure and building data solutions on Azure.
- Strong experience designing, tuning, and operating distributed data processing systems at scale.
- Deep knowledge of data engineering best practices including version control, CI/CD, automated testing, DevOps/DataOps, and observability.
- Proven ability to lead cross-functional technical initiatives and influence architectural direction.
- Strong problem-solving, debugging, and analytical skills in complex, multi-system environments.
- Ability to thrive in agile, dynamic, and collaborative engineering teams.
- Experience with Databricks Unity Catalog, Delta Live Tables, or Databricks Workflows.
- Skilled in advanced data modeling (dimensional, data vault, semantic layers).
- DataOps experience including pipeline observability, monitoring, and automated quality.
- Experience with metadata management and governance platforms (Unity Catalog, Purview, Collibra, Alation).
- Experience with streaming frameworks (Kafka, Event Hubs, Kinesis) and Spark Structured Streaming.
- Experience contributing to architecture review boards, technical councils, or data governance forums.
Top Skills
Sql,Python,Spark,Databricks,Delta Lake,Unity Catalog,Databricks Jobs,Databricks Workflows,Delta Live Tables,Azure,Kafka,Event Hubs,Kinesis,Spark Structured Streaming,Purview,Collibra,Alation,Ci/Cd,Version Control,Devops,Dataops,Observability
Similar Jobs
.png){kind=avatar}
Fintech • Machine Learning • Software • Financial Services
Provide 1st to 3rd level support for users, manage servers and networks, troubleshoot issues, and maintain IT operations.
Top Skills:
Active DirectoryAdobeEmailJavaLinuxmacOSMdmO365Windows
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
The role involves assessing information security risks, implementing cyber security technologies, managing compliance, and providing training to teams.
Top Skills:
Cyber Security TechnologyInformation Security ComplianceIt Asset SecuritySecurity Solutions
Cloud • Security • Software • Cybersecurity • Automation
As a Staff Backend Engineer, you will enhance GitLab's Developer Experience by optimizing tooling, CI/CD pipelines, and testing frameworks, while collaborating with teams to identify pain points and drive improvements.
Top Skills:
GoRuby
What you need to know about the Edinburgh Tech Scene
From traditional pubs and centuries-old universities to sleek shopping malls and glass-paneled office buildings, Edinburgh's architecture reflects its unique blend of history and modernity. But the fusion of past and future isn't just visible in its buildings; it's also shaping the city's economy. Named the United Kingdom's leading technology ecosystem outside of London, Edinburgh plays host to major global companies like Apple and Adobe, as well as a growing number of innovative startups in fields like cybersecurity, finance and healthcare.
