Design, build, and operate scalable batch and streaming data pipelines on the Databricks Lakehouse. Develop and optimize Spark (PySpark/Scala) jobs and SQL transformations, implement Delta Lake patterns, orchestration, CI/CD, monitoring, governance, and documentation. Partner with analysts and data scientists to deliver curated, well-governed data products.
As a Software Engineer II at JPMorgan Chase within our Corporate Investment Bank , Payments Technology team, you’ll design, build, and operate scalable data pipelines and analytics workloads on the Databricks Lakehouse platform. We’re looking for a Databricks Engineer to design, build, and operate scalable data pipelines and analytics workloads on the Databricks Lakehouse platform. You’ll partner with data analysts, data scientists, and application teams to deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products.
Job responsibilities
- Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators (e.g., Workflows, Airflow, ADF).
- Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency.
- Build and maintain curated data models (bronze/silver/gold), data quality checks, and automated testing.
- Implement CI/CD for notebooks and code (Git-based workflows), and automate deployments across environments.
- Manage and tune Databricks clusters, jobs, and configurations; monitor production workloads and resolve incidents.
- Integrate multiple data sources (cloud storage, relational DBs, APIs, event streams) and implement robust ingestion patterns.
- Apply data governance and security best practices (access controls, secrets management, lineage/metadata, auditing).
- Create clear documentation for pipelines, data contracts, and operational runbooks.
- Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards.
- Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation
Required qualifications, skills and capabilities
- Experience with building data pipelines on Databricks and/or Apache Spark in production.
- Strong coding skills in Python (PySpark) and SQL (Scala a plus).
- Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM).
- Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data Factory, etc.).
- Familiarity with cloud data platforms (AWS/Azure/GCP) and storage (S3/ADLS/GCS).
- Solid understanding of data engineering fundamentals: data modeling, ETL/ELT patterns, reliability, observability, and performance tuning.
- Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, testing, troubleshooting, or documentation) with demonstrated ability to critically evaluate and validate AI-generated outputs.
- Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations
Preferred qualifications
- Experience with streaming (Structured Streaming, Kafka/Event Hubs/Kinesis).
- Experience implementing data quality frameworks (Great Expectations, DQ) and data testing in CI.
- Exposure to Unity Catalog (or similar) for governance and fine-grained permissions.
About UsJ.P. Morgan is a global leader in financial services, providing strategic advice and products to the world’s most prominent corporations, governments, wealthy individuals and institutional investors. Our first-class business in a first-class way approach to serving clients drives everything we do. We strive to build trusted, long-term partnerships to help our clients achieve their business objectives.
We recognize that our people are our strength and the diverse talents they bring to our global workforce are directly linked to our success. We are an equal opportunity employer and place a high value on diversity and inclusion at our company. We do not discriminate on the basis of any protected attribute, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis protected under applicable law. We also make reasonable accommodations for applicants’ and employees’ religious practices and beliefs, as well as mental health or physical disability needs. Visit our FAQs for more information about requesting an accommodation.
Similar Jobs
HR Tech • Information Technology • Professional Services • Sales • Software
Own the full enterprise SaaS sales cycle in the UK, from outbound prospecting and discovery through negotiation and close. Build relationships with companies of 1,000+ employees, manage multiple stakeholders, partner with Sales Engineering on demonstrations, maintain Salesforce data, attend industry events, and collaborate cross-functionally to drive revenue growth.
Top Skills:
SaaSSalesforce
HR Tech • Information Technology • Professional Services • Sales • Software
Manage multiple complex UK payrolls end to end for outsourced-service customers. Provide expert guidance on UK payroll legislation, statutory payments, pensions, benefits in kind, share schemes, taxation, reconciliations, HMRC submissions, and year-end processing. Resolve discrepancies, advise enterprise customers, build trusted relationships, mentor colleagues, and contribute to documentation and continuous improvement.
Top Skills:
AsanaAuto EnrolmentHmrcNational InsuranceP11DPayePayroll ApisPayroll IntegrationsReal Time Information (Rti)SlackZendesk
HR Tech • Information Technology • Professional Services • Sales • Software
Own the full sales cycle for HiBob’s small mid-market segment, targeting organizations with 200–500 employees. Responsibilities include outbound prospecting, pipeline generation, relationship development with senior decision-makers, product demonstrations, commercial negotiations, forecasting, and closing SaaS deals. The role also involves cross-functional collaboration with Sales Engineering, Marketing, Customer Success, and Partnerships, along with participation in events, conferences, and partner networking.
Top Skills:
Salesforce
What you need to know about the Edinburgh Tech Scene
From traditional pubs and centuries-old universities to sleek shopping malls and glass-paneled office buildings, Edinburgh's architecture reflects its unique blend of history and modernity. But the fusion of past and future isn't just visible in its buildings; it's also shaping the city's economy. Named the United Kingdom's leading technology ecosystem outside of London, Edinburgh plays host to major global companies like Apple and Adobe, as well as a growing number of innovative startups in fields like cybersecurity, finance and healthcare.
