As a Senior Data Engineer, you'll design and implement ETL pipelines, develop data processing solutions, support analytics, and ensure data quality.
PP - Data Engineer B. - Job0688Summary
We are seeking a senior Data Engineer to join the Credit Platform Data team to design, build, and maintain robust data pipelines and ETL processes. This role is critical to ingesting, transforming, and loading data from diverse sources into the data warehouse, enabling analytics and product decisions.
You will work with product managers, analysts, and stakeholders to translate data requirements into scalable, reliable solutions, drive automated data quality checks, and improve system performance. The role offers hands-on engineering with Python, SQL, PySpark and data modeling to support a growing credit platform.
Responsibilities- Design and implement scalable ETL pipelines to ingest and transform data for the data warehouse.
- Develop and maintain Python-based data processing using PySpark and Pandas.
- Write efficient SQL for data extraction, transformation, and validation.
- Build and maintain data models and schemas to support analytics and reporting.
- Implement automated data quality checks and monitoring for pipeline reliability.
- Troubleshoot, debug, and resolve data issues across ingestion and transformation stages.
- Collaborate with product managers, analysts, and stakeholders to gather and refine data requirements.
- Participate in design and code reviews to ensure best practices and performance.
- Maintain documentation for data flows, schemas, and operational procedures.
- Automate repeatable tasks and testing related to ETL and data pipelines using shell scripting and Unix tools.
- Bachelor's degree in Computer Science, Engineering, or a related field.
- 3+ years of proven experience as a Data Engineer or similar role, with a strong background in database development, ETL processes, and software development.
- Proficiency in SQL and scripting languages such as Python, with experience working with relational databases.
- Proficiency in PySpark, Pandas or other data processing libraries.
- Familiarity with data warehousing concepts and tools, such as AWS Redshift, Google BigQuery, or Snowflake, and experience optimizing performance for large-scale data processing.
- Experience with data modeling, schema design, and optimization techniques for scalability.
- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize data processing pipelines for scale.
- Experience with Automation test cases is a must too.
- Experience with Unix/Linux operating systems and shell scripting.
- Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
- Self-motivated and proactive, with a passion for continuous learning and professional development.
- ETL strong experience, including a deep understanding of data extraction, transformation,and loading processes (must-have requirement).
- Proficiency in Python programming language for data processing tasks .
- Strong knowledge and experience in data modeling to design efficient and scalable database structures ..
- Proficiency in PySpark, Pandas, or other data processing libraries commonly used in Python
- Knowledge on development and engineering and software lifecycle is required
Top Skills
Aws Redshift
Google Bigquery
Pandas
Pyspark
Python
Snowflake
SQL
Unix/Linux
Similar Jobs
Information Technology • Business Intelligence • Consulting
The senior SDET will design automation frameworks for mobile and backend testing, ensure high-quality releases, and integrate automated tests into CI/CD workflows.
Top Skills:
AppiumCypressJunitKotlinPostmanSelenium
Information Technology • Business Intelligence • Consulting
Design and develop scalable cloud-native backend services using .NET Core, C#, AWS, and Apache Kafka. Collaborate with teams and mentor junior engineers.
Top Skills:
.Net CoreAngularApache KafkaAWSC#JavaScriptKubernetesPythonTensorFlowTerraform
Information Technology • Business Intelligence • Consulting
The Senior Android Developer will lead the development of Android applications using Kotlin, ensuring high-quality coding and agile collaboration with design and product teams.
Top Skills:
Android SdkKotlin
What you need to know about the Edinburgh Tech Scene
From traditional pubs and centuries-old universities to sleek shopping malls and glass-paneled office buildings, Edinburgh's architecture reflects its unique blend of history and modernity. But the fusion of past and future isn't just visible in its buildings; it's also shaping the city's economy. Named the United Kingdom's leading technology ecosystem outside of London, Edinburgh plays host to major global companies like Apple and Adobe, as well as a growing number of innovative startups in fields like cybersecurity, finance and healthcare.
