Design, build, and maintain scalable Python-based microservices and backend APIs. Develop ETL and data transformation pipelines, manage relational and NoSQL schemas, optimize performance, implement async patterns, ensure API security, collaborate across teams, and troubleshoot production data platform issues.
We are looking for an experienced Senior Python Data Engineer with strong expertise in Python development, microservices architecture, data engineering, and backend API development. The ideal candidate will have hands-on experience building scalable Python-based applications and data services, designing robust REST APIs, performing complex data transformations, and developing high-performance backend frameworks to support enterprise data-driven applications.
This role requires strong proficiency in Python web frameworks, asynchronous programming, microservices architecture, ETL processes, database management, and modern software engineering best practices. Exposure to big data technologies, DevOps practices, and cloud-native architectures will be highly advantageous.
RequirementsKey Responsibilities
- Design, develop, and maintain scalable Python-based web frameworks and backend services.
- Develop robust RESTful APIs using Python frameworks such as FastAPI, Flask, or Django.
- Build and maintain microservices-based applications supporting enterprise data platforms.
- Design and implement data transformation pipelines for data ingestion, processing, and enrichment.
- Develop backend systems to efficiently serve datasets through APIs and other interfaces.
- Collaborate with cross-functional teams to deliver scalable and secure data-driven applications.
- Design and maintain relational and NoSQL database schemas.
- Develop efficient ETL (Extract, Transform, Load) processes for enterprise data requirements.
- Optimize application and database performance to ensure scalability and reliability.
- Implement asynchronous programming techniques using asyncio and optimize concurrent connections and I/O operations.
- Ensure data integrity, consistency, and quality across data transformation pipelines.
- Participate in code reviews and contribute to software engineering best practices.
- Troubleshoot, debug, and resolve production issues across applications and data pipelines.
- Collaborate closely with stakeholders and engineering teams throughout the SDLC.
- Strong hands-on expertise in Core Python.
- Experience building enterprise-grade Python applications.
- Strong understanding of:
- Python programming concepts
- Object-oriented programming
- Asynchronous programming (asyncio)
- Concurrent connections handling
- I/O optimization techniques
- Clean code and software design principles
Hands-on experience with one or more of the following:
- FastAPI
- Flask
- Django
- Pyramid
Strong understanding of:
- RESTful API development
- Routing
- Authentication mechanisms
- Testing frameworks
- Database integrations
- Framework architecture and scalability
Strong experience with:
- Microservices-based application design
- Service-oriented architectures
- API integrations
- Distributed systems
- Enterprise backend services
Knowledge of:
- API Gateways
- Service communication patterns
- Scalable backend architectures
Experience implementing:
- OAuth
- JWT Authentication
- API Security Best Practices
- Authorization mechanisms
- Encryption standards
- Secure API development practices
Strong hands-on experience with:
- Data transformation pipelines
- ETL processes
- Data cleansing and enrichment
- Data aggregation techniques
- Data quality and integrity management
Experience in:
- Dataset preparation
- Data processing workflows
- Enterprise data platforms
Experience working with:
Relational Databases- PostgreSQL
- MySQL
- SQL Server (preferred)
- MongoDB
- Redis (good to have)
Strong knowledge of:
- Database schema design
- Performance optimization
- Transactions management
- Efficient data retrieval techniques
Hands-on experience with:
- SQLAlchemy
- Django ORM
Ability to perform:
- CRUD operations
- Query optimization
- Transaction handling
- Data modelling
Experience with:
- CI/CD pipeline development and maintenance
- Git-based workflows
- Version control best practices
- Branching strategies
- Code reviews
- Production deployments
Exposure to:
- Apache Spark
- Hadoop
- Apache Kafka
Understanding of:
- Data warehousing concepts
- Large-scale data processing architectures
Strong proficiency in:
- SQL
- Shell Scripting
- Python scripting
Exposure to:
- AWS
- Microsoft Azure
- Google Cloud Platform (GCP)
- Docker
- Kubernetes
Knowledge of:
- Data governance principles
- Data security best practices
- Compliance frameworks
- Enterprise data management practices
- Experience with R Programming Language.
- Experience working on enterprise-scale data engineering initiatives.
- Exposure to cloud-native architectures and containerized deployments.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management capabilities.
- Strong debugging and troubleshooting skills.
- Ability to work effectively in Agile and collaborative environments.
- Strong ownership mindset and attention to detail.
- Ability to adapt quickly to evolving business and technical requirements.
- Strong documentation and technical communication skills.
Similar Jobs
Fintech • Payments • Software • Financial Services • Automation
Design, build, and maintain Python-based microservices and high-performance REST APIs. Develop ETL/data transformation pipelines, model relational and NoSQL schemas, implement ORMs, enforce security (OAuth/JWT), and productionize services with CI/CD, testing, and observability in an enterprise data platform.
Top Skills:
Api GatewayAsyncioCi/CdDjango OrmETLFastapiFlaskGitJwtMicroservicesNoSQLOauthPythonRestful ApisShell ScriptingSQLSqlalchemy
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Drive new business across North/West Africa from UAE: manage full sales cycle, build pipeline with SDRs/SEs, sell CrowdStrike solutions to C‑level, handle renewals and cross/up‑sell, and exceed quota while maintaining competitive product knowledge.
Top Skills:
Cloud-Native Security PlatformsCrowdstrikeEndpoint Detection And Response (Edr)SaaS
Cloud • Information Technology • Security • Software • Cybersecurity
The Account Executive will drive sales growth by managing customer relationships, leading complex sales cycles, and articulating Cloudflare's value proposition, focusing on new business acquisition in the Digital Natives sector.
Top Skills:
Cloud NetworkingEdge ComputingGoogle SuiteMsft SuiteSalesforceSecurityTableau
What you need to know about the Edinburgh Tech Scene
From traditional pubs and centuries-old universities to sleek shopping malls and glass-paneled office buildings, Edinburgh's architecture reflects its unique blend of history and modernity. But the fusion of past and future isn't just visible in its buildings; it's also shaping the city's economy. Named the United Kingdom's leading technology ecosystem outside of London, Edinburgh plays host to major global companies like Apple and Adobe, as well as a growing number of innovative startups in fields like cybersecurity, finance and healthcare.
