Senior Data Engineer (AWS Databricks)
Share
Job Location:
Monthly Salary:
Posted on:
5 hours ago
Vacancies:
1 Vacancy
Job Summary
We are seeking a Data Engineer to join our team focused on data platform integration and pipeline engineering on Databricks. This role will work closely with AI Engineers to enable model development and deployment by building secure reliable and scalable data pipelines and integrations.
The role is not focused on data transformation or analytics modelling. Instead it concentrates on ingestion orchestration connectivity and operational pipelines that support AI and advanced analytics workloads within Databricks environments deployed into client-managed accounts.
Experience Required: 5 yrs
Key Responsibilities
Databricks-Centric Data Engineering
- Build and maintain data pipelines that ingest data into Databricks on AWS.
- Configure and manage Databricks jobs and workflows to support AI workloads.
- Integrate Databricks with upstream source systems and downstream AI services.
- Ensure data is accessible and performant for AI training and inference use cases.
Pipeline Engineering (Non-Transformational)
- Design and implement pipelines focused on:
- Data ingestion
- Data movement
- Orchestration and scheduling
- Develop pipelines using Python and Databricks-native tooling (e.g. Databricks Jobs workflows).
- Ensure pipelines are production-ready with monitoring logging and alerting.
AWS Environment Integration
- Work within client-owned AWS environments collaborating with DevOps engineers on infrastructure provisioning.
- Integrate Databricks pipelines with cloud services such as:
- Ensure pipelines align with client security and governance requirements.
Security Governance & Compliance
- Build and operate pipelines that meet SOC 1 compliance requirements including:
- Access controls and permissions
- Audit logging and traceability
- Controlled deployment processes
- Support data governance standards within Databricks environments.
Delivery & Operations
- Deploy pipeline code via GitHub-based CI/CD pipelines.
- Support operational monitoring and incident response for data pipelines.
- Document pipeline designs dependencies and operational processes.
Qualifications :
Core Technical Skills
- Strong experience as a Data Engineer with a focus on Databricks-based platforms.
- Hands-on experience with Databricks on AWS including:
- Jobs and workflows
- Cluster configuration (user-level understanding)
- Proficiency in Python for pipeline development.
- Experience using GitHub and CI/CD workflows.
Engineering & Delivery
- Experience building production-grade ingestion and orchestration pipelines.
- Ability to work effectively in client-facing delivery environments.
- Strong documentation and collaboration skills.
Nice to Have
- Experience with Databricks Unity Catalog.
- Exposure to event-driven or streaming data architectures.
- Familiarity with MLOps concepts and AI lifecycle support.
Remote Work :
No
Employment Type :
Full-time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Blend360 is an award-winning provider of data, analytics, and talent solutions for Fortune 500 companies. The company has made the Inc. 5000 list of Fastest Growing Companies every year they have been in business and has been awarded a world-class ranking in client satisfaction for th ... View more
