Data Engineer

Chicago, IL - USA

Monthly Salary: $ 129000 - 144000

Posted on: 30+ days ago

Vacancies: 1 Vacancy

Job Summary

Summary

The Data Engineer will help build and maintain the data pipelines models and infrastructure that power analytics business-intelligence and machine-learning initiatives across the company. Youll work alongside senior engineers and cross-functional partners to turn raw data into reliable well-documented datasets that drive informed decisions and innovative products. This role reports to the Senior Manager Data Engineering.

Duties and Responsibilities

Build E2E Azure Databricks-based data solutions.
Design develop and maintain scalable ETL and streaming data pipelines on Azure Databricks leveraging Apache Spark Delta Lake and Azure Data Lake Storage (ADLS Gen2) to enable reliable lakehouse architectures and ensure efficient ingestion transformation and storage of data
Build and optimize data models and schemas for analytics reporting and operational data stores
Build and optimize Delta Lake / Lakehouse patterns (Bronze/Silver/Gold) including schema evolution and time travel
Develop high-quality PySpark / Spark SQL transformations optimize joins partitioning caching and shuffle behavior.
Implement and maintain data quality frameworks including data validation monitoring and alerting mechanisms.
Collaborate closely with data architects data analysts BI engineers and product teams to align data engineering activities with business goals.
Contribute to CI/CD pipelines supported by version control linting automated testing security scanning and monitoring
Troubleshoot and resolve complex Azure Databricks platform data infrastructure and pipeline issues ensuring minimal downtime and optimal performance.
Follow coding standards participate in code reviews and document your work to foster knowledge sharing.

Salaries vary by location and are based on numerous factors including but not limited to the relevant market skills experience and education of the selected candidate. If an estimated salary range for this role is available it will be provided in our Target Salary Range section. Our compensation package also includes bonus eligibility and a comprehensive benefits program. Benefits information can be found at Salary Range

$129000 - $144000 if located in Illinois

Qualifications

To perform this job successfully an individual must be able to perform the Duties and Responsibilities (Duties) above satisfactorily and meet the requirements below. The requirements listed below are representative of the minimum knowledge skill and/or ability required. Reasonable accommodations will be made to enable individuals with disabilities to perform the essential functions of the job. If you need such an accommodation please email (current employees should contact Human Resources).

Education and/or Experience:

Required:

Bachelors degree in Computer Science Engineering Data Science or a related field
A minimum of 3 years of hands-on experience designing building and operating data solutions.
Knowledge of Databricks architecture and core components including Databricks Lakehouse Delta Lake Databricks SQL Apache Spark Clusters Unity Catalog Databricks Workflows (Jobs) and Databricks Notebooks
Proficiency with Python SQL and Apache Spark for data processing
Proven experience building reusable metadata-driven data ingestion frameworks using Python and Scala
Familiarity with cloud data-platform components such as object storage metadata / data catalog services batch/streaming/CDC ingestion & processing
Proven experience with data modeling schema design and performance tuning of large-scale data systems
Experience working with AI & BI engineers to deliver high-quality data products to stakeholders
Skilled at crafting compelling data narratives through tables reports dashboards and other visualization tools
Understanding of data engineering best practices: code repositories CI/CD pipelines test automation monitoring and alerting systems
Strong problem-solving and analytical skills with excellent attention to detail
Excellent communication skills and experience collaborating with technical and business stakeholders

Preferred:

Experience building data pipelines in an Azure Databricks environment
Hands-on experience integrating Azure Databricks with Azure DevOps Azure Blob Storage / ADLS Gen2 Azure Key Vault and Azure Data Factory
Familiarity with enterprise data modeling tools such as ERwin Data Modeler including the ability to interpret and apply logical and physical data models to analytical and lakehouse architectures
Familiarity with Infrastructure as Code (IAC)
Experience working with regulated/sensitive data and controls
Experience working in an Agile delivery model

Other Skills and Abilities:

The following will also be required of the successful candidate:

Strong organizational skills
Strong attention to detail
Good judgment
Strong interpersonal communication skills
Strong analytical and problem-solving skills
Able to work harmoniously and effectively with others
Able to preserve confidentiality and exercise discretion
Able to work under pressure
Able to manage multiple projects with competing deadlines and priorities

Sidley Austin LLP is an Equal Opportunity Employer

#LI-Hybrid

#LI-OE1

Required Experience:

SummaryThe Data Engineer will help build and maintain the data pipelines models and infrastructure that power analytics business-intelligence and machine-learning initiatives across the company. Youll work alongside senior engineers and cross-functional partners to turn raw data into reliable well-d...

Summary

Duties and Responsibilities

Build E2E Azure Databricks-based data solutions.
Design develop and maintain scalable ETL and streaming data pipelines on Azure Databricks leveraging Apache Spark Delta Lake and Azure Data Lake Storage (ADLS Gen2) to enable reliable lakehouse architectures and ensure efficient ingestion transformation and storage of data
Build and optimize data models and schemas for analytics reporting and operational data stores
Build and optimize Delta Lake / Lakehouse patterns (Bronze/Silver/Gold) including schema evolution and time travel
Develop high-quality PySpark / Spark SQL transformations optimize joins partitioning caching and shuffle behavior.
Implement and maintain data quality frameworks including data validation monitoring and alerting mechanisms.
Collaborate closely with data architects data analysts BI engineers and product teams to align data engineering activities with business goals.
Contribute to CI/CD pipelines supported by version control linting automated testing security scanning and monitoring
Troubleshoot and resolve complex Azure Databricks platform data infrastructure and pipeline issues ensuring minimal downtime and optimal performance.
Follow coding standards participate in code reviews and document your work to foster knowledge sharing.

$129000 - $144000 if located in Illinois

Qualifications

Education and/or Experience:

Required:

Bachelors degree in Computer Science Engineering Data Science or a related field
A minimum of 3 years of hands-on experience designing building and operating data solutions.
Knowledge of Databricks architecture and core components including Databricks Lakehouse Delta Lake Databricks SQL Apache Spark Clusters Unity Catalog Databricks Workflows (Jobs) and Databricks Notebooks
Proficiency with Python SQL and Apache Spark for data processing
Proven experience building reusable metadata-driven data ingestion frameworks using Python and Scala
Familiarity with cloud data-platform components such as object storage metadata / data catalog services batch/streaming/CDC ingestion & processing
Proven experience with data modeling schema design and performance tuning of large-scale data systems
Experience working with AI & BI engineers to deliver high-quality data products to stakeholders
Skilled at crafting compelling data narratives through tables reports dashboards and other visualization tools
Understanding of data engineering best practices: code repositories CI/CD pipelines test automation monitoring and alerting systems
Strong problem-solving and analytical skills with excellent attention to detail
Excellent communication skills and experience collaborating with technical and business stakeholders

Preferred:

Experience building data pipelines in an Azure Databricks environment
Hands-on experience integrating Azure Databricks with Azure DevOps Azure Blob Storage / ADLS Gen2 Azure Key Vault and Azure Data Factory
Familiarity with enterprise data modeling tools such as ERwin Data Modeler including the ability to interpret and apply logical and physical data models to analytical and lakehouse architectures
Familiarity with Infrastructure as Code (IAC)
Experience working with regulated/sensitive data and controls
Experience working in an Agile delivery model

Other Skills and Abilities:

The following will also be required of the successful candidate:

Strong organizational skills
Strong attention to detail
Good judgment
Strong interpersonal communication skills
Strong analytical and problem-solving skills
Able to work harmoniously and effectively with others
Able to preserve confidentiality and exercise discretion
Able to work under pressure
Able to manage multiple projects with competing deadlines and priorities

Sidley Austin LLP is an Equal Opportunity Employer

#LI-Hybrid

#LI-OE1

Required Experience:

Key Skills

Apache Hive
S3
Hadoop
Redshift
Spark
AWS
Apache Pig
NoSQL
Big Data
Data Warehouse
Kafka
Scala

Apply Now

About Company

Sidley Austin LLP

Sidley is a global law firm, collaborating across disciplines and borders to help clients in more than 70 countries achieve business objectives.

View Profile View Profile

AI AutoApply

Apply to 100+ jobs with one click

AI Resume Builder

Create an ATS-ready CV in minutes

AI Cover Letter

Write a personalized letter instantly

Data Engineer

Chicago, IL - USA

Job Summary

Summary

Duties and Responsibilities

Summary

Duties and Responsibilities

Key Skills

About Company

Related Jobs