Data Engineer

We are seeking a skilled Data Engineer to join our team and take ownership of designing, maintaining, and improving our data delivery processes. This role is pivotal to organizing and unlocking internal and vendor datasets, ensuring reliable, accurate, and fast data pipelines to meet our business needs.You will work alongside an experienced team, including cross-training existing team members for awareness and troubleshooting capabilities, while primarily being responsible for delivering efficient and robust ETL solutions.Key Responsibilities:ETL raw data (primarily CSV and JSON formats) into database tables (MySQL and PostgreSQL) using Airflow, Python, Pandas, Polars, and Linux CLI utilities.Create, maintain, and optimize database tables for performance, leveraging indexes, basic partitioning, and other techniques.Design, build, and maintain reliable, accurate, and high-performance data pipelines.Write and maintain complex SQL queries for data analysis and transformation.Work with APIs, Docker, GitHub, and other tools to support data engineering tasks.Collaborate with cross-functional teams to meet business objectives related to data delivery and reporting.Optionally work with AWS services like S3 to move data in and out of the cloud ecosystem.

Proficiency in Python and Airflow (creating and managing DAGs). Experience with ETL processes and working with structured and semi-structured data formats (CSV, JSON). Strong SQL skills with the ability to write and optimize complex queries. Familiarity with database systems such as MySQL and PostgreSQL. Experience with Docker, GitHub, and APIs.