Data Engineer

Design & Develop Scalable Data Pipelines: Architect and build efficient data ingestion pipelines that empower the business with timely and actionable insights to drive strategy and operations.

Collaborate with Cross-Functional Teams: Work closely with technical and business teams to gather requirements and translate them into technical specifications and solutions.

Optimize Data Architecture: Continuously refine and improve the performance, reliability, and scalability of data pipelines to ensure seamless integration of data across systems.

Ensure Data Integrity: Perform rigorous data quality checks and uphold best practices to maintain the accuracy, consistency, and integrity of data.

Build ETL Solutions: Develop ETL frameworks to collect, transform, and integrate data from diverse sources such as Kafka, Scylla, PostgreSQL, MongoDB, APIs, and various file formats.

Adopt Best Practices: Implement best-in-class engineering practices around reporting and analysis, ensuring data integrity, testing, validation, and comprehensive documentation.

Bachelor's degree in Computer Science, Information Systems, or a related technical field, or equivalent work experience.

3+ years of hands-on experience with cloud-based data technologies, including message queues, event grids, relational databases, NoSQL databases, data warehouses, and big data technologies (e.g., Spark).

Proficiency in Spark (Java, Python, SQL): Expertise in developing and optimizing Spark-based applications for large-scale data processing.

Advanced SQL Skills: Ability to create complex queries, manage database structures, and ensure optimal performance.

Experience with Docker and Kubernetes: Familiarity with deploying applications using modern containerisation and orchestration tools.

DevOps and CI/CD: Solid understanding of modern DevOps practices, including Git, continuous integration, and continuous deployment pipelines.

Agile Development Experience: Comfortable working in an Agile environment, particularly using Scrum methodology.

API Frameworks & OOP: Experience with API frameworks and object-oriented programming to integrate services and improve data workflows.

Business Acumen: Ability to collaborate closely with leadership to deliver innovative solutions tailored to evolving business needs.

Strong Communication Skills: Excellent verbal and written communication abilities, with a knack for explaining complex technical concepts to both technical and non-technical stakeholders.

Ability to leverage tools like langchain, llama index ..etc to build llm based applications.