Empresa:

Inetum

Descrição da Função

We are looking for a Data Engineer to join our Inetum Team and be part of a work culture focused on innovation!

About the Role

The Data Engineer is responsible for building and maintaining the infrastructure that supports the organization's data architecture. The role involves creating and managing data pipelines using Airflow for data extraction, processing, and loading, ensuring their maintenance, monitoring, and stability.

The engineer will work closely with data analysts and end-users to provide accessible and reliable data.

Technical Skills:

• Python

• Experience in creating APIs in Python

• PySpark

• Spark Environment Architecture

• SQL, Oracle Data Base

• Experience in creating and maintaining distributed environments using Hadoop and Spark

• Hadoop ecosystem - HDFS + Yarn

• Containerization - Docker is Mandatory

• Data Lakes - Experience in organizing and maintaining data lakes - S3 is preferred

• Experience with Parquet file format

• Apache Airflow - Experience in both pipeline development and deploying Airflow in distributed environment

• Apache Kakfa

• Experience in automating applications deployment using DevOps tools - Jenkins is Mandatory, Ansible is a plus

• Candidate must be able to use Unix, must know how to use Unix commands to check processes, to read files, processes and run bash commands.

Main Tasks:

• Responsible for maintaining the infrastructure that supports the current data architecture

• Responsible for creating data pipelines in Airflow for data extracting, processing and loading

• Responsible for data pipelines maintenance, monitoring and stability

• Responsible for providing data access to data analysts and end-users

• Responsible for DevOps infrastructure

• Responsible for deploying Airflow dags to production environment using DevOps tools

• Responsible for code and query optimization

• Responsible for data pipelines maintenance, monitoring and stability

• Responsible for code review

• Responsible for improving the current data architecture and DevOps processes

• Responsible for delivering data in useful and appealing ways to users

• Responsible for performing and documenting analysis, review and study on specified regulatory topics

• Responsible for understanding business change and requirement needs, assess the impact and the cost.

Language Skills

• English

Send your CV & join our team!

Observações

Lisboa (Portugal)