Empresa:

Decskill

Descrição da Função

Decskill, founded in 2014 as an IT Consulting Company, places paramount importance on its greatest asset: its people. Our main mission is to deliver value through knowledge and talent, and we achieve this by fostering a culture of excellence and investing in the development and well-being of our people. With over 600 dedicated professionals and offices in Lisbon, Porto, Madrid and Luxembourg, Decskill operates across three core areas:

• DECSKILL TALENT: We believe that our people are key to our success. Through Talent, we empower our team to embrace the digital transformation challenges of our clients. We collaborate with clients to drive innovation, ensuring project success and business growth.

• DECSKILL BOOST: Equipping our team with the latest tools and methodologies, we optimize Time-to-Market and deliver innovative solutions exceeding client expectations.

• DECSKILL CONNECT: Our team collaborates closely with clients to implement and manage IT infrastructures that generate long-term value.

At Decskill, we believe that by nurturing and empowering our people to confront the challenges of digital transformation, we create value not only for our clients but also for our entire ecosystem, fostering a digital community dedicated to growth and progress.

We are looking for a Senior Data Engineer, for a project based in Lisbon.

Responsabilities:

• Responsible for maintaining the infrastructure that supports the current data architecture
• Responsible for creating data pipelines in Airflow for data extracting, processing and loading
• Responsible for data pipelines maintenance, monitoring and stability
• Responsible for providing data access to data analysts and end-users
• Responsible for DevOps infrastructure
• Responsible for deploying Airflow dags to production environment using DevOps tools
• Responsible for code and query optimization
• Responsible for data pipelines maintenance, monitoring and stability
• Responsible for code review
• Responsible for improving the current data architecture and DevOps processes
• Responsible for delivering data in useful and appealing ways to users
• Responsible for performing and documenting analysis, review and study on specified regulatory topics
• Responsible for understanding business change and requirement needs, assess the impact and the cost

Requirements:

• Experience in Python
• Experience in creating APIs in Python
• Experience in PySpark
• Experience in Spark Environment Architecture
• Experience in SQL, Oracle Data Base
• Experience in creating and maintaining distributed environments using Hadoop and Spark
• Hadoop ecosystem - HDFS + Yarn
• Containerization - Docker is Mandatory
• Data Lakes - Experience in organizing and maintaining data lakes - S3 is preferred
• Experience with Parquet file format
• Apache Airflow - Experience in both pipeline development and deploying Airflow in distributed environment
• Experience inApache Kakfa
• Experience in automating applications deployment using DevOps tools - Jenkins is Mandatory, Ansible is a plus
• Fluent in English (written and oral)

If you're interested in this job please send your CV to or with reference "LM/Senior Data Engineer".

Decskill is committed to equality and non-discrimination with all our talents. We recruit and promote talent, based on diversity and inclusion, regardless of age, gender, ethnicity, race, nationality or any other form of discrimination incompatible with the dignity of the human being

Observações

Not Specified (Portugal)