Airswift has been tasked by one of our major oil & gas clients to seek a Data Engineer to work within their facilities located in Houston, TX for an initial 12-month contract, likely to be extended.
Within this position, you will be responsible for develop and implement digital solutions and products, as these are built on successful access to high-quality data. Alongside this, will define the data lifecycle, and gather and clean business data.
Day to day responsibilities:
- Apply strong expertise in data mining and information retrieval to design, develop, optimize, and maintain data architecture and pipelines
- Work with internal customers to understand their data requirements, develop and model data structure, and design and build the ingestion process to provide access to data from operational and enterprise source systems
- Design and develop data integration and data pipelines (ETL)
- Plan and deliver reliable, good practice data integration strategies and approaches
Previous education and experience required:
- Bachelor’s degree in Computer Science, MIS, or Engineering
- 4+ years of experience working in data engineering or architecture role
- Strong experience with: Python, Scala, SQL, Java
- Strong experience in multiple database technologies such as: Distributed Processing (Spark, Hadoop, EMR) Traditional RDBMS (MS SQL Server, Oracle, MySQL, PostgreSQL) Cloud Database Services (DynamoDB, Elasticache for Redis, Snowflake)
- Experience in traditional data warehousing / ETL tools (Informatica, Talend, Pentaho, DataStage)
- Exposure to Cloud platforms such as: AWS, Azure, Google Platform or Databricks
- Experience with data ingestion, analysis and processing pipelines using Kafka, Kafka Streams, Spark Streaming and similar cloud native technologies
- Experience with data processing systems with Hadoop, Spark, Storm, Impala, etc.
- Understanding of understanding of traditional ETL tools & RDBMS, End to End Data Pipeline
- Experience working on a collaborative Agile product team