Data Engineer / Ingénieur Data H/F

QuantHouse has developed a BigData platform to manage its historical Market Data. This platform is used to automate the delivery of historical Market Data over the Internet. You will join the R&D team to improve, maintain and optimize the Scala-based Spark jobs used to ingest and process Market Data. Ingest jobs are running every day to integrate the latest batch of market data into the platform. Extraction jobs are executed on-demand, whenever a customer orders historical data. The technical environment is Hadoop (MapR distribution), Spark, Scala, Java, YARN, Elasticsearch, Oozie and a data volume of 200+ TB.

DESCRIPTION

• Ingest
• Improve, optimize and maintain the Ingest Scala job running in Spark
• Process referential data (financial instrument characteristics) into hdfs/parquet and into Elasticsearch
• Process quotation data (trades & order books) into hdfs/parquet and into Elasticsearch, filtering and merging data depending on referential information
• Enrich content based on external data sources during the ingest process
• Generate ingest reports and alerts
• Extract
• Improve, optimize and maintain the Extract Scala job running in Spark
• Based on client purchase (list of instruments, duration), extract from HDFS/Parquet the data matching the order
• Provide Level 2 support
• Improve the high availability of the platform
• Work with Web Application developers to interface with the BigData backend
• Work closely with Production team to help monitor the platform’s health and generate performance reports
• Work with our Product team to specify and develop new features for the platform
• Work with R&D Technology and Performance team to improve the Java tools used to access market data

QUALIFICATIONS

• Minimum 2 years of successful work experience with Scala and Java performing ETL jobs
• Demonstrable experience with Hadoop (MapR or Cloudera distribution)
• Experience working on applications scheduled by YARN and run by Spark
• Experience developing REST API to expose a service
• Knowledge of market data is a plus
• Experience with Linux, Shell, Python scripting
• Self-driven, ability to work independently
• Team spirit and desire to collaborate and contribute