Job Details

• Data and problem analysis
• Convert the solution to Spark and other big data tools based implementation
• Understand data patterns
• Based on access/write pattern choose or convince team on storage sub-system
• Tuning performance optimization of Spark data pipelines
• Automation of day to day tasks
• Build data pipelines and ETL using heterogeneous sources to Hadoop using Kafka, Flume, Sqoop, Spark Streaming etc.

• 6-12 years of experience working in Hadoop Ecosystem and big data technologies
• Must be very comfortable with reading and writing Scala, Python or Java code
• Well versed with Spark development (SQL and/or Spark) - should able to express complex logic if SQL is not possible
• Sound Knowledge of HBase or any NoSQL database e.g. Cassandra, MongoDB.
• Good OS and HDFS knowledge
• Work experience on AWS on any other cloud would be an added advantage
• Experience in batch or real time data streaming
• Ability to dynamically adapt to conventional big-data frameworks and open source tools if project demands
• Knowledge of design strategies for developing scalable, resilient, always-on data lake
• Experience in agile(scrum) development methodology
• Strong development/automation skills
• Some understanding of data architecture principles, including data access patterns and data modelling
• Previous experience in Financial Services organisations an advantage
• Excellent communication and presentation skills
• A Tertiary degree in IT or similar subject

Reference ID - GH/Technology/JC//18062020//91299/Big Data Developer -Manager