· Implement Spark using Scala and utilize data frames and Spark-SQL APIs for faster data processing.
· Apply Neural Network concepts like LSTM, GRU to create better attribute classifiers for data monitoring.
· Use Spark to consume data from Kafka and convert that to a common format using Scala.
· Handle one network level and push data to different source like salesforce, Account-DB.
· Work on a Kafka cluster to demonstrate the incoming and outgoing events via Spark streaming.
· Implement Elasticsearch, Logstash, Kibana, and Beats (Elastic Stack) to optimize centralization.
· Build POC application to monitor the process flow using Tensorboard in the TensorFlow application.
· Work with Spark to consume data from Kafka and convert that to a common format using Scala.
· Develop Spark code and Spark-SQL/ Streaming for faster testing and processing of data.
· Read/ write data from various file formats (JSON, text, parquet, Schema RDD) using Spark-SQL.
· Import the data from different sources like HDFS/HBase into Spark RDD.
· Create data pipeline using Kafka, HBase, Spark and Hive to ingest, transform and analyze customer behavior.
This position requires a minimum of Bachelor’s degree or equivalent in Computer Science, Computer Information Systems, and Information Technology or related.