Spark Streaming with Twitter and Kafka for sentiments of tweets on COVID-19 and India

Last updated on May 24, 2021

Due to the second wave of covid-19, There is an unprecedented surge of covid-19 in india and we wanted to analyze the sentiment of public on the basis of tweeter. We have used stanford core NLP to analyze the tweets. For visualization, we only sent tweets sentiment and collected 100k+ tweet’s sentiment in 20hrs.

Dependencies:-

- kafka_2.13-2.8.0
- elasticsearch-7.12.1
- kibana-7.12.1-linux-x86_64
- Logstash-7.4.2
- sparkVersion = "2.4.7"
- scalaVersion := "2.11.12"

I have created spark stream application which reads tweets via twitter java library and broadcast via kafka producer named TopicA. Logstash takes TopicA and insert it into Elastic Search and results are visualized into Kibana.In this I used keywords “india”, “covid-19”, “oxygen”, and “remdesivir”.

Machine Learning

Related