Skip to content

beyhangl/StructuredSparkStreaming

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Structured Spark Streaming with Kafka and Redis

This project uses structured streaming architecture that comes with spark 2.0+ of real time data read from kafka.

Redis is used as cache architecture and this cache is renewed periodically.We used redis for cache Because Spark streaming does not have cache architecture, such a structure was used.

Project includes end-to-end realtime system. You can change the sql queries according to your project and change the join operations.

Some jars were used in the project and these jars were found on the repos.

Please inform me about the issue when there is a missing jar.