"# Spark"
This is repo i used to explore new libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem and mainly focused on Spark processes.
This project is organized into modules. Modules may be independent or have dependencies on other modules within spark. When possible, dependencies on external projects are minimized.
The following modules currently exist.
The data module provides logical abstractions on top of storage subsystems (e.g. HDFS) that let users think and operate in terms of records, datasets, and dataset repositories. If you're looking to read or write records directly to/from a storage system, the data module is for you.
Example code demonstrating how to use Spark-batch
To build
mvn install
Steps to setup and run spark-batch in intellij
Download and store winutils.exe in local folder EX: C:\hadoop\bin.
set HADOOP_HOME = C:\hadoop\;
update PATH = %PATH%;%HADOOP_HOME%\bin;
clone this repo and open in intellij
SparkJobRunner.scala is the main class.
Right click on SparkJobRunner.scala and run!!