This script is designed to install a CentOS 6 (and other yum-based OS) cluster with Hadoop DFS, YARN, MapReduce, Apache Hive and Spark of N nodes (one of them master). The original script was done by Pablo Rebollo (from the University of Puerto Rico) and was further improved by me.
To use this script, the first step is to clone this repository.
git clone https://github.com/omarpr/hadoop_install
Then, modify the following lines of the file hadoop_install.sh. In them, you will specify the name of the user account (this is important for permissions) and the IP of the master node and each data node.
CUSER=omar.soto2
Modify the nodes file to add the IPs of all the nodes of the cluster. The first line of the file will correspond to the master node. The other lines will be identified as dataXX node.
136.145.216.XXX
136.145.216.XXX
136.145.216.XXX
136.145.216.XXX
...
Copy the script to all the servers that you will have in the cluster. To install it, just run the script:
sudo ./hadoop_install.sh
Load the .bash_profile to have the environment variables ready:
. ~/.bash_profile
To start the NameNode and the YARN (only on master node), use this:
hadoop-daemon.sh start namenode
yarn-daemon.sh start resourcemanager
(Remember to format the HDFS)
hdfs namenode -format cluster
Run the hadoop_install.sh on each data node (and load .bash_profile) and start the DataNode and NodeManager, using this:
hadoop-daemon.sh start datanode
yarn-daemon.sh start nodemanager
HDFS Web Application
http://[Master IP]:50070
YARN Web Application
http://[Master IP]:8088
List directory contents
hdfs dfs -ls /
Make directory recursively
hdfs dfs -mkdir -p /a/path
Balance HDFS with threshold 1%
export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"
hdfs balancer -threshold 1
Run PySpark console
pyspark --master yarn --deploy-mode client --conf='spark.executorEnv.PYTHONHASHSEED=223'
Submit PySpark application
spark-submit [code.py]
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
sc = SparkContext(appName="[Application Name]")
sc.setLogLevel("WARN")
ssc = StreamingContext(sc, 60)
# Code to execute every 60 seconds.
ssc.start()
ssc.awaitTermination()