-
Notifications
You must be signed in to change notification settings - Fork 0
/
deploying-spark-cluster.txt
68 lines (42 loc) · 1.87 KB
/
deploying-spark-cluster.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
Spark cluster deployment and sanity check
** follow instructions to install az here https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt
One time setup:
* install using the above link
* az login
* az account set --subscription "IEM - 096224"
1. in the Azure portal, go to HDinsight clusters, and verify there is none active
2. run the deploy script in spark/conf/ex2
az deployment group create --resource-group ddscourse-infrastructure-group --template-file ./template.json --parameters parameters.json
3. run
ssh-copy-id sshuser@<master host name>
4. connect to the Checker, http://homework-tester.westeurope.cloudapp.azure.com/96224/submit/hw/2
and upload test.py
verify you get the link to see the logs
5. use the above link to see the logs.
If no connection, check ssh connectivity.
To Delete the cluster:
az hdinsight delete --name noam-spark --resource-group ddcourse-infrastructure-group -y --no-wait
--------------
$ cat start_spark_cluster
#!/bin/bash
az deployment group create --resource-group ddscourse-infrastructure-group --template-file ./template.json --parameters parameters.json
if [ $? -ne 0 ]; then
echo "Subject:Cron job: create spark cluster FAILED" | sendmail [email protected]
fi
vmadmin@dds2020s-kafka:~$ cat stop_spark_cluster
#!/bin/bash
az hdinsight delete --name noam-spark --resource-group ddscourse-infrastructure-group -y --no-wait
if [ $? -ne 0 ]; then
echo "Subject:Cron job: Delete spark cluster FAILED" | sendmail [email protected]
fi
times in UTC !!
$ crontab -l
# Edit this file to introduce tasks to be run by cron.
# m h dom mon dow command
30 12 * * * /home/vmadmin/start_spark_cluster
40 12 * * * /home/vmadmin/stop_spark_cluster
==============
sending mail from linux shell:
generate google application password
sudo apt install ssmtp
configure the smtp.conf