The design concepts of this DFS largely come from the Google File System.
Read about Google File System here.
It has 3 components:
Master
- This is the main controller node that the client initially connects to. It is responsible for maintaining the meta-data of files.Slave
- These are the storage nodes that are controlled by the master node. Data is stored and replicated on these nodes.Client
- The program through which the user uploads/lists files.
- Allows client to load, fetch, delete, and list files
- Communication framework used is RMI
- Design can be scaled for any number of slaves
- Heartbeat check of workers at regular intervals
- Recovery of master meta-data from logs, in case the master goes down
Java 8 and Maven should be set up
Step 1: For running DFSMaster.java, add the below VM arguments accordingly and run
--master-registry-host <eg: localhost>
--master-registry-port <eg: 8085>
--logPath <eg: /Users/akashshrivastva/Desktop/dfs-master/log/master.log>
Step 2: Run DFSSlave.java, add the below VM arguments accordingly and run
--master-registry-host <eg: localhost>
--master-registry-port <eg: 8085>
--registry-port <eg: 8081> [NOTE: This is the slave registry port. In case you are running both the master
and slave on same machine, this port number needs to be different from that of master's.]
--service-name <eg: worker-1> [NOTE: If you are running multiple slaves on the same machine, give each of them
a different service name]
--slave-data-path <eg: /Users/akashshrivastva/Desktop/dfs>
Step 3: Run DFSClient.java, add the below VM arguments accordingly and run
--master-registry-host <eg: localhost>
--master-registry-port <eg: 8085>