Skip to content

An experimental distributed map reduce system based on Google's MapReduce, written in Rust!

Notifications You must be signed in to change notification settings

123vivekr/distributed-map-reduce

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Distributed Map Reduce

An experimental implementation of Map Reduce based on Google's Map Reduce paper written in Rust.

        ┌─────────────────┐                           ┌─────────────────────┐
        │ Coordinator     │                           │      Worker         │
        │                 │                           │                     │
        └─────────┬───────┘                           └──────────┬──────────┘
                  │                                              │
                  │                                              │
                  │Provisions                                    │Provisions
                  │                                              │
                  │                                              │
┌─────────────────┼─────────────────┐     ┌──────────────────────▼────────────────────┐
│                 │                 │     │                                           │
│                 │                 │     │                                           │
│                 │                 │     │              ┌───────────────────┐        │
│                 │                 │     │              │   Worker Node 1   │        │
│                 │                 │     │      ┌◄──────┤                   │        │
│                 │                 │     │      │       └───────────────────┘        │
│                 │                 │     │      │                                    │
│         ┌───────▼──────────┐      │     │      │       ┌───────────────────┐        │
│         │                  │      │     │      │       │  Worker Node 2    │        │
│         │ Coordinator Node ◄──────┼─────┼──────┤◄──────┤                   │        │
│         └──────────────────┘      │     │      │       └───────────────────┘        │
│                                   │     │      │                                    │
│                                   │     │      │       ┌───────────────────┐        │
│                                   │     │      │       │   Worker Node 3   │        │
│                                   │     │      └◄──────┤                   │        │
│                                   │     │              └───────────────────┘        │
│                                   │     │                                           │
│                                   │     │                                           │
│                                   │     │                                           │
└───────────────────────────────────┘     └───────────────────────────────────────────┘
         Coordiantor Plane                              Worker Plane

Instructions to Run

First start the coordinator

cargo run --bin coordinator -- <n_reduce> <filenames>

Example: cargo run --bin coordinator -- 5 *.txt

Then start the workers

cargo run --bin worker -- 127.0.0.1:12345 <number_of_workers>

Example: cargo run --bin worker -- 127.0.0.1:12345 10

About

An experimental distributed map reduce system based on Google's MapReduce, written in Rust!

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages