-
-
Notifications
You must be signed in to change notification settings - Fork 778
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #566 from FedML-AI/dev/v0.7.0
Dev/v0.7.0
- Loading branch information
Showing
81 changed files
with
480 additions
and
758 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
69 changes: 69 additions & 0 deletions
69
.github/workflows/smoke_test_cross_silo_fedavg_ldp_linux.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| # This is a basic workflow to help you get started with Actions | ||
|
|
||
| name: LDP-Linux | ||
|
|
||
| # Controls when the workflow will run | ||
| on: | ||
| # Triggers the workflow on push or pull request events but only for the master branch | ||
| schedule: | ||
| # Nightly build at 12:12 A.M. | ||
| - cron: "12 12 */1 * *" | ||
| pull_request: | ||
| branches: [ master, test/v0.7.0, dev/0.7.0 ] | ||
|
|
||
| # Allows you to run this workflow manually from the Actions tab | ||
| workflow_dispatch: | ||
|
|
||
| # A workflow run is made up of one or more jobs that can run sequentially or in parallel | ||
| jobs: | ||
| cross-silo-ldp-test: | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| working-directory: python | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ ubuntu-latest] | ||
| arch: [X64] | ||
| python-version: ['3.8'] | ||
| client-index: ['0', '1', '2'] | ||
| # exclude: | ||
| # - os: macos-latest | ||
| # python-version: '3.8' | ||
| # - os: windows-latest | ||
| # python-version: '3.6' | ||
| runs-on: [self-hosted, runner-linux, devops] | ||
| timeout-minutes: 15 | ||
| steps: | ||
| - uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| - uses: actions/checkout@v3 | ||
| - name: pip install -e ./ | ||
| run: | | ||
| pip install -e ./ | ||
| - name: server - cross-silo - ldp | ||
| run: | | ||
| cd examples/cross_silo/mqtt_s3_fedavg_ldp_mnist_lr_example | ||
| run_id=cross-silo-ho-${{ format('{0}{1}{2}{3}', github.run_id, matrix.os, matrix.arch, matrix.python-version) }} | ||
| echo ${run_id} | ||
| bash run_server.sh $run_id | ||
| if: ${{ matrix.client-index == '0' }} | ||
|
|
||
| - name: client 1 - cross-silo - ldp | ||
| run: | | ||
| cd examples/cross_silo/mqtt_s3_fedavg_ldp_mnist_lr_example | ||
| run_id=cross-silo-ho-${{ format('{0}{1}{2}{3}', github.run_id, matrix.os, matrix.arch, matrix.python-version) }} | ||
| echo ${run_id} | ||
| bash run_client.sh 1 $run_id | ||
| if: ${{ matrix.client-index == '1' }} | ||
|
|
||
| - name: client 2 - cross-silo - ldp | ||
| run: | | ||
| cd examples/cross_silo/mqtt_s3_fedavg_ldp_mnist_lr_example | ||
| run_id=cross-silo-ho-${{ format('{0}{1}{2}{3}', github.run_id, matrix.os, matrix.arch, matrix.python-version) }} | ||
| echo ${run_id} | ||
| bash run_client.sh 2 $run_id | ||
| if: ${{ matrix.client-index == '2' }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
60 changes: 60 additions & 0 deletions
60
.../examples/cross_silo/mqtt_s3_fedavg_defense_mnist_lr_example/config/krum/gpu_mapping.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # You can define a cluster containing multiple GPUs within multiple machines by defining `gpu_mapping.yaml` as follows: | ||
|
|
||
| # config_cluster0: | ||
| # host_name_node0: [num_of_processes_on_GPU0, num_of_processes_on_GPU1, num_of_processes_on_GPU2, num_of_processes_on_GPU3, ..., num_of_processes_on_GPU_n] | ||
| # host_name_node1: [num_of_processes_on_GPU0, num_of_processes_on_GPU1, num_of_processes_on_GPU2, num_of_processes_on_GPU3, ..., num_of_processes_on_GPU_n] | ||
| # host_name_node_m: [num_of_processes_on_GPU0, num_of_processes_on_GPU1, num_of_processes_on_GPU2, num_of_processes_on_GPU3, ..., num_of_processes_on_GPU_n] | ||
|
|
||
|
|
||
| # this is used for 10 clients and 1 server training within a single machine which has 4 GPUs | ||
| mapping_default: | ||
| ChaoyangHe-GPU-RTX2080Tix4: [3, 3, 3, 2] | ||
|
|
||
| # this is used for 4 clients and 1 server training within a single machine which has 4 GPUs | ||
| mapping_config1_5: | ||
| host1: [2, 1, 1, 1] | ||
|
|
||
| # this is used for 10 clients and 1 server training within a single machine which has 4 GPUs | ||
| mapping_config2_11: | ||
| host1: [3, 3, 3, 2] | ||
|
|
||
| # this is used for 10 clients and 1 server training within a single machine which has 8 GPUs | ||
| mapping_config3_11: | ||
| host1: [2, 2, 2, 1, 1, 1, 1, 1] | ||
|
|
||
| # this is used for 4 clients and 1 server training within a single machine which has 8 GPUs, but you hope to skip the GPU device ID. | ||
| mapping_config4_5: | ||
| host1: [1, 0, 0, 1, 1, 0, 1, 1] | ||
|
|
||
| # this is used for 4 clients and 1 server training using 6 machines, each machine has 2 GPUs inside, but you hope to use the second GPU. | ||
| mapping_config5_6: | ||
| host1: [0, 1] | ||
| host2: [0, 1] | ||
| host3: [0, 1] | ||
| host4: [0, 1] | ||
| host5: [0, 1] | ||
| # this is used for 4 clients and 1 server training using 2 machines, each machine has 2 GPUs inside, but you hope to use the second GPU. | ||
| mapping_config5_2: | ||
| gpu-worker2: [1,1] | ||
| gpu-worker1: [2,1] | ||
|
|
||
| # this is used for 10 clients and 1 server training using 4 machines, each machine has 2 GPUs inside, but you hope to use the second GPU. | ||
| mapping_config5_4: | ||
| gpu-worker2: [1,1] | ||
| gpu-worker1: [2,1] | ||
| gpu-worker3: [3,1] | ||
| gpu-worker4: [1,1] | ||
|
|
||
| # for grpc GPU mapping | ||
| mapping_FedML_gRPC: | ||
| hostname_node_server: [1] | ||
| hostname_node_1: [1, 0, 0, 0] | ||
| hostname_node_2: [1, 0, 0, 0] | ||
|
|
||
| # for torch RPC GPU mapping | ||
| mapping_FedML_tRPC: | ||
| lambda-server1: [0, 0, 0, 0, 2, 2, 1, 1] | ||
| lambda-server2: [2, 1, 1, 1, 0, 0, 0, 0] | ||
|
|
||
| #mapping_FedML_tRPC: | ||
| # lambda-server1: [0, 0, 0, 0, 3, 3, 3, 2] |
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 2 additions & 1 deletion
3
...ample/custom_data_and_model/run_client.sh → ...fedavg_ldp_mnist_lr_example/run_client.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| #!/usr/bin/env bash | ||
| RANK=$1 | ||
| python3 torch_client.py --cf config/fedml_config.yaml --rank $RANK --role client | ||
| RUN_ID=$2 | ||
| python3 torch_client.py --cf config/fedml_config.yaml --rank $RANK --role client --run_id $RUN_ID |
4 changes: 2 additions & 2 deletions
4
...ist_lr_example/step_by_step/run_server.sh → ...fedavg_ldp_mnist_lr_example/run_server.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| python3 torch_server.py --cf config/fedml_config.yaml --rank 0 --role server | ||
| RUN_ID=$1 | ||
| python3 torch_server.py --cf config/fedml_config.yaml --rank 0 --role server --run_id $RUN_ID |
File renamed without changes.
File renamed without changes.
Empty file.
12 changes: 0 additions & 12 deletions
12
...ss_silo/mqtt_s3_fedavg_local_dp_mnist_lr_example/custom_data_and_model/build_mlops_pkg.sh
This file was deleted.
Oops, something went wrong.
66 changes: 0 additions & 66 deletions
66
...o/mqtt_s3_fedavg_local_dp_mnist_lr_example/custom_data_and_model/config/fedml_config.yaml
This file was deleted.
Oops, something went wrong.
3 changes: 0 additions & 3 deletions
3
...s/cross_silo/mqtt_s3_fedavg_local_dp_mnist_lr_example/custom_data_and_model/run_server.sh
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.