Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move data dir to models #1099

Merged
merged 7 commits into from
Jul 25, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Jul 24, 2023

Description

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@dagardner-nv dagardner-nv requested review from a team as code owners July 24, 2023 23:08
@dagardner-nv dagardner-nv self-assigned this Jul 24, 2023
@dagardner-nv dagardner-nv added bug Something isn't working non-breaking Non-breaking change 2 - In Progress ! - Hotfix Related to a Hotfix Release labels Jul 24, 2023
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets wait for confirmation from packaging that this looks good before merging

@pdmack
Copy link
Contributor

pdmack commented Jul 25, 2023

@mdemoret-nv @dagardner-nv does this look right?
Trying to understand the change.

-rw-rw-r-- 1 root root 755468 Jul 25 00:47 /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/data/bert-base-cased-hash.txt
-rw-rw-r-- 1 root root 213450 Jul 25 00:47 /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/data/bert-base-cased-vocab.txt
-rw-rw-r-- 1 root root 796289 Jul 25 00:47 /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/data/bert-base-uncased-hash.txt
-rw-rw-r-- 1 root root 231508 Jul 25 00:47 /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/data/bert-base-uncased-vocab.txt
-rw-rw-r-- 1 root root 755468 Jul 25 00:39 /workspace/models/data/bert-base-cased-hash.txt
-rw-rw-r-- 1 root root 213450 Jul 25 00:39 /workspace/models/data/bert-base-cased-vocab.txt
-rw-rw-r-- 1 root root 796289 Jul 25 00:39 /workspace/models/data/bert-base-uncased-hash.txt
-rw-rw-r-- 1 root root 231508 Jul 25 00:39 /workspace/models/data/bert-base-uncased-vocab.txt
lrwxrwxrwx 1 root root     38 Jul 25 00:39 /workspace/models/training-tuning-scripts/log-parsing-models/resources/bert-base-cased-hash.txt -> ../../../data/bert-base-cased-hash.txt
lrwxrwxrwx 1 root root     38 Jul 25 00:39 /workspace/models/training-tuning-scripts/sid-models/resources/bert-base-cased-hash.txt -> ../../../data/bert-base-cased-hash.txt
lrwxrwxrwx 1 root root     39 Jul 25 00:39 /workspace/models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt -> ../../../data/bert-base-cased-vocab.txt
lrwxrwxrwx 1 root root     40 Jul 25 00:39 /workspace/models/training-tuning-scripts/sid-models/resources/bert-base-uncased-hash.txt -> ../../../data/bert-base-uncased-hash.txt
lrwxrwxrwx 1 root root     41 Jul 25 00:39 /workspace/models/training-tuning-scripts/sid-models/resources/bert-base-uncased-vocab.txt -> ../../../data/bert-base-uncased-vocab.txt

@dagardner-nv
Copy link
Contributor Author

@pdmack we moved the data dir to models/data and replaced it with a symlink such that in the source tree you have: morpheus/data -> ../models/data
Then we updated the symlinks under the models/training-tuning-scripts dir.

When we perform an install setuptools follows the symlink and copies the actual files into site-packages.

What you are seeing is that if you use the docker/run_container_release.sh script it mounts the source tree's models dir over the images dir:

-v $PWD/models:/workspace/models

@mdemoret-nv and I talked about this and we wanted to come up with something that would work for both users in a bare-metal install, a local docker build or a container downloaded from NGC.

Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 96f8df0 into nv-morpheus:branch-23.07 Jul 25, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
! - Hotfix Related to a Hotfix Release bug Something isn't working non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

3 participants