Add -m flag to gsutil step; add dockstore branch filters to facilitate development

[mmorgantaylor]

This PR addresses spec-ops issue #235 - Use -m flag in final gsutil mv of files in ImportGenomes.

Additionally, this PR adds branch filters to the dockstore.yml file that will help with development. The filter for each workflow indicates which branch(es) will show up for that workflow in dockstore. If we don't include these filters, dockstore will run checks of ALL workflows on ALL branches, which causes timeouts. We could remove these filters later (before merging to master) or not, but for now this could help us develop on ah_var_store. Note that we'll need to add feature branches to that file as we work on them.

This workflow was tested in Terra and the upload succeeded.

Also confirmed that if one file fails, the entire process throws an error code (i.e. -m flag will not cause failures to silently pass) - in example below, test_file_list.txt was a list of 6 files, including 1 file that did not exist.

➜  cat test_file_list.txt | gsutil cp -I gs://dsp-fieldeng-dev/test_cp/
Copying file://test1.txt [Content-Type=text/plain]...
Copying file://test2.txt [Content-Type=text/plain]...
Copying file://test3.txt [Content-Type=text/plain]...
CommandException: No URLs matched: test4.txt
➜  cat test_file_list.txt | gsutil -m cp -I gs://dsp-fieldeng-dev/test_cp/
If you experience problems with multiprocessing on MacOS, they might be related to https://bugs.python.org/issue33725. You can disable multiprocessing by editing your .boto config or by adding the following flag to your command: `-o "GSUtil:parallel_process_count=1"`. Note that multithreading is still available even if you disable multiprocessing.

CommandException: No URLs matched: test4.txt
Copying file://test1.txt [Content-Type=text/plain]...
Copying file://test5.txt [Content-Type=text/plain]...
Copying file://test2.txt [Content-Type=text/plain]...
Copying file://test3.txt [Content-Type=text/plain]...
Copying file://test6.txt [Content-Type=text/plain]...
- [5/5 files][   37.0 B/   37.0 B] 100% Done
Operation completed over 5 objects/37.0 B.
CommandException: 1 file/object could not be transferred.

[kcibul]

one comment to address, but I need to know more about the dockstore thing before having a opinion on that. maybe at standup today?

[kcibul]

Did you ever determine the change in behavior when adding the -m w.r.t exit codes?

For example, trying gsutil cp file1.txt filethatdoesntexist.txt gs://destination vs the same thing with -m. We can likely handle either case, but we should know (and maybe put a comment here about it)

[mmorgantaylor]

yes, I addressed this in the PR comment! but I can add a comment to the code, sure

[kcibul]

ha -- no I just missed the PR comment and went right to the code! ------------------------------- Kristian Cibulskis Director, Technology - SpecOps Broad Institute of MIT and Harvard [email protected]

…

On Thu, Feb 25, 2021 at 9:51 AM M. Morgan Taylor ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In scripts/variantstore/wdl/ImportGenomes.wdl <#7104 (comment)>: > @@ -168,10 +168,10 @@ task CreateImportTsvs { --mode GENOMES \ -SNM ~{sample_map} \ --ref-version 38 - - gsutil cp metadata_*.tsv ~{output_directory}/metadata_tsvs/ - gsutil cp pet_*.tsv ~{output_directory}/pet_tsvs/ - gsutil cp vet_*.tsv ~{output_directory}/vet_tsvs/ + + gsutil -m cp metadata_*.tsv ~{output_directory}/metadata_tsvs/ yes, I addressed this in the PR comment! but I can add a comment to the code, sure — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#7104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK3RAZ6QGZC7DLW4MQ55BLTAZPXZANCNFSM4YFMJBJA> .