Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add -m flag to gsutil mv step #7129

Merged
merged 1 commit into from
Mar 4, 2021
Merged

Add -m flag to gsutil mv step #7129

merged 1 commit into from
Mar 4, 2021

Conversation

mmorgantaylor
Copy link
Member

I added -m to gsutil cp in a previous PR but missed the gsutil mv step post-bq load - so here that is. tested it from the command line, works well. also confirmed that it will throw an error if one (or more) files has an error:

$ cat test_files_bucket.txt | gsutil -m mv -I gs://dsp-fieldeng-dev/test_mv/

If you experience problems with multiprocessing on MacOS, they might be related to https://bugs.python.org/issue33725. You can disable multiprocessing by editing your .boto config or by adding the following flag to your command: `-o "GSUtil:parallel_process_count=1"`. Note that multithreading is still available even if you disable multiprocessing.

Copying gs://dsp-fieldeng-dev/test_cp/test1.txt [Content-Type=text/plain]...
Copying gs://dsp-fieldeng-dev/test_cp/test2.txt [Content-Type=text/plain]...
CommandException: No URLs matched: gs://dsp-fieldeng-dev/test_cp/test4.txt
Copying gs://dsp-fieldeng-dev/test_cp/test3.txt [Content-Type=text/plain]...
Copying gs://dsp-fieldeng-dev/test_cp/test5.txt [Content-Type=text/plain]...
Copying gs://dsp-fieldeng-dev/test_cp/test6.txt [Content-Type=text/plain]...
Removing gs://dsp-fieldeng-dev/test_cp/test1.txt...
Removing gs://dsp-fieldeng-dev/test_cp/test2.txt...
Removing gs://dsp-fieldeng-dev/test_cp/test3.txt...
Removing gs://dsp-fieldeng-dev/test_cp/test5.txt...
Removing gs://dsp-fieldeng-dev/test_cp/test6.txt...
- [5/5 files][   37.0 B/   37.0 B] 100% Done
Operation completed over 5 objects/37.0 B.
CommandException: 1 file/object could not be transferred.

Copy link
Contributor

@ahaessly ahaessly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mmorgantaylor mmorgantaylor merged commit 611d0cb into ah_var_store Mar 4, 2021
@mmorgantaylor mmorgantaylor deleted the gsutil_m_mv branch March 4, 2021 19:14
kcibul pushed a commit that referenced this pull request Mar 9, 2021
mmorgantaylor added a commit that referenced this pull request Apr 6, 2021
This was referenced Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants