Improve DUM Upload Performance and Avoid Replication of Files in the Cloud

[jordanpadams]

💡 Description

Currently, execution is taking too long to upload data. We need to figure out ways we can improve throughput. TBD on this task. This may be dependent on system CPUs available.

⚔️ Sub-tasks

• As a user, I want to skip upload of files already in S3 (nucleus staging bucket) #92
• As a user, I want to skip upload of files that are already in the Registry #99
• As a user, I want an end summary report in logs to show statistics of files uploaded #98
• As a user, I want to force an upload of file that is already in S3 or the Registry #100

[jordanpadams]

Removing from release plan for time being until we hear more from SBN-PSI about performance concerns.

[viviant100]

SBN-PSI noted that --num-threads flag did improve the performance considerably.

[jordanpadams]

📆 04/2024 status: In work. On schedule.

[jordanpadams]

📆 05/2024 status: In work. Delayed 1 sprint due to delays in NASA-PDS/registry#185. No impact on build expected

[jordanpadams]

📆 06/2024 status: In worked. Delayed an additional sprint because blocked by completion of NASA-PDS/registry#185. Will be done by build.

[jordanpadams]

📆 07/2024 status: In worked. Delayed an additional sprint, blocked by completion of NASA-PDS/registry#185. Will be done by build.

[jordanpadams]

Closing for now. Will have second iteration on this task to have registry in the loop

[jordanpadams]

Follow-on here: #151