As a user, I want status summary reports during a long running execution (batching functionality)

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

Node Operator

💪 Motivation

...so that I can regular check the overall status of an execution in either file upload "chunks"

📖 Additional Details

From @mdrum

2. Perhaps allow for batching of some sort. Rather than accepting all files and generating one big report at the end, you would want to split it up into groups of 1000 files or something and create mid-reports along the way, with a final tally being generated at the end. We could do this on our end, of course, but then it would mean we would have to combine the mid-reports manually ourselves. Something to think about.

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Initial thoughts: new flag(s) to allow someone to specify reporting type: (cumulative, batch, both), and select the batching "chunks" (default: 1000 files)

🎉 I&T

No response

[jordanpadams]

Status: @collinss-jpl encountered a potential snag in the batching feature, limits maximum batch size (250-300 files). looking to do some more profiling to determine the limitations.