server,discovery: fetch as many Os as possible during discovery before a shorter timeout

[kyriediculous]

What does this pull request do? Explain your changes. (required)
This PR changes discovery to take into account all possible orchestrators as long as they respond to a discovery request within 1 second. Previously we would stop gathering responses once we received a small subset of responses.

Specific updates (required)

• change numOrchs to be equal to the Orchestrator pool size
• Decreased the timeout to 1 second
• Changed the suspension time to poolSize/numSessions (not sure if this is better but at least this avoids poolSize/numOrchs which would always be 1)
• Changed the cut-off point at which we do a refresh to be at less than 20% of the original selected set.

How did you test each of these updates (required)
https://github.com/livepeer/internal-project-tracking/issues/124

Does this pull request close any open issues?
Fixes #1853

Checklist:

• Read the contribution guide
• make runs successfully
• All tests in ./test.sh pass
• README and other documentation updated
• Pending changelog updated

[yondonfu]

A few minor comments, but largely looks good and just missing a changelog entry.

As the next step, let's analyze the following things (the ability to fetch work distribution data should be useful) mentioned in #1853 in tests on the network:

We should experiment with this and observe the following data points:

• Distribution of work before and after these changes
• Frequency of session list refreshes
• Impact of a fixed 1 second delay of discovery on the first segment submission of a stream

[yondonfu]

Looks like we're introducing a min working set size here to determine when to pop Os off the suspension queue. I think an alternative that would avoid explicitly introducing a min working set size variable/function is the following:

---

Make the following update to the discovery loop:

// This loop runs until either all orchestrators have responded or a timeout
for i := 0; i < numAvailableOrchs && !timeout; i++ {
    // ...
}

Then, only the caller of GetOrchestrators() needs to be aware of the min working set size (which is the case right now outside of this PR) instead of making the GetOrchestrators() function aware of a min working set size variable/function.

The end result should be the same as what is currently implemented in this PR. The caller specifies the min working set size via numOrchestrators and GetOrchestrators() will fetch as many responses as possible from all Os, but if a bunch of Os are suspended such that the number of non-suspended Os is less than numOrchestrators, we'll pop Os off the suspension queue. The caller would be BroadcastSessionsManager which sets its numOrchs field to the min working set size which is passes as numOrchestrators in the GetOrchestrators() call.

WDYT?

[kyriediculous]

What you mention makes sense, the main reason I went with this (and I'm open to alternatives) is because if we select as many O's as possible e.g. 50 , and 5 don't respond. We'll still have a working set of 45 which seems sufficient enough to not have to pop on suspended O's until there's 50 O's again. It might make more sense to do that only when the working set would otherwise get too small.

[yondonfu]

Right. So if numOrchestrators reflected a min working set size that is smaller than 50 (i.e. 8), which is the # of available Os in this scenario, defined in BroadcastSessionsManager then we wouldn't pop any Os from the suspension queue in the scenario that you described.

[yondonfu]

A variant of this was merged to master previously so closing.