Task loading endlessly and workers disconnected

[claudiogreco]

Hello,

I am running a Mephisto data collection task hosted using Heroku. I am collecting annotations, but several workers (about 20% of the total number of workers) are complaining about the fact that they just saw a loading page or were disconnected at some point. In the terminal, I am seeing messages like: "Onboarding agent onboarding_2806 disconnected or errored, final status timeout.". The configuration file contains the following variables which might be relevant in order to solve the issue: allowed_concurrent: 1, max_num_concurrent_units: 0, and maximum_units_per_worker: 1. Would setting the variable max_num_concurrent_units equal to a value in the range [250-500] as suggested in the FAQs solve the issue? If yes, would it be possible to change that variable in the configuration file without having to restart the task from scratch?

Thank you very much in advance!

[JackUrb]

HI @claudiogreco - on 1.0 this is almost definitely something that max_num_concurrent_units would help resolve (assuming you're using the default database type singleton). Unfortunately this option can only be configured on launch.

You raise a good point through, considering we have suggested ranges on these, we may move to set them as per-blueprint defaults for our common tasks.

Also, for performance-related issues in the future, you may find our dashboarding tutorial provides extra context, at the very least with things you can share with us for better debugging assistance.

[claudiogreco]

Thank you very much for the information.