Prevent oversampling of study submissions by any single worker

[meta-paul]

For some studies (i.e. Tasks spanning multiple TaskRuns) it may be desired to limit number of submissions received from the same worker (for example, to prevent bias in annotation tasks dataset).

The new TaskRunArgs parameter max_submissions_per_worker addresses this need.

[meta-paul]

@JackUrb @xing-liu This is a feature to prevent oversampling. Not part of TaskReviewApp feature, but due to the size of that other PR I've used that branch as the base for now.

[JackUrb]

Is this the same as maximum_units_per_worker?

[meta-paul]

@JackUrb Similar idea, but for a different provider setup. In Prolific we don't know ahead of time who will be working on a unit before releasing it. So sidelining an overcontributing worker should be done differently (via participant group, not at the time of unit assignment because we have no control over that).

[JackUrb]

I see - I think in general we should actually try to merge this concept. In #773 I noted that the current way of doing things (based on a count and a "no access" page) wasn't great, and that qualification-based would be preferred. I don't think we should have two separate arguments for this.

[meta-paul]

@JackUrb I can attempt at parameter unification there. Should we do this before merging TaskReview app code, or it can be done later?

[JackUrb]

I don't think it should be merged as is, and also don't think this feature should block task review.

[meta-paul]

@JackUrb I've nested my logic under your paramtere name maximum_units_per_worker. That way one parameter will cover both use cases of Mturk and Prolific. Let me know if that makes sense.

(By the way, disregard failing styling check. Not sure why it's failing, as if Github is running on another version of the code - couldn't find failing lines within this branch)

[JackUrb]

This consolidation now makes a lot more sense. Just be sure to test it out both on MTurk and prolific.

[mojtaba-komeili]

Had a quick look and it seems reasonable to me. I also see Jack is onboard with the changes. Feel free to merge after adding some tests please.

[JackUrb]

Overall this looks good to me. Thanks for unifying the logic between Prolific and MTurk providers, I think the flow is more obvious now for both.

One future TODO (#773) would be to push an exclusionary qualification to MTurk in the same way you currently do to prolific for qualification groups. This could be an option for a v1.3 feature, and would involve creating per-task-run exclusionary qualifications by default for each MTurk TaskRun. This PR makes the flow for that feature request more straightforward.