Multiprocessing support for off policy algorithms

[araffin]

Description

closes #179

Add support for multiprocessing, only HerReplayBuffer not supported (done in #654 ).

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)
• I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[araffin]

@yonkshi @SonSang could you also take a look and review this PR?

[Miffyli]

Would you have any performance metrics to share vs. single-threaded (agent performance, training time)? I will try to take a look before end of the week :)

[araffin]

so I've got something qualitative so you get the general idea of the pros/cons of multi envs.
This is SAC on Pendulum on cpu with (in order):

• one env
• two envs
• four envs
• height envs
• four envs and gradient_steps=2 (instead of 1)

Sample efficiency:

Wall clock time

more envs = less sample efficient and faster wall clock time but you can retrieve some sample efficiency by having more gradients steps , in that case you will lose some speed (especially in that experiment with my laptop on cpu only)

the slower the env, the bigger the gap with using one env only will be

[Miffyli]

LGTM but need to fix the typing thing :)

[araffin]

I reintroduced the next_obs = deepcopy(new_obs_) because it prevents many errors (and not only with dict observations)