Cleanup IP race condition on EC2Architect

[JackUrb]

Summary

At the moment, the EC2Architect could run into three failure scenarios.

1. You may have already allocated all 5 elastic IPs associated with your account by having many jobs running
2. You may try to connect to a server before the association is complete, leading to the wrong IP being returned before an attempted push.
3. The attempt to connect to the server could fail due to having used that same IP for a different server before, thus clashing with an entry in known_hosts
   This PR addresses all three in one sweep.

Implementation

We now only associate an IP to an instance when trying to push content to that instance, and then free that IP afterwards. This way in reserving the IP we know exactly what IP is going to be used, we can clear it from known hosts, and can stay under the 5 IP limit.

I also create the try_server_push to consolidate some logic used in both server pushes to one place.

Testing

Launched a test task on sandbox with the new setup, ensured that the server loaded properly and reserved IPs were not present after launch.

[codecov-commenter]

Codecov Report

Merging #606 (4877639) into main (bc3526f) will decrease coverage by 0.05%.
The diff coverage is 11.76%.

@@            Coverage Diff             @@
##             main     #606      +/-   ##
==========================================
- Coverage   63.34%   63.28%   -0.06%     
==========================================
  Files          86       86              
  Lines        8208     8221      +13     
==========================================
+ Hits         5199     5203       +4     
- Misses       3009     3018       +9     

Impacted Files	Coverage Δ
...histo/abstractions/architects/ec2/ec2_architect.py	35.76% <0.00%> (+0.51%)	⬆️
...ephisto/abstractions/architects/ec2/ec2_helpers.py	17.18% <12.00%> (+0.46%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc3526f...4877639. Read the comment docs.