Skip to content

Latest commit

 

History

History
291 lines (227 loc) · 18.3 KB

README.md

File metadata and controls

291 lines (227 loc) · 18.3 KB

Python Plays Grand Theft Auto 5 - Reboot

This is a reboot of the project from 2017, but with a whole new approach. This time, instead of collecting data by hand and training a classification model, we opted to create a whole new system for the data collecting, training and playing, and to do all of that live. Our models are now using regression (with other output types being still possible), training data is collected automatically and we stream models during their training phase so anyone can join and watch the progress.

Our AI, besides having multiple alter-egos (models), is still called Charles, like in 2017.

For a full explanation, refer to the System page, but for a quick summary - we’re using a central server that all other parts connect to. The Data Collectors run in separate GTA5 instances and our NPCs are collecting and Balancing the data, which is sent through the Server to the Trainer. The Trainer buffers these data, trains the model and updates the Player. The Player is using the Dual-Camera system to play so we can watch the 3rd person camera (also called cinematic camera) while the model is fed the Hood Camera.

For more information, refer to these other pages:



Stream and current stream layout:

The progress is being streamed on Twitch: https://www.twitch.tv/sentdex. Below we keep a list of dates and streamed models.

Stream's layout consists of: layout.jpg

  • top-left tile is the 3rd person camera of the car driving, the main camera to observe Charles
  • this main tile, in its lower-right corner, shows the current model and when it's been created
  • top-right tile called Hood Camera is exactly what the models "see" - the input to the convolutional backbone of the model
  • middle-right tile called Convcam shows the reshaped output of the CNN backbone and lets us observe how the CNN part trains
  • bottom-right tile called Player Console shows current driving predictions along with additional information
  • bottom-middle tile called Server/Trainer Console shows training progress along with some basic training information
  • bottom-left tile called Tensorboard shows the loss of the training process


Driving examples:

 



Model list:



Streaming timeline

May 1st:

May 2nd:

May 7th:

May 8th:

May 14th:

May 16th:

  • Stopped streaming model_0006_inceptionresnetv2_v1 at batch 60500
  • The stream has been offline for several days since we had to send back the Comino machine and setting up another one took longer than we expected

May 28th:

  • After new machine was set and new models developed, we restarted streaming
  • Started streaming model_0009_irv2_cr_tl_v1 from scratch

May 29th:

July 5th:

  • Stopped streaming model_0004_inceptionresnetv2_v3 at batch 752000
  • The stream has been stopped as there have been no further improvements in the model and to catch up with documentation and do some off-stream development


Project timeline

This is not a full project log and contains only the key changes that are visible on the stream (and a few from before the stream has started). There are many more things going on "behind the scenes".

2017:

November 2021:

  • The idea of bringing back this project has born

Early 2022:

  • The project has started the development phase of the main system, the initial models, and the game mod

April 8th:

April 9th:

April 10th:

April 11th:

April 15th:

April 19th:

April 22nd:

April 24th:

April 30th:

May 1st:

  • The Convcam has been created
  • Started streaming on Twitch.tv
  • Started streaming model_0003_xception_v2 from scratch

May 2nd:

May 3rd:

May 7th:

May 8th:

May 13th:

May 14th:

May 14th:

May 16th:

  • Stopped streaming model_0006_inceptionresnetv2_v1 at batch 60500
  • Stream has been offline for several days since we had to send back the Comino machine and setting up another one took longer than we expected

May 22nd:

May 23th:

May 24th:

May 25th:

May 26th:

May 28th:

May 29th:

May 30th:

May 31th:

June 1th:

June 3rd:

June 4th:

June 9th:

June 10th:

June 13th:

July 5th:

  • Stopped streaming model_0004_inceptionresnetv2_v3 at batch 752000
  • The stream has been stopped as there have been no further improvements in the model and to catch up with documentation and do some off-stream development