Environment not registered in OpenAI Gym

[arsenious]

Hi

I would really like to use this environment for Deep RL reserach purposes. But I'm not able to get it to work. Please help. Thanks

Using TensorFlow backend.
[2017-08-28 17:41:07,956] Making new env: MazeExplorer-v0
Traceback (most recent call last):
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 137, in spec
return self.env_specs[id]
KeyError: 'MazeExplorer-v0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/USERNAME/PycharmProjects/MazeExplorer/MazeTest.py", line 5, in
env = gym.make('MazeExplorer-v0')
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 161, in make
return registry.make(id)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 118, in make
spec = self.spec(id)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 147, in spec
raise error.UnregisteredEnv('No registered env with id: {}'.format(id))
gym.error.UnregisteredEnv: No registered env with id: MazeExplorer-v0

Process finished with exit code 1

[arsenious]

@mryellow

[arsenious]

Also, when I changed it to
env = gym.make('MazeExplorerEat-v0')

Using TensorFlow backend.
[2017-08-28 17:50:04,419] Making new env: MazeExplorerEat-v0
Traceback (most recent call last):
File "/home/USERNAME/PycharmProjects/MazeExplorer/MazeTest.py", line 5, in
env = gym.make('MazeExplorerEat-v0')
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 161, in make
return registry.make(id)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 119, in make
env = spec.make()
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 85, in make
cls = load(self._entry_point)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 17, in load
result = entry_point.load(False)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/pkg_resources/init.py", line 2316, in load
return self.resolve()
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/pkg_resources/init.py", line 2322, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym_mazeexplorer/envs/init.py", line 1, in
from gym_mazeexplorer.envs.mazeexplorer_env import MazeExplorerEnv
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym_mazeexplorer/envs/mazeexplorer_env.py", line 8, in
import mazeexp as mx
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/mazeexp/init.py", line 1, in
from mazeexp.engine.mazeexp import MazeExplorer
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/mazeexp/engine/mazeexp.py", line 10, in
import config
ImportError: No module named 'config'

[arsenious]

I'm using Python 3.5 so I think it was an import issue
Changing the line import config -> import mazeexp.engine.config
worked

But this new error comes and I can't file assets folder

Using TensorFlow backend.
[2017-08-28 18:00:26,573] Making new env: MazeExplorerEat-v0
Traceback (most recent call last):
File "/home/USERNAME/PycharmProjects/MazeExplorer/MazeTest.py", line 5, in
env = gym.make('MazeExplorerEat-v0')
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 161, in make
return registry.make(id)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 119, in make
env = spec.make()
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 85, in make
cls = load(self._entry_point)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym/envs/registration.py", line 17, in load
result = entry_point.load(False)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/pkg_resources/init.py", line 2316, in load
return self.resolve()
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/pkg_resources/init.py", line 2322, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym_mazeexplorer/envs/init.py", line 1, in
from gym_mazeexplorer.envs.mazeexplorer_env import MazeExplorerEnv
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/gym_mazeexplorer/envs/mazeexplorer_env.py", line 8, in
import mazeexp as mx
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/mazeexp/init.py", line 1, in
from mazeexp.engine.mazeexp import MazeExplorer
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/mazeexp/engine/mazeexp.py", line 11, in
import mazeexp.engine.config
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/mazeexp/engine/config.py", line 160, in
"player": pyglet.image.load(os.path.join(script_dir, 'assets', 'player7.png')),
File "/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/pyglet/image/init.py", line 178, in load
file = open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/home/USERNAME/anaconda2/envs/tensorflow35/lib/python3.5/site-packages/mazeexp/engine/assets/player7.png'

[mryellow]

I'm using Python 3.5

That's possibly an issue, believe I had to stick with 2.7 due to dependencies somewhere along the line. Give it a go though.

"player": pyglet.image.load(os.path.join(script_dir, 'assets', 'player7.png')),

Okay so that exists in https://github.com/mryellow/maze_explorer/tree/master/mazeexp/engine/assets ... Have seen this error before, guess it's a dev/production environment issue.

https://github.com/mryellow/maze_explorer/blob/807cef1e7ceadfa8fe9d4d863ef2aeed989af618/MANIFEST.in#L2

That part should be including the assets files, confirmed they show in the distribution zip file...

I'll see if I can duplicate it and fix the build config.

A work-around which should work would be to clone the gym and environment repo, then pip install -e .. That will link direct to the source and the file will be in the expected path.

I was able to get it working with adding import gym_mazeexplorer to examples/openai_gym.py:

python ~/src/tensorforce_fork/examples/openai_gym.py -m logs/agent001 -s logs/agents/agent001 --monitor-video 100 MazeExplorerEat-v0 -a DQNAgent -c logs/agent.json -n logs/network_flat_lstm.json

agent.json

{
  "preprocessing": null,
  "exploration": {
    "type": "epsilon_anneal",
    "epsilon": 1.0,
    "epsilon_final": 0.01,
    "epsilon_timesteps": 2e6,
    "start_after": 5000
  },

  "batch_size": 64,
  "memory_capacity": 1000000,
  "memory": "prioritized_replay",
  "update_frequency": 4,
  "first_update": 5000,
  "repeat_update": 1,

  "target_update_frequency": 10000,

  "discount": 0.95,
  "learning_rate": 0.0025,
  "optimizer": {
    "type": "rmsprop",
    "momentum": 0.95,
    "epsilon": 0.01
  },
  "tf_saver": true,
  "tf_summary": null,
  "log_level": "info",

  "update_target_weight": 1.0,
  "double_dqn": false,
  "clip_loss": 0.0
}

network_flat_lstm.json

[
    {
        "type": "flatten"
    },
    {
        "type": "dense",
        "size": 64
    },
    {
        "type": "dense",
        "size": 64
    },
    {
        "type": "lstm"
    }
]

Although classic DQN has trouble with the reward function phrased how it is currently (where only apples/poison/walls have rewards but no constant proximity signal like Karpathy used).

Not certain the memory config is making it into agent with current build, complains about Configuration values not accessed: first_update, update_frequency, repeat_update, memory, batch_size, memory_capacity, tf_saver.

[mryellow]

Deployed a new version 0.0.10 which includes the asset files in both the source and binary distributions.

https://github.com/mryellow/maze_explorer/blob/ab8a25ccd05105d2fe57e0213d690cfc07e45827/setup.py#L13

[arsenious]

Thank you for the help! I'll check it out and let you know. I'm working on Deep RL for maze solving and I think this project has great potential as a benchmark environment

[mryellow]

great potential as a benchmark environment

I see the apples/poison thing as being very similar to DeepMind's Labyrinth but with a much smaller input state space, making it good for smaller/faster experiments.

Think really need to reign in the default reward function for benchmark use, something easily learnable which provides meaningful scores or reward graph.

Currently running with -10 on wall collisions, without setting a terminal state (so it keeps running into walls). Which results in a reward graph starting at around -1000 for 500 timesteps and approaching 0 as epsilon exploration gets out of the way. Something which starts at 0 and continues to improve all the way to super-human is probably needed to make it a good benchmark.

Has been handy running it this way for discovering bugs in velocity code or wall collisions, but think need to cut all those "running into a wall" experiences by setting a terminal flag and having more prioritised experiences relating to finding apples rather than avoiding walls.

I'll do a few more experiments between these wall rewards and Karpathy's wall proximity based reward. Aiming for something which can be effectively trained on a classic DQN without A3C (his proximity reward results in smooth forward reward seeking behaviour after 1m experiences, where wall reward can result in a lot of spinning in circles).

Then when it comes to reward functions which are a little more rarely rewarding (like second exploration/escape game mode) there is some A3C testing needed. However I had TF device errors on first attempt running that with TensorForce, not sure what needs changing in the config.