[BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses #246

JackUrb · 2020-09-14T20:14:46Z

Overview

This change is moving Mephisto over from using argparser (and related helpers) to configure its core operations to using Hydra. Hydra will allow for configurable, committable files that allow sharing exact experimental runs with ease. It'll also allow easier access to deep Mephisto configuration options directly from the command line, or through those configuration scripts.

There's no clean way to introduce these changes into Mephisto other than one breaking branch, and as such this PR contains all of the components for migrating Mephisto over. The commits in order roughly follow the Implementation checklist. Most are direct refactors that choose to retain existing behavior where possible, but introduce Hydra to facilitate a complete and total removal of ARG_STRING, Argparse, and all related helpers.

Implementation

Directly translates existing argparse options to dataclasses for use in hydra
Update all classes to reference their related argument-providing dataclasses
Update operator flow to work from a Hydra config, rather than the ARG_LIST
Update each of the examples in turn to have Hydra-based scripts rather than argparse based scripts.
Updates task READMEs to show the correct hydra usage.
Updates all testing to use Hydra code.
Delete all ARG_STRING, argparse, and related code (add_args_to_group, much of argparse_parser, etc).
Creates guide for transitioning from pre-hydra to post hydra

Testing

Tests:

pytest test
====================================================================================== test session starts ======================================================================================
platform darwin -- Python 3.8.3, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/jju/mephisto
plugins: hydra-core-1.0.0, requests-mock-1.7.0
collected 105 items                                                                                                                                                                             

test/test_mephisto.py .                                                                                                                                                                   [  0%]
test/core/test_database.py sssssssssssssssssssssss.......................                                                                                                                 [ 44%]
test/core/test_operator.py ....                                                                                                                                                           [ 48%]
test/core/test_supervisor.py ......                                                                                                                                                       [ 54%]
test/core/test_task_launcher.py ....                                                                                                                                                      [ 58%]
test/providers/mturk/test_mturk.py .                                                                                                                                                      [ 59%]
test/providers/mturk_sandbox/test_mturk_provider.py ssss.....                                                                                                                             [ 67%]
test/server/architects/test_heroku_architect.py sss...                                                                                                                                    [ 73%]
test/server/architects/test_local_architect.py sss...                                                                                                                                     [ 79%]
test/server/architects/test_mock_architect.py sss...                                                                                                                                      [ 84%]
test/server/blueprints/test_mock_blueprint.py ssssssss........                                                                                                                            [100%]

=============================================================== 61 passed, 44 skipped, 10 warnings in 134.81s (0:02:14) ===============================================================

`mephisto register <>` test

>>> mephisto register mock
dest        type    default         help                                        choices    required
----------  ------  --------------  ------------------------------------------  ---------  ----------
name        str     MOCK_REQUESTER  Name for the requester in the Mephisto DB.             True
force_fail  bool    False           Trigger a failed registration                          False
>>> mephisto register mock name=TEST_MOCK_2 force_fail=True
Forced failure test exception was set
>>> mephisto register mock name=TEST_MOCK_2
Registered successfully.
>>> mephisto requesters
  requester_id  provider_type    requester_name    registered
--------------  ---------------  ----------------  ------------
             1  mock             MOCK_REQUESTER    False
             2  mock             TEST_MOCK_2       False

pringshia

Can't wait until this gets shipped!

Most of the comments here are mostly be just being annoying with some nits.

docs/quickstart.md

mephisto/core/operator.py

pringshia · 2020-09-16T18:09:07Z

mephisto/core/operator.py

+            'Operator.parse_and_launch_run_wrapper has been deprecated in favor '
+            'of using Hydra for argument configuration. See the docs TODO in order '
+            'to upgrade.'
+        )


yay for developer friendly UX

mephisto/data_model/blueprint.py

mephisto/utils/scripts.py

examples/parlai_chat_task_demo/parlai_test_script.py

mephisto/core/hydra_config.py

examples/static_react_task/run_task.py

omry · 2020-09-16T20:41:28Z

Awesome!

omry

Awesome.

omry · 2020-09-17T07:43:23Z

docs/quickstart.md

+        access-key-id=$ACCESS_KEY\
+        secret-access-key=$SECRET_KEY


You can access environment variables from the config:

key: ${env:ACCESS_KEY}

See OmegaConf docs.

omry · 2020-09-17T07:48:31Z

examples/parlai_chat_task_demo/README.md

+- `mephisto.task.task_title` - The title of the task showed to workers in their queue
+- `mephisto.task.task_description` - The short description for the task that may be showed inline while they're searching for tasks
+- `mephisto.task.task_tags` - Any searchable tags that would make it easy to find your task
+- `mephisto.task.task_reward` - The reward paid out in dollars for completing your task
+- `mephisto.task.task_name` - This task name will be used to consolidate your data across runs. Generally we suggest using `descriptive-string-pilot-#` when piloting and `descriptive-task-string` for your main collection. More details on this are in the (TODO) Mephisto argument instructions.


You can generate the config node with --cfg job --package mephisto.
I often use those as documentation snippets, try it out to see the output.
Example here.

Unfortunately there is no support for help strings yet.

I've managed pulling out help strings from the dataclass metadata in #249. Might be worth considering pulling help from metadata in Hydra? (of course this relies on being able to get the dataclass itself rather than the config).

omry · 2020-09-17T07:50:21Z

examples/parlai_chat_task_demo/conf/custom_simple.yaml

@@ -0,0 +1,13 @@
+#@package _global_


You can also use #@ package mephisto here to drop one nesting level.

omry · 2020-09-17T07:51:32Z

examples/parlai_chat_task_demo/conf/custom_simple.yaml

+      used to demonstrate the functionalities around using Mephisto 
+      for ParlAI tasks."
+    task_reward: 0.3
+    task_tags: "dynamic,chat,testing"


You can use yaml lists:

task_tags: [dynamic,chat,testing]

or:

task_tags: - dynamic - chat - testing

examples/parlai_chat_task_demo/conf/custom_simple.yaml

omry · 2020-09-17T07:55:03Z

examples/parlai_chat_task_demo/parlai_test_script.py

+defaults = [
+    {"mephisto.blueprint": BLUEPRINT_TYPE},
+    {"mephisto.architect": 'local'},
+    {"mephisto.provider": 'mock'},
+    "conf/base",
+    {"conf": "example"},
+]
+


If you have a scriptconfig.yaml file in your search path you can put the defaults in it in yaml format (which is cleaner). it will get used even if your TestScriptConfig does not have the defaults declared.

The only difficulty here is that BLUEPRINT_TYPE is only loaded up at runtime, as we import it from another class.

omry · 2020-09-17T07:55:53Z

examples/parlai_chat_task_demo/parlai_test_script.py

+@dataclass 
+class TestScriptConfig(RunScriptConfig):
+    defaults: List[Any] = field(default_factory=lambda: defaults)
+    task_dir: str = TASK_DIRECTORY


consider using OmegaConf resolvers for things like that.

Considered, but what is the use case for using a resolve being more advantageous? Sure, in both cases I can use ${task_dir:} but if I actually want to access cfg.task_dir I'll still need task_dir: ${task_dir:} somewhere right?

examples/static_react_task/README.md

JackUrb added 13 commits September 8, 2020 11:30

First data classes

b76b39b

A lot more dataclasses

379c2e8

rest of the dataclasses

7762143

Mephisto check clears

dda5dfc

Getting blueprint args to parse

71db773

Add all the hydra configs, config loading POC

da8557d

Configuration change

39696d2

Operator now launches static HTML

9a56f5e

Static HTML example working again\!

fd22763

Static with onboarding migrated

546a729

React static task works now too

78273d0

ParlAI tasks are now working too

0380caf

Argparse, but for dataclasses

fe1949a

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 14, 2020

JackUrb added 12 commits September 14, 2020 16:40

test_operator progress

ab9972f

Fixing test_task_launcher

92e34d1

test supervisor now works

7adf99d

Operator test runs

843bb9c

Provider has wrong signature for architect setup

7012ba0

Architect tests now working

9ff3f47

Blueprint tests work

d112c5b

Removing argparse parsing

697a991

Fixing mephisto cli

1df0a71

Fixing some architects

1c82cd6

Removing add_args_to_group

2cd3670

Database test accurate param form

9c26b96

JackUrb changed the title ~~[WIP] [BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses~~ [BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses Sep 15, 2020

JackUrb marked this pull request as ready for review September 15, 2020 15:00

JackUrb requested a review from pringshia September 15, 2020 15:00

Getting the scripts to work again

4e46345

JackUrb added 5 commits September 15, 2020 14:33

cleaning readmes

d781980

Updating readmes

7076912

Updating script configuration

333dba3

hydra migration docs

3be2f0a

documentation for readability

ae094c4

pringshia approved these changes Sep 16, 2020

View reviewed changes

Addressing pratik's comments

6982e00

JackUrb merged commit b6161ae into master Sep 16, 2020

JackUrb deleted the dataclasses-hydra branch September 16, 2020 19:52

This was referenced Sep 16, 2020

Hydra quick extra fixes #247

Merged

Cli command for mephisto wut #249

Merged

omry reviewed Sep 17, 2020

View reviewed changes

JackUrb mentioned this pull request Sep 17, 2020

Some additional hydra suggestions #250

Merged

EricMichaelSmith mentioned this pull request Oct 1, 2020

Unable to run turn_annotations_static task facebookresearch/ParlAI#3141

Closed

JackUrb mentioned this pull request Jan 27, 2021

[TODO Roundup] Make Operator more configurable #94

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses #246

[BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses #246

JackUrb commented Sep 14, 2020 •

edited

Loading

pringshia left a comment

pringshia Sep 16, 2020

omry commented Sep 16, 2020

omry left a comment

omry Sep 17, 2020

omry Sep 17, 2020

JackUrb Sep 17, 2020 •

edited

Loading

omry Sep 17, 2020

omry Sep 17, 2020

omry Sep 17, 2020

JackUrb Sep 17, 2020

omry Sep 17, 2020

JackUrb Sep 17, 2020 •

edited

Loading

[BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses #246

[BREAKING CHANGES] Argument Configuration with Hydra and Dataclasses #246

Conversation

JackUrb commented Sep 14, 2020 • edited Loading

Overview

Implementation

Testing

Tests:

mephisto register <> test

pringshia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omry commented Sep 16, 2020

omry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackUrb Sep 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackUrb Sep 17, 2020 • edited Loading

Choose a reason for hiding this comment

JackUrb commented Sep 14, 2020 •

edited

Loading

`mephisto register <>` test

JackUrb Sep 17, 2020 •

edited

Loading

JackUrb Sep 17, 2020 •

edited

Loading