v1.4.0

Overview

In this release we added Data Porter feature that allows to move full/partial data dumps between different isntances of Mephisto. This enables things like:

• backups/restores of Mephisto data
• consolidating task data across different Mephisto instances
• sharing task data with your colleagues

We also introduced a simple (forward-only) workflow for database migrations, enabled custom code snippets in Form Composer, and updated numerous library versions.

Last but not least, mephisto.ai documentation got a major update as well.

Included PRs

• Bump pyarrow from 9.0.0 to 14.0.1 by @dependabot in #1083
• Bump @cypress/request and cypress in /examples/form_composer_demo/webapp by @dependabot in #1106
• Bump @cypress/request and cypress in /packages/react-form-composer by @dependabot in #1105
• Bump ip from 2.0.0 to 2.0.1 by @dependabot in #1103
• Bump aiohttp from 3.8.6 to 3.9.2 by @dependabot in #1101
• Bump follow-redirects from 1.15.2 to 1.15.4 by @dependabot in #1094
• Bump pillow from 10.0.1 to 10.2.0 by @dependabot in #1099
• Fixing agent desync on RemoteProcedure tasks by @JackUrb in #1063
• Mock architect actions recording by @JackUrb in #1064
• Script to write out qualifications by @JackUrb in #1069
• Enabled custom code insertions for FormComposer by @meta-paul in #1108
• Display qualifications granted to a worker in TaskReview app by @meta-paul in #1127
• Update mephisto.ai documentation by @meta-paul in #1128
• Data porter feature by @meta-paul in #1153
• Bump webpack-dev-middleware from 5.3.3 to 5.3.4 by @dependabot in #1130
• Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /mephisto/review_app/client by @dependabot in #1129
• Bump xmltodict from 0.12.0 to 0.13.0 by @dependabot in #1126
• Bump @babel/core from 7.17.7 to 7.24.0 by @dependabot in #1118
• Bump follow-redirects from 1.15.2 to 1.15.6 in /examples/static_react_task/webapp by @dependabot in #1116
• Bump actions/setup-python from 2 to 5 by @dependabot in #1113
• Bump actions/setup-node from 2 to 4 by @dependabot in #1111
• Bump dorny/paths-filter from 2 to 3 by @dependabot in #1109
• Bump react-shadow from 19.1.0 to 20.4.0 by @dependabot in #1121
• Bump follow-redirects from 1.15.5 to 1.15.6 in /mephisto/review_app/client by @dependabot in #1114
• Bump follow-redirects from 1.15.5 to 1.15.6 in /packages/mephisto-task-multipart by @dependabot in #1112
• Bump react-player from 2.9.0 to 2.16.0 by @dependabot in #1148
• Bump @storybook/addon-links from 6.5.16 to 8.0.10 by @dependabot in #1151
• Bump aiohttp from 3.9.2 to 3.9.4 by @dependabot in #1146
• Bump idna from 3.4 to 3.7 by @dependabot in #1143
• Bump ejs from 3.1.9 to 3.1.10 in /mephisto/review_app/client by @dependabot in #1150
• Bump follow-redirects from 1.15.4 to 1.15.6 by @dependabot in #1154
• Bump @storybook/react from 6.5.16 to 8.0.10 by @dependabot in #1152
• Bump ejs from 3.1.8 to 3.1.10 by @dependabot in #1149
• Bump mypy from 0.971 to 1.10.0 by @dependabot in #1147
• Bump tar from 6.1.13 to 6.2.1 by @dependabot in #1142
• Added unittests for Data Porter command options by @meta-paul in #1156
• Bump codecov/codecov-action from 3 to 4 by @dependabot in #1115
• Bump express from 4.18.2 to 4.19.2 by @dependabot in #1135
• Bump pyyaml from 5.4.1 to 6.0.1 by @dependabot in #1125
• Bump express from 4.18.3 to 4.19.2 in /mephisto/review_app/client by @dependabot in #1136
• Bump pdoc from 10.0.4 to 14.4.0 by @dependabot in #1123

Full Changelog: v1.3.0...v1.4.0

v1.3.0

Overview

This release introduces FormComposer task generator. It aims to simplify creating React front-end for form-based Tasks. Main highlights:

• Task front-end can be defined via JSON configuration (no need for custom code)
• Multiple form versions are supported within a single Task
• Multi-version forms can be generated within the same Task, based on variables' values
• Expiring (presigned) URLs are supported for S3 files (for data safety)
• Task Review app support includes preview of uploaded media files

Included PRs

• Mephisto v1.3 by @meta-paul in #1102

Full Changelog: v1.2.1...v1.3.0

v1.2.1

Overview

This release rounded out TaskReview app functionality. Now you can conveniently give feedback and bonuses to individual workers while performing review of their submissions. This feature works with all currently supported providers.

Included PRs

• V1.2.1 dev by @meta-paul in #1100

Full Changelog: v1.2.0...v1.2.1

Mephisto 1.2.0

Overview

This release is focused on adding a new provider (Prolific) and better data review experience.
It has a larger diff than usual, and we will return to a more frequent release cadence afterwards.

See the list of changes when upgrading from v1.1.0 below. Note that only essential dependency updates are listed.

Big Changes

• Added integration with Prolific provider, as an alternative to Mturk for data collection. Expect some minor additions in the subsequent releases, but the core functionality is already there. #1008
• Added UI-driven review of collected task results, a major upgrade over the previously used command-line script. Support for bonuses and feedback for workers to be added in the subsequent releases. When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #1058
• Added dockerization support for running Mephisto. #1008

Included PRs

Feature + UX Improvements

• Getting complete MTurk pay for individual units by @JackUrb in #446
• Adding gold units and functionality by @JackUrb in #923
• including hydra_configs in package (for 1.1.1) by @JackUrb in #942
• Adding missing hook registration for MTurk Sandbox wrap_crowd_source by @JackUrb in #987
• Reorganizing pyproject.toml to use Extras over Groups by @JackUrb in #1002
• Added AWS credentials pre-commit hook by @meta-paul in #1018
• Update heroku_architect.py by @JackUrb in #1025
• ec2 architect uninterruptable shutdown by @JackUrb in #1023
• Permissive socket upload by @JackUrb in #1026
• Adding auth implementation of prolific client by @JackUrb in #1028
• Add Prolific provider (#1008) by @meta-paul in #1012
• Improve logger in Python code by @meta-paul in #1045
• Laying down basic strokes of standard get_data response by @JackUrb in #1065
• Prolific workspace management by @JackUrb in #1067
• Update heroku_architect.py by @JackUrb in #1062
  1076
• Create a reusable app for task review by @meta-paul in #1058
• Prevent oversampling of study submissions by any single worker by @meta-paul in #1080

Bugfixes

• Fixed Static task to not explicitly check for objData.file1 by @gzhihongwei in #966
• Refactoring QA from unit assign to use in onboarding by @JackUrb in #960
• dropped the self from cant_cancel_expirations by @arjunan-k in #971
• Fixing missing events in wrap_crowd_source.js by @JackUrb in #988
• InaccessiblePrometheusServer Exception by @mojtaba-komeili in #1016
• Updated error message to manually delete EC2 server by @gzhihongwei in #1007
• quick fix: change order of checks in operator by @PReithofer in #1039
• Small fixes from local branch by @JackUrb in #1054
• fix race condition in generate_units by @PReithofer in #1056

Docs + Examples

• Corrected documentation inconsistencies and typos by @gzhihongwei in #965
• Adding citation note to readme by @JackUrb in #989
• Fix typo in operations/README.md by @eltociear in #992
• Add additional python versions to test by @pringshia in #925
• fix rendering of assignment url for mock tasks by @edwardguo61 in #1005
• New testing option for ec2 (reuse existing server) by @JackUrb in #1036
• Small typo fix for running the static_test_script by @benmatselby in #1019
• Add unittests for Prolific integration by @meta-paul in #1046
• running black across codebase by @JackUrb in #1043
• Add unittests for Prolific integration (2) by @meta-paul in #1047
• Updated Prolific integrations docs by @meta-paul in #1049
• Clean up TODO comments by @meta-paul in #1050
• Update copyright headers across the codebase by @meta-paul in #1075

Dependencies

• Update to React 18 by @pringshia in #905
• Upgrading codecov version by @JackUrb in #1027

New Contributors

• @gzhihongwei made their first contribution in #965
• @arjunan-k made their first contribution in #971
• @eltociear made their first contribution in #992
• @edwardguo61 made their first contribution in #1005
• @benmatselby made their first contribution in #1019
• @meta-paul made their first contribution in #1018
• @PReithofer made their first contribution in #1039

Full Changelog: v1.1.0...v1.2.0

Mephisto 1.1.0

Overview

This release contains a number of Stability and Quality of Life improvements from v1.0.0 and v1.0.2. The big focus has been on collection success rate, documentation, and various bugfixes.

For those updating from 1.0, see the 1.0.2 changes as well for a complete changelog.

Breaking changes

• Mephisto no longer supports Python 3.7. This is as of #898, wherein to get the ParlAI peer imports working we needed to bump to ParlAI 1.7.0, which no longer supports Python 3.7.

Big Changes

• When launching on MTurk, Mephisto will request that you run the mephisto scripts mturk cleanup script inline before launching. This exercise prevents having long-running tasks that haven't been cleaned up properly. #918
• When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #919

Big Fixes

• Lots of documentation, thanks @Etesam913!
• Significant fixes to proper execution and state tracking in Mephisto, which should improve stability across all job types (#875, #878, #902, #922)
• Major fix for Heroku tasks using the useMephistoLiveTask hook, which was causing all messages to be double sent in both directions. (#921)

Included PRs

Feature + UX Improvements

• Enhance logging messages with rich by @Etesam913 in #869
• Added ability to customize html-static-task-example submit button by @Etesam913 in #879
• Adds ability to clear onboarding for a worker by @Etesam913 in #886
• Adding more general function with better name by @mojtaba-komeili in #893
• Making MTurk cleanup run daily per requester by @JackUrb in #918
• 12 hour inactivity shutdown by @JackUrb in #919

Bugfixes

• [Bugfix] Fixing double update bug on heroku by @JackUrb in #921
• Fixing Worker Pool race condition for assignments by @JackUrb in #875
• ParlAI Assignment Fixes pt 2 by @JackUrb in #878
• Resolving static return status bug by @JackUrb in #902
• Make self.metadata backward-compatible by @chiehminwei in #861
• Finding OSTYPE if not exists by @mojtaba-komeili in #892
• fix breaks in create_validation_function for screening units by @macota in #864
• Fix mephisto web cli command by @Etesam913 in #865
• Removing threadsafety as we already lock connections behind threads by @JackUrb in #897
• Extending some 1.1 changes by @JackUrb in #922

Testing & DevX

• Add path filter by @Etesam913 in #868
• Add cypress test to simple static task by @Etesam913 in #881
• fix #874: Upgrade pip and create main_data_directory in Dockerfile by @mjkanji in #917
• pre-commit install by @mojtaba-komeili in #894
• Some fixes made to run a live demo of Mephisto functionality by @JackUrb in #909

Docs + Examples

• Adds cypress section to frontend docs by @Etesam913 in #860
• Adds tips/feedback section to docs by @Etesam913 in #838
• Add tests to tips example by @Etesam913 in #833
• Add searchbar to docs by @Etesam913 in #858
• Update searchbar colors by @Etesam913 in #863
• Auto Generate Blueprint Documentation by @Etesam913 in #870
• Auto Generate Architect Documentation by @Etesam913 in #871
• Auto Generate Requester Documentation by @Etesam913 in #872
• Auto Generate Provider Documentation by @Etesam913 in #873
• Update deploy docs github action to auto-generate docs by @Etesam913 in #877
• Add Screening Example by @Etesam913 in #883
• Creates a readme for simple_static_task by @Etesam913 in #882
• Improve onboarding example by @Etesam913 in #887
• Add info on link_task_source to docs by @Etesam913 in #885
• Update documentation to reflect that make_qualification_dict takes 3 arguments. by @xksteven in #908

Dependencies

• Bump terser from 5.12.1 to 5.14.2 in /docs/web by @dependabot in #853
• Update @blueprintjs/core version by @pringshia in #876
• Bump oauthlib from 3.2.0 to 3.2.1 by @dependabot in #901
• Bump protobuf from 3.19.4 to 3.19.5 by @dependabot in #903
• Bump joblib from 1.1.0 to 1.2.0 by @dependabot in #904
• Upgrading to poetry 1.2, resolving dependency issues by @JackUrb in #898
• Upgrade yarn 3.2.1->3.2.4 by @pringshia in #920

New Contributors

• @chiehminwei made their first contribution in #861
• @mjkanji made their first contribution in #917

Full Changelog: v1.0.3...v1.1.0

v1.0.3: Last version, but ParlAI-stable

Overview

v1.0.2 shipped with a bug that broke ParlAI-chat related tasks. This release only includes the fix for that (#859), but is otherwise equivalent to v1.0.2.

v1.0.2: Key Bugfixes and QoL Upgrades

Overview

Some issues still remaining from v1.0.0 (and even sooner) have been ironed out, so this release aims to fix those as well as include some new QoL upgrades.

Highlights

• A status mismatch issue related to collection slowdowns on long-running tasks has been squashed (#830)
• Adding testing for our example files (#795, #800)
• Upgrading completely to Poetry as our package manager (#779)
• Core work for new worker addons package, to be included in coming release notes (#791, #785)

Additional Changes

• MTurk improvements and fixes:
  • #810: Functionality to specify the lifetime of a task
  • #780: Fix to HIT tags that were being improperly submitted as individual characters
  • #767: Fix to storing and restoring AWS credentials
• Bugfix for deploying to the HerokuArchitect via git (#824)
• Locking down the version of Node that our router uses to solve issues deploying on the EC2Architect (#762)
• Cleanup to the Operator.shutdown method to allow disconnected onboardings to be cleared (#769)
• Developer QoL:
  • #814: Consolidating the AgentState classes and introducing a metadata attribute
  • #770: Making Agent and OnboardingAgent pull from the same base class for code deduplication
• Bugfixes to the RemoteProcedureBlueprint:
  • #763: Fixing an issue that prevented reloading from restoring the state
  • #798, #782: Cleaning up on the RemoteProcedure examples
  • #784: Adding a new RemoteProcedure example using a text model-in-the-loop

1.0.1 Release: minor 1.0.0 Bugfixes

Overview

Following the 1.0.0 release, a few bugs were surfaced. The largest involved an update of mephisto-task already published, and this release syncs the underlying Mephisto library to refer to the same version.

Full Changelog

• #728 Fixes an issue preventing live tasks from working over https
• #749 Adds back some missing mephisto-task package versions that were preventing correct builds at runtime
• #756 Fixes a small bug in the HTMLStaticBlueprint
• #745 Clarifying correct typing for data passed via handleSubmit in our StaticBlueprint tasks
• #754 fixes docker imaging issue if mephisto metrics install has already been run locally
• #747 Set some defaults for max_num_concurrent_tasks to encourage people to use this option more consistently

Mephisto 1.0.0 Release

This PR captures all of the work the team has done for Mephisto up to now, setting our first stable API, prioritizing user-friendly features, and providing a massively updated documentation site. Read more about this release on the blog.

EC2 Architect, Documentation, and Various Bugfixes

Overall this update doesn't change any existing semantics for Mephisto, but includes some new functionality (including launches on EC2 instead of Heroku!) and some small bugfixes. In total, this release contains the following:

• EC2 architect and followup fixes (#603, #606, #609, #618, #622)
• Documentation overhaul (#611, #612, #613)
• Early implementation for using Gold labelling worker quality control (#585)
• Bugfixes for existing blueprints (#588, #594, #637)
• Various lint and dependency updates