Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial/beta version of speech recognition plugin. #117

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

Bankminer78
Copy link
Contributor

@Bankminer78 Bankminer78 commented Jul 21, 2024

This is a PR to begin work on a speech-recognition plugin for jsPsych.

An initial version of the speech recognition plugin has been created. It uses Huggingface's Transfomer.js module to import a speech recognition model based on OpenAI's whisper-tiny. Due to build issues, the import of the transformer.js module and the subsequent inference is done through a direct eval() method. The plugin outputs as trial data the transcript of the audio, and the timestamp of the first utterance of the words in the choices parameter (if recognized). The following is a list of things to do next:

  • Change logic so that only the first utterance of any of the words in choices is sent out and not just the first utterance of the first words by order as they have been set as a parameter.
  • Change the use of eval to avoid security issues
  • See if a build with the Transformers.js library is possible.
  • Set up more graceful model caching.
  • Add more items to this list.

An example file with functionality has been added to the plugin.

Copy link

changeset-bot bot commented Jul 21, 2024

⚠️ No Changeset found

Latest commit: 94cc98d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants