An end-to-end testing framework for A-Frame, built on Playwright
anime-test-example-vr-video.mp4
The following instructions are suitable if you have an existing project that you want to write tests for, and
- You want to maintain the tests in the same repo as the project
- You have a folder
node_modules
at the root of your project (if you don't, either create it, or usenpm init
to set it up) - You are happy for the test scripts to be stored in a folder names
tests
at the root of the project. - You don't already have a
playwright.config.js
file in your root directory.
If this is all the case, you can install as follows:
npm install --save-dev aframe-e2e-testing
This will install all required dependencies, including playwright & playwright browser binaries. When it completes, you should see this message.
==================
aframe-e2e-testing
==================
Welcome to aframe-e2e-testing
Starter configuration has been created in playwright.config.js
Two sample scripts can be found in tests/examples/
To check everything is working, run: npx playwright test
or (for headed mode): npx playwright test --headed
For more details see the README.md file in aframe-e2e-testing
Happy Testing!
==================
Now you can run npx playwright test
or npx playwright test --headed
to see the example scripts running.
To create some scripts for your project...
-
Copy one or both of the example scripts to e.g.
/tests/my_basic_tests/
-
Modify this line to point to a suitable URL
await page.goto('https://aframe.io/aframe/examples/showcase/ui/');
for local development, this is probably something like this:
await page.goto('http://localhost:8080/');
(you can also set up a base URL in
playwright.config.js
) -
Modify the script itself to interact with the entities on your page. See the A-Frame Utilities API for details on what you can do.
-
You'll probably want to delete the example scripts - or you could keep them for reference, and skip running them by adding a testIgnore setting in
playwright.config.js
:testIgnore: '**/examples/**',
If you want your tests somewhere other than /tests
just move the files, and update the value of testDir
in playwright.config.js
If you had a pre-existing playwright-config.js
file in your root directory, or pre-existing clashing filenames in tests/examples/
, they won't have been overwritten. You can get copy the default playwright config and sample scripts from GitHub, and manually set them up wherever you want.
For more details, including config, trace & debugging options please refer to the playwright docs
As you write tests, you may find you need to add or modify the A-Frame Utilities
The currently recommended way to do this is to:
-
fork the
aframe-e2e-testing
repo and clone it locally as a sub-module of your project, like this (filling in your username to point to the correct fork)git submodule add https://github.com/<username>/aframe-e2e-testing
or clone a specific branch like this...
git submodule add -b <branch> https://github.com/<username>/aframe-e2e-testing
-
in the root directory of your project, run
npm install --save-dev .\aframe-e2e-testing
You should now be able to
- run tests using
npx playwright test
- modify your fork of
aframe-e2e-testing
in a version-controlled manner, with changes usable by test scripts in the parent project.
When you, or someone else, clones your project, they'll also clone the correct modified version of aframe-e2e-testing
If you do make changes to aframe-e2e-testing
please consider submitting a PR to incorporate them upstream, so that other users can benefit from them.
This repository includes some example tests, which run against various A-Frame examples.
After cloning the repository, run:
npm install
npx playwright install
Limit your changes in the aframe-e2e-testing
submodule to changes that you want to share upstream. Test scripts and utilities that are specific to your project should be developed in your main repository, outside this sub-module.
You may wish to keep your test scripts in a separate repository from your product code.
In this case, you could do this by simply forking the aframe-e2e-testing
repo, and developing your scripts directly there.
However, that will make it hard to share any updates to A-Frame Utilities upstream, as there's no clear separation between the test scripts you've developed, and any potentially re-usable updates that you have made to A-Frame Utilities.
So, even if you want test scripts to be maintained in a separate repository from your code, it's still recommended to follow the approach described above in "Get Started", to keep project-specific code separate from re-usable code.
These example scripts include some pauses for obervability / demonstration reasons when running in headed mode, but I have been careful to put these pauses in places where the script is not dependent on them for successful execution. In most cases it is bad practice to depend on pauses in test scripts for them to run reliably - see expect.toReturn()
below for an alternative to pauses in test scripts.
This repository contains a range of utility functions that are specifically helpful for test automation of A-Frame applications. It also contains guidelines for extensions to this API for either generic or application-specific purposes.
API Documentation can be found here: https://diarmidmackenzie.github.io/aframe-e2e-testing/out/
To generate the documentation:
From the root directory, run: npx jsdoc -R src/README.md src
We have a few useful extensions to the Expect library.
expect(x).toBeApprox(value, dps)
- value: the value to check against
- dps (optional, default: 9): the number of decimal places to check.
Useful for comparing positions or rotations that may have minor floating point deviations.
expect(x).toBeNakedId()
Checks that a supplied entity identifier is a naked id, and does not begin with #. Useful for input checking on functions that expect an entity Id, rather than a selector.
expect(x).toReturn(object, options)
- x: should be a function that returns an object. Can be an async function.
- object: a set of key/value pairs that are expected to be returned by the function.
- options: an object containing key/value pairs specifying additional matching options:
- debug (default: false): output logs to help debug unexpected results.
- timeout (default: 5000): the number of msecs to wait for the function to return matching values.
- retry (default: 200), the number of msecs to wait between one call of the function and the next.
- dps (default: -1, which implies a perfect match), how many DPs of accuracy to check for.
This function will call the supplied function x repeatedly, based on the retry setting, until it returns all the key/value pairs specified in the object - or until a timeout is reached.
This can be extremely useful in test scripts where you want to check that something updates to a new value, but you don't know exactly when it will happen (e.g. because it depends on network replication from another client, or depends on raycasting which typically runs on a timer and therefore is not instantaneous).
Example usage:
var getMaterial = async () => await A.getEntityMaterial("#box")
await expect(getMaterial).toReturn({color: A.color(0xffffff)})
Using expect.toReturn() is strongly recommended over padding your scripts with additional timers, which tend to lead to flaky, heard-to-main test scripts that don't run as quickly as they could.
Playwright is a modern framework for end-to-end testing in the browser, with a wide range of advantages.
While selenium/web driver still have many strong advocates, more modern tools such as Playwright & Cypress provide an easier learning curve, and are being widely adopted in the industry, with a great deal of success.
Unfortunately Cypress has a significant limitation that a single test cannot run against multiple browser instances, which means that Cypress cannot straightforwardly test multi-user experiences.
Playwright does not have this limitation, which is why it was selected as the base for this framework.
The test framework in this repository was originally developed in collaboration with Virtonomy (https://virtonomy.io).
I am grateful to them for giving me permission to release these components as Open Source under the MIT License.
VR tests currently only run against Chromium. Firefox / Webkit not yet supported for VR tests.
Non-VR tests typically work against all browsers.
If you hit problems, please raise an issue here.
In particular, making this code available as an npm module is brand new, and there may be some teething issues in this area - so if you hit problems please do let me know & I'll try to help.