Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External test data #1557

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Conversation

HeroicKatora
Copy link
Member

@HeroicKatora HeroicKatora commented Sep 23, 2021

This is a draft for a system that would allow external packagers to run our test suite based on the published packages, without requiring us to add that test data into the packages. This is based on a recent idea of mine which I published as xtest-data. Comment of all sorts welcome on the approach.

I'm hoping to quickly arrive at a stable version—maybe even 1.0—of the linked crate by prototyping it with image or its sub-crates. These should be a decent reflection of real world testing and we certainly have a relatively heavy test suite in terms of file sizes. One particular pain point which I haven't been able to integrate yet would be to move data out-of-tree altogether but I imagine this to be feasible with submodules—indeed, submodules should already work without requiring everyone to download all data—but more interesting would be git lfs etc.

How to run the tests

  • Manually package the crate: cargo package --no-verify
  • Unpack the crate at some location:
    gunzip -c target/package/image-0.23.14.crate | tar -C /tmp --extract --file -
    cd /tmp/image-0.23.14
  • Do the dance for the crate (note that only reference_images works atm):
    # Note the error output here, no network connection yet.
    cargo test --tests check_references -- --nocapture
    # Instruct the crate to auto-setup the directories
    CARGO_XTEST_DATA_FETCH=1 cargo test --tests check_references -- --nocapture

Overall the simplifications seem to work out nicely. A bit of code and
distracting imports removed.
The publicly available version of this has a lock issue that I know
about and has been fixed in a local dependency version. At least that's
what I'm trying to test out with this bump and a local patch.crates-io
section.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant