-
Notifications
You must be signed in to change notification settings - Fork 28.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_runner: support programmatically running --test
#44241
Conversation
CC @nodejs/test_runner |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm in favor (though still not in full capacity to engage) - this is also still missing tests, would be useful to test concurrent invocations
6a2fd8f
to
b52fccb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, wondering if the API of "provide a stream for output and return a promise for undefined" is what users would want/expect.
Additionally there should be a way to get more than one failure probably?
what other alternatives are there? we can return a stream synchronously but then test failures won't throw/reject - they will simply be reported in the stream
in such a case - the failure describes the failed subtests |
recapping a conversation I had with @benjamingr:
@nodejs/test_runner WDYT? |
LGTM! |
This comment was marked as outdated.
This comment was marked as outdated.
PR-URL: #44241 Fixes: #44023 Fixes: #43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
This needs to be backported to v16.x |
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#TBD Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#44873 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#44873 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#44873 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#44873 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#44873 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs#44241 Backport-PR-URL: nodejs#44873 Fixes: nodejs#44023 Fixes: nodejs#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: #44241 Backport-PR-URL: #44873 Fixes: #44023 Fixes: #43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [node](https://github.com/nodejs/node) | stage | minor | `16.18.1-bullseye` -> `16.19.0-bullseye` | --- ### Release Notes <details> <summary>nodejs/node</summary> ### [`v16.19.0`](https://github.com/nodejs/node/releases/tag/v16.19.0): 2022-12-13, Version 16.19.0 'Gallium' (LTS), @​richardlau [Compare Source](nodejs/node@v16.18.1...v16.19.0) ##### Notable Changes ##### OpenSSL 1.1.1s This update is a bugfix release and does not address any security vulnerabilities. ##### Root certificates updated to NSS 3.85 Certificates added: - Autoridad de Certificacion Firmaprofesional CIF [`A626340`](nodejs/node@A62634068) - Certainly Root E1 - Certainly Root R1 - D-TRUST BR Root CA 1 2020 - D-TRUST EV Root CA 1 2020 - DigiCert TLS ECC P384 Root G5 - DigiCert TLS RSA4096 Root G5 - E-Tugra Global Root CA ECC v3 - E-Tugra Global Root CA RSA v3 - HiPKI Root CA - G1 - ISRG Root X2 - Security Communication ECC RootCA1 - Security Communication RootCA3 - Telia Root CA v2 - vTrus ECC Root CA - vTrus Root CA Certificates removed: - Cybertrust Global Root - DST Root CA X3 - GlobalSign Root CA - R2 - Hellenic Academic and Research Institutions RootCA 2011 ##### Time zone update to 2022f Time zone data has been updated to 2022f. This includes changes to Daylight Savings Time (DST) for Fiji and Mexico. For more information, see <https://mm.icann.org/pipermail/tz-announce/2022-October/000075.html>. ##### Other Notable Changes - \[[`33707dcd03`](nodejs/node@33707dcd03)] - **dgram**: add dgram send queue info (theanarkh) [#​44149](nodejs/node#44149) Dependency updates: - \[[`3b2b70d792`](nodejs/node@3b2b70d792)] - **deps**: upgrade npm to 8.19.3 (npm team) [#​45322](nodejs/node#45322) Experimental features: - \[[`1e0dcd1ee0`](nodejs/node@1e0dcd1ee0)] - **cli**: add `--watch` (Moshe Atlow) [#​44366](nodejs/node#44366) - \[[`8c73279ebb`](nodejs/node@8c73279ebb)] - **util**: add default value option to parsearg (Manuel Spigolon) [#​44631](nodejs/node#44631) ##### Commits - \[[`bbef3c42f6`](nodejs/node@bbef3c42f6)] - **build**: add version info to timezone update PR (Darshan Sen) [#​45021](nodejs/node#45021) - \[[`cc2c7648e0`](nodejs/node@cc2c7648e0)] - **build**: support Python 3.11 (Luigi Pinca) [#​45191](nodejs/node#45191) - \[[`ac24c80663`](nodejs/node@ac24c80663)] - **build**: remove redundant condition from common.gypi (Richard Lau) [#​45076](nodejs/node#45076) - \[[`03dcbe3030`](nodejs/node@03dcbe3030)] - **build**: fix bad upstream merge (Stephen Gallagher) [#​44642](nodejs/node#44642) - \[[`1e0dcd1ee0`](nodejs/node@1e0dcd1ee0)] - **cli**: add `--watch` (Moshe Atlow) [#​44366](nodejs/node#44366) - \[[`96d131665e`](nodejs/node@96d131665e)] - **cluster**: use inspector utils (Moshe Atlow) [#​44592](nodejs/node#44592) - \[[`704836033a`](nodejs/node@704836033a)] - **crypto**: update root certificates (Luigi Pinca) [#​45490](nodejs/node#45490) - \[[`5a776d4a69`](nodejs/node@5a776d4a69)] - **deps**: update timezone to 2022f (Richard Lau) [#​45613](nodejs/node#45613) - \[[`3b2b70d792`](nodejs/node@3b2b70d792)] - **deps**: upgrade npm to 8.19.3 (npm team) [#​45322](nodejs/node#45322) - \[[`9fbc8b21db`](nodejs/node@9fbc8b21db)] - **deps**: update corepack to 0.15.1 (Node.js GitHub Bot) [#​45331](nodejs/node#45331) - \[[`87e3d002ca`](nodejs/node@87e3d002ca)] - **deps**: update corepack to 0.15.0 (Node.js GitHub Bot) [#​45235](nodejs/node#45235) - \[[`e972ff7b13`](nodejs/node@e972ff7b13)] - **deps**: V8: backport [`bbd800c`](nodejs/node@bbd800c6e359) (Chengzhong Wu) [#​44947](nodejs/node#44947) - \[[`af9d8217c0`](nodejs/node@af9d8217c0)] - **deps**: V8: cherry-pick [`b953542`](nodejs/node@b95354290941) (Chengzhong Wu) [#​44947](nodejs/node#44947) - \[[`38202d321b`](nodejs/node@38202d321b)] - **deps**: update undici to 5.12.0 (Node.js GitHub Bot) [#​45236](nodejs/node#45236) - \[[`7c0da6adf9`](nodejs/node@7c0da6adf9)] - **deps**: update archs files for OpenSSL-1.1.1s (RafaelGSS) [#​45274](nodejs/node#45274) - \[[`1149ead6f7`](nodejs/node@1149ead6f7)] - **deps**: upgrade openssl sources to OpenSSL\_1\_1\_1s (RafaelGSS) [#​45274](nodejs/node#45274) - \[[`cd54bce4f5`](nodejs/node@cd54bce4f5)] - **deps**: update timezone (Node.js GitHub Bot) [#​44950](nodejs/node#44950) - \[[`2901abe4f0`](nodejs/node@2901abe4f0)] - **deps**: update undici to 5.11.0 (Node.js GitHub Bot) [#​44929](nodejs/node#44929) - \[[`c80cf97033`](nodejs/node@c80cf97033)] - **deps**: update corepack to 0.14.2 (Node.js GitHub Bot) [#​44775](nodejs/node#44775) - \[[`33707dcd03`](nodejs/node@33707dcd03)] - **dgram**: add dgram send queue info (theanarkh) [#​44149](nodejs/node#44149) - \[[`c708d9bb94`](nodejs/node@c708d9bb94)] - **doc**: fix typo in parseArgs default value (Tobias Nießen) [#​45083](nodejs/node#45083) - \[[`5a0efa05d2`](nodejs/node@5a0efa05d2)] - **node-api**: handle no support for external buffers (Michael Dawson) [#​45181](nodejs/node#45181) - \[[`db31de634e`](nodejs/node@db31de634e)] - **readline**: refactor to avoid unsafe regex primordials (Antoine du Hamel) [#​43475](nodejs/node#43475) - \[[`fbc52e5729`](nodejs/node@fbc52e5729)] - **src**: disambiguate terms used to refer to builtins and addons (Joyee Cheung) [#​44135](nodejs/node#44135) - \[[`953072d3db`](nodejs/node@953072d3db)] - **src**: let http2 streams end after session close (Santiago Gimeno) [#​45153](nodejs/node#45153) - \[[`54608d8dc3`](nodejs/node@54608d8dc3)] - **src**: split property helpers from node::Environment (Chengzhong Wu) [#​44056](nodejs/node#44056) - \[[`6733556783`](nodejs/node@6733556783)] - **test**: add test to validate changelogs for releases (Richard Lau) [#​45325](nodejs/node#45325) - \[[`821d832cef`](nodejs/node@821d832cef)] - **test**: mark test-watch-mode\* as flaky on all platforms (Pierrick Bouvier) [#​45049](nodejs/node#45049) - \[[`02a18eac69`](nodejs/node@02a18eac69)] - **test**: fix test-runner-inspect (Moshe Atlow) [#​44620](nodejs/node#44620) - \[[`197df63f74`](nodejs/node@197df63f74)] - **test**: add a test to ensure the correctness of timezone upgrades (Darshan Sen) [#​45299](nodejs/node#45299) - \[[`42e9d8016a`](nodejs/node@42e9d8016a)] - **test**: fix textdecoder test for small-icu builds (Richard Lau) [#​45225](nodejs/node#45225) - \[[`6d736a56d8`](nodejs/node@6d736a56d8)] - **test**: fix watch mode test flake (Moshe Atlow) [#​44739](nodejs/node#44739) - \[[`543d3d2bf3`](nodejs/node@543d3d2bf3)] - **test**: deflake watch mode tests (Moshe Atlow) [#​44621](nodejs/node#44621) - \[[`97f6caf4eb`](nodejs/node@97f6caf4eb)] - **test**: split watch mode inspector tests to sequential (Moshe Atlow) [#​44551](nodejs/node#44551) - \[[`499750ff7a`](nodejs/node@499750ff7a)] - **test**: update list of known globals (Antoine du Hamel) [#​45255](nodejs/node#45255) - \[[`64d343af74`](nodejs/node@64d343af74)] - **test_runner**: support using `--inspect` with `--test` (Moshe Atlow) [#​44520](nodejs/node#44520) - \[[`99ee5e484d`](nodejs/node@99ee5e484d)] - **test_runner**: fix `duration_ms` to be milliseconds (Moshe Atlow) [#​44450](nodejs/node#44450) - \[[`37e909251c`](nodejs/node@37e909251c)] - **test_runner**: support programmatically running `--test` (Moshe Atlow) [#​44241](nodejs/node#44241) - \[[`0ae5694f88`](nodejs/node@0ae5694f88)] - **tools**: update certdata.txt (Luigi Pinca) [#​45490](nodejs/node#45490) - \[[`891368cefd`](nodejs/node@891368cefd)] - **tools**: remove faulty early termination logic from update-timezone.mjs (Darshan Sen) [#​44870](nodejs/node#44870) - \[[`543493c242`](nodejs/node@543493c242)] - **tools**: fix timezone update tool (Darshan Sen) [#​44870](nodejs/node#44870) - \[[`c77f660b75`](nodejs/node@c77f660b75)] - **tools**: fix `create-or-update-pull-request-action` hash on GHA (Antoine du Hamel) [#​45166](nodejs/node#45166) - \[[`58c30dd049`](nodejs/node@58c30dd049)] - **tools**: update gr2m/create-or-update-pull-request-action (Luigi Pinca) [#​45022](nodejs/node#45022) - \[[`749a4b3e5e`](nodejs/node@749a4b3e5e)] - **tools**: use Python 3.11 in GitHub Actions workflows (Luigi Pinca) [#​45191](nodejs/node#45191) - \[[`6f541d99a5`](nodejs/node@6f541d99a5)] - **tools**: have test-asan use ubuntu-20.04 (Filip Skokan) [#​45581](nodejs/node#45581) - \[[`e7ed56f501`](nodejs/node@e7ed56f501)] - **tools**: make license-builder.sh comply with shellcheck 0.8.0 (Rich Trott) [#​41258](nodejs/node#41258) - \[[`cc819b4bf8`](nodejs/node@cc819b4bf8)] - **tools**: fix typo in `avoid-prototype-pollution` lint rule (Antoine du Hamel) [#​44446](nodejs/node#44446) - \[[`254358c81e`](nodejs/node@254358c81e)] - **tools**: refactor `avoid-prototype-pollution` lint rule (Antoine du Hamel) [#​43476](nodejs/node#43476) - \[[`8c73279ebb`](nodejs/node@8c73279ebb)] - **util**: add default value option to parsearg (Manuel Spigolon) [#​44631](nodejs/node#44631) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC41NS4wIiwidXBkYXRlZEluVmVyIjoiMzQuNTUuMCJ9--> Reviewed-on: https://git.walbeck.it/mwalbeck/docker-cyberchef/pulls/143 Co-authored-by: renovate-bot <[email protected]> Co-committed-by: renovate-bot <[email protected]>
PR-URL: nodejs/node#44241 Backport-PR-URL: nodejs/node#44873 Fixes: nodejs/node#44023 Fixes: nodejs/node#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs/node#44241 Backport-PR-URL: nodejs/node#44873 Fixes: nodejs/node#44023 Fixes: nodejs/node#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>
PR-URL: nodejs/node#44241 Fixes: nodejs/node#44023 Fixes: nodejs/node#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> (cherry picked from commit 59527de13d39327eb3dfa8dedc92241eb40066d5)
PR-URL: nodejs/node#44241 Fixes: nodejs/node#44023 Fixes: nodejs/node#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> (cherry picked from commit 59527de13d39327eb3dfa8dedc92241eb40066d5)
PR-URL: nodejs/node#44241 Fixes: nodejs/node#44023 Fixes: nodejs/node#43675 Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]> (cherry picked from commit 59527de13d39327eb3dfa8dedc92241eb40066d5)
Fixes: #44023
FIxes: #43675
The meeting discussed in #44023 did not end up taking place, but there was consunsus on adding a programmatic (and configurable) way to run
--test
.this does not yet support configuring inside the test files themselves, that will wait for the discussions regarding global node/test runner configuration (#43973, #43895)
todo: