Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🌱 Enhance test output and management in ValidateTestReturn #3810

Merged
merged 5 commits into from
Jan 30, 2024

Conversation

spencerschrock
Copy link
Contributor

What kind of change does this PR introduce?

test helper enhancement

What is the current behavior?

  • Test management (t.Fail(), t.Errorf, or similar) is done by all the callers
  • The only provided details are that the number of statements is mismatched, so it takes longer to investigate the test failure.

What is the new behavior (if this is a feature change)?**

  • All of the test management is handled by ValidateTestReturn.
  • t.Log is used to print the details seen by the test. These details are only shown when a test fails, or -v is passed (like the e2e tests).
    • This helps significantly when answering the question "what are the details the code is currently throwing?"
--- FAIL: TestGithubTokenPermissions (0.00s)
    --- FAIL: TestGithubTokenPermissions/don't_penalize_job-level_writes (0.00s)
        ~/scorecard.git/checks/permissions_test.go:461: INFO: topLevel permissions set to 'read-all'
        ~/scorecard.git/checks/permissions_test.go:461: DEBUG: not a publishing workflow: .github/workflows/github-workflow-permissions-run-multiple-writes.yaml
        ~/scorecard.git/checks/permissions_test.go:461: DEBUG: not a releasing workflow: .github/workflows/github-workflow-permissions-run-multiple-writes.yaml
        ~/scorecard.git/checks/permissions_test.go:461: DEBUG: not a GitHub Pages deployment workflow: .github/workflows/github-workflow-permissions-run-multiple-writes.yaml
        ~/scorecard.git/checks/permissions_test.go:461: DEBUG: not a SARIF workflow, or not an allowed one
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'packages' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'actions' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'statuses' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'checks' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'security-events' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'deployments' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: WARN: jobLevel 'contents' permission set to 'write'
        ~/scorecard.git/checks/permissions_test.go:461: don't penalize job-level writes: (-expected +actual)  utests.TestReturn{
              	Error:         nil,
              	Score:         10,
            - 	NumberOfWarn:  6,
            + 	NumberOfWarn:  7,
              	NumberOfInfo:  1,
              	NumberOfDebug: 4,
              }
  • Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

Special notes for your reviewer

You'll see three sort of changes:

  1. removal of test management in each spot, because ValidateTestReturn handles it:
-if !scut.ValidateTestReturn(t, tt.name, &tt.expected, &r, &dl) {
-	t.Fail()
-}
+scut.ValidateTestReturn(t, tt.name, &tt.expected, &r, &dl)
  1. passing in a "testing.T" for e2e tests so we don't get panics. GinkgoTB is how we do it. (The Should(BeTrue()) part relates to 1. above)
-scut.ValidateTestReturn(nil, "repo rules accessible", &expected, &result, &dl).Should(BeTrue())
+scut.ValidateTestReturn(GinkgoTB(), "repo rules accessible", &expected, &result, &dl)
  1. The changes to ValidateTestReturn that let it all happen.

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

NONE

Copy link

codecov bot commented Jan 18, 2024

Codecov Report

Merging #3810 (414d087) into main (19047e8) will decrease coverage by 4.79%.
The diff coverage is 83.33%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3810      +/-   ##
==========================================
- Coverage   75.53%   70.74%   -4.79%     
==========================================
  Files         232      232              
  Lines       15763    15772       +9     
==========================================
- Hits        11906    11158     -748     
- Misses       3121     3929     +808     
+ Partials      736      685      -51     

@spencerschrock spencerschrock marked this pull request as ready for review January 18, 2024 22:44
@spencerschrock spencerschrock requested a review from a team as a code owner January 18, 2024 22:44
@spencerschrock spencerschrock requested review from justaugustus and laurentsimon and removed request for a team January 18, 2024 22:44
@spencerschrock spencerschrock merged commit 83ff808 into ossf:main Jan 30, 2024
38 checks passed
@spencerschrock spencerschrock deleted the test-helper-log branch January 30, 2024 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants