Bring CLOMonitor Score to 100%

[eddie-knight]

Per discussion in slack, this issue is to track the efforts necessary to bring argo-workflow CLOMonitor score to 100% as part of the CNCF security slam.

Below is a checklist of action items. I will add comments if there is anything that can't be addressed or if I am unsure of how to address it. It is possible to exclude checks with a written justification if that becomes necssary.

CLOMonitor report

Summary

Repository: argo-workflows
URL: https://github.com/argoproj/argo-workflows
Checks sets: CODE
Score: 88

Checks passed per category

Category	Score
Documentation	100%
License	75%
Best Practices	100%
Security	80%
Legal	n/a

Checks

Documentation [100%]

• Changelog (docs)
• Contributing (docs)
• Maintainers (docs)
• Readme (docs)

License [75%]

• Apache-2.0 (docs)
• Approved license (docs)
• License scanning (docs)

Best Practices [100%]

• Artifact Hub badge (docs)
• Contributor License Agreement (docs) EXEMPT
• Developer Certificate of Origin (docs)
• OpenSSF badge (docs)
• Recent release (docs)

Security [80%]

• Binary artifacts (docs)
• Code review (docs)
• Dangerous workflow (docs)
• Dependency update tool (docs)
• Maintained (docs)
• Software bill of materials (SBOM) (docs)
• Security policy (docs)
• Signed releases (docs)
• Token permissions (docs)

For more information about the checks sets available and how each of the checks work, please see the CLOMonitor's documentation.

[eddie-knight]

The Token-Permissions check is actively being refined right now, so we shouldn't put too much thought into that until this issue closes: ossf/scorecard#2338

[agilgur5]

• Signed releases (docs)

This was completed in #9837

• License scanning (docs)

• I believe Snyk's security scans do license checks as well, so I think this might just need a badge on the README to satisfy.
  • EDIT: not quite, see my comment below
  • EDIT2: Completed in docs: add FOSSA License Badge for CLOMonitor #12032

And there's a new one now:

• OpenSSF Scorecard Badge (docs)
  • EDIT: Completed in docs: Add OpenSSF Scorecard status badge to README.md #11897

Assigning to myself right now as I may be working on the related SLSA Level 3, Level 4 checks soon. But anyone can feel free to add as well!

[agilgur5]

I believe Snyk's security scans do license checks as well, so I think this might just need a badge on the README to satisfy.

Welp, never mind, apparently Snyk does not support this for Go deps (see also the CLOMonitor issue: cncf/clomonitor#50 (comment)). So I integrated FOSSA same as Argo CD (see also #12023 that was made automatically).

So am gonna make a PR for that badge for CLOMonitor and that should bring our non-security scores at 100%. EDIT: See #12032

I'm going to open a separate issue for the security scores / OpenSSF checks so that we can close this one out and that one can be more focused. There's also a few other newer OpenSSF checks as well, and Token-Permissions seems to be refined now. EDIT: See #12031

[agilgur5]

The Token-Permissions check is actively being refined right now, so we shouldn't put too much thought into that until this issue closes: ossf/scorecard#2338

@eddie-knight curious if the SECURITY-INSIGHTS.yml checks ("Dependencies policy", "Security insights", and "Self-Assessment) are going through similar refinement this time around? They aren't listed under OpenSSF Scorecard's checks right now.

I also haven't seen any project (including k8s itself) adopt a SECURITY-INSIGHTS.yml other than some of the OpenSSF related repos.

[eddie-knight]

Definitely @agilgur5. The SI was adopted by CLOMonitor (not via Scorecard) at the beginning of this month in order to streamline the hygiene checks you listed there.

If there's a clarity gap I'd love to get more insight from you so that I can make sure to improve the related supporting material for the Slam.

[agilgur5]

The SI was adopted by CLOMonitor (not via Scorecard) at the beginning of this month in order to streamline the hygiene checks you listed there.

Is it going to be added to Scorecard? It's a bit confusing that most, but not all, of the security checks are via Scorecard

If there's a clarity gap

Not really a clarity gap, I understand what SECURITY-INSIGHTS.yml is for and have read the spec etc. I am bit surprised that CLOMonitor is pushing it despite current lack of adoption though, hence why I asked (especially since Token-Permissions went through some refinement too, so there is history for that).

More specifically, the lack of adoption for it currently (including in Scorecard) combined with how much information there is to fill out for it (plus yet another file at the root of the repo) makes the effort/value ratio feel not really worthwhile right now.
Especially as SECURITY-INSIGHTS.yml does not really improve any security on its own as it's just a metadata file; feels like there are (much?) more valuable uses of time for the same effort (e.g. adding SLSA Level 3+ provenance, adding CodeQL, fixing vulns per #12031, adding step-security/harden-runner, etc).

[eddie-knight]

I can't speak on the Scorecard roadmap (but there are folks in the CNCF Slack #security-slam channel who can). I believe there is discussion of extending Scorecard to reference the SI in situations where less-predictable things such as SBOM artifacts are needing to be detected.

That's the idea behind integrating it into CLOMonitor— we tried to capture the results of those discussions in a CLOMonitor GitHub issue for posterity. You can see there was debate around whether the new checks should be in the Documentation or Code check set... but the intent at the end of the day is to use these new checks to find the harder-to-detect elements that TAG Security recommends.

[agilgur5]

Yea I guess I am confused as to the difference between CLOMonitor's security checks and Scorecard's checks; one would think they'd be the same.

I see that Security Policy and SBOMs are also CLOMonitor-only.
It could make sense if Scorecard was all quantitative things and CLOMonitor was all metadata, but that's also not the case right now since Scorecard has "OpenSSF Best Practices Badge", "License", etc (also "Maintained", which is more of an opinionated check).
Some standardization and/or explicit delegation of checks between the two would be good to see.

More specifically, the lack of adoption for it currently (including in Scorecard) combined with how much information there is to fill out for it (plus yet another file at the root of the repo) makes the effort/value ratio feel not really worthwhile right now.

Well ok this got completed in CD argoproj/argo-cd#16135 so the effort is now significantly reduced as Workflows would be very similar

[eddie-knight]

CLOMonitor currently runs a subset of scorecard checks and run a few additional checks that have been recommended by either TAG Security or maintainers of graduated CNCF projects.

To compare and contrast a bit... In scorecard, nobody is reasonably expected to have a full 10/10 score, so checks can't be optionally skipped. In contrast, projects can skip CLOMonitor checks (including the checks that harness Scorecard) by providing a written justification in the CLOMonitor config.