Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add nested workspace traversal #36

Draft
wants to merge 2 commits into
base: 1.0-dev
Choose a base branch
from
Draft

Add nested workspace traversal #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
58d6120
feat: Add support for multiple/nested workspace traversal under new r…
MLSTRM Mar 15, 2024
ec2ffcf
Add test for nested workspace resolution and update snapshots
MLSTRM Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ $ yarn CycloneDX make-sbom
(choices: "application", "framework", "library", "container", "platform", "device-driver", default: "application")
--reproducible Whether to go the extra mile and make the output reproducible.
This might result in loss of time- and random-based values.
--recursive Scan all nested workspaces within the current project, rather than just the one in the current working directory.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scan all nested workspaces within the current project, rather than just the one in the current working directory.

my questions:

  • Could you explain how this is a use case?
  • If the current workspace had no dependency to any other workspace, why would you want the other workspaces be part of the BOM?
  • If the current workspace had a dependency on any other workspace, is this not already in the SBOM?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My use case comes from a multi-application project, specifically multiple serverless aws lambda functions, which work together to perform a single task.
As they cannot be deployed separately, it is more useful/accurate to me to have a single SBOM for the overall project, rather than one per function.
Currently the dependencies declared within each sub-workspace are not included at the top level, and without generating multiple separate SBOMs and somehow merging them, I cannot currently represent the full usage state of my application.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. I do not see this as being a feature of the MVP.
So this will probably be not merged for some time.

Workspaces are a way of organizing work, not some architectural or design decision.
Therefore, they actually have no representation in an SBOM.

If you had a product containing of several components, each being an independent application, then you should be using either BOM-Links connecting your product's components/services with eachother, or use a merge-tool to combine individual SBOMs into one.


━━━ Details ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Expand Down
19 changes: 14 additions & 5 deletions sources/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ import {
Configuration,
type Plugin,
Project,
ThrowReport
ThrowReport,
type Workspace
} from '@yarnpkg/core'
import { type PortablePath, ppath } from '@yarnpkg/fslib'
import { Command, Option } from 'clipanion'
Expand Down Expand Up @@ -73,6 +74,10 @@ class SBOMCommand extends BaseCommand {
description: 'Whether to go the extra mile and make the output reproducible.\nThis might result in loss of time- and random-based values.'
})

recursive = Option.Boolean('--recursive', false, {
description: 'Resolve dependencies from all nested workspaces within the current one.'
})

async execute (): Promise<void> {
const configuration = await Configuration.find(
this.context.cwd,
Expand All @@ -86,6 +91,9 @@ class SBOMCommand extends BaseCommand {

if (this.production) {
workspace.manifest.devDependencies.clear()
if (this.recursive) {
project.workspaces.forEach((w: Workspace) => { w.manifest.devDependencies.clear() })
}
const cache = await Cache.find(project.configuration)
await project.resolveEverything({ report: new ThrowReport(), cache })
} else {
Expand All @@ -97,14 +105,15 @@ class SBOMCommand extends BaseCommand {
outputFormat: parseOutputFormat(this.outputFormat),
outputFile: parseOutputFile(workspace.cwd, this.outputFile),
componentType: parseComponenttype(this.componentType),
reproducible: this.reproducible
reproducible: this.reproducible,
recursive: this.recursive
})
}
}

function parseSpecVersion (
specVersion: string | undefined
): OutputOptions['specVersion'] {
): OutputOptions[ 'specVersion' ] {
if (specVersion === undefined) {
return CDX.Spec.Version.v1dot5
}
Expand All @@ -119,7 +128,7 @@ function parseSpecVersion (

function parseOutputFormat (
outputFormat: string | undefined
): OutputOptions['outputFormat'] {
): OutputOptions[ 'outputFormat' ] {
if (outputFormat === undefined) {
return CDX.Spec.Format.JSON
}
Expand All @@ -136,7 +145,7 @@ function parseOutputFormat (
function parseOutputFile (
cwd: PortablePath,
outputFile: string | undefined
): OutputOptions['outputFile'] {
): OutputOptions[ 'outputFile' ] {
if (outputFile === undefined || outputFile === '-') {
return stdOutOutput
} else {
Expand Down
15 changes: 8 additions & 7 deletions sources/sbom.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import { PackageURL } from 'packageurl-js'
import {
type BuildtimeDependencies,
type PackageInfo,
traverseWorkspace
traverseWorkspaces
} from './traverseUtils'

const licenseFactory = new CDX.Factories.LicenseFactory()
Expand All @@ -55,6 +55,7 @@ export interface OutputOptions {
outputFile: PortablePath | typeof stdOutOutput
componentType: CDX.Enums.ComponentType
reproducible: boolean
recursive: boolean
}

export async function generateSBOM (
Expand All @@ -74,9 +75,9 @@ export async function generateSBOM (
bom.metadata.timestamp = new Date()
}

const allDependencies = await traverseWorkspace(
const allDependencies = await traverseWorkspaces(
project,
workspace,
outputOptions.recursive ? project.workspaces : [workspace],
config
)
const componentModels = new Map<LocatorHash, CDX.Models.Component>()
Expand Down Expand Up @@ -153,9 +154,9 @@ async function addMetadataTools (bom: CDX.Models.Bom): Promise<void> {
*/
function serialize (
bom: CDX.Models.Bom,
specVersion: OutputOptions['specVersion'],
outputFormat: OutputOptions['outputFormat'],
reproducible: OutputOptions['reproducible']
specVersion: OutputOptions[ 'specVersion' ],
outputFormat: OutputOptions[ 'outputFormat' ],
reproducible: OutputOptions[ 'reproducible' ]
): string {
const spec = CDX.Spec.SpecVersionDict[specVersion]
if (spec === undefined) { throw new RangeError('undefined specVersion') }
Expand Down Expand Up @@ -218,7 +219,7 @@ function getAuthorName (manifestRawAuthor: unknown): string | undefined {
*/
function packageInfoToCycloneComponent (
pkgInfo: PackageInfo,
reproducible: OutputOptions['reproducible']
reproducible: OutputOptions[ 'reproducible' ]
): CDX.Models.Component {
const manifest = pkgInfo.manifest
const component = componentBuilder.makeComponent(
Expand Down
108 changes: 55 additions & 53 deletions sources/traverseUtils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,12 @@ export interface PackageInfo {

// Modelled after traverseWorkspace in https://github.com/yarnpkg/berry/blob/master/packages/plugin-essentials/sources/commands/info.ts#L88
/**
* Recursively traveses workspace and its transitive dependencies.
* Recursively traverses workspaces and their transitive dependencies.
* @returns Packages and their resolved dependencies.
*/
export async function traverseWorkspace (
export async function traverseWorkspaces (
project: Project,
workspace: Workspace,
workspaces: Workspace[],
config: Configuration
): Promise<Set<PackageInfo>> {
// Instantiate fetcher to be able to retrieve package manifest. Conversion to CycloneDX model needs this later.
Expand All @@ -67,62 +67,64 @@ export async function traverseWorkspace (
cacheOptions: { skipIntegrityCheck: true }
}

const workspaceHash = workspace.anchoredLocator.locatorHash

/** Packages that have been added to allPackages. */
const seen = new Set<LocatorHash>()
const allPackages = new Set<PackageInfo>()
/** Resolved dependencies that still need processing to find their dependencies. */
const pending = [workspaceHash]

while (true) {
// pop to take most recently added job which traverses packages in depth-first style.
// Doing probably results in smaller 'pending' array which makes includes-search cheaper below.
const hash = pending.pop()
if (hash === undefined) {
// Nothing left to do as undefined value means no more item was in 'pending' array.
break
}

const pkg = project.storedPackages.get(hash)
if (pkg === undefined) {
throw new Error(
'All package locator hashes should be resovable for consistent lockfiles.'
)
}
for (const workspace of workspaces) {
const workspaceHash = workspace.anchoredLocator.locatorHash

/** Packages that have been added to allPackages. */
const seen = new Set<LocatorHash>()
/** Resolved dependencies that still need processing to find their dependencies. */
const pending = [workspaceHash]

while (true) {
// pop to take most recently added job which traverses packages in depth-first style.
// Doing probably results in smaller 'pending' array which makes includes-search cheaper below.
const hash = pending.pop()
if (hash === undefined) {
// Nothing left to do as undefined value means no more item was in 'pending' array.
break
}

const fetchResult = await fetcher.fetch(pkg, fetcherOptions)
let manifest: Manifest
try {
manifest = await Manifest.find(fetchResult.prefixPath, {
baseFs: fetchResult.packageFs
})
} finally {
fetchResult.releaseFs?.()
}
const packageInfo: PackageInfo = {
package: pkg,
manifest,
dependencies: new Set()
}
seen.add(hash)
allPackages.add(packageInfo)

// pkg.dependencies has dependencies+peerDependencies for transitive dependencies but not their devDependencies.
for (const dependency of pkg.dependencies.values()) {
const resolution = project.storedResolutions.get(
dependency.descriptorHash
)
if (typeof resolution === 'undefined') {
throw new Error('All package descriptor hashes should be resolvable for consistent lockfiles.')
const pkg = project.storedPackages.get(hash)
if (pkg === undefined) {
throw new Error(
'All package locator hashes should be resovable for consistent lockfiles.'
)
}
packageInfo.dependencies.add(resolution)

if (!seen.has(resolution) && !pending.includes(resolution)) {
pending.push(resolution)
const fetchResult = await fetcher.fetch(pkg, fetcherOptions)
let manifest: Manifest
try {
manifest = await Manifest.find(fetchResult.prefixPath, {
baseFs: fetchResult.packageFs
})
} finally {
fetchResult.releaseFs?.()
}
const packageInfo: PackageInfo = {
package: pkg,
manifest,
dependencies: new Set()
}
seen.add(hash)
allPackages.add(packageInfo)

// pkg.dependencies has dependencies+peerDependencies for transitive dependencies but not their devDependencies.
for (const dependency of pkg.dependencies.values()) {
const resolution = project.storedResolutions.get(
dependency.descriptorHash
)
if (typeof resolution === 'undefined') {
throw new Error('All package descriptor hashes should be resolvable for consistent lockfiles.')
}
packageInfo.dependencies.add(resolution)

if (!seen.has(resolution) && !pending.includes(resolution)) {
pending.push(resolution)
}
}
}
}

return allPackages
}
};
Loading
Loading