-
Notifications
You must be signed in to change notification settings - Fork 17.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: stamp the pseudo-version in builds generated by go build
#50603
Comments
At least speaking personally, for cases like mvdan/sh#519, my intent is to show something like It's true that something like a proper module version might be more useful; a git commit hash doesn't give any hint as to how old a version is, whereas a semver version prefix or a timestamp can give a starting point. So, in principle, I agree with you: 1.18 is a big step forward, but it's still unfortunate that the main module version remains as However, in practice, I still agree with Jay's comment in #29228 (comment); we shouldn't make such a "locally inferred version" look like a normal version, because it's reasonably likely to be wrong or cause confusion with users.
Could you give some examples? I can only think of very unlikely scenarios, such as manually corrupting the module download cache after downloading some dependencies. That cache is read-only by default, and With the main module in a git checkout, I can think of multiple scenarios which seem more likely:
I think that, if we are to implement something like this, the versions must be somehow different from the canonical and unique versions that get computed from fully published commits and tags. This would make it very clear that the versions are inferred from local state, and not guaranteed to be correct. As a simplistic example, imagine that tagging |
To add a more concrete example: if we made the change proposed here, and locally inferred versions looked like fully published versions, I would have a harder time trusting the output of |
I was thinking of the case where since Go itself doesn't expose its own concept of a version to the program, the users themselves are forced to create their own concepts of a version, either through things like Let me rephrase my point. Go can't enforce any useful properties for the user's notion of a version because it doesn't know about it, and as such if we make The user does gain something in the latter case though. They don't have to create build wrappers.
I very much agree with this, with one caveat. If the locally checked-out version is identical to a published release, I would expect the version to match the release. If the locally checked-out version can not be guaranteed to match any release, then yes, it should be published with something like Unfortunately, I can't imagine how this would work without internet access, and quite often a prerequisite of automated systems running |
Hold on, another thought. If we always add the commit hash, and some other metadata to the main module version for local builds, essentially always making them a fully qualified Go pseudo-version, then they will always be different from the published version, so there's no potential for confusion there. Even better, in semver terms these builds will sort before the published version, which is probably what people want. For this, what I said earlier about
can no longer be true, but perhaps that is ok as long as we come up with a documented and stable convention that describes versioning for local builds (as opposed to just dumping a "devel" in the metadata field). |
The main caveat here, I think, is unpublished tags. If I create a local, unpublished tag for, say, That may or may not be a significant issue, though: if we always use a pseudo-version, we'll at least have the commit hash as a common point of reference even if the base versions differ. |
@4ad right, a local build can't always know what is or isn't published, as requiring a network roundtrip takes us back to square one. Your idea of trying to stick to semver, and always using some form of pseudo-version which includes a hash, sounds good. With one caveat, though: the commit hash isn't enough to make the version unambiguous, because I can have infinite kinds of uncommitted changes that do not change the HEAD commit hash. @bcmills good point about tags still messing with pseudo-versions, but at least if we always include a timestamp and some form of unique hash, then I think we're good. With the caveat above about uncommitted changes :) |
We do have another hash available to us, though, which changes whenever any input Go code changes: the build IDs used for the build cache. I seem to recall that one such ID is embedded into binaries, too. Not ideal, as such a hash also includes build parameters like GOOS or -tags, which don't normally affect versions. But at least it fixed the problem with uncommitted files in VCS. |
Yes, uncommitted changes should be explicit in the pseudo-version, but I think we can suffix |
the new buildinfo already records whether the workspace is clean with |
we could use one of So main will always have a version like
|
What is the main motivation of encoding the local version in pseudo-version style rather than keeping those extra info (timestamp?) as extra metadata fields - if it's not guaranteed that they are always available in the origin or proxies? It seems like the BTW, I feel like the main module's version isn't sufficient to describe a tool's behavior in certain cases - |
The main motivation is that
Emphasis mine. It's not rather than, It doesn't replace the existing metadata fields. If you want to read the metadata, you should read it from those fields instead of parsing the pseudo-version. However, that metadata is useful in disambiguating builds produced by
This sounds like an argument to always use the build ID as the version suffix instead of the VCS hash.
I hope so too, but again, I think that discussion is out of scope for this thread, which is more about bringing |
(devel)
in binaries, which is not very usefulgo build
This proposal has been added to the active column of the proposals project |
As a kind of experience report, I'm using custom build-scripts for years, solely to embed version information of the main module into the executable. When building, I attach the following information:
With this kind of information, I'm able to build version-strings however I like. Usually I try to match go's pseudo-version strings, but when I need to stay compatible with the version-scheme from other, non-go projects, I can do so as well.
The version is printed when the application is started with a I don't have anything against adding a build ID, but it wouldn't really solve my problem. My use cases are:
The previously raised issue that git-tags might only be local was never really an issue in my experience. Version-tags aren't made lightheartedly and are always immediately pushed to the server in the projects I work on. I really hope we can this kind of information into a go-executable one day, because I would finally be able to get rid of all my build- and tool-scripts. |
It sounds like @mvdan and @bcmills have some hesitation around the fact that these pseudo-versions would not correspond to any publicly available version, even though they look like those. That does seem like a reason not to do this. We now have Git VCS info separately in the builds (as of Go 1.18; try go version -m). Do we need to add a second way to record that information? |
go build
go build
We can make it unambiguously distinct, for example instead of
No, we certainly only need one way to encode VCS info. The suggestion to put VCS info in the metadata field of the pseudo-version was to match the |
With replace statements in go.mod and the new workspace mode in Go 1.18 it is possible to build Go programs that include local versions of modules besides the main module. For those modules the go tool also lists The new build vcs metadata only helps identify the main module. Adding vcs metadata for all local modules seems valuable and not currently supported. The format suggested above by @4ad would be more informative than |
This may be true, but the concern above seems to be adding vcs metadata that looks like a pseudo-version. It need not, and it probably should not. We can always add that separately; maybe that should be a different proposal. (I think this is the first comment to bring up VCS info for replaced modules that point to other local repos.) |
A thought: if we're only concerned about having a reliable way to always get some useful version for the main module, I think it could be an API of its own, like I personally will be implementing logic like that to replace Another option, if we want this to also work for library modules, would be If the above sounds interesting, I'll happily develop the idea further and create a new proposal. I realise it's not the same as this proposal, but I also think it could be a different solution to the same end-user problem :) |
@mvdan I would very much like to see something like that to replace the boiler place build lines that exist in code at work. |
We use the semantic version of the binaries in order to compute API compatibility between different binaries. I am afraid that if your Now, one might object to using the binary version in this way and perhaps recommend using a separately maintained API version instead that is separate from the binary version. I would tend to agree except this is outside my control. I do not have the operational liberty to change this. |
Thank you for all your ideas. This is definitely an important feature. After considering all of your ideas and discussing with @matloob and @rsc, we have landed upon this format: v[tag]+[optional dirty]When the current commit matches a tagged version. [optional dirty]dirty: there are uncommitted changes in the build. Example:
[Pseudo version]+[optional dirty]When the current commit does not match a tagged version. Example:
If you have any feedback, please let us know! |
The proposed format makes sense to me when either the work tree is dirty or the currently-selected commit doesn't exactly match a tag. It's not clear to me from the most recent comment whether there would also be a special case to use the regular version number alone when the current commit exactly matches a tag. Ideally I'd like for that case to be indistinguishable from the treatment of libraries, but I don't know if there are hurdles that make that infeasible. If the outcome were to use the proposed format in the dirty or not-tagged case but to use the tag-derived version number when at tag exactly matches then I think that would be sufficient for the codebase I help maintain in my day job to drop its own redundant version number tracking and use the toolchain-generated version information exclusively, which would be great because we'd then get more reliable information for non-release builds (that are always just called "v1.2.3-dev" for us today). |
One potential caveat: Some version constraint systems use lexical ordering to decide which is the newest between two prerelease versions that have the same base version. In that case I don't think that's a blocking concern -- it's questionable whether it's ever meaningful to sort development builds relative to release builds anyway -- but wanted to raise it anyway since it was a hazard I've run into in the past in a different context. |
|
From #50603 (comment)
More specific example: Let's assume we found a vulnerability in a binary built from a clean checkout of When creating a CVE, we should use The version in the build info is also used by the Go telemetry. To collect the telemetry, the version string should be explicitly listed in the configuration. I don't think the binary with version |
I guess
Could you expand on that? |
It should be one dash followed by dot separated components: https://semver.org/#spec-item-9 |
The current pseudoversions use different base versions based on existing tags in the current of parent commits:
The dot separator Given that we already have existing pseudoversions that can identify commits, I'm not sure why we need a separate format for local builds, beyond marking it as either a local clean or dirty build (maybe build metadata |
I see your point. It would be useful to drop the pseudo version if the current commit matches a tag. My understanding is that they may need to be separate. A local build might inherently be different than one from one that is go install'd and such we don't want to confuse the two. I will discuss more with @rsc and @matloob and circle back.
Interesting point. Like you said, I don't think it would be meaningful to sort a local build against other pre-release versions, but let's add a -0 to have it be less than any pre-releases. So we have v1.2.3-0-devel.
vX.0.0 if there is no such tag.
Good point! Let's have it sorted after. Upon further investigation, copying the psuedo-version behavior from go install makes sense here which bumps the version if the commit is newer than the last release. So we will perform this operation:
This we're not completely sure on. But I suppose a combination of '.' and '-' makes more sense. Taking into account the above points, maybe instead we should do: v[X.Y.(Z+1)]-0.[timestamp].devel-[commit]+[optional dirty]
I agree with you. I think it makes sense to closely follow the existing structure from: https://cs.opensource.google/go/go/+/master:src/cmd/vendor/golang.org/x/mod/module/pseudo.go @hyangah Thanks for the help guys. I will update my original comment to reflect these changes. |
I want to reiterate what @hyangah said - as of 2022, and when the proposal was accepted, we had already reached consensus that a local build of a commit on a semver tag should result in a module version reflecting just that semver string. See #50603 (comment) for example. I don't think new information has come to light since then, so I'd be very confused if we suddenly decided to implement something else. |
Further, if a proposed pseudo-version of a clean local commit with a semver tag is syntactically different from the semver tag, then how does that affect version.Compare? It should return |
@hyangah @mvdan About stamping the tagged version without any suffixes when doing a go build in a clean repo: I've been thinking about this and I can't prove to myself that we'd get the same build running I'm also not totally sure about how module pruning affects a module loaded from It's likely there's no issue there and I'm overthinking it. It might also be that these issues don't matter because the version is not meant treated as 100% accurate. |
I believe version.Compare only compares go versions. If we cannot guarantee the same behavior of go install and go build with @matloob's concerns then it might make sense to add a +build suffix? (similar idea to what Sean had suggested with +local) |
That is my understanding from the previous consensus. I don't have a strong opinion on a suffix like It's also worth noting that one could often use the presence and contents of the stamped |
I too am confused why we would define a new pseudo-version syntax instead of using the existing algorithm and derivation code. Let's talk more about this next time we meet, @matloob and @samthanawalla. |
@rsc @matloob and I discussed this today. I have revised my original comment to reflect the decision we came to. See #50603 (comment) When the current commit matches a tagged version we will use v[tag]. When the current commit does not match a tagged version or there are uncommitted changes, we'll use the existing pseudo version format. along with an optional dirty tag. (We dropped the devel and the +build) |
Are uncommitted changes only considered for the directory where the |
We were planning to rely on what the VCS state is, I.e. git status. However as a compromise, would v[tag]+dirty work for your use case? If your current commit matches a tagged version but you have uncommitted changes, you would get v[tag]+dirty instead of [pseudoversion]+dirty. |
I would prefer to be able to push a binary as is from my machine if there are no changes in the the given go module. Other than a go.work file and possible env vars there shouldn't be any dirty files affecting the go binary itself so marking the binary as dirty when there are no changes in the module itself seems overly cautious.
I am not sure I know enough about Go to know anything else other than the go.work and env vars that introduce additional states. I would love to learn a bit more about what else affects a go module. At the end of the day, having the version stamped will be a win even if I can only get the non-dirty version out of CI or a clean clone. |
By additional states I meant to say version control states. I do get your point but a tag is more a property of the repo and not just the main module. Changes to the repo as a whole will reflect accordingly in the stamped version. While it may be overly cautious, I don't think we will support this use case as of now. But that could change in the future if necessary. Updated #50603 comment to include v[tag]+dirty use case. |
@DavidGamba I would be interested in understanding your use case better. Our expectation is that those planning to do a build of a go program at a given version would check out the appropriate version and do a clean build. Our plan is to reuse the |
The more and more I think about this, the more I get the feeling that having a completely precise solution is either extremely hard or impossible. If a module has a replace directive pointing to a local directory outside of the module, then changing that replacing directory content could result in a different binary. To make things more complicated, this replacing directory might be outside of a repo where the module is. It seems to me that the current approach is making a decent compromise. It will cover the more common case where the replacing module is in the same repo. Of course, it might add +dirty if part of a repo completely unrelated to module is being changed. Regarding monorepo, it seems that the current approach will always have |
My use case is not a major use case since official builds will come from CI. It is just in those cases where you want to push something out that doesn't have a pipeline yet. Cloning a clean repo for those is not a major inconvenience and worst case the binary will just have the +dirty label. |
cmd/go embeds dependency version information in binaries, which is very useful. From Go 1.18 onwards, cmd/go also embeds VCS information in binaries, which makes it even more useful than it was before.
As #37475 mentions, people place version information in binaries using
-ldflags='-X foo=bar'
, which requires an additional build wrapper. The new VCS stamping feature of cmd/go should alleviate the need for external wrapper, but I am afraid it comes short.The version information, in the sense of Go's pseudo version is not recorded for the main module when doing
go build
:The version is recorded as expected when doing
go install
:I am afraid this limitation of cmd/go will continue to force people to use external build wrappers that set
-ldflags
, which is rather unfortunate.I am not the first to want main module version information in binaries, this has been already asked for in various issues, for example in #29814, which was closed as a duplicate of #37475, but it really wasn't a duplicate, as #37475 is about VCS information, and #29814 is about semantic versioning. Other examples of people asking for this feature are mvdan/sh#519 and #29228 (comment) where various workarounds were proposed.
Speaking of workarounds, the only workaround that I know that currently works would be to create a local module proxy and pass
GOPROXY
togo install
, but that is an extremely high-overhead workaround, andgo install
is not a replacement forgo build
anyway, sincego install
comes with some rather severe limitations regarding how vendoring works and what you can put in go.mod, andgo install
doesn't support controllingGOBIN
when cross-compiling.I realize that Git tags are a local concept, and by doing the "wrong" git operations one could come up with a different pseudo-version for the same source code. I am afraid I don't have any solution or suggestion regarding this git misfeature, except to note that even in this case the hash information is recorded correctly, and in every case by the virtue of having access to the local source code the programmer can always do some local operation that has the potential to cause a version mislabeling. Git is just more prome to do this by accident, but the ability is there, always.
I don't have any stats to back this up, but from my experience most corporate source code is built by
go build
, notgo install
, and it would be great if somehow Go's notion of versioning would be stamped bygo build
.CC @bcmills @mvdan @rsc
The text was updated successfully, but these errors were encountered: