Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport of Fix AP upgrade version issue into release/1.17.x #27377

Merged

Conversation

hc-github-team-secure-vault-core
Copy link
Contributor

Backport

This PR is auto-generated from #27277 to be assessed for backporting due to the inclusion of the label backport/1.17.x.

The below text is copied from the body of the original PR.


This is a subtle issue introduced by #26449 while fixing a different issue. It only impacts Vault Enterprise but is generally simpler to reason about.

That PR was a correct fix, however the effectiveSDKVersion was only ever being set on active nodes not on perf standbys in Enterprise.

The big change here is:

  • Get rid of the custom Set method and instead always set the effectiveSDKVersion from Core during SetupCluster which is necessary on all types of servers
  • As a belt-and-braces measure, default RaftBackend.effectiveSDKVersion back to the binary version so even if we fail to plumb this from core in some edge case in the future it's still the right value in almost all cases (other than testing environments).

I've manually tested this locally. To reproduce the issue it fixes you need to:

  1. Attempt an AP upgrade in Vault Enterprise from any version before 1.15.8 to any version after 1.15.8
  2. Ensure you don't manually set autopilot_upgrade_version in config on the new servers (doing so bypasses the bug).

In this case you'll see the new servers join but remain "target version" because their Echo's are sending an empty version string. You also see some logs about empty upgrade_versions which are legit in this case but are also confusing because you can see them even after the fix due to timing (if AP runs before an Echo has come in but after node joined raft on the leader you'll see that log even on a non-broken cluster and then you don't see a corresponding "OK" log after wards).

TODO

  • also fix the logging confusion where we complain about no upgrade version due to timing and then don't log that it's OK again
  • review this scenario with others and test more cases to be sure this is correct.
  • See if there is a meaningful way we could test this in CI without a flaky complicated end-to-end test just for plumbing...

Overview of commits

@hc-github-team-secure-vault-core hc-github-team-secure-vault-core force-pushed the backport/fix/autopilot-upgrade-ii/adversely-handy-calf branch from 60fd3ba to cf0e7b3 Compare June 5, 2024 20:02
@github-actions github-actions bot added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Jun 5, 2024
@raskchanky raskchanky added this to the 1.17.1 milestone Jun 5, 2024
Copy link

github-actions bot commented Jun 5, 2024

Build Results:
All builds succeeded! ✅

Copy link

github-actions bot commented Jun 5, 2024

CI Results:
All Go tests succeeded! ✅

@banks
Copy link
Member

banks commented Jun 6, 2024

I used the wrong process for these backports.

@banks banks closed this Jun 6, 2024
@banks
Copy link
Member

banks commented Jun 6, 2024

Ooops no, this is current version so we still need this one.

@banks banks reopened this Jun 6, 2024
@banks
Copy link
Member

banks commented Jun 6, 2024

⚠️ This is waiting on a PR to release branch in Ent that will comment out a test that will otherwise break the release branch when this merges. We shouldn't appove until that is merged see https://github.com/hashicorp/vault-enterprise/pull/5976#issuecomment-2152273601

@banks banks merged commit b0ea89b into release/1.17.x Jun 6, 2024
153 checks passed
@banks banks deleted the backport/fix/autopilot-upgrade-ii/adversely-handy-calf branch June 6, 2024 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants