Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal error: concurrent map writes with RegisterFlagCompletionFunc #1320

Closed
MichaelMure opened this issue Jan 26, 2021 · 4 comments · Fixed by #1438
Closed

fatal error: concurrent map writes with RegisterFlagCompletionFunc #1320

MichaelMure opened this issue Jan 26, 2021 · 4 comments · Fixed by #1438

Comments

@MichaelMure
Copy link
Contributor

Cobra has a global map here to store completion function when those exists.

This global map is not protected in any way, which leads to fatal error: concurrent map writes when handling multiple root command concurrently. Yes, it's a little odd situation but it's a useful one to generate documentation/completion faster (ie, concurrently). Most of those documentation/completion generator will actually mutate things here and there in the commands, which makes them not thread-safe. Now, this global map for completion functions also make things no, thread-safe, even if you duplicated entirely your root command.

I see the following possible solutions:

  • move this completion function (no map anymore) directly into flag.Flag. No more global, no more indirection, but that's another package.
  • move the map into cobra.Command, to at least make be thread-safe when you have a copy of your Command
  • add a mutex or use sync.Map.

Example code to trigger the issue:

package main

import (
	"fmt"
	"sync"

	"github.com/spf13/cobra"
)

func main() {
	cmd := cobra.Command{}

	N := 2
	options := make([]string, N)

	var wg sync.WaitGroup
	for i := 0; i < N; i++ {
		flag := fmt.Sprintf("flag%d", i)
		cmd.Flags().StringVarP(&options[i], flag, "", "value", "usage")
		wg.Add(1)
		go func() {
			defer wg.Done()
			err := cmd.RegisterFlagCompletionFunc(flag, nil)
			if err != nil {
				panic(err)
			}
		}()
	}

	wg.Wait()

}

Originally posted by @krobelus in #1290 (comment)

@Luap99
Copy link
Contributor

Luap99 commented Jan 26, 2021

The issue is not limited to the flagCompletionFunctions map. You can also easily reproduce with to following since the flags are also stored as map.

package main

import (
	"fmt"
	"sync"

	"github.com/spf13/cobra"
	"github.com/spf13/pflag"
)

func main() {
	cmd := cobra.Command{}

	N := 20

	var wg sync.WaitGroup
	for i := 0; i < N; i++ {
		flag := fmt.Sprintf("flag%d", i)
		wg.Add(1)
		go func() {
			defer wg.Done()
			cmd.Flags().StringP(flag, "", "value", "usage")
		}()
	}

	wg.Wait()

	cmd.Flags().VisitAll(func(f *pflag.Flag) {
		fmt.Println(f.Name)
	})

}

@github-actions
Copy link

This issue is being marked as stale due to a long period of inactivity

@MichaelMure
Copy link
Contributor Author

Still an issue.

@silenceshell
Copy link
Contributor

we met this problem too.

Luap99 added a commit to Luap99/cobra that referenced this issue Jul 2, 2021
The flag completion functions should not be stored in the root cmd.
There is no requirement that the root cmd should be the same when
`RegisterFlagCompletionFunc` was called. Storing the flags there does
not work when you add the the flags to your cmd struct before you add the
cmd to the parent/root cmd. The flags can no longer be found in the rigth
place when the completion command is called and thus the flag completion
does not work.

Also spf13#1423 claims that this would be thread safe but we still have a map
which will fail when accessed concurrently. To truly fix this issue use a
RWMutex.

Fixes spf13#1437
Fixes spf13#1320

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/cobra that referenced this issue Jul 2, 2021
The flag completion functions should not be stored in the root cmd.
There is no requirement that the root cmd should be the same when
`RegisterFlagCompletionFunc` was called. Storing the flags there does
not work when you add the the flags to your cmd struct before you add the
cmd to the parent/root cmd. The flags can no longer be found in the rigth
place when the completion command is called and thus the flag completion
does not work.

Also spf13#1423 claims that this would be thread safe but we still have a map
which will fail when accessed concurrently. To truly fix this issue use a
RWMutex.

Fixes spf13#1437
Fixes spf13#1320

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/cobra that referenced this issue Jul 2, 2021
The flag completion functions should not be stored in the root cmd.
There is no requirement that the root cmd should be the same when
`RegisterFlagCompletionFunc` was called. Storing the flags there does
not work when you add the the flags to your cmd struct before you add the
cmd to the parent/root cmd. The flags can no longer be found in the rigth
place when the completion command is called and thus the flag completion
does not work.

Also spf13#1423 claims that this would be thread safe but we still have a map
which will fail when accessed concurrently. To truly fix this issue use a
RWMutex.

Fixes spf13#1437
Fixes spf13#1320

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/cobra that referenced this issue Jul 2, 2021
The flag completion functions should not be stored in the root cmd.
There is no requirement that the root cmd should be the same when
`RegisterFlagCompletionFunc` was called. Storing the flags there does
not work when you add the the flags to your cmd struct before you add the
cmd to the parent/root cmd. The flags can no longer be found in the rigth
place when the completion command is called and thus the flag completion
does not work.

Also spf13#1423 claims that this would be thread safe but we still have a map
which will fail when accessed concurrently. To truly fix this issue use a
RWMutex.

Fixes spf13#1437
Fixes spf13#1320

Signed-off-by: Paul Holzinger <[email protected]>
jpmcb pushed a commit that referenced this issue Jul 2, 2021
* Fix flag completion

The flag completion functions should not be stored in the root cmd.
There is no requirement that the root cmd should be the same when
`RegisterFlagCompletionFunc` was called. Storing the flags there does
not work when you add the the flags to your cmd struct before you add the
cmd to the parent/root cmd. The flags can no longer be found in the rigth
place when the completion command is called and thus the flag completion
does not work.

Also #1423 claims that this would be thread safe but we still have a map
which will fail when accessed concurrently. To truly fix this issue use a
RWMutex.

Fixes #1437
Fixes #1320

Signed-off-by: Paul Holzinger <[email protected]>

* Fix trailing whitespaces in fish comp scripts

Signed-off-by: Paul Holzinger <[email protected]>
marckhouzam added a commit to marckhouzam/cobra that referenced this issue Oct 23, 2023
Different problems have been reported about flag completion registration.
These two tests are the cases that were not being verified but had been
mentioned as problematic.

Ref:
- spf13#1320
- spf13#1438 (comment)

Signed-off-by: Marc Khouzam <[email protected]>
marckhouzam added a commit that referenced this issue Oct 28, 2023
Different problems have been reported about flag completion registration.
These two tests are the cases that were not being verified but had been
mentioned as problematic.

Ref:
- #1320
- #1438 (comment)

Signed-off-by: Marc Khouzam <[email protected]>
Dav-14 added a commit to formancehq/cobra that referenced this issue Mar 15, 2024
* Create unit test illustrating unknown flag bug (spf13#1854)

Created a unit test that tests the unknown flag
error message when the unknown flag is located
in different arg positions.

* Update stale.yml (spf13#1863)

* fix: force ForEach-Object to return array in pwsh completion (spf13#1850)

Fixes spf13#1847

* Makefile: add target richtest (spf13#1865)

Don't require contributors to install richgo but keep it as an option and for CI

* build(deps): bump golangci/golangci-lint-action from 3.2.0 to 3.3.1 (spf13#1851)

Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.2.0 to 3.3.1.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](golangci/golangci-lint-action@v3.2.0...v3.3.1)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update kubescape org (spf13#1874)

Signed-off-by: David Wertenteil <[email protected]>

* ci: deprecate go 1.15 (spf13#1866)

Remove testing for go 1.15 to allow CI to pass, but don't force projects to upgrade.

* fix: conflict import name with variable (spf13#1879)

`template` is an import in `cobra.go` file and also used as a variable
name, which masks the library in the scope of that function.

* Update badge route (spf13#1884)

Based on
badges/shields#8671

* fix: func name in doc strings (spf13#1885)

Corrected the function name at the start of doc strings, as per the convention
outlined in official go documentation: https://go.dev/blog/godoc

* completions: do not detect arguments with dash as 2nd char as flag (spf13#1817)

Fixes spf13#1816

Previously, arguments with a dash as the second character (e.g., 1-ff00:0:1)
were detected as a flag by mistake. This resulted in auto completion misbehaving
if such an argument was last in the argument list during invocation.

* build(deps): bump github.com/inconshreveable/mousetrap (spf13#1872)

* Add documentation about disabling completion descriptions (spf13#1901)

* Improve MarkFlagsMutuallyExclusive example in User Guide (spf13#1904)

* Update shell_completions.md (spf13#1907)

align documentation with the code : completions.go:452

* build(deps): bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 (spf13#1902)

* Removes stale bot from GitHub action (spf13#1908)

Signed-off-by: John McBride <[email protected]>

* Add keeporder to shell completion (spf13#1903)

This allows programs to request the shell to maintain the order of completions that was returned by the program

* Add support for PowerShell 7.2+ (spf13#1916)

PowerShell 7.2 has changed the way arguments are passed to executables.
This was originally an experimental feature in 7.2, but as of 7.3 it is
built-in. A simple "" is now sufficient for passing empty arguments, no
back-tick escaping is required.

Fixes spf13#1849

Signed-off-by: Oldřich Jedlička <[email protected]>
Co-authored-by: Oldřich Jedlička <[email protected]>

* ci: deprecate go 1.16 (spf13#1926)

* ci: test Golang 1.20 (spf13#1925)

* update copyright year (spf13#1927)

* Update projects_using_cobra.md (spf13#1932)

Signed-off-by: Florent Poinsard <[email protected]>

* Document suggested layout for subcommands (spf13#1930)

Signed-off-by: Luiz Carvalho <[email protected]>

* Allow sourcing zsh completion script (spf13#1917)

Although it is not the recommended approach, sourcing a completion
script is the simplest way to get people to try using shell completion.
Not allowing it for zsh has turned out to complicate shell completion
adoption.  Further, many tools modify the zsh script to allow sourcing.

This commit allows sourcing of the zsh completion script.

Signed-off-by: Marc Khouzam <[email protected]>

* Update main image to better handle dark background (spf13#1883)

Fixes spf13#1880

Signed-off-by: Marc Khouzam <[email protected]>
Co-authored-by: Deleplace <[email protected]>

* Fix typo in fish completions (spf13#1945)

* build(deps): bump golangci/golangci-lint-action from 3.4.0 to 3.5.0 (spf13#1971)

* Fix grammar: 'allows to' (spf13#1978)

The use in generated bash completion files is getting flagged by
Lintian (the Debian package linting tool).

Signed-off-by: Taavi Väänänen <[email protected]>

* test: make fish_completions_test more robust (spf13#1980)

Use temporary files instead of assuming the current directory is
writable. Also, if creating a temporary file still returns an error,
prevent the test from failing silently by replacing `log.Fatal` with
`t.Fatal`.

* powershell: escape variable with curly brackets (spf13#1960)

This fixes an issue with program names that include a dot, in our case
`podman.exe`. This was caused by the change in commit 6ba7ebb.

Fixes spf13#1853

Signed-off-by: Paul Holzinger <[email protected]>

* build(deps): bump golangci/golangci-lint-action from 3.5.0 to 3.6.0 (spf13#1976)

* Move documentation sources to site/content (spf13#1428)

* Add 'one required flag' group (spf13#1952)

* golangci: enable 'unused' and disable deprecated replaced by it (spf13#1983)

* doc: fix typo, Deperecated -> Deprecated (spf13#2000)

* minor corrections to unit tests (spf13#2003)

* build(deps): bump golangci/golangci-lint-action from 3.6.0 to 3.7.0 (spf13#2021)

* command: temporarily disable G602 due to securego/gosec#1005 (spf13#2022)

* ci: test golang 1.21 (spf13#2024)

* Customizable error message prefix (spf13#2023)

* feat: add getters for flag completions (spf13#1943)

* Add notes to doc on preRun and postRun condition (spf13#2041)

* build(deps): bump actions/setup-go from 3 to 4 (spf13#1934)

* build(deps): bump github.com/cpuguy83/go-md2man/v2 from 2.0.2 to 2.0.3 (spf13#2047)

* Allow running persistent run hooks of all parents (spf13#2044)

Currently, only one of the persistent pre-runs and post-runs is executed.
It is always the first one found in the parents chain, starting at this command.
Expected behavior is to execute all parents' persistent pre-runs and post-runs.

Dependent projects implemented various workarounds for this:
- manually building persistent hook chains (in every hook).
- applying some kind of monkey-patching on top of Cobra.

This change eliminates the necessity for such workarounds
by allowing to set a global variable EnableTraverseRunHooks.

Tickets:
- spf13#216
- spf13#252

Signed-off-by: Volodymyr Khoroz <[email protected]>

* Fix linter errors (spf13#2052)

When using golangci-lint v1.55.0 some new errors were being reported.

Signed-off-by: Marc Khouzam <[email protected]>

* Don't complete --help flag when flag parsing disabled (spf13#2061)

Fixes spf13#2060

When a command sets `DisableFlagParsing = true` it requests the
responsibility of doing all the flag parsing. Therefore even the
`--help/-f/--version/-v` flags should not be automatically completed
by Cobra in such a case.

Without this change the `--help/-h/--version/-v` flags can end up being
completed twice for plugins: one time from cobra and one time from the
plugin (which has set `DisableFlagParsing = true`).

Signed-off-by: Marc Khouzam <[email protected]>

* Add tests for flag completion registration (spf13#2053)

Different problems have been reported about flag completion registration.
These two tests are the cases that were not being verified but had been
mentioned as problematic.

Ref:
- spf13#1320
- spf13#1438 (comment)

Signed-off-by: Marc Khouzam <[email protected]>

* Replace all non-alphanumerics in active help env var program prefix (spf13#1940)

* Replace all non-alphanumerics in active help env var program prefix

There are other characters besides the dash that are fine in program
names, but are problematic in environment variable names. These include
(but are not limited to) period, space, and non-ASCII letters.

* Another change in docs to mention non-ASCII-alphanumeric instead of just dash

* build(deps): bump actions/checkout from 3 to 4 (spf13#2028)

* Support usage as plugin for tools like kubectl (spf13#2018)

In this case the executable is `kubectl-plugin`, but we run it as:

    kubectl plugin

And the help text should reflect the actual usage of the command.

To create a plugin, add the cobra.CommandDisplayNameAnnotation:

    rootCmd := &cobra.Command{
        Use: "plugin",
        Annotations: map[string]string{
            cobra.CommandDisplayNameAnnotation: "kubectl plugin",
        }
    }

Internally this change modifies CommandPath() for the root command to
return the command display name instead of the command name. This is
used for error messages, help text generation, and completions.

CommandPath() is expected to have spaces and code using it already
handle spaces (e.g replacing with _), so hopefully this does not break
anything.

Fixes: spf13#2017

Signed-off-by: Nir Soffer <[email protected]>

* Improve API to get flag completion function (spf13#2063)

The new API is simpler and matches the `c.RegisterFlagCompletionFunc()`
API.  By removing the global function `GetFlagCompletion()` we are more
future proof if we ever move from a global map of flag completion
functions to something associated with the command.

The commit also makes this API work with persistent flags by using
`c.Flag(flagName)` instead of `c.Flags().Lookup(flagName)`.

The commit also adds unit tests.

Signed-off-by: Marc Khouzam <[email protected]>

* feat: expose GetCompletions (was getCompletions)

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: David Wertenteil <[email protected]>
Signed-off-by: John McBride <[email protected]>
Signed-off-by: Oldřich Jedlička <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Luiz Carvalho <[email protected]>
Signed-off-by: Marc Khouzam <[email protected]>
Signed-off-by: Taavi Väänänen <[email protected]>
Signed-off-by: Paul Holzinger <[email protected]>
Signed-off-by: Volodymyr Khoroz <[email protected]>
Signed-off-by: Nir Soffer <[email protected]>
Co-authored-by: Brian Pursley <[email protected]>
Co-authored-by: Enrico Candino <[email protected]>
Co-authored-by: Norman Dankert <[email protected]>
Co-authored-by: Unai Martinez-Corral <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Wertenteil <[email protected]>
Co-authored-by: Yash Ladha <[email protected]>
Co-authored-by: Seonghyeon Cho <[email protected]>
Co-authored-by: Dominik Roos <[email protected]>
Co-authored-by: Shihta Kuan <[email protected]>
Co-authored-by: janhn <[email protected]>
Co-authored-by: Ggg6542 <[email protected]>
Co-authored-by: John McBride <[email protected]>
Co-authored-by: Gyanendra Mishra <[email protected]>
Co-authored-by: Oldřich Jedlička <[email protected]>
Co-authored-by: Oldřich Jedlička <[email protected]>
Co-authored-by: Florent Poinsard <[email protected]>
Co-authored-by: Luiz Carvalho <[email protected]>
Co-authored-by: Marc Khouzam <[email protected]>
Co-authored-by: Deleplace <[email protected]>
Co-authored-by: Tom Payne <[email protected]>
Co-authored-by: Taavi Väänänen <[email protected]>
Co-authored-by: Branch Vincent <[email protected]>
Co-authored-by: Paul Holzinger <[email protected]>
Co-authored-by: Martijn Evers <[email protected]>
Co-authored-by: gocurr <[email protected]>
Co-authored-by: Jun Nishimura <[email protected]>
Co-authored-by: Nuno Adrego <[email protected]>
Co-authored-by: Souma <[email protected]>
Co-authored-by: Alexandru-Claudius Virtopeanu <[email protected]>
Co-authored-by: Haoming Meng <[email protected]>
Co-authored-by: vkhoroz <[email protected]>
Co-authored-by: Ville Skyttä <[email protected]>
Co-authored-by: Nir Soffer <[email protected]>
Co-authored-by: Geoffrey Ragot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants