-
Notifications
You must be signed in to change notification settings - Fork 86
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
persist store paths, and migrate repo to v3 (#3429)
### Summary This PR bumps the repo version from v2 to v3, where we introduce the following changes: 1. Change the execution store and job store default paths to no longer include the node name in the path. To clarify, v2 paths look like `~/.bacalhau/QmUBgU7xHKK44RuTHgrvnJfoSdZJS4fddT197iyTF5qjEV-compute/executions.db` and `QmUBgU7xHKK44RuTHgrvnJfoSdZJS4fddT197iyTF5qjEV-requester/jobs.db`, whereas in v3 they are `~/.baclahau/compute_store/executions.db` and `~/.bacalhau/orchestrator_store/jobs.db`. This change is needed to allow users to change our node names to longer depend on libp2p keys. 1. Persists the execution store and job store paths in `config.yaml` so that users won't lose their state if we change our default path names in the future 2. Adds a new `Node.Name` config, generates a node name still based on libp2p, and persist it. This will simplify an upcoming PR that allows users to define their own node name through the cli 3. Introduce migration scripts that will migrate v2 repos by creating `config.yaml` if it doesn't exist, persist the store paths, and rename old path names. No such migration will happen if the user provided their own `config.yaml` with store paths configured. ### `bacalhau config show` I've also introduced `bacalhau config show` command to print out the current configuration which includes configs in `config.yaml`, env variables and default values. ### Few issues encountered 1. The repo is initialized in the `RootCmd` before any flags are registered. This means only default configurations are used when initializing the repo and any env variables or flags will be ignored. For example, if I run `bacalhau serve --compute-execution-store-path my-awesome-path` for the very first time, then bacalhau will still initialize the repo with default value, create `~/.bacalhau/compute-store` directory, and create `config.yaml` with `~/.bacalhau/compute-store` instead of my provided one. It is difficult to do things differently with our current setup as we initialize the repo in any command, but these flags are only available for `serve` command 2. We use global `viper` instance to set our config. I had to reset the config in my tests before each run. It might be safer to not use the global instance 4. I've tried creating a repo per node in devstack, but couldn't get that to work because of our global configs. Configurations of the first repo initialized are also passed to the second repo's config, and we have test cases that fetch info of the very first repo initialized, such as https://github.com/bacalhau-project/bacalhau/blob/badd1ce5fd9508bd615f9a8e08bd8ce44d2eceb7/pkg/test/devstack/timeout_test.go#L112 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced new files to enhance configuration management, including displaying default and current settings. - Added functionality for handling private keys within the libp2p network. - Implemented a new approach for repository migration, including updating repository versions and handling specific configurations. - Enhanced CLI commands for better configuration setup and display. - **Refactor** - Updated the logic flow in configuration setup, removing unnecessary imports and functions. - Simplified the initialization of repository and node configuration, including passing context parameters where needed. - Adjusted error messaging for clearer debugging. - Streamlined setup and teardown processes in tests. - **Bug Fixes** - Fixed exclusion patterns in pre-commit hooks to correctly ignore specified directories. - **Chores** - Updated imports and removed unused code across multiple files for cleaner codebase management. - **Documentation** - Added summaries and comments for clarity in understanding the purpose and impact of changes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Loading branch information
Showing
67 changed files
with
1,079 additions
and
514 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
package config | ||
|
||
import ( | ||
"github.com/bacalhau-project/bacalhau/pkg/config" | ||
"github.com/spf13/cobra" | ||
"github.com/spf13/viper" | ||
"gopkg.in/yaml.v3" | ||
) | ||
|
||
func newDefaultCmd() *cobra.Command { | ||
showCmd := &cobra.Command{ | ||
Use: "default", | ||
Short: "Show the default bacalhau config.", | ||
RunE: func(cmd *cobra.Command, args []string) error { | ||
return defaultConfig(cmd) | ||
}, | ||
} | ||
showCmd.PersistentFlags().String("path", viper.GetString("repo"), "sets path dependent config fields") | ||
return showCmd | ||
} | ||
|
||
func defaultConfig(cmd *cobra.Command) error { | ||
// clear any existing configuration before generating the default. | ||
config.Reset() | ||
defaultConfig, err := config.Init(cmd.Flag("path").Value.String()) | ||
if err != nil { | ||
return err | ||
} | ||
cfgbytes, err := yaml.Marshal(defaultConfig) | ||
if err != nil { | ||
return err | ||
} | ||
cmd.Println(string(cfgbytes)) | ||
return nil | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
package config | ||
|
||
import ( | ||
"github.com/bacalhau-project/bacalhau/pkg/config" | ||
"github.com/spf13/cobra" | ||
"github.com/spf13/viper" | ||
"gopkg.in/yaml.v3" | ||
) | ||
|
||
func newShowCmd() *cobra.Command { | ||
showCmd := &cobra.Command{ | ||
Use: "show", | ||
Short: "Show the current bacalhau config.", | ||
RunE: func(cmd *cobra.Command, args []string) error { | ||
return showConfig(cmd) | ||
}, | ||
} | ||
showCmd.PersistentFlags().String("path", viper.GetString("repo"), "sets path dependent config fields") | ||
return showCmd | ||
} | ||
|
||
func showConfig(cmd *cobra.Command) error { | ||
// clear any existing configuration before generating the current. | ||
currentConfig, err := config.Init(cmd.Flag("path").Value.String()) | ||
if err != nil { | ||
return err | ||
} | ||
cfgbytes, err := yaml.Marshal(currentConfig) | ||
if err != nil { | ||
return err | ||
} | ||
cmd.Println(string(cfgbytes)) | ||
return nil | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.