-
Notifications
You must be signed in to change notification settings - Fork 86
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adds a new publisher type that allows for publishing to the local disk of the compute node. This allows users to test out the publisher functionality without needing to set up a remote storage service making it easier to get started with Bacalhau. Currently this implemention is does not include features such as content lifetimes, or secure access, and so is not recommended for production use yet. This publisher is useful for testing and development purposes. The local publisher comprises two parts. The publisher is responsible for taking the output from a job, compressing it and moving it to a location where it can be access by the second part. This part is the HTTP server which delivers the content back to the caller. By default the HTTP server listens on port 6001, but this can be changed by setting the `--local-publisher-port` flag on the compute node. The HTTP server will serve the content from the directory specified by the `local-publisher-directory` flag. If this is not set, the HTTP server will serve the content from a subdirectory of the configured bacalhau storage directory. Finally the `--local-publisher-address` flag can be used to specify the address that the HTTP server listens on. The default for this varies by environment, in test and development environments it uses `localhost`, in production environments it uses `public` to obtain a public address. Of course, you can set these values in config in preference to using the command line flags if the defaults are not suitable.
- Loading branch information
Showing
33 changed files
with
672 additions
and
43 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
package configflags | ||
|
||
import "github.com/bacalhau-project/bacalhau/pkg/config/types" | ||
|
||
var LocalPublisherFlags = []Definition{ | ||
{ | ||
FlagName: "local-publisher-address", | ||
DefaultValue: Default.Node.Compute.LocalPublisher.Address, | ||
ConfigPath: types.NodeComputeLocalPublisherAddress, | ||
Description: `The address for the local publisher's server to bind to`, | ||
}, | ||
{ | ||
FlagName: "local-publisher-port", | ||
DefaultValue: Default.Node.Compute.LocalPublisher.Port, | ||
ConfigPath: types.NodeComputeLocalPublisherPort, | ||
Description: `The port for the local publisher's server to bind to (default: 6001)`, | ||
}, | ||
{ | ||
FlagName: "local-publisher-directory", | ||
DefaultValue: Default.Node.Compute.LocalPublisher.Directory, | ||
ConfigPath: types.NodeComputeLocalPublisherDirectory, | ||
Description: `The directory where the local publisher will store content`, | ||
}, | ||
} |
43 changes: 43 additions & 0 deletions
43
docs/docs/setting-up/other-specifications/publishers/local.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
--- | ||
sidebar_label: Local | ||
--- | ||
|
||
# Local Publisher Specification | ||
|
||
Bacalhau's Local Publisher provides a useful option for storing task results on the compute node, allowing for ease of access and retrieval for testing or trying our Bacalhau. | ||
|
||
:::danger | ||
|
||
The Local Publisher should not be used for Production use as it is not a reliable storage option. For production use, we recommend using a more reliable option such as an S3-compatible storage service. | ||
::: | ||
|
||
## Local Publisher Parameters | ||
The local publisher requires no specific parameters to be defined in the publisher specification. The user only needs to indicate the publisher type as "local", and Bacalhau handles the rest. Here is an example of how to set up a Local Publisher in a job specification. | ||
|
||
```yaml | ||
Publisher: | ||
Type: local | ||
``` | ||
## Published Result Specification | ||
Once the job is executed, the results are published to the local compute node, and stored as compressed tar file, which can be accessed and retrieved over HTTP from the command line using the `get` command. TAhis will download and extract the contents for the user from the remove compute node. | ||
|
||
### Result Parameters | ||
- URL `(string)`: This is the HTTP URL to the results of the computation, which is hosted on the compute node where it ran. | ||
Here's a sample of how the published result might appear: | ||
|
||
```yaml | ||
PublishedResult: | ||
Type: local | ||
Params: | ||
URL: "http://192.168.0.11:6001/e-c4b80d04-ff2b-49d6-9b99-d3a8e669a6bf.tgz" | ||
``` | ||
|
||
In this example, the task results will be stored on the compute node, and can be referenced and retrieved using the specified URL. | ||
|
||
|
||
## Caveats | ||
|
||
- By default the compute node will attempt to use a public address for the HTTP server delivering task output, but there is no guarantee that the compute node is accessible on that address. If the compute node is behind a NAT or firewall, the user may need to manually specify the address to use for the HTTP server in the `config.yaml` file. | ||
- There is no lifecycle management for the content stored on the compute node. The user is responsible for managing the content and ensuring that it is removed when no longer needed before the compute node runs out of disk space. | ||
- If the address/port of the compute node changes, then previously stored content will no longer be accessible. The user will need to manually update the address in the `config.yaml` file and re-publish the content to make it accessible again. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.