Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added low-level mechanism to run weaver apps. #651

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

mwhittaker
Copy link
Member

This PR introduces a new low-level mechanism, run.Run, to run a weavelet when the topology of the application (e.g., the address of every listener and component) is static and predetermined.

To draw an analogy, git distinguishes between plumbing and porcelain. Plumbing commands are complicated low-level commands. Porcelain commands are simple, high-level commands that are built on top of plumbing commands. Most users use porcelain commands most (or all) of the time, but plumbing commands exist to make more complex git operations possible (though not necessarily convenient).

For Service Weaver, weaver single, weaver multi, weaver ssh, weaver kube, and weaver gke are porcelain commands. We don't offer any plumbing commands, besides writing a deployer completely from scratch. This PR aims to introduce a generally useful plumbing command.

Motivation

Here's our vision for how this PR will be useful. The details are not final, but I want to share something concrete to motivate the PR. We'll add a user-facing plumbing command on top of the run package, let's say weaver run. weaver run will take a topology file describing the topology of the application, say topo.toml:

deployment_id = "v1"
topology = {
    "Main": ["localhost:9000"],
    "Odd": ["localhost:9001"],
    "Even": ["localhost:9002"],
}
groups = [
    {
        name: "main",
        components: ["Main"],
        listeners: {"collatz": "localhost:8000"},
        address: "localhost:9000",
    },
    {
        name: "odd",
        components: ["Odd"],
        address: "localhost:9001",
    },
    {
        name: "even",
        components: ["Even"],
        address: "localhost:9002",
    },
]

Then, you can run an application manually like this:

$ weaver run weaver.toml topo.toml odd  # run "odd" group
$ weaver run weaver.toml topo.toml even # run "even" group
$ weaver run weaver.toml topo.toml main # run "main" group

This command will be generally useful, but our motivating use case is to enable people to run Service Weaver applications on Kubernetes with a level of control not possible with weaver kube.

This PR introduces a new low-level mechanism, `run.Run`, to run a
weavelet when the topology of the application (e.g., the address of
every listener and component) is static and predetermined.

To draw an analogy, git distinguishes between [plumbing and
porcelain][git]. Plumbing commands are complicated low-level commands.
Porcelain commands are simple, high-level commands that are built on top
of plumbing commands. Most users use porcelain commands most (or all)
of the time, but plumbing commands exist to make more complex git
operations possible (though not necessarily convenient).

For Service Weaver, `weaver single`, `weaver multi`, `weaver ssh`,
`weaver kube`, and `weaver gke` are porcelain commands. We don't offer
any plumbing commands, besides writing a deployer completely from
scratch. This PR aims to introduce a generally useful plumbing command.

> Motivation

Here's our vision for how this PR will be useful. The details are not
final, but I want to share something concrete to motivate the PR.
We'll add a user-facing plumbing command on top of the `run` package,
let's say `weaver run`. `weaver run` will take a topology file
describing the topology of the application, say `topo.toml`:

```toml
deployment_id = "v1"
topology = {
    "Main": ["localhost:9000"],
    "Odd": ["localhost:9001"],
    "Even": ["localhost:9002"],
}
groups = [
    {
        name: "main",
        components: ["Main"],
        listeners: {"collatz": "localhost:8000"},
        address: "localhost:9000",
    },
    {
        name: "odd",
        components: ["Odd"],
        address: "localhost:9001",
    },
    {
        name: "even",
        components: ["Even"],
        address: "localhost:9002",
    },
]
```

Then, you can run an application manually like this:

```shell
$ weaver run weaver.toml topo.toml odd  # run "odd" group
$ weaver run weaver.toml topo.toml even # run "even" group
$ weaver run weaver.toml topo.toml main # run "main" group
```

This command will be generally useful, but our motivating use case is to
enable people to run Service Weaver applications on Kubernetes with a
level of control not possible with `weaver kube`.

[git]: https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
@mwhittaker mwhittaker self-assigned this Oct 5, 2023
mwhittaker added a commit that referenced this pull request Oct 5, 2023
```console
$ weaver exec --help
Execute a Service Weaver application.

Usage:
  weaver exec <config file> <topology file> <node>

Flags:
  -h, --help	Print this help message.

Description:
  "weaver exec" is low-level command to run a Service Weaver application when
  the topology of the application (e.g., the address of every listener and
  component) is static and predetermined.

  It is more convenient to deploy an application using a deployer like "weaver
  multi" or "weaver kube". "weaver exec" gives you more control over how you
  deploy your application, but this comes at the cost of being more cumbersome
  to use.

Topology Files:
  To deploy an application using "weaver exec", you must specify the topology
  of your application in a TOML file. For example, consider the following
  topology file for a Service Weaver application with components Main, Foo, and
  Bar where the Main component exports a listener named lis.

      ┌────────────────────topology.toml──────────────────────┐
      │ id = "v1"                                             │
      │                                                       │
      │ [nodes.main]                                          │
      │ address = "localhost:9000"                            │
      │ components = ["github.com/ServiceWeaver/weaver/Main"] │
      │ listeners.lis = "localhost:8000"                      │
      │                                                       │
      │ [nodes.foobar]                                        │
      │ address = ":9001"                                     │
      │ dial_address = "localhost:9001"                       │
      │ components = [                                        │
      │     "github.com/example/app/Foo",                     │
      │     "github.com/example/app/Bar",                     │
      │ ]                                                     │
      └───────────────────────────────────────────────────────┘

  A topology file begins with a unique deployment id (e.g., "v1"). Next is a
  series of node definitions. Every node has a name (e.g., "main", "foobar")
  and the following fields:

      - address:      The address of the node.
      - dial_address: The dialable address of the node (defaults to address).
      - components:   A list of the components hosted on the node.
      - listeners:    A map from listener names to listener addresses.

  In the example above,

  - The "main" node runs on address "localhost:9000" with dialable address
    "localhost:9000". It hosts the Main component and exports listener
    "lis" on "localhost:8000".
  - The "foobar" node listens on address ":9001" with dialable address
    "localhost:9001". It hosts components Foo and Bar.

  Note that nodes are a logical concept. You can run multiple nodes on the same
  physical machine, or you can run multiple nodes on different machines.

  To deploy an application with "weaver exec", you provide the weaver config
  file, the topology file, and the name of the node to execute. To run the
  example above, we would run the following two commands:

      $ weaver exec weaver.toml topology.toml foobar # run the foobar node
      $ weaver exec weaver.toml topology.toml main   # run the main node
```

See #651 for context.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant