Adding vbs support #2666

nyospe · 2024-02-27T16:30:06Z

Closes EspressoSystems/versioned-binary-serialization#6

This PR:

Enables versioned-binary-serialization in HotShot, including support through tide-disco and surf-disco.
Sets up a path for making HotShot generic over updatability versioning in the application layer.

This PR does not:

replace the hardcoded version 0.1 that was added for Updatability

Key places to review:

all implementators of ConnectedNetwork; note that the versions of WebServerNetwork and CombinedNetworks are specific to that network, and not the same as the versions of payloads (broadcast_message, direct_message) over those networks.
network control messages for Libp2pNetwork - I'm not sure I got that right, and it might need its own unique version for those.

https://github.com/EspressoSystems/versioned-binary-serialization/issues/6

ss-es

I think this makes sense to me overall -- left some minor comments + questions to check my understanding!

ss-es · 2024-02-27T16:58:01Z

crates/constants/src/lib.rs

@@ -1,6 +1,6 @@
 //! configurable constants for hotshot

-use serde::{Deserialize, Serialize};
+use versioned_binary_serialization::version::{StaticVersion, Version};


Just to check my understanding: is Version a value-level version and StaticVersion a type-level version?

Yes. We need the type-level version in a number of places... if Rust's const generics were more mature, I'd be using it as a binding in place of the MAJOR/MINOR thing as well... but having a value Version is also useful in a number of places.

ss-es · 2024-02-27T17:03:29Z

crates/constants/src/lib.rs

 /// Constant for protocol version 0.1.
 pub const VERSION_0_1: Version = Version { major: 0, minor: 1 };

+/// Constant for protocol static version 0.1.
+pub const STATIC_V_0_1: StaticVersion<0, 1> = StaticVersion;


nit/minor (feel free to ignore): could we call this STATIC_VERSION_0_1?

It got really verbose in places, I wanted a shorter name. But the _0_1 thing is going to go away as soon as I finish working out a few more things, in favor of generic _BASE and _UPDATE containers, and there will be a ::StaticVersion on those.

Is this better?

ss-es · 2024-02-27T17:03:56Z

crates/constants/src/lib.rs

+/// Constants for WebServerNetwork and WebServer
+/// The Web CDN is not, strictly speaking, bound to the network; it can have its own versioning.
+/// Web Server CDN Version (major)
+pub const WEB_SERVER_MAJOR: u16 = 0;


nit/minor (feel free to ignore): could we call this WEB_SERVER_MAJOR_VERSION?

crates/example-types/src/node_types.rs

ss-es · 2024-02-27T19:03:00Z

crates/hotshot-qc/src/bit_vector_old.rs

@@ -263,12 +263,12 @@ mod tests {
            // Check the QC and the QCParams can be serialized / deserialized
            assert_eq!(
                qc,
-                bincode::deserialize(&bincode::serialize(&qc).unwrap()).unwrap()
+                Serializer::<0, 1>::deserialize(&Serializer::<0, 1>::serialize(&qc).unwrap()).unwrap()


Since this a round-trip tests that should really work in any version, should this (and L271 in the same file) be Serializer::CURRENT_STATIC_VERSION? (Should we have a CURRENT_STATIC_VERSION? We could alias it to STATIC_VERSION_0_1, but then we wouldn't have to update it here.)

There does need to be a CURRENT_BASE_VERSION, and a CURRENT_UPGRADE_VERSION, and a compile-time check to determine if they are the same (e.g. no upgrade available), but aliasing won't work. I'm working on a solution using a generic type to impl on the core types, but I haven't worked it out yet. I left this (and a few other places) as-is because they're not actually hooked up to the real versioning.

ss-es · 2024-02-27T19:30:45Z

crates/libp2p-networking/tests/counter.rs

@@ -64,7 +64,7 @@ pub async fn counter_handle_network_event(
    match event {
        IsBootstrapped => {}
        GossipMsg(m, _) | DirectResponse(m, _) => {
-            if let Ok(msg) = bincode_opts().deserialize::<CounterMessage>(&m) {
+            if let Ok(msg) = Serializer::<0, 1>::deserialize::<CounterMessage>(&m) {


I think this (+ the other <0,1> and STATIC_V_0_1 in this file) are hardcoded to a version, even though the functions they're called in are not explicitly versioned.

... this one, specifically, might be a problem. I was less stringent about using real versions in tests, but I'm not sure this is uniform in all directions. I'll take a closer look.

ss-es · 2024-02-27T19:34:06Z

crates/orchestrator/src/lib.rs

+
+/// Orchestrator is not, strictly speaking, bound to the network; it can have its own versioning.
+/// Orchestrator Version (major)
+pub const ORCHESTRATOR_MAJOR: u16 = 0;


nit/minor (feel free to ignore): I think this should be ORCHESTRATOR_MAJOR_VERSION

ss-es · 2024-02-27T19:37:01Z

crates/testing/tests/memory_network.rs

@@ -184,7 +187,7 @@ async fn memory_network_direct_queue() {
    // Send messages
    for sent_message in first_messages {
        network1
-            .direct_message(sent_message.clone(), pub_key_2)
+            .direct_message(sent_message.clone(), pub_key_2, STATIC_V_0_1)


Same comments re: hard-coding versions without versioning function

ss-es · 2024-02-27T19:41:14Z

crates/testing/tests/unit/message.rs


-    assert_eq!(version.major, major_version_read);
-    assert_eq!(version.minor, minor_version_read);
+    let version_read = Version::deserialize(&serialized_message).unwrap().0;


I think this actually changes the test -- it's now testing a property of the Serializer (round-trip version de/serialize = identity), where before it was testing that the binary representation actually had the version in the right place.

I think the previous test was good to have somewhere, though you probably already have these tests in vbs. I think we could just delete this file?

Yes, there's tests in vbs... This probably should be deleted.

ss-es · 2024-02-27T19:44:01Z

crates/types/src/message.rs

@@ -34,9 +33,6 @@ use std::{fmt::Debug, marker::PhantomData};
 #[derive(Serialize, Deserialize, Clone, Debug, Derivative, PartialEq, Eq, Hash)]
 #[serde(bound(deserialize = "", serialize = ""))]
 pub struct Message<TYPES: NodeType> {


Just checking for my own understanding: NodeType is versioned right? (So Message is implicitly versioned through that?)

Not right now, no. I'm working on a way to do that...

Sorry, a bit of clarification. Nothing implements the trait Versioned, which has been added in vbs; it was meant to be added alongside Serialize and Deserialize as a feature macro, e.g.

#[derive(Serialize, Deserialize, Versioned)] #[base_version(1, 3)] #[max_version(1, 7)] struct MyStruct { }

but I've come to realize that we usually don't really know the actual version in crates that have generic types...

We may end up needing a crate-level version and a mechanism for mapping the versions of crates to the actual version, but that's not trivial. Another solution might be a tagged, but never versioned, repository containing a single "version" crate that gets bound in the executable target's Cargo.lock, and injects versions into our library crates.

Message is not inherently bound to a specific protocol version, though a change to Message will force versioning once Versioned is implemented. Not by way of NodeType, though.

I'm still trying to come up with a nice way to inject versions into the consensus by way of something coupled to NodeType, but giving it generic parameters is catastrophically contagious.

nyospe added 4 commits February 23, 2024 16:34

WIP

48bcb63

Still chasing a bug

2e1ab26

One down, one to go

8e3f2a7

... and done.

ec2c030

nyospe requested review from bfish713, DieracDelta, elliedavidson and ss-es February 27, 2024 16:30

Cleanup and finishing touches

1a89903

ss-es reviewed Feb 27, 2024

View reviewed changes

nyospe added 3 commits February 27, 2024 16:47

Merge branch 'main' into adding_vbs_support

060419c

Fixing the post-merge code, and one missed issue in the pre-merge code.

c84a82f

Some clarifying name changes.

cb63b1b

nyospe requested a review from ss-es February 28, 2024 18:03

nyospe added 6 commits February 28, 2024 16:17

Merge branch 'main' into adding_vbs_support

719a409

Merge branch 'main' into adding_vbs_support

3fc412c

Removing bincode dependency where applicable

c7a5849

Merge branch 'main' into adding_vbs_support

95ee913

cleanup of some (not quite all) remaining bincode invocations

a08d2a0

dependency versioning tags and cleanup

7879e9e

nyospe marked this pull request as ready for review March 1, 2024 21:27

nyospe added 2 commits March 4, 2024 14:43

Merge branch 'main' into adding_vbs_support

5ebeeee

Merge branch 'main' into adding_vbs_support

a713900

nyospe requested a review from lukaszrzasik as a code owner March 7, 2024 22:13

nyospe added 6 commits March 7, 2024 17:16

bump version

14c1731

Merge branch 'main' into adding_vbs_support

5946d58

Merge branch 'main' into adding_vbs_support

1bd0934

Bumping anticipated version

a2f9324

Merge branch 'main' into adding_vbs_support

6d28476

Now with new VBS

489c8e7

nyospe and others added 16 commits March 14, 2024 18:28

Merge branch 'main' into adding_vbs_support

08a648a

branch => tag on dependencies

3e78443

Update lock?

495f57d

Merge branch 'main' into adding_vbs_support

1242ea3

Cargo.lock update

a7fa618

Merge branch 'main' into adding_vbs_support

eb11d5d

Apparently just * lint runs with --features="hotshot-testing" enabled

447db83

integrate new storage type

6560903

fix build, tie up stragglers, modifying tasks next

5f39339

merge latest stable tabs

0471188

Merge branch 'main' into adding_vbs_support

957dcd4

merge latest tags and fix method signatures

398cc17

Update *-disco

4ecf438

Merge branch 'jp/issue-2621' into adding_vbs_support

17b7099

Dependencies updated to tags

a4a76ca

sync to main and fix the early merge consequences

f659c48

bfish713 approved these changes Mar 19, 2024

View reviewed changes

nyospe merged commit b0adec3 into main Mar 19, 2024
13 checks passed

nyospe deleted the adding_vbs_support branch March 19, 2024 18:31

rob-maron mentioned this pull request Mar 20, 2024

[Push CDN] Add whitelist update binary #2815

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding vbs support #2666

Adding vbs support #2666

nyospe commented Feb 27, 2024

ss-es left a comment

ss-es Feb 27, 2024

nyospe Feb 27, 2024

ss-es Feb 27, 2024

nyospe Feb 27, 2024

nyospe Feb 28, 2024

ss-es Feb 27, 2024

ss-es Feb 27, 2024

nyospe Feb 27, 2024

ss-es Feb 27, 2024 •

edited

Loading

nyospe Feb 27, 2024

ss-es Feb 27, 2024

ss-es Feb 27, 2024

ss-es Feb 27, 2024

nyospe Feb 27, 2024

ss-es Feb 27, 2024

nyospe Feb 27, 2024

nyospe Feb 27, 2024

nyospe Feb 27, 2024

Adding vbs support #2666

Adding vbs support #2666

Conversation

nyospe commented Feb 27, 2024

This PR:

This PR does not:

Key places to review:

ss-es left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ss-es Feb 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ss-es Feb 27, 2024 •

edited

Loading