-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Encoding traits, Path + Entry impls #23
Conversation
Hi, I am having a look at this at the moment and the implications for usage in iroh-willow. My question is: Is it necessary to couple the Encoder and Decoder traits to ufotofu? My guess is that the reasoning is that with this design With the trait coupled to Edit: After another look at ufotofu, I think I can implement the traits for my channel struct. Will report back once I tried it out. |
To me, the true cost is the write buffer itself. You need to allocate it, and you need to copy its contents into the channel primitive. The ufotofu design allows for zero-copy serialisation, where you write directly to the channel (even if its internal buffer is smaller than the total size of the encoding) without intermediate allocations. This is a win, and the resulting code is also more (or, at least, quite) elegant (for example, the entry encoder). The big drawback is having to adopt ufotofu, of course. For obvious reasons I don't consider that as a drawback conceptually, but I cannot talk away the real engineering cost (and risk?), especially since the ufotofu codebase is immature and still undergoing some changes. In particular, you'd need to code against the I'd be up for a call to help with ufotofu integration and just general high-bandwidth sharing. |
Not necessarily. My encoder trait encodes into My It does not use MaybeUninitialized, but that is an ortogonal concern. I do like the design of the ufotofu traits, I am just unclear if the coupling to these new and still unstable traits is required for the quite low-level encoder trait. Edit: Links to the traits in iroh-willow: traits: https://github.com/n0-computer/iroh/blob/willow/iroh-willow%2Fsrc%2Futil%2Fcodec.rs usage in channel send: (exact signatures are different from what I wrote pseudocodish above but conceptually the same) Note that I am not opposed to the traits introduced here conceptually, but interested if the encoding can be made less coupled to the io traits without perf or ergonomic downsides. |
I'm not sure whether we are talking past each other. I'm basically interested in the setting where the buffer of the channel (say, the OS-managed buffer for to write tcp data into) is smaller than the encoding. As far as I'm understanding the sentence I quoted, you'd require a channel that can allocate arbitrarily large buffers on demand, which is what the ufotofu design avoids. Then again, the code you linked uses In any case, the ufotofu design is pretty much the same as that of
Agreed. I keep switching between being happy or unhappy with the ufotofu approach. But irrespective of those feelings, it will stay the way it is, because it should have the ability to work with uninitialised memory. I just wish it was less of a pain...
Yeah, I share the sentiment, but I didn't find a satisfying way to do it. I think the ability for the channel to dictate fragment sizes is crucial, and that is what the "io[-like] traits" are all about. In my opinion, there is simply a void in terms of popular traits/apis in the rust ecosystem for general, io-like traits. And ufotofu fills that void in a - to me - satisfying manner. It just isn't a popular API (yet?). Feel free to reach out on discord if you want to schedule a call. |
Now with zero-copy Path changes merged in. |
@Frando Just published ufotofu version 0.2.0 fyi, that one should be fairly stable to code against. |
compact_width
modulemax_power
moduleEncoder
andDecoder
traits which work with Ufotofu consumers and producers.Encoder
+Decoder
on PathRcEncoder
+Decoder
on Entryearthstar
crate with nascentNamespaceId
andIdentityId
structsArbitrary
on these types so we could use realistic values in fuzz tests as per Willow parameters for testing? #17.This is already enough to review, I will do the relative encodings in a separate branch off of this.
Still needs: