-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Pitch] Apache avro serialization #74
Comments
Also ... I f you could point me out to an example of a json-resialized |
other random thoughts... Questions:
vs
OR EVEN .... This would be very efficient in terms of space
|
Hello, I wanted to make sure you knew this was seen, but haven't had a chance to write a full response, will hopefully respond by this Friday. |
The JSON scheme was made for ergonomics of use, not for size, and you're correct, it's that the
The outer-most container isn't a
The name is again kept for human readability. If you were writing a binary format you wouldn't need to store it, just the accession or something like it to be able to get back to the full
I don't know if this is explicitly called out in the specification, but I think (and may be wrong) that readers should support and recognize reading both separate and compound values like We don't have any example libraries with clusters in them yet, but I converted one of the SpectraST libraries to JSON. It includes several attribute sets. I think there are still problems with the conversion though, so I'll work on this more in the coming week. |
Hi y'all!
I started a (VERY EARLY PROTOTYPE) that implements serialization to apache avro.
I think it would be a good alternative to json with more efficient disk usage.
https://github.com/jspaezp/avrospeclib
I am still implementing the schema using pydantic and deriving form it the
avro schema.
Some disk usage metrics on a reasonably large speclib I have
Read-write speeds
let me know if there is any interest in adopting it!
best!
The text was updated successfully, but these errors were encountered: