-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backwards compatibility? (new structure definitions but old json serialization) #997
Comments
So, the simplest thing to do is use optionals in the new version of the structure. That works if that new version simply extends the previous one. But if some types are changed or some fields are removed, that won't work. The next option is using a variant. We support any type with the interface similar to that of Finally, if the struct is not the root of the target, then you can write a Update: |
Thanks for the suggestions. Currently using a But this seems to not correctly handle the |
That example doesn't deal with variant, so I'm not sure what you mean. Do you have a variant as a member of a described struct? |
Yes I have a described struct containing a variant. What is needed to handle that? Is there an example you can point me to? |
It should work: https://godbolt.org/z/1YzKs4s4e |
It does not work for me if I use https://godbolt.org/z/Gc7Tq9PzK Passing the contextual tag there causes it to not hold the alternative |
The problem with your approach is that your You should either throw an exception (since you're using a throwing overload) or use a non-throwing overload and return a Also, I've noticed that you are using contexts incorrectly. When you are using a two context overload of BTW, why do you need that custom overload? Currently it doesn't do anything that the library isn't doing already. And the library actually does more, e.g. it has special handling for optionals. |
The custom overload is there to handle backward compatibility in a generic way. As new parameters are added I want to decode them as default if they are missing in the json serialization. Consider the following use case struct C {
std::variant<A, B> variant;
double new_param{14.1}; // new parameter added
};
BOOST_DESCRIBE_STRUCT(C, (), (variant, new_param))
// json string generated by old describe, need backward compatibility
std::string_view json = R"foo({"variant":{"d":3.14E0}})foo"; Is there a way to distinguish between boost json trying the different variants versus the case where a new field has been added? Can I look at |
I see. You are writing a templated overload seemingly intended for all described classes, but in fact you only need it for C. So, make it just for C. That way you don't need a context. Also, if it's an overload for a specific type, you can easily implement it using e.g. C c;
c.a = value_to<A>( obj.at("a") );
c.b = value_to<B>( obj.at("b") ); But if you don't want to do that, you can take the implementation for described classes from the library and slightly change it. I actuall did it here: https://godbolt.org/z/frbhPxqe5 The main interesting point is usage of template<>
struct is_required<
boost::describe::descriptor_by_pointer<C_Ds, &C::new_param> >
: std::false_type
{}; And then inside the overload I use it like this: auto const found = obj->find(D.name);
if( found == obj->end() )
{
if constexpr( !is_required< decltype(D) >::value ) // ignore missing new fields
return;
throw std::runtime_error("missing field");
} |
I would probably want to make struct A {
int i;
};
BOOST_DESCRIBE_STRUCT(A, (), (i))
struct B {
int i;
double d;
};
BOOST_DESCRIBE_STRUCT(B, (), (i,d))
struct C {
std::variant<A, B> variant;
double new_param = 14.1; // new parameter added
}; |
Oh, so you want to support these added fields in other structs too? Then, you do need to make the overload a template. |
Backwards support means I need to support added and removed fields in data structures. I can see boost 1.83 does support |
Ok, so you have several structs with a few members which were added, and a few which were removed. In addition, those structs are nested inside each other, right? To be honest, in this case I'd write custom BTW, this issue accidentally helped me to catch a bug: #999. |
I opened #991 for this. My use case was more about forward-compatibility and ignoring new fields the program doesn't know about, but it's kind of the same. |
At some point we can't provide a good enough generic solution. Generic solutions are for generic problems. This is why we allow you to fully customise behaviour with This problem of supporting older versions of a struct isn't very generic. Hence, I believe, the most efficient (in terms of time spent on writing it and in terms of ease of updating it in the future) is a If you wish to have some sort of generic approach, you can write a generic But., semantially, you expect a "variant" of versions of described struct: Finally, this approach of converting to/from type A instead of type B, followed/preceded by conversion bewtween B and A seems to arise quite often. And we probably want to support it in JSON explicitly. This is what #989 is about. If that is implemented, then user code would look something like this: namespace ns {
struct S_v0_1 { ... };
BOOST_DESCRIBE_STRUCT(S_v0_1, (), (...))
struct S_v0_2 { ... };
BOOST_DESCRIBE_STRUCT(S_v0_2, (), (...))
struct S_v0_2 { ... };
BOOST_DESCRIBE_STRUCT(S_v0_2, (), (...))
struct S_v1_0 { ... };
BOOST_DESCRIBE_STRUCT(S_v1_0, (), (...))
struct S : S_v1_0
{
...
explicit S( variant<S_v1_0, S_v0_2, S_v0_1>&& ); // for value_to
explicit operator variant<S_v1_0, S_v0_2, S_v0_1>(); // for value_from
};
} // namespace ns
namespace boost::json {
template<>
struct serialize_as<ns::S> { using type = variant<ns::S_v1_0, ns::S_v0_2, ns::S_v0_1>; };
} // namespace boost::json |
I agree this is not an easy one! The problem I have with the variant approach you suggest is that, while quite elegant, it doesn't scale very well for big structs and/or lots of versions. If I ever completely shuffle my struct this would serve me well, but not for adding fields one by one. But thanks for the clear examples, they helped me implement exactly what I wanted: #include <cstdio>
#include <boost/describe/class.hpp>
#include <boost/json/value_to.hpp>
#include <boost/json/parse.hpp>
struct A {
int a = 0;
int b = 1;
};
BOOST_DESCRIBE_STRUCT(A, (), (a, b))
struct JsonLenient {};
constexpr JsonLenient json_lenient;
template<typename T>
T tag_invoke(boost::json::value_to_tag<T>, const boost::json::value &v, JsonLenient) {
printf("debug: lenient tag_invoke for %s\n", typeid(T).name());
const boost::json::object& obj = v.as_object();
T result;
using Ds = boost::describe::describe_members<T,
boost::describe::mod_any_access | boost::describe::mod_inherited>;
size_t count = 0;
boost::mp11::mp_for_each<Ds>([&](auto D) {
auto it = obj.find(D.name);
if (it == obj.end()) {
printf("debug: default-constructed field %s\n", D.name);
return;
}
result.*D.pointer = boost::json::value_to<
std::decay_t<decltype(result.*D.pointer)>>(it->value());
++count;
});
if (count != obj.size())
printf("debug: extra members\n");
return result;
}
int main()
{
A a;
a = boost::json::value_to<A>(boost::json::parse(R"( {} )"), json_lenient);
printf(" *** a=%d b=%d\n\n", a.a, a.b);
a = boost::json::value_to<A>(boost::json::parse(R"( {"a":10} )"), json_lenient);
printf(" *** a=%d b=%d\n\n", a.a, a.b);
a = boost::json::value_to<A>(boost::json::parse(R"( {"a":10,"b":11} )"), json_lenient);
printf(" *** a=%d b=%d\n\n", a.a, a.b);
a = boost::json::value_to<A>(boost::json::parse(R"( {"a":10,"b":11,"c":12} )"), json_lenient);
printf(" *** a=%d b=%d\n\n", a.a, a.b);
}
The Maybe you could consider integrating something similar upstream ? It feels like a generic solution to a generic problem, because it just makes described structs parsing much less strict without any notion of versions/migrations (and all the headaches that comes with that). It would help to keep the simple cases simple. And for cases where you're not just adding ints and strings, you could add the variant trick on top of that. |
My solution to getting backwards compatibility was to use a templated |
What is the best way to achieve backward compatibility with boost json? My data structures has changed, but I still need to interact with json data generated by previous versions of my structs. What is the best way to achieve this? Any examples? Simply trying to invoke
boost::json::value_to
will throw an exception.I'm using boost json and boost describe version 1.83.
The text was updated successfully, but these errors were encountered: