Umi Template Library as a JSON schema

[samuelduchesne]

It's a tree! (actually, a DAG)

Umi Template Libraries are essentially a tree of objects that have parents referencing children. For example, a OpaqueConstruction object references multiple OpaqueMaterial objects; OpaqueMaterial can be shared between OpaqueConstruction objects.

The problem

The library files themselves don’t have data validation (such as limits on the min, max, choices, uniqueness, etc.; documentation of each attribute, units, etc.) which makes it, for now, the role of the developer to check if values are “valid” for ArchSim/EnergyPlus. Each new tool being developed needs new validations (archetypal has its own set of validation functions).

By using a python package called pydantic, we can define traditional python classes (in fact very similar to dataclasses) that can take attributes and a rich array of validator functions (handled by pydantic). Each time a class is initialized with data, the values are validated against the rules of the attribute.

Trying to create a MaterialLayer that is less than 3mm? An error would be raised. Objects can be serialized via model.json(…). Inversely, they can be parsed from json via model.parse_file(…). Now, archetypal does a lot of that kind of validation, because I coded those in, not because of the schema. A schema would standardize that.

Standardization is key

Once all your python classes are defined (one for Materials, one for Schedules, one for Zones, etc,), pydantic can create a proper JSON Schema file that is compliant with JSON Schema Core, JSON Schema Validation and OpenAPI. That is where things become VERY interesting. You can use tools that create clients in various language in one command. You need a C# library, ingest the schema and out comes a C# library with all the same validation as the original python system. Need a javascript library? Same thing.

Requirements

What de we need?

• Ability to load in-memory a tree representation of all available objects in a template library. That library can either be a serialized template library file or a database.
• Versioning the schema: It should be easy for a tool to know what version of UMI Template Libraries it is compatible with.
• Upgrade-Downgrade template library versions.

Questions

• What is the best way a serializing a graph?
• What does pydantic do when there is a nested class and you serialize

Prisma

• What about prisma? I heard it is cool.

[szvsw]

I made a little demo here, with accompanying code here.

It still relies on serializing to JSON, which Pydantic does handle by creating deeply nested objects (rather than the old, tabular style with objects identified by ref). The problem with that is of course that when deserializing, you would lose all referential relationships and end up with a fully unique tree... except for a little trick I implemented in the class validator, which, makes it so that you can read the nested tree and still get all relationships out!

I also implemented a simple method for outputting the old "flat"/tabular style of serialization in a much more concise way than the existing archetypal representation. Deserializing from that would really need more work, but ultimately if we want to go that route again, we really should use a db.

You can also use instantly output a JSONSchema successfully, and then use datamodel-codegen to successfully generate a pydantic model representation - see here for the autogenerated pydantic models!. This automodel can successfully deserialize a previously serialized library, however the resulting objects won't be correctly "shared" since the deserialization relies on a code level trick to understand that objects which share ids should only be created once and then shared. Pydantic / JSON schema don't really support the notion of foreign key deserialization natively, as that is really a database level idea.

Unfortunately, SQLModel does not support pydantic v2 yet, though progress on it seems to be coming along based off of github issues/pr.

That would really be the ideal approach, as then we don't need to worry so much about deserialization etc - the schema and database handle all that automatically. In some sense, this should probably wait until SQLModel + pydantic v2 compatibility is online, since that would be an extremely seamless, natural solution.

There's a lot more to talk about here, including the idea of updating classes fields dynamically from a new schema etc.

Now an entirely separate but valid, though more challenging approach, would be to define our own meta schema - i.e. the schema for the schema, in otherwords, how UTL objects can be defined and extended.

Let's talk more on a call.

[samuelduchesne]

Would we write the schema.py "manually" and generate a JSON schema to be published on SchemaStore.org (or others)? By writing it manually, we can keep the rich functions such as computing the assemblies R-value.

[szvsw]

@samuelduchesne Yep, that's the main idea I think, since there are so many rich functions which are part of the library. It doesn't really make sense to conceive of a pure data schema in this case.

[szvsw]

@samuelduchesne I added a computed field example to the pydantic version (diffusion coefficient calc'd from k/rho*cp). JSON Schema adds this as "readOnly" field to the material schema; datamodel codegen turns it into a normal field unfortunately - I think it's nice to still have it within the json schema, but we don't want datamodel to use it when outputting the models; there is probably some workaround for having it ignore certain fields, or including it as a different type of object within the json schema.

[szvsw]

@samuelduchesne One kind of cool thing about computed fields - the computed properties get serialized with the library! See e.g. here:

https://github.com/samuelduchesne/archetypal/blob/1b95a65f08d4b37228ff3ebde040e869de6cac2e/archetypal/template/schema/flatlib.json#L2C5-L10C10

[samuelduchesne]

That's great!

[szvsw]

Oh wow - just saw that Prisma has a python adapter now: https://prisma-client-py.readthedocs.io/en/stable/

And it plays with pydantic!

We use Prisma at my startup and its fantastic. I'm going to try to play around with that as well.

[szvsw]

@samuelduchesne I did find this fork of SQLModel which is most of the way to pydantic v2 compatibility, not entirely though: https://github.com/honglei/sqlmodel/

I pushed an example of setting up tables for a template library would look like here: https://github.com/samuelduchesne/archetypal/tree/pydantic-schema/archetypal/template/db/schema.py

I also uploaded a sample db. in the same folder.

I'll try the prisma adapter later in the week. All of these approaches have benefits and drawbacks to think through, some may be mergeable, etc etc. I have a feeling the Prisma approach might be the winner, not sure yet though.

[samuelduchesne]

How can we handle units in a way that can be validated for each field?

Incompatibility

For instance, if one has a schedule with temperatures, can a warning or a ValueError be raised if the magnitude is "incoherent" with the units of the field.

Carry units

Can we carry units information throughout the deserialization -> serialization -> deserialization loop?

[szvsw]

Yeah there is a lot to discuss here - notice in the Pure Pydantic version (/schema folder) I added the json_schema_extra to numerical fields, which add that info to the serialized schema (which can be passed along with the serialized library). The JSON Schema already maintains minimum and maximums though, so isn't that enough to ensure unit coherence? I.e. the autogenerated pydantic models from the schema still have correct range validators.

[szvsw]

@samuelduchesne see e.g. here:

archetypal/archetypal/template/schema/schema.json

Lines 93 to 116 in 34252fd

	"MaterialBase": {
	"description": "Material Definition used in construction assemblies and windows",
	"properties": {
	"id": {
	"format": "uuid4",
	"title": "Object ID",
	"type": "string"
	},
	"Conductivity": {
	"description": "Thermal conductivity 'k', in W/(m*K) of material",
	"maximum": 5,
	"minimum": 0,
	"title": "Thermal Conductivity",
	"type": "number",
	"units": "W/(m*K)"
	},
	"SpecificHeat": {
	"description": "Specific Heat 'C_p', in J/(kg*K) of material",
	"maximum": 2000,
	"minimum": 100,
	"title": "Specific Heat",
	"type": "number",
	"units": "J/(kg*K)"
	}

[szvsw]

@samuelduchesne I also added real time graph tracking to the Pure pydantic approach. I think pure pydantic will probably be the one that wins out...

[szvsw]

Alright, so I added a few more things:

You can now generate a graph of the schema itself, not just of the library:

This would allow you to automatically construct a database schema if desired with full join/link table specification.

I also improved (a little bit) the graph layout for libraries (note, the nodes below are actual objects, not classes - there just happens to be only one of a few of the objects so it looks somewhat similar to the graph above):

I also added the ability to dereference an object, so in other words if you try to create a Construction by just passing {"id": "202321-239234..."} it will attempt to find the correct object provided it exists; this, coupled with the schema level graph, will allow us to automatically climb up the tree, instantiating leaf-level objects first, then objects that reference them, and so on very efficiently.

This will mean that the tabular form will be a valid and compact way of representing the library (as it always has been, since it is effectively a relational database but in json)

[szvsw]

@samuelduchesne Alright, I added in deserialization of flat libraries, leveraging the schema graph (i.e. which classes are leaf classes etc).

Overall, the development is way more elegant/less verbose with the Pure Pydantic version than SQLModel; Prisma will also have some serious challenges, the main one being that it doesn't natively support constraints on fields (you need to add the CHECK statements as SQL to the migrations). Even though Prisma is so nice for managing the data schema and migrations, I don't think it has the support we want for things like validation etc. without us adding in a bunch of other workarounds; prisma is great when you are doing lots of complex query construction, but we don't have a lot of that.

I am pretty sure Pure Pydantic (supplemented with nx) will be the way to go.

[samuelduchesne]

So if I understand correctly:

• it's too complicated to have a db-first approach.
• computed fields are cool because they serialize to JSON.
• you implemented serializing/deserializing to and from flat JSONs and nested JSONs.
• you carry a network graph for each library to determine if objects are linked or not. The object hash is the id I suppose?
• a json schema graph can be created
• a library graph can be created

Next Steps/What about?

• Combining libraries and operations on libraries: LibA += LibB? LibA |= LibB
• Implement missing objects such as schedules, zone loads, etc.
• DesignDay schedules for schedules (that's a very important feature which archetypal.template does not support yet)

[szvsw]

It's a tree! (actually, a DAG)

I updated the title - technically it's a Directed Acyclic Graph, or DAG, not a tree. Every node in a tree has exactly one parent (except the root), whereas in DAGs, nodes can have multiple parents (which is what we have). If we had a tree, then Pydantic model dump would be trivially all we need. Because we have a DAG, we have our slightly more interesting requirements for data storage (ie tabular/relational joins, or just storing flat nodes/edge lists, etc)