Can we remove the dependence of the DB on jagged arrays?

[opotowsky]

Our dependence on jagged arrays means we have a max numpy version of 1.23.5.

What is the basis for needing jagged arrays? Can we restructure something to remove this?

We have a ticket open for this here.

@john-science @onufer @albeanth @mgjarrett @jakehader @sombrereau

[albeanth]

Do you know where we use jagged arrays?

[jakehader]

I'm not entirely sure the basis, but I have to assume it's to reduce DB size with zero values / sparse data in an array / matrix (e.g., Nones, NaNs)? @youngmit might come to our rescue here 😉. If you search "jagged" in the repository there are several instances in the DB implementation docs and comments @albeanth.

[mgjarrett]

There aren't actually any jagged arrays stored in the database. But Database3 sometimes creates a jagged numpy array before flattening it into a 1D array and storing that 1D array (along with offsets) in the database.

So we would probably just need to build our own JaggedArray class that would be a special type of list. You would create a JaggedArray instance, append a bunch of numpy arrays to the list, and then concatenate the whole thing into a flattened 1D numpy array with offsets and spit that back out to be dumped into the database.

[mgjarrett]

I'm looking more closely, and it appears that we also create jagged numpy arrays for convenience when reading the data back out of the HDF5.

armi/armi/bookkeeping/db/database3.py

Lines 1361 to 1363 in 827d223

	data = numpy.array(layout.location)[layoutIndicesForType][
	indexInData
	]

It would take some work to refactor all of this to not require a jagged numpy array, but it's probably not too complicated.

[john-science]

The problem I originally had in this ticket was identifying, conclusively, all the locations we use jagged arrays.

That would make this work significantly easier.

[opotowsky]

Ok this isn't as urgent as I thought. I tried to recreate the errors I thought I had on linux with numpy 1.23.5 and am unable to do so.

It's still a good idea to put into a longer term plan so we aren't locked into ever-aging versions of numpy, but instead of a this-year thing it's a next year thing ;-)

[john-science]

I would like to note that because we can't use the latest versions of NumPy (due specifically to jagged arrays), ARMI cannot support Python 3.12: #246

That's a pretty big deal.

[albeanth]

@john-science do you think this should be apart of ARMI v0.4?

[john-science]

I very much do, yes. I think I would have to fight pretty hard to get that put on the calendar though.

[ntouran]

Jagged arrays are used to store pin detailed parameter data on assemblies that have differing numbers of pins. So if you have a CR with 17 pins and a fuel assembly with 127 pins, that ends up using jagged arrays. We can definitely refactor the way these data are stored in the DB in order to support the deprecation, but being able to store pin detailed param data on multi-num pin cores is an essential feature for the framework.

[john-science]

Check out Michael's PR on the topic: #1726

[mgjarrett]

I was able to get something working a couple of months ago: https://github.com/terrapower/armi/tree/jaggedArrays but then interest died down so I put it on the back burner.

It's not a finished product yet, but I think it's enough to show notionally that this can be done without a huge overhaul. I basically just added a new class to database3.py. I'd definitely like to have more eyes on it and spend some more time testing corner cases and making sure everything is implemented efficiently.

There are a couple unit tests that are passing on my local machine with the old version of numpy, but fail when running with numpy 1.26.4 on GitHub actions. I think these failures could probably be resolved pretty easily, but I haven't gotten around to it yet.

[opotowsky]

🦸‍♂️

[john-science]

We have a ticket open for this, and a PR.

So, we are definitely doing this soon, and I will close this discussion.