Preserve order of recording extractor custom properties in add_electrodes()

[weiglszonja]

Resolve #792

[CodyCBakerPhD]

Why are rel_x and rel_y after the custum ones though? I would have expected them before since they are pre-defined in NWB schema

[CodyCBakerPhD]

How does this interact with two recording properties that have different columns? And if you write them a in a different order? Is the order still intuitive?

The order I would have expected to see on the electrodes table is (i) pre-defined NWB columns (group, location, x, y, z, if present, etc) then (ii) ones added automatically to a format by SI, then (iii) custom ones set after recording interface initialization

Also while I think the approach is fine, making use of the implicit ordering property of insertion of keys into a dict - I'll just remind people Python wasn't always like that 😉making it even more explicit would mean allowing the user to specify the column order via an argument

[weiglszonja]

How does this interact with two recording properties that have different columns? And if you write them a in a different order? Is the order still intuitive?

This is a good point and I didn't think about in for my use case, I would say in this case it could be based on precedence:

        self.recording_1.set_property(key="custom_property_x", values=[0.1] * self.num_channels)
        self.recording_1.set_property(key="custom_property_y", values=[0.2] * self.num_channels)
        self.recording_1.set_property(key="custom_property_z", values=[0.3] * self.num_channels)
        self.recording_1.set_property(key="custom_property_1", values=["value_1"] * self.num_channels)

        self.recording_2.set_property(key="custom_property_2", values=["value_2"] * self.num_channels)

        add_electrodes(recording=self.recording_2, nwbfile=self.nwbfile)

        add_electrodes(recording=self.recording_1, nwbfile=self.nwbfile)

Then I would expect the order of custom properties to be ["custom_property_2", "custom_property_x", etc.]

The order I would have expected to see on the electrodes table is (i) pre-defined NWB columns (group, location, x, y, z, if present, etc) then (ii) ones added automatically to a format by SI, then (iii) custom ones set after recording interface initialization

I think this is still the order but within (iii) the custom ones will be added in the order set() returns them.
That is what I tried to undo with "remembering" the "order" in a sense how they were added originally to the recording extractor.

Also while I think the approach is fine, making use of the implicit ordering property of insertion of keys into a dict - I'll just remind people Python wasn't always like that 😉making it even more explicit would mean allowing the user to specify the column order via an argument

This is a good point, but I think we should also make sure we are adding columns (i) -> (ii) -> (iii) and then there could be an optional argument that could change this order,

but as you pointed out what should be the behavior when there are 2 or more recording properties that have different columns?

add_electrodes(recording=self.recording_2, nwbfile=self.nwbfile)
add_electrodes(recording=self.recording_1, nwbfile=self.nwbfile, order=["custom_property_x", "custom_property_y", "custom_property_z")

[CodyCBakerPhD]

I think this behavior and all concerns could be abated if we use the metadata["Ecephys"]["Electrodes"]: List[dict] as the defining order of all columns, with the one condition being that all properties on the recording extractor must be defined in that metadata structure

Then we can always take the order in that list as the defining order to write, including 'reserving' a column by adding it and populating default 'empty' values for that data type even if it doesn't exist in the first recording extractor triggering the call to add_electrodes

[weiglszonja]

Then we can always take the order in that list as the defining order to write, including 'reserving' a column by adding it and populating default 'empty' values for that data type even if it doesn't exist in the first recording extractor triggering the call to add_electrodes

I'm not sure how to do this, I started refactoring to have the electrodes metadata populated (see e1f2c6c) but I think I'll need help to rewrite current logic. I only managed to do it for now when adding properties by columns (see the test in 403b63a).

For the Turner data these changes achieve the ordering of "custom" attributes as they appear in the electrodes metadata:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.26%. Comparing base (fa6edd5) to head (c01ef14).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #793      +/-   ##
==========================================
+ Coverage   91.23%   91.26%   +0.03%     
==========================================
  Files         121      121              
  Lines        6309     6331      +22     
==========================================
+ Hits         5756     5778      +22     
  Misses        553      553              

Flag	Coverage Δ
unittests	91.26% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...c/neuroconv/tools/spikeinterface/spikeinterface.py	94.66% <100.00%> (+0.25%)	⬆️

[CodyCBakerPhD]

Suggested change

	"location",
	"group",
	"group_name",
	"channel_name",
	"rel_x",
	"rel_y",
	"channel_name",
	"group",
	"group_name",
	"location",
	"x",
	"y",
	"z",
	"rel_x",
	"rel_y",

The one column we can't control the order of is naturally the id, but what do you think about this ordering?

Also could you make up some x/y/z values (possibly using some recent real life examples as a good baseline) to match the location?

[CodyCBakerPhD]

Having the tests is a good first start

I see you have some for the single extractor case, but if you can add some for the multi extractor case you're unsure about (such that the new tests are failing) then it can help by making the problem more well defined (we know what to achieve, just not how to achieve it)

At that point I can take a look and suggest a strategy