Add Linopy to PyPSA-Eur

[pz-max]

Changes proposed in this Pull Request

Provide new solve_networks option linopy=True, which uses Linopy as a backend. Many constraints had to be rewritten due to the xarray concept in Linopy. All constraints are tested against the previous implementation.

Test status per constraint for single country ["BE"] :

• add_battery_constraints ✅ . Same objective as previous pypsa-eur implementation.
• add_operational_reserve_margin ✅ . Tested and results in same objective as previous pypsa-eur implementation: 1.849977447e+09. The test can be reproduced by adding the following lines to the tutorial config.yaml:

 operational_reserve:
    activate: true
    epsilon_load: 0.02
    epsilon_vres: 0.02
    contingency: 400000 

• add_BAU_constraints ✅ . Tested and results in the same objective result as the previous implementation: 6.8079564e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:

scenario:
  opts: [Co2L-BAU-24H]
  
electricity:
  voltages: [220., 300., 380.]
  co2limit: 100.e+6
  BAU_mincapacities:
    solar: 10000
    onwind: 0
    OCGT: 100000
    offwind-ac: 0
    offwind-dc: 0

• add_SAFE_constraints ✅ . Tested and results in the same objective result as the previous implementation: 3.4869299e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:

scenario:
  opts: [Co2L-SAFE-24H]
  
electricity:
  SAFE_reservemargin: 0.5

• add_EQ_constraints ✅ . Tested and provides the same objective result as the previous implementation: 3.1334..e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:

scenario:
  opts: [Co2L-EQ0.7-24H]

• add_CCL_constraints ✅ . Tested for one country. Provides the same objective result as the previous implementation: 3.1453661e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:

scenario:
  opts: [Co2L-CCL-24H]
electricity:
  agg_p_nom_limits: data/agg_p_nom_minmax.csv

and with the following data in agg_p_nom_minmax.csv

country,carrier,min,max
BE,onwind,10000,
BE,offwind-ac,,
BE,offwind-dc,,
BE,solar,20000,

Test status per constraint for single country ["BE", "DE"] :

• add_CCL_constraints
• add_battery_constraints
• add_operational_reserve_margin
• add_BAU_constraints
• add_SAFE_constraints
• add_EQ_constraints

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

[pz-max]

One observed issue is that PyPSA renames generators that are extendable to generator-ext and non-extendable generators as generator-fix. Xarray cannot easily deal with it when performing functions e.g. groupby along carriers. The reason PyPSA renames the arrays in that way is its performance. Not writing np.nan for expansion problems for non-extendable generators increases speed.

What can be investigated:

• Does the compat.py work without any issues when e.g. -ext stuff is disregarded? (could help to make Linopy compat.py interface smoother). If the test demonstrates better usability it might make sense to reconsider some of the functions which use -fix and -ext syntax in pypsa/optimization
• How much performance drop do we experience when disregarding this -ext stuff in PyPSA (Tip from Fabian H. to change this: "you can go in pypsa/descriptors.py and reset the index naming of get_extendables_i to the c, same with get_non_extendables_i and committables")
• Does it help performance using instead of np.nan (float 64) -> pd.NaN (int 8)
• Does it help usability to define extra_functionality constraints in PyPSA-Eur by Linopy directly vs compat.py?

Update:

• I am ignoring most of the above
• -ext is probably not an issue. Xarray just requires clear dimensions, coords and naming and data. RHS or LHS inputs sometimes require .rename() or redefining the axis .rename_axis()
• Currently, I am disregarding the compat.py avoiding an extra interface layer

[pz-max]

FYI @FabianHofmann, I made all PyPSA-Eur functions run for a single country ["BE"].

I noted that the CCL constraint is a bit more tricky. Because there is a groupby with multi-index, which groupby_sum in Linopy does not support yet:

pypsa-eur/scripts/solve_network.py

Lines 167 to 178 in a22dc92

	p_nom_per_cc = (
	pd.DataFrame(
	{
	"p_nom": linexpr((1, get_var(n, "Generator", "p_nom"))),
	"country": gen_country,
	"carrier": n.generators.carrier,
	}
	)
	.dropna(subset=["p_nom"])
	.groupby(["country", "carrier"])
	.p_nom.apply(join_exprs)
	)

. The current code hacks around and makes it run for one country. It also seems that groupby in xarray with multi-index is not really supported/adequately addressed.

My code 100% breaks if it is applied to multiple countries e.g. ["BE", "DE"] for the CCL constraint.
I will test all other constraints to see if ["BE", "DE"] works before going back to the CCL stuff.

[pz-max]

Only CCL constraint breaks now for multiple countries. If the 'groupsum()' for the multiindex would work this could be solved with ease. As alternative,

• an interactive process to generate the lhs expression might solve it too e.g. Fix linopy implementation of nominal capacity per-bus/carrier constraints PyPSA#524

[pz-max]

Thanks to @FabianHofmann for the little co-hacking session for the CCL constraint. Everything is now fixed and tested. As of now, Fabian H. will eventually review this PR in the next 1-2 weeks.

Open review questions:

• Any code changes?
• Deprecation status on Linopy==False implementation?
• Any extra documentation necessary?

[pz-max]

Closing because of reimplementation in #625

[fneum]

Thanks, @pz-max! @FabianHofmann used your branch as a starting point (preserving commits) and said your contribution made it easy to implement the switch to linopy!

[pz-max]

Yep @fneum, I was with Igor and FabianH when we moved the Linopy stuff to the merge branch :)
Congrats on the merge 🎉 🎉 🎉