Linopy transition

[ekatef]

Closes #494

Integrate linopy following PyPSA/pypsa-eur#625

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[ekatef]

Currently works locally with cbc. Points which need attention:

• there is a mismatch between batteries efficiency and links which has temporary resolved by using a single efficiency value, but needs additional investigation
• what does mean replacement p with p_set in add_operational_reserve_margin_constraint()?
• the solver log has disappeared from the logger files, while output of solver logging into a console presents if running solving from VSCode but is skipped is using a plain terminal
• there is a fuzzy feeling that double-check is needed on transforming to xarray; in particular, the current calculation of ext_carrier_i differ from PyPSA-Eur implementation https://github.com/PyPSA/pypsa-eur/blob/6505d78f4e9a1362e8309bcd5b4b8b99ac60337c/scripts/solve_network.py#L350

[ekatef]

I can reproduce CI error locally when using lgpk, while the workflow works with cbc

[pz-max]

Cool! Just a comment. I would not follow my Linopy PR only but also check if the PyPSA-EUR main changed some parts.

[davide-f]

Hello @ekatef, that's great news! :D
To answer to a question, the use of p_set is the more appropriate way to get access to the demand time series of load.

Regarding solver log, it may make sense to check the latest pypsa-eur implementation as I guess they have accounted for that.

As a remark, this morning we talked with Leon and it turns out that to merge this PR requires changes to the -sec version, which may be delicate also for some of their projects.
So, it is a good idea to investigate and prepare this PR, but it can be merged only when the alignment can be performed also with the Sec version.
What is your feeling?

[ekatef]

Hey @davide-f!

Thanks a lot for the detailed analysis :)

Ah, I see: the load is an input, not a result. So, n.loads_t.p_set is in fact more appropriate than n.loads_t.p. Thanks for the explanation!

Regarding a potential merge of this PR, absolutely agree that it's crucial to ensure that the changes introduced don't break anything. Actually, I perceive this work as a kind of experiment to estimate how much effort would be needed for transition to linopy interface. If you see a potential to align this with requirements of PyPSA-Sec at some stage, that is great and happy to adjust this PR accordingly.

My general impression is that linopy transition is not as challenging as it could be expected from the first sight (thanks to @pz-max PR in PyPSA-Eur and clear instructions)

[ekatef]

Some adjustments were done to get closer to PyPSA-Eur implementation. However, glpk solution still breaks. The reason seems to be the following command is given to a solver: 'glpsol --lp /var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T/linopy-problem-p4tiu2uu.lp --output /var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T/linopy-solve-7tkku64p.sol --solver_logfile /Users/ek/_github_/pypsa-earth/logs/NG_BJ/solve_network/elec_s_6_ec_lcopt_Co2L-4H_solver.log'

According to a warning, there is no --solver_logfile option available for glpsol, while it is transferred intorun_glpk as solver_options (not sure yet about the reason).

[ekatef]

Results of additional testing:

1. now highs is captured and used;
2. some introduced changes make a testing problem infeasible, and it looks like the reason for it is that the problem formulation somehow gets lost:

INFO:linopy.solvers:Log file at /tmp/highs.log.
ERROR:   getOptionIndex: Option "solver_logfile" is unknown
Presolving model
Problem status detected on presolve: Infeasible
Model   status      : Infeasible
Objective value     :  0.0000000000e+00
HiGHS run time      :          0.00
WARNING:linopy.constants:Optimization failed: 
Status: warning
Termination condition: infeasible
Solution: 0 primals, 0 duals
Objective: nan
Solver model: available
Solver message: infeasible

3. there are some troubles with logfile options for cbc:

No match for solver_logfile - ? for list of commands
No match for logs/MY/solve_network/elec_s_10_ec_lcopt_Co2L-2H_solver.log - ? for list of commands

[davide-f]

IT may be good to revive this PR and finalize it when possible, to be able to use the latest pypsa version.
How far do you think we are to finalize?

I am aware that you have a lot on your desk right now, no pressure at all :) we can give priorities and tackle them one by one

[ekatef]

IT may be good to revive this PR and finalize it when possible, to be able to use the latest pypsa version. How far do you think we are to finalize?

I am aware that you have a lot on your desk right now, no pressure at all :) we can give priorities and tackle them one by one

@davide-f agree :) I'd expect that something about a week of focused work should be enough to resolve the remained questions. As discussed, #919 and #903 seem to have higher priorities now. Also, not sure what is the current status of PyPSA-Earth-Sec in terms of pyomo/linopy dependencies.

Anyway, after #919 and #903 will be finished, happy to get back to this PR. Ping me please in case this would get urgent.

[ekatef]

Testing with the updated PyPSA version (0.25.2 is used locally).

Clustering is failing currently:

1. the default SCIP solver (copy-pasted from PyPSA-Eur) is not included into PyPSA-Earth environment;
2. ipopt is installed but not seen by linopy for some reason.

CI is failing due to conda being unhappy: Operation too slow. Less than 30 bytes/sec transferred the last 60 seconds.

[ekatef]

The major points left:

• do not revise implementation of add_operational_reserve_margin (in PyPSA-Earth decomposed on two functions which look a way more clear as compared with PyPSA-Eur);
• check implementation of add_battery_constraints. Currently, it's different in PyPSA-Earth and PyPSA-Eur, which may it be the effect of the sec-merge (which imply the need to revise implementation after PyPSA-Earth merge);
• add_RES_constraints must be refactored and reimplemented;
• solve_network should be checked: that may be the point where PyPSA-Earth may need a slightly different approach as one used in PyPSA-Eur may be not the most clear one from the usability perspective.

[ekatef]

Update: testing successful in the base-case configuration, but there is a problem when fetching isolated networks (something with p key which is expected but not found in the provided data structures).

[ekatef]

A couple of notes on linopy grammar:

1. pypsa.Network() doesn't initialise model property -> an explicit call n.optimize.create_model() is needed
2. Multiplication is not commutative in the current linopy implementation: n.model["Generator-p"] * n.snapshot_weightings.generators works, while n.snapshot_weightings.generators * n.model["Generator-p"]` throws an error.
3. Not sure what is the way to make groupby(some_grouper).sum() work on linopy variables.

[ekatef]

Local testing on reproducibility of objective value

main version

INFO:main:Objective function: 5813422825.0
INFO:main:Objective constant: 2319283737.695124

linopy version

INFO:main:Objective function: 5473150322.268269
INFO:main:Objective constant: 2319359288.9025226

That means ~6% difference for the objective function and basically same value for the objective constant (1e-3%, but here is rather a question of numerical precision). So, the PR seems to work properly.

As a usability note, that is not obvious at all from the CI logs. The PR changes format of solver traceback due to different conventions of lopf and linopy, which makes finding correspondance quite tricky.

[ekatef]

@finozzifa would be very grateful if you would have time to check this PR.

Pre-commit is currently unhappy due to excessively long lines in the configuration file. But that won't be fixed in this PR.

Another concern is backward compatibility. I'm currently looking into that, but not sure if it'll be possible to find a code solution for that.