Improving load_osm module

[d3netxer]

The load_osm module is used from parsing OSM PBFs and making transportation graphs. It can be slow to run for large graphs.

I timed the load_osm functions and found out that the generateRoadsGDF method takes the most time (about 90% of the time), specifically finding all of the intersections.

At first I thought that the generateRoadsGDF method was not that important and could be skipped. However I ran more tests, and found out that the generateRoadsGDF is indeed necessary.

This is due to how GOSTnets parses the PBF file. The first step of the load_osm process is creating the the OSM_to_network object . During the creation of this object, the OSM PBF file will be parsed and OSM ways will be imported. The ways can not be used directly for a transportation network because they are not always split at intersections. See below for an example:

Therefore running the GOSTNet's generateRoadsGDF method is necessary. See this example for the graph after intersections are generated:

One possible avenue forward is to try to speed up the generateRoadsGDF method. I have not looked deeply into this.

Another avenue is to look at other libraries for inspiration. GOSTnets uses the GDAL OSM driver to parse the PBF and it cannot import the individual nodes that make up the ways. Comparing this with OSMNX, OSMNX uses Overpass queries, and OSMNX can parse both the ways and the nodes. OSMNX creates separate edges between each node pair initially. See ex:

Then OSMNX can optionally simplify where it will merge edges except at intersections. See example after OSMNX simplification:

I also tested Pyrosm, and it will split up the roads like OSMNX does. Pyrosm is able to read from OSM PBF files and save an OSMNX compatible NetworkX graph. The Pyrosm result would still benefit from using the OSMNX simplification method. Additional testing is needed for large graph as it seems like Pyrosm can need a lot of memory in these cases.