Error running Pyomo with threading

[alma-walmsley]

Summary

I am trying to run several models in parallel using python's builtin threading module. However, when trying to solve, the nl_writer (repn>plugins>nl_writer.py) does not appear to be able to handle writing files in parallel. In particular, it uses a shared resource AMPLRepn.ActiveVisitor, and a locking system which blocks any sort of parallel access from happening:

def __enter__(self):
    assert AMPLRepn.ActiveVisitor is None
    AMPLRepn.ActiveVisitor = self.visitor
    self.pause_gc = PauseGC()
    self.pause_gc.__enter__()
    return self

def __exit__(self, exc_type, exc_value, tb):
    self.pause_gc.__exit__(exc_type, exc_value, tb)
    assert AMPLRepn.ActiveVisitor is self.visitor
    AMPLRepn.ActiveVisitor = None

If two threads attempt to write separate problem files at the same time, the first thread will set ActiveVisitor, while the other will fail at the assert statement, since ActiveVisitor was set by the first (since it is shared among threads).

Note that this problem won't necessarily show up all of the time, since it relies on two threads trying to write .nl files at the same time (can be a rare occurrence) - it's more likely to happen with many threads and/or large problem files (?)

I am not sure if there is any easy way around this - the way this is implemented does appear to be an intentional design decision.

Steps to reproduce the issue

Below is an example python script which runs into the error pretty much all the time on my machine.

import pyomo.environ as pyo
import threading


def thread_function():
    model = pyo.ConcreteModel()
    # create a lot of variables and bind them to constraints
    # (large model - will spend a lot time trying to write the problem file)
    # ie. x0 = 1, x1 = 1, x2 = 1, ...
    for i in range(200):
        model.add_component(f'x{i}', pyo.Var(domain=pyo.NonNegativeReals))
        model.add_component(f'c{i}', pyo.Constraint(expr=model.component(f'x{i}') == 1))

    # solve the model
    solver = pyo.SolverFactory('ipopt')
    solver.solve(model)


if __name__ == '__main__':
    threads = list()
    # create 500 threads to get a greater chance of hitting the error
    for i in range(500):
        x = threading.Thread(target=thread_function)
        threads.append(x)
        print(f"Thread {i} Start")
        x.start()

    for index, thread in enumerate(threads):
        thread.join()
        print(f"Thread {index} Done")
    
    print("Main Done")

Error Message

line 16, in thread_function
    solver.solve(model)
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/solvers.py", line 597, in solve
    self._presolve(*args, **kwds)
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/solver/shellcmd.py", line 224, in _presolve
    OptSolver._presolve(self, *args, **kwds)
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/solvers.py", line 706, in _presolve
    ) = self._convert_problem(
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/solvers.py", line 757, in _convert_problem
    return convert_problem(
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/convert.py", line 99, in convert_problem
    problem_files, symbol_map = converter.apply(*tmp, **tmpkw)
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/solvers/plugins/converter/model.py", line 184, in apply
    (problem_filename, symbol_map_id) = instance.write(
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/core/base/block.py", line 1991, in write
    (filename, smap) = problem_writer(self, filename, solver_capability, io_options)
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/repn/plugins/nl_writer.py", line 348, in __call__
    info = self.write(model, FILE, ROWFILE, COLFILE, config=config)
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/repn/plugins/nl_writer.py", line 391, in write
    with _NLWriter_impl(ostream, rowstream, colstream, config) as impl:
  File "/home/alma/.local/lib/python3.10/site-packages/pyomo/repn/plugins/nl_writer.py", line 548, in __enter__
    assert AMPLRepn.ActiveVisitor is None
AssertionError

Information on your system

Pyomo version: 6.7.0
Python version: 3.11.7
Operating system: Windows (running WSL)
How Pyomo was installed: pip from within conda environment

[svkawtikwar]

Have you tried switching to concurrent.futures package? It does process based parallelization rather than thread based (it is definitely more limiting but may work for your use case)

[jsiirola]

This is a known issue with the NLv2 writer (at least it was reported to me offline last week). I think I have a solution that should remove the need for the singleton without significantly impacting performance. In the meantime, there are a couple workarounds:

1. As @svkawtikwar pointed out, use process-based parallelism (instead of thread-based)
2. Switch back to the NLv1 writer by calling

from pyomo.opt import WriterFactory
doc = WriterFactory.doc('nl')
WriterFactory.unregister('nl')
WriterFactory.register('nl', doc)(WriterFactory.get_class('nl_v1'))

[alma-walmsley]

fantastic :D