Cannot set logging directory when using a saved model

[hom-bahrani]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in a new virtualenv with only neuralprophet installed, directly from github:

    git clone <copied link from github>
    cd neural_prophet
    pip install .

  • Checked the Answered Questions on the Github Disscussion board: https://github.com/ourownstory/neural_prophet/discussions
    If you have the same question but the Answer does not solve your issue, please continue the conversation there.
  • Checked that your issue isn't already filed: https://github.com/ourownstory/neural_prophet/issues
    If you have the same issue but there is a twist to your situation, please add an explanation there.
  • Considered whether your bug might actually be solveable by getting a question answered:
    • Please post a package use question
    • Please post a forecasting best practice question
    • Please post an idea or feedback

Describe the bug

I trained a model in Google Colab and then exported this model to GCP for inference using the neuralprophet save and load methods. I deployed the model to both a VM instance and Cloud functions and noticed the same error. When loading the saved model I think the MetricsLogger directory is not being set, and as a result it seems like a read only directory is being used for the logger when I try to call the predict method. I replicated the error in a virtual env on my mac

Traceback (most recent call last):
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/loggers/tensorboard.py", line 232, in log_metrics
    self.experiment.add_scalar(k, v, step)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/loggers/logger.py", line 54, in experiment
    return get_experiment() or DummyExperiment()
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
    return fn(*args, **kwargs)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/loggers/logger.py", line 52, in get_experiment
    return fn(self)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/loggers/tensorboard.py", line 175, in experiment
    self._fs.makedirs(self.root_dir, exist_ok=True)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/fsspec/implementations/local.py", line 50, in makedirs
    os.makedirs(path, exist_ok=exist_ok)
  File "/usr/local/bin/../../../Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/bin/../../../Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/content'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 150, in <module>
    predicted_value = get_predictions()
  File "main.py", line 134, in get_predictions
    preds = loaded_model.predict(future_df)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/neuralprophet/forecaster.py", line 883, in predict
    dates, predicted, components = self._predict_raw(df_i, df_name, include_components=decompose)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/neuralprophet/forecaster.py", line 2926, in _predict_raw
    result = self.trainer.predict(self.model, loader)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 950, in predict
    self._predict_impl, model, dataloaders, datamodule, return_predictions, ckpt_path
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in _predict_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1154, in _run
    self._log_hyperparams()
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1222, in _log_hyperparams
    logger.log_hyperparams(hparams_initial)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
    return fn(*args, **kwargs)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/loggers/tensorboard.py", line 211, in log_hyperparams
    self.log_metrics(metrics, 0)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
    return fn(*args, **kwargs)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/neuralprophet/logger.py", line 29, in log_metrics
    super(MetricsLogger, self).log_metrics(metrics, step)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
    return fn(*args, **kwargs)
  File "/Users/hombahrani/Code/prophet_version_test/prophet_module/lib/python3.7/site-packages/pytorch_lightning/loggers/tensorboard.py", line 236, in log_metrics
    raise ValueError(m) from ex
ValueError: 
 you tried to log -1 which is currently not supported. Try a dict or a scalar/tensor.

[hom-bahrani]

I realised the /content directory in the Traceback is coming from colab, I can get around this by hardcoding the directory to match the deployment env e.g. /tmp which is used in Cloud functions and can be written to, alternatively I can also train and deploy in the same VM instance. I like the idea of training in colab though as its free

[hom-bahrani]

Not sure if this would be easy to implement, but it would be nice to pass in the name of the logging directory when loading a model

[leoniewgnr]

hi @hom-bahrani, we see that this is a relevant feature, but currently have no capacity to work on this.
Would you like to work on this?

[linhsinp]

Hi there, our team is also having this issue in the process of deploying a project. Unfortunately we cannot spend time on the subject right now. Just a heads-up. Thank you for providing your thoughts @hom-bahrani - it helped us identify the problem.