Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSUControl shuts down PSU on Raspberry Pi just few seconds after startup #265

Open
ArkBrj opened this issue Apr 25, 2023 · 2 comments
Open

Comments

@ArkBrj
Copy link

ArkBrj commented Apr 25, 2023

Hello,
I have spent several hours debugging why PSUControl shuts down my printer right after startup of OctoPi and finally found the root cause. Strictly speaking the problem is not on the PSUControl's side, but I think PSUControl should at least document my findings on a wiki page (I could not find any) and maybe even have some workaround.
The root cause of my problems was that after PSUControl's wait for idle timeout starts the system time gets changed by NTP and PSUControl's wait gets terminated prematurely. In my case the system time advanced by more than one day while the idle timeout was set for 30 mins, Instead of waiting for 30 mins it finished in few seconds and led to printer shutdown.
I am using official OctoPi on Raspberry Pi with few plugins but I do not think they matter.
What does matter is that Raspberry Pi does not maintain system time while being off. Instead, on startup it restores the time as it was at the moment of the last shutdown, which could have been hours, days or months ago. Later on, NTP sets the correct time but this can happen seconds or even minutes later depending on many reasons. Especially if only WiFi connectivity is available.
So if it happens that PSUControl starts its idle timer before NTP advances system time, PSUControl's wait might take just few seconds.

PSUControl uses threading.Event.wait function to wait for timeout. Right now OctoPi with all Raspbian updates uses Python 3.9.2. According to the documentation, Python 3 is supposed to use monotonic clock for waits so they should not be affected by changes in the system time. However it is easy to see that this is not the case, at least for python 3.9.2.
Simple repro on my octopi:

  1. Open 2 ssh sessions to OctoPi
  2. In one session run "date; python -c "from threading import Event; Event().wait(3600)"; date". This is supposed to print the current date/time, wait for 1 hour (3600 secs) and then print the current date/time again.
  3. In another session just change the system data advancing it by couple of hours: "sudo date -s '2023-04-24 19:00'" (change the actual data/time accordingly).
  4. Look at the first session and see that wait finishes the moment you change the system date/time instead of waiting for an hour.

As I mentioned above, this is strictly not PSUControl's problem and seems to be genuine bug in Python.
However, given the fact that Raspberry Pi is a widely used platform for OctoPrint, maybe PSUControl can have some kind of protection logic to make sure that it does not turn off the printer due to sudden changes in the system date/time?

@kantlivelong
Copy link
Owner

Not really a bug in Python or the plugin. This should help you though https://raspberrypi.stackexchange.com/questions/94635/how-can-i-delay-the-startup-of-systemd-services-until-the-datetime-is-set-no-rt. Have the OctoPrint service depend on systemd-time-wait-sync.service

@ArkBrj
Copy link
Author

ArkBrj commented Apr 25, 2023

It is for sure a bug in Python that periodically gets fixed and then reappears again. For example, a similar bug with condition variable: https://bugs.python.org/issue23428
I am aware about possibility to start octoprint only after time sync happens. But before I do this I need to see what happens if my Internet connection is down, which happens from time to time. Will octoprint start? How soon will it start? A lot to investigate...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants