Real Time Safety Questions

[jbeck28]

This isn't exactly an issue, so I apologize if this is the wrong place to ask this... seemed like it might start a conversation though.

I see that there is no mention of preempt_rt patch as in the Universal_Robots_ROS_Driver, and the question I have is whether it would be as easy as setting the thread priority in the main thread inside egm_hardware_interface_main.cpp to achieve lower jitter?

The reason I ask is I've had significant problems running the driver I wrote (similar to this one!) on a low-power Intel NUC, where it previously ran fine on a faster Xeon desktop (both with Preempt_RT patch, and priority set in the main thread). What tends to happen is that when a large (10,000+ point) trajectory is published for execution, the main thread hangs for roughly 3 seconds on the waitForMessage() function defined in abb_libegm/egm_controller_interface.h.

I am also curious if the io_service thread should have realtime priority in the event the main thread does? This didn't solve the problem I am having with my driver, but thought it would be worth asking anyway.

Thanks @jontje for putting this together! Looking forward to testing it!

-Josh

[gavanderhoorn]

Is this about safety or performance?

[jbeck28]

Both, I think. I'm pretty ignorant of the subject, but I am assuming you are asking if this is about real-time safety, or real-time performance? It seems like they are pretty related, but I'm mostly concerned with ensuring that the control period is 4ms as desired.

Did I answer your question? I apologize if I've misunderstood.

[jbeck28]

Just going to throw a bit more information out there. I adjusted the timeout for waitForMessage() so as to be more permissive of timing jitter, and put a code block inside of the main control loop in egm_hardware_interface_main.cpp which prints the control period if it exceeds 10ms. I had found that it regularly exceeded 10ms, often spiking to 30ms. This was not the case with the proprietary EGM driver that I had been using. So, I placed the following code block inside the same file as above, to give the main thread real time priority, as was the case with the driver I had been using:

std::ifstream realtime_file("/sys/kernel/realtime", std::ios::in);
  bool has_realtime;
  realtime_file >> has_realtime;
  if (has_realtime)
  {
    const int max_thread_priority = sched_get_priority_max(SCHED_FIFO);
    if (max_thread_priority != -1)
    {
      // We'll operate on the currently running thread.
      pthread_t this_thread = pthread_self();

      // struct sched_param is used to store the scheduling priority
      struct sched_param params;

      // We'll set the priority to the maximum.
      params.sched_priority = max_thread_priority;

      int ret = pthread_setschedparam(this_thread, SCHED_RR, &params);
      if (ret != 0)
      {
        ROS_ERROR_STREAM("Unsuccessful in setting main thread realtime priority. Error code: " << ret);
      }
      // Now verify the change in thread priority
      int policy = 0;
      ret = pthread_getschedparam(this_thread, &policy, &params);
      if (ret != 0)
      {
        std::cout << "Couldn't retrieve real-time scheduling paramers" << std::endl;
      }

      // Check the correct policy was applied
      if (policy != SCHED_FIFO)
      {
        ROS_ERROR("Main thread: Scheduling is NOT SCHED_FIFO!");
      }
      else
      {
        ROS_INFO("Main thread: SCHED_FIFO OK");
      }

      // Print thread scheduling priority
      ROS_INFO_STREAM("Main thread priority is " << params.sched_priority);
    }
    else
    {
      ROS_ERROR("Could not get maximum thread priority for main thread");
    }
  }  

In this case, the control period never exceeds 10ms. However, I had found that the proprietary driver I was using before when run on a slower computer, behaved very poorly when a large trajectory was published for execution. The control period would exceed the timeout on the robot controller and the "Event Message 41822: No Data From UDPUC Device" error would appear. This same behavior was experienced about 50% of the time with this driver as well, when the main thread had real-time priority. However this was not experienced when thread priorities were not changed.

Just to clarify, I am using Ubuntu 18.04 with kernel 5.4.93-rt51. Do you @gavanderhoorn have any direction as to where one may look to solve this? In your assessment, would improving timing performance of this driver be a worthwhile endeavor? I may be wrong in my assumption that the spikes in control period hurt performance in my application, but if this is something that others may be interested in, I'm more inclined to try my hand at solving it.

Thanks,
Josh