-
Notifications
You must be signed in to change notification settings - Fork 7.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LoadProhibited in FreeRTOS Event List processing (IDFGH-2300) #242
Comments
Thanks. I had a look and #233 does not tell me much. I compile in Debug, I have similar issues (different backtrace, but landing in same line in list.c) since some time, not only on recent IDF. |
Hi @nathanjel , Thanks for reporting this. It looks like the same as an issue that we're also tracking internally, we've seen the same crash but only under high network load conditions. We're continuing to investigate and will update you as soon as we have some more information. Angus |
Exactly - this happens when the device is "bombarded" with messages during load testing. (a few hundred bytes MQTT message every 1ms). The software itself is generallly a port from our original implementation on TI's CC3200, where load testing was never an issue. Results improved after we implemented rate limiting on ESP32 port (rejecting messages when messages/sec limit exceeded) but never reached the stability of our original implementation - see all my other bug reports. Did You consider upgrading FreeRTOS from 8.2 to a newer version? |
I tried these suggestions - no luck. |
Seem to happen easier if code is running on core 1 and it happens both on incoming and outgoing packets. Queues are used throughout the system so I doubt that it has anything to do with the problem. |
It seems to remain a theme in my app now, if under load.
|
Two interesting observations
The code says the list was not empty (queue.c from FreeRTOS)
And GDB shows that at the time of dump, the list was empty
This makes absolute sense, if one realizes that locking on the |
This one is definitely a race condition, related to locking issue in a multi-processor environment. In FreeRTOS/tasks.c tick handler Now, if the |
Seems my pull request came in at the same time as this Anyhow, now the issue does not occur. Thanks. |
Thanks for your PR, we have been discussing ways to fix this issue and your looks like a solid one. We will need to have a more detailed look, but I do see a chance this will be merged as a more reliable solution. |
Thumbs up for more reliable. "reducing the window where the race condition can happen" does not make me sleep like a baby at night. |
Thank You. My initial solution that I applied internally to be able to continue work was very similar to the one in 01c4a98, but the code was more messy and thus I was not offering it to the wider community. The one proposed is elegant, and as the additional list check is being done with the protection of Adding the xTaskMutex around all the invocations indeed seems more robust, but it also is heavyweight as the overhead counted in instructions executed is much larger than the workaround, and it messes more with FreeRTOS internals. I think both are just patches needed to work around fundamental single-core driven FreeRTOS design concepts. In my head I picture a solution that drops the queue-tasklist coupling completly and replaces it with something - maybe a lock-free ringbuffer that would store queue "events", for consideration by the scheduler... |
this issue still exists (sdk v2.1). no heavy load on the network is required - for me it happens when I send a single SSL request just after wifi reconnects details |
I am getting this issue as well
|
@rebeliousconformist |
Hi david, Also, with the CAN bus task off, the MQTT connects to the rabbitMQ server perfectly. With the CAN bus readout task switched off. the MQTT event loop works fine. When the MQTT is called from a Task, then it sends only 22-23 messages. I presume it is getting preempted by either the Wifi event loop or the MQTT event loop. Also when the semaphores are removed, then the errors go away. Ill post my code soon |
Hi @rebeliousconformist Any luck with the test code? Could you please post some bits of your test code? |
Hi David, Regards, |
Hi Nachiket, Thanks for your help and sorry for not responding earlier. Will try to reproduce the issue with your code! |
Hi @david-cermak the event loop crash error is resolved in the latest developments. |
Hi @rebeliousconformist Thanks for the update! Closing the underlying internal tracker then. |
#8336 looks like a similar issue with event groups in esp-idf v4.4, does |
Hi, this stills happens in v4.4.2 release. To trigger it you can use led_strip examlple (esp_idf/examples/peripherals/rmt) combined with the connection example. You have to make a freertos task that constantly turn on and turn off the ws2812 rgb leds in intervals of 100 ms. Meanwhile the rgb task is loop, you have to try to connect to wi-fi. This error causes two behaviours, the first is the Loadprohibited guru handler, the second one is that the program freezes at rmt_write_sample |
Also, when config the RMT driver I use a clock diviser of 4 instead of 2 as the API suggests. |
In a complex UART/MQTT/WebSockets/SSL scenario, under heavy load (UART data in/out processing every 10ms, several os Tasks, bombarded with MQTT messages and sends few MQTT messages per second, using both CPU cores) i randomly run into this repeatable error, which seems to occur as invalid pointer is dereferenced in FreeRTOS queue processing code. The 0x1d could hardly be a valid pointer given ESP32 memory map.
Down to sys_sem_signal it's only the semafor queue object being referenced, and error occurs later down the stream as the semafor owning task is extracted and system-wide event list adjusted, I would suspect the error is occuring somewhere in FreeRTOS internals.
No idea how to fix this so far, no workarounds.
The text was updated successfully, but these errors were encountered: