AsyncClient blocks thread when connection drops; cannot reconnect

[buckle2000]

Whenever the connection drops, this library takes up 100% CPU on my computer. I tried increasing the delay to reconnection_delay=10, reconnection_delay_max=3600, but it blocks the thread.

How to reproduce the bug

1. Run the script below
2. Cut off the Internet connection
3. Wait

import asyncio
import socketio

async def main():
    sio = socketio.AsyncClient()
    await sio.connect("https://socket-io-tweet-stream.now.sh/socket.io/")
    await sio.wait()

if __name__ == "__main__":
    asyncio.run(main())

[buckle2000]

Here is the result of python -m cProfile -s tottime test.py (only first lines shown)

         460101747 function calls (460093534 primitive calls) in 273.832 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        4   82.674   20.668  222.914   55.729 asyncio_client.py:479(_read_loop_websocket)
 45976708   57.839    0.000   89.199    0.000 asyncio_client.py:322(_receive_packet)
       64   50.583    0.790   50.583    0.790 {method 'poll' of 'select.epoll' objects}
 45976714   24.969    0.000   36.484    0.000 packet.py:17(__init__)
183931306/183931303   21.252    0.000   21.252    0.000 {built-in method builtins.isinstance}
 45976719   13.797    0.000   22.469    0.000 __init__.py:1424(info)
 45976714    9.554    0.000    9.555    0.000 client_ws.py:209(receive)
 45976721    8.672    0.000    8.672    0.000 __init__.py:1677(isEnabledFor)
45996747/45995319    4.161    0.000    4.162    0.000 {built-in method builtins.len}

[miguelgrinberg]

Questions:

• Do you have a server running at the connection address, or do you need the connection to fail to reproduce?
• Are you cutting off the network on the client or the server?
• How are you physically doing this (i.e. unplugging ethernet) or turning off ethernet/wi-fi from your network settings?
• What operating system is this client running on?

I also have a request. Could you run your client with logging and capture a few lines when it reaches the high-consuming CPU state? Just change to this to log:

sio = socketio.AsyncClient(engineio_logger=True)

[buckle2000]

• There must be a valid server at the address at first. Otherwise, there will be an exception saying "connection failed".
• Both. The server went down randomly and produced the same issue.
• Both.
• Linux

I modified my test script to your instruction. The output:

# disconnect from internet
# a few seconds later:
Sending packet PING data None
# a few seconds later, the screen is filled with infinite copies of this message without delay inbetween:
Received packet NOOP data None
Received packet NOOP data None
Received packet NOOP data None
Received packet NOOP data None
Received packet NOOP data None
Received packet NOOP data None
Received packet NOOP data None
Received packet NOOP data None

Also, it's possible to recover if I reconnect to the Internet before the first Received packet NOOP data None. It's impossible after that.

[buckle2000]

reconnection=False doesn't help as well.

[miguelgrinberg]

This isn't a reconnection problem. The network reads after you interrupted the network are not failing for you, they keep returning the last packet over and over instead of detecting that the network is down, which really makes no sense at all. Anything unusual about your networking set up? I suggest you try a different computer with different hardware to see if you get the normal behavior, which is that the read should raise an exception when the connection or the network are gone.

[buckle2000]

I think it isn't a problem of my network. The packet is NOOP, not PONG.

[buckle2000]

I tested on a cheapest droplet on Digital Ocean with Python 3.6.9 and Linux 4.15.0-66-generic Ubuntu. The result is the same.

First, I ran the following script with python3 test.py.

import asyncio
import socketio

async def main():
    sio = socketio.AsyncClient(engineio_logger=True)
    await sio.connect("https://socket-io-tweet-stream.now.sh/socket.io/")
    await sio.wait()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Then I blocked all the IP address of socket-io-tweet-stream.now.sh with route add -host IP reject.

What follows is the same as on my computer, infinite debug messages and 100% CPU load.

P.S. My computer has Python 3.8.0 and Linux 5.4.5-arch1-1.

[miguelgrinberg]

The aiohttp websocket client appears to not be detecting disconnects. I have to debug this more, but it does appear that this is a bug in aiohttp. I haven't found a workaround yet. Possibly related: aio-libs/aiohttp#2309 and aio-libs/aiohttp#4153.

[miguelgrinberg]

@buckle2000 can you install python-engineio master branch and retry? I believe I have found a workaround.

[buckle2000]

It is fixed. The output is now:

Sending packet PING data None
PONG response has not been received, aborting
Unexpected error "WebSocket read returned None", aborting
Waiting for write loop task to end
Exiting ping task
Exiting write loop task
Waiting for ping loop task to end
Exiting read loop task
Attempting polling connection to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3
HTTP GET request to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3&t=1577078282.7167127 failed with error .
Attempting polling connection to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3
HTTP GET request to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3&t=1577078290.682973 failed with error .
Attempting polling connection to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3
Task exception was never retrieved
future: <Task finished name='Task-18' coro=<TCPConnector._resolve_host() done, defined at /home/user/.local/lib/python3.8/site-packages/aiohttp/connector.py:774> exception=gaierror(-3, 'Temporary failure in name resolution')>
Traceback (most recent call last):
  (traceback omitted)
socket.gaierror: [Errno -3] Temporary failure in name resolution
HTTP GET request to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3&t=1577078299.8015127 failed with error .
Attempting polling connection to https://socket-io-tweet-stream.now.sh/socket.io/?transport=polling&EIO=3
Task exception was never retrieved
future: <Task finished name='Task-21' coro=<TCPConnector._resolve_host() done, defined at /home/user/.local/lib/python3.8/site-packages/aiohttp/connector.py:774> exception=gaierror(-3, 'Temporary failure in name resolution')>
Traceback (most recent call last):
  (traceback omitted)
socket.gaierror: [Errno -3] Temporary failure in name resolution
# ... like this forever

[buckle2000]

Thanks for making this library. Don't forget to release the fix to PyPI.