Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error with Twitch integration when updating #101488

Closed
pablogonzalezalba opened this issue Oct 5, 2023 · 28 comments
Closed

Error with Twitch integration when updating #101488

pablogonzalezalba opened this issue Oct 5, 2023 · 28 comments

Comments

@pablogonzalezalba
Copy link

The problem

Hello,
I have successfully connected the new Twitch integration on 2023.10 and at the time of integration, all my followed channels were retrieved with the correct status (streaming/offline).
However, since yesterday, they haven't updated. Checking the logs I have the following:

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:732
First occurred: 03:57:31 (123858 occurrences)
Last logged: 20:20:13

    Update for sensor.bengineering fails
    Update for sensor.annacramling fails
    Update for sensor.hueywj fails
    Update for sensor.amazonian fails
    Update for sensor.benjamin_wheeler fails

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 732, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 991, in async_device_update
    await self.async_update()
  File "/usr/src/homeassistant/homeassistant/components/twitch/sensor.py", line 133, in async_update
    followers = (await self._client.get_channel_followers(self._channel.id)).total
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ChannelFollowersResult' object has no attribute 'total'

After enabling debug logging I also found this (it repeats for each account I follow):

2023-10-05 00:02:05.647 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.lafistreams fails
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 732, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 991, in async_device_update
    await self.async_update()
  File "/usr/src/homeassistant/homeassistant/components/twitch/sensor.py", line 139, in async_update
    await self._async_add_user_attributes()
  File "/usr/src/homeassistant/homeassistant/components/twitch/sensor.py", line 163, in _async_add_user_attributes
    sub = await self._client.check_user_subscription(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twitchAPI/twitch.py", line 1645, in check_user_subscription
    return await self._build_result('GET', 'subscriptions/user', param, AuthType.EITHER, [AuthScope.USER_READ_SUBSCRIPTIONS], UserSubscription)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twitchAPI/twitch.py", line 516, in _build_result
    response = await self._api_request(method, session, _url, auth_type, auth_scope, data=body_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twitchAPI/twitch.py", line 441, in _api_request
    req = await session.request(method, url, headers=headers, json=data)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.twitch.tv:443 ssl:default [Try again]

What version of Home Assistant Core has the issue?

core-2023.10.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Twitch

Link to integration documentation on our website

https://www.home-assistant.io/integrations/twitch/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

@home-assistant
Copy link

home-assistant bot commented Oct 5, 2023

Hey there @joostlek, mind taking a look at this issue as it has been labeled with an integration (twitch) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of twitch can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign twitch Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)


twitch documentation
twitch source
(message by IssueLinks)

@iHaveAstream
Copy link

Hi,

very same issue here, also with HA 2023.10.0.
Has worked without any (error) logs for a few hours todqy, but since 1 1/5 hrs permanently increasing.
Reloading the Twitch Integration fixes it for a while but then it's gonna happen again.

@yyolk
Copy link
Contributor

yyolk commented Oct 5, 2023

I'm also receiving the error, I figured I'd be hitting this with my 200+ channels I follow (most of which haven't been online in ages, forgot that I followed). After enabling debug logging for about 5 minutes I get a 300+mb debug log 🫠. I also saw that the OP has 123858 occurences, and thought that was huge, but then saw in my logs, 427231 occurrences hehe.

I think the problem is throttling, from the debug it's clear that is occuring:

2023-10-04 15:32:46.762 WARNING (MainThread) [twitchAPI.twitch] reached rate limit, waiting for reset
2023-10-04 15:32:46.763 WARNING (MainThread) [twitchAPI.twitch] reached rate limit, waiting for reset
2023-10-04 15:32:46.820 WARNING (MainThread) [twitchAPI.twitch] reached rate limit, waiting for reset
2023-10-04 15:32:46.837 WARNING (MainThread) [twitchAPI.twitch] reached rate limit, waiting for reset
2023-10-04 15:32:46.837 WARNING (MainThread) [twitchAPI.twitch] reached rate limit, waiting for reset

I also see this where the homeassistant.helpers.entity WARNs around the updates taking too long they occur directly after lines about reaching rate limit above, so I'm thinking they're directly correlated.

2023-10-04 15:41:46.466 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.deadmau5 is taking over 10 seconds
2023-10-04 15:41:46.466 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.whippy is taking over 10 seconds
2023-10-04 15:41:46.466 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.playstation is taking over 10 seconds

AFAIK each channel is updated every minute, the normal frequency for sensor updates.

I can also confirm, that "reloading" the integration fixes it temporarily. I imagine it's re-creating the session that has it's throttle reset.

I see similar stack traces in the OP in my logs, but I am pretty certain that's a result of not catching a case where the twitch client from pyTwitchAPI library doesn't get the response and is instead throttled or the API responds immediately with an empty body before throttling or something so in looking at the object that holds the response will throw those errors.

The second error where the connection is outright refused, I am also pretty certain is due to throttling, but since the API doesn't respond, there might be a case where the client doesn't know why it can't connect and defers to throwing a generic ClientConnectionError exception.

@LordBoos
Copy link
Contributor

LordBoos commented Oct 7, 2023

I have the same problem and I only follow 27 channels. It works for an hour or few hours and then stops updating.

@Th3M1k3y
Copy link

Th3M1k3y commented Oct 8, 2023

Is a bearer token used when contacting the API?

In their API documentation they mention up to 800 calls pr minute if a bearer token is provided with the call.

@joostlek
Copy link
Member

joostlek commented Oct 8, 2023

Yes, we use a bearer token. But we're doing more requests for one channel. I will try to optimize this and split out the current sensor into more sensors before the next release as I think this is the only way to proper fix this.

@Th3M1k3y
Copy link

Th3M1k3y commented Oct 8, 2023

How many calls pr channel is it making?

Searching for a solution I found a thread saying disabling pooling and creating an automation to update using homeassistant.update_entity at a lower rate would fix it. But following 84 channels, and updating every /5 minute is still making it hit the limit. I also tried disabling all the entities I am not interested in monitoring in the hope that would help, but also no luck with that.

@ZerZex
Copy link

ZerZex commented Oct 8, 2023

How many calls pr channel is it making?

Searching for a solution I found a thread saying disabling pooling and creating an automation to update using homeassistant.update_entity at a lower rate would fix it. But following 84 channels, and updating every /5 minute is still making it hit the limit. I also tried disabling all the entities I am not interested in monitoring in the hope that would help, but also no luck with that.

It will probably come down to making the calls more efficient, like the get-streams API, instead of sending a single login name you can query up to 100 at a time. And since the max people you can follow on twitch is 2000 I believe, the calls definitely need to be more efficient/robust.

@morremeyer
Copy link

morremeyer commented Oct 10, 2023

I am running into this with only 4 channels that I follow, so it does not look like an issue that only appears when following a lot of channels.

@joostlek Thanks for looking into it ❤️

@Steltek
Copy link

Steltek commented Oct 10, 2023

Chiming in: I disabled polling on the integration and set up an automation to poll 2 channels every 5 minutes. It still failed. Strangely enough, it starts working again after a reload, as if it got stuck on something that's not rate limiting. (Is it not refreshing auth tokens properly or something? ...)

Going to try scheduling an automated reload of the integration to see what that does...

Update: Regularly reloading the integration, combined with the more limited polling, indeed appears to make it work.

@storm1er
Copy link

@Steltek

Going to try scheduling an automated reload of the integration to see what that does...

How do you do that?
So we can have a workaround displayed here

@Steltek
Copy link

Steltek commented Oct 13, 2023

@Steltek

Going to try scheduling an automated reload of the integration to see what that does...

How do you do that? So we can have a workaround displayed here

Create an automation with a time pattern trigger, and in the actions add a call to "Home Assistant Core Integration: Reload config entry" and select the Twitch channel entity you want to be updated. (I'm not sure if a single reload call is enough to reload the entire integration or if you need one for each channel. Since I'm only monitoring 2 channels, I just added a call for each.)

@iHaveAstream
Copy link

I do it that way now, every hour. Still get around 1k errors. I do follow 9 channels.

@Steltek
Copy link

Steltek commented Oct 14, 2023

I do it that way now, every hour. Still get around 1k errors. I do follow 9 channels.

How frequently are you polling them? (I poll 2 channels every 5 minutes, and reload the integration every hour, with no errors.)

@iHaveAstream
Copy link

I didn't change something. What's the default? And where can I change it?

@Jaeegerr
Copy link

Funnywise had the Same error and was reading here also. Since i made an Account on Twitch ONLY for the Integration and all works it without Problems with around 15 Entities.
Dont Know if twitch has a Limit in there where you could only respond to certain amounts of Channels.

@BreeOge
Copy link

BreeOge commented Oct 26, 2023

I am having this issue as well.. It continues the last 11 lines, over and over again. My last log had over 2414906 lines of errors.
Tried to add it as code below, but it would not format correctly.

2023-10-22 19:50:43.307 DEBUG (MainThread) [homeassistant.components.twitch] User is not subscribed
2023-10-22 19:50:43.375 DEBUG (MainThread) [homeassistant.components.twitch] User is not subscribed
2023-10-22 19:50:43.877 DEBUG (MainThread) [homeassistant.components.twitch] User is not subscribed
2023-10-22 19:50:43.894 DEBUG (MainThread) [homeassistant.components.twitch] User is not subscribed
2023-10-22 19:52:00.944 ERROR (MainThread) [homeassistant.components.sensor] Platform twitch does not generate unique IDs. ID 404702041 already exists - ignoring sensor.alzerngameing
2023-10-22 20:55:58.034 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.ti_tessarofilipe is taking over 10 seconds
2023-10-22 20:56:18.001 WARNING (MainThread) [homeassistant.components.sensor] Updating twitch sensor took longer than the scheduled update interval 0:00:30
2023-10-22 20:56:48.002 WARNING (MainThread) [homeassistant.components.sensor] Updating twitch sensor took longer than the scheduled update interval 0:00:30
2023-10-22 20:57:18.003 WARNING (MainThread) [homeassistant.components.sensor] Updating twitch sensor took longer than the scheduled update interval 0:00:30
2023-10-22 20:57:48.004 WARNING (MainThread) [homeassistant.components.sensor] Updating twitch sensor took longer than the scheduled update interval 0:00:30
2023-10-22 20:57:59.793 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.ti_tessarofilipe fails
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
sock = await self._connect_sock(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 628, in sock_connect
return await fut
^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 668, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('146.75.82.214', 443)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 732, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 991, in async_device_update
await self.async_update()
File "/usr/src/homeassistant/homeassistant/components/twitch/sensor.py", line 139, in async_update
await self._async_add_user_attributes()
File "/usr/src/homeassistant/homeassistant/components/twitch/sensor.py", line 173, in _async_add_user_attributes
follows = await self._client.get_followed_channels(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twitchAPI/twitch.py", line 1849, in get_followed_channels
return await self._build_iter_result('GET', 'channels/followed', param, AuthType.USER, [AuthScope.USER_READ_FOLLOWS], FollowedChannelsResult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twitchAPI/twitch.py", line 483, in _build_iter_result
response = await self._api_request(method, session, _url, auth_type, auth_scope, data=body_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twitchAPI/twitch.py", line 441, in _api_request
req = await session.request(method, url, headers=headers, json=data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
raise last_exc
File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.twitch.tv:443 ssl:default [Connect call failed ('146.75.82.214', 443)]
2023-10-22 21:21:58.073 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.alexmann0014 is taking over 10 seconds
2023-10-22 21:22:18.051 WARNING (MainThread) [homeassistant.components.sensor] Updating twitch sensor took longer than the scheduled update interval 0:00:30
2023-10-22 21:22:48.052 WARNING (MainThread) [homeassistant.components.sensor] Updating twitch sensor took longer than the scheduled update interval 0:00:30
2023-10-22 21:43:01.454 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.breeoge fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 732, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 991, in async_device_update
await self.async_update()
File "/usr/src/homeassistant/homeassistant/components/twitch/sensor.py", line 133, in async_update
followers = (await self._client.get_channel_followers(self._channel.id)).total
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

@wizbowes
Copy link

+1. I have many subscriptions, but all but 2 are disabled. Still failing every 30 seconds on those 2.

@ScottG489
Copy link
Contributor

If this is throttling related, a more robust approach may be to implement a "push" model using webhooks:
https://dev.twitch.tv/docs/eventsub/

@igorsantos07
Copy link
Contributor

igorsantos07 commented Jan 13, 2024

I get baffled how some simple integrations go into the HA Core and are broken for months (or "since forever" as can be seen by the amount of integration bugs abandoned here) and some other very complex integrations and almost flawless end up in HACS as "secondary citizens", such as LG devices.

At the start, I thought naively there would be a "promotion process" of some sort to move in-dev integrations from HACS into the Core once they're considered useful and stable... but now I see it's just a matter of where the person is comfortable coding and allowed to do PRs (or leaving bugs).

I know this is open source and all, not to shoot the messenger developer, Joost is definitely busy with school, life happens, etc... Been there a bunch of times.
But hey, this bug was posted shortly after a blog mention about an official integration and no one pulled it out, reverted the changes, or fixed it in three months? There's a completely broken integration in HA. But I guess it's so rarely used (125 installations so far) it's not worth thinking this reduces the overall quality perception of HA.

@Krispkiwi
Copy link

Yeah It's broken in my instance too same issue, hopefully it gets fixed soon

@joostlek
Copy link
Member

Does this still happen at the moment? Something major has been fixed in a 2023.3 hotfix

@iHaveAstream
Copy link

Actually, it seems to be fixed...

@ScottG489
Copy link
Contributor

Fixed for me as well for now. However, I think it would take a while for the issue to crop up so best to give it at least a few days to confirm.

@LordBoos
Copy link
Contributor

LordBoos commented Apr 1, 2024

I'm running it for several days now and everything seems OK. I have 30 followed channels and all are updating correctly.

@stefmde
Copy link

stefmde commented Apr 1, 2024

I had the Twitch HA Integration disabled myself because my plugin showed too much nonsens due to the bug in the Integration. When you guys says it might be fixed then I will try again. Hope it works again. It was really useful before for me. And thanks for the update :)

@joostlek
Copy link
Member

joostlek commented Apr 1, 2024

I'll close this issue then

@joostlek joostlek closed this as completed Apr 1, 2024
@ScottG489
Copy link
Contributor

I just thought I'd say that I haven't had any issues since the fix. Thanks @joostlek!

@github-actions github-actions bot locked and limited conversation to collaborators May 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests