Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deco S7 - Fetch keys 401 unauthorized #130

Open
Nedevski opened this issue Dec 25, 2022 · 14 comments
Open

Deco S7 - Fetch keys 401 unauthorized #130

Nedevski opened this issue Dec 25, 2022 · 14 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@Nedevski
Copy link

Version of the custom_component

v2.2.3

Deco Model

Deco S7 (1.3.0 Build 20220609 Rel. 64814)

Describe the bug

Just got 2 different Deco networks and I'm trying to add them to 2 different HA instances (one of them is fresh).
I get the same result, I am using the settings below.
The IP of the root node is correct, the password is correct.
At my main HA instance I got some 2-3 second delay the first time I tried to add it and then it was an immediate error afterwards
Checked the logs and they don't provide any meaningful info for me, just straight 401's

I saw the other issue, however my error messages are different and that's why I'm opening a new issue.

image

Debug log

image

[Warning] Error testing credentials: 401, message='Unauthorized', url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')
[Error] Fetch keys client response error: 401, message='Unauthorized', url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')
@amosyuen
Copy link
Owner

Follow steps in Additional Debugging https://github.com/amosyuen/ha-tplink-deco/blob/master/DEBUGGING.md and find what the request for fetch keys looks like

@Nedevski
Copy link
Author

Nedevski commented Dec 26, 2022

Here is the debug log, nothing of value, I think

2022-12-26 18:30:06.704 DEBUG (MainThread)
[homeassistant.components.websocket_api.http.connection] [140208908303520]
Sending {"id":85,"type":"result","success":true,"result":[{"name":"custom_components.tplink_deco",
"message":["Error testing credentials: 401, message='Unauthorized',
url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')"],
"level":"WARNING","source":["custom_components/tplink_deco/config_flow.py",81],
"timestamp":1672072193.8828318,"exception":"","count":1,"first_occurred":1672072193.8828318},
{"name":"custom_components.tplink_deco",
"message":["Fetch keys client response error: 401, message='Unauthorized',
url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')"],"level":"ERROR",
"source":["custom_components/tplink_deco/api.py",137],"timestamp":1672072193.8812113,
"exception":"","count":1,"first_occurred":1672072193.8812113}]}

I logged in to the Deco Web UI with dev console open and these are the only requests I'm getting:
image

Also when I'm logged in the Web UI and I try to add the integration I am not logged out.
Even if I logout of the Web UI and I try again, still no luck.

Maybe it's just incompatible with this firmware?

EDIT: I tried clearing all cache, cookies and website data, same result.

@Nedevski
Copy link
Author

Ok, I did the advanced debugging part, I did find some data={something} in one of the requests and I ran it through the command.

Quick note - the command in the documentation did not work: jQuery.encrypt.encryptManager.encryptor.AesDecrypt(decodeURIComponent(data));

However this was successful:
jQuery.encrypt.encryptManager.encryptor.aes.decrypt(decodeURIComponent(data));

The output was:
{"operation":"read"}

@amosyuen
Copy link
Owner

Hmm, since you didn't see an http call that looks like login?form=keys then seems like the authentication method is different in that model and firmware.

You should only get logged out if the authentication suceeds, so it's expected that you didn't get logged out since it failed.

It is rather difficult to reverse engineer the API remotely, so we'll probably have to wait for a programmer who has a deco with incompatible firmware to look into it. For now I'll add your model and firmware to the README as incompatible.

@amosyuen amosyuen added the bug Something isn't working label Dec 26, 2022
@amosyuen amosyuen changed the title "Unexpected error" when adding the integration Deco S7 - "Unexpected error" when adding the integration Dec 26, 2022
@wzaatar
Copy link

wzaatar commented Dec 28, 2022

Same error with a Deco X60 mesh…

@amosyuen amosyuen changed the title Deco S7 - "Unexpected error" when adding the integration Deco S7 - Fetch keys 401 unauthorized Dec 28, 2022
@amosyuen amosyuen mentioned this issue Dec 29, 2022
@bsimmo
Copy link
Contributor

bsimmo commented Dec 30, 2022

Still happens with 1.3.3 firmware?
I guess more will go this way too.

@Nedevski
Copy link
Author

Nedevski commented Jan 3, 2023

Yes, still happens with 1.3.3 firmware. This is not an issue on the Deco side, they just seem to use different requests with the newer Deco models.

I am a developer myself, but not with Python but with .NET. I am still pretty new to creating/modifying integrations, so if someone that has more experience is willing to help, I can look into this.

@amosyuen
Copy link
Owner

amosyuen commented Jan 3, 2023

@Nedevski I can help with the HA integration part, the difficult part is reverse engineering TP-Link's calls and encryption scheme. In the old decos they do some encryption with RSA and AES, generating payloads with matching signs. Honestly I only figured it out from looking at other libraries / blog posts. Hopefully they haven't changed the encryption scheme too much.

@amosyuen amosyuen added the help wanted Extra attention is needed label Feb 2, 2023
@fgsilva12
Copy link

Anyone working on a solution for the Deco S7?

@Nedevski
Copy link
Author

Nedevski commented Dec 23, 2023

I just had a small breaktrough. So it looks like after you are logged in, you poll 3 different endpoints - network, device, client.

All requests are in the form of endpoint?form=performance&id=some_long_key and they return some encrypted response string. I just saw my previous comment about using the jquery encryptManager and I modified it a bit.

jQuery.encrypt.encryptManager.encryptor.aes.decrypt("some_encrypted_response_string");

Without any additional parameters I was able to get a full readable JSON with all my devices:

      {
        "linked_device_info": {
          "signal_level": {
            "band2_4": 0,
            "band5": 0
          },
          "connection_type": [
            "wired"
          ],
          "device_id": ""
        },
        "ip": "0.0.0.0",
        "mac": "XX-XX-XX-XX-XX-XX",
        "name": "UGhvbmU=",
        "online": false,
        "owner_id": "1",
        "remain_time": 0,
        "enable_priority": false,
        "interface": "main",
        "client_type": "phone",
        "wire_type": "wired",
        "connection_type": "wired",
        "up_speed": 0,
        "down_speed": 0
      },

I will look into it a bit more later today and will report if I find anything. If someone is a bit more familiar with encryption in general and is willing to waste a bit of time on this - ping me.

@aronkahrs-us
Copy link

aronkahrs-us commented Jan 27, 2024

seems like the authentication method is different in that model and firmware.

As far as I understand the new authentication method has 4 "steps" to login.

Step 1:
Some sort of "enable", it sends a POST request with 2 parameters (code and asyn) and in the form data it sends enable
code is 16 and asyn is 0

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    data = 'enable'

    response = requests.post(
        '<here goes the host>/?code=16&asyn=0',
        headers=headers,
        data=data,
    )

Step 2:
Get the RSA ee and nn, it sends a POST request with 3 parameters (code , asyn and id) and in the form data it sends get
code is 16, asyn is 0 and id seems to be a random, 32 character, alphanumeric string

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    data = 'get'

    response = requests.post(
        f'<here goes the host>/?code=16&asyn=0&id={USER_ID}',
        headers=headers,
        data=data,
    )

Step 3:
The actual login, here we send the password (RSA encrypted), it sends a POST request with 3 parameters (code , asyn and id) and in the form data it sends the RSA encrypted password
code is 7, asyn is 0 and id seems to be a random, 32 character, alphanumeric string

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    ee = keys[1]
    nn = keys[2]
    data = str(rsa_encrypt(int(nn,16),int(ee,16),password.encode()))

    response = requests.post(
        f'<here goes the host>/?code=16&asyn=0&id={USER_ID}',
        headers=headers,
        data=data,
    )

Step 3:
This step sets something, but I haven't figured out what, it sends a POST request with 3 parameters (code , asyn and id) and in the form data it sends set followed by something RSA encrypted
code is 16, asyn is 0 and id seems to be a random, 32 character, alphanumeric string

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    data = "set <Something encrypted>"

    response = requests.post(
        f'<here goes the host>/?code=7&asyn=0&id={USER_ID}',
        headers=headers,
        data=data,
    )

Every step returns 00000 if the request was successful, if not, it returns three, 5 numbers codes(error codes I assume), an encrypted code and 00000 at the end

All requests are in the form of endpoint?form=performance&id=some_long_key and they return some encrypted response string.

That id is the random, 32 character, alphanumeric string used to login

If I manage to get anything else I'll share it here, hope this helps.

@aronkahrs-us
Copy link

Update

This step sets something, but I haven't figured out what

It sets the AES Key (i supose to decrypt on the server side?), so basically it sends the key and the iv as a RSA encrypted string.

The string it encrypt is f"k={key}&i={iv}", so the data in the request ends up something like this:

data = "set "+str(rsa_encrypt(int(nn,16),int(ee,16),f"k={key}&i={iv}".encode()))

@aronkahrs-us
Copy link

Update 2

id seems to be a random, 32 character, alphanumeric string

Turns out it's not random, it's an encrypted form of the password:

var n = s.su.encrypt(e[3], s.encrypt.MD5(r), e[4])
return localStorage.setItem("id", n)

I don't know which encryption it uses.

@aronkahrs-us
Copy link

doLogin function

I'm going to share the controller.js file that has the doLogin function

!function(s) {
    s.su.moduleManager.define("localLogin", {
        services: ["ajax"],
        models: ["localLogin", "localLoginControl"],
        stores: [],
        views: ["localLoginView"],
        deps: ["login", "main"],
        listeners: {
            "ev_on_launch": function(e, n, t, o, r, l, a) {
                s.encrypt.encryptManager.cleanStorage(),
                s.su.encryptor = s.encrypt.encryptManager.genEncryptor(),
                localStorage.removeItem("id"),
                l.main.keyDictionary = "",
                l.main.getTmpKey()
            }
        },
        init: function(n, e, t, o, r, l) {
            this.control({
                "#local-login-pwd": {
                    "keyup": function(e) {
                        13 == e.keyCode && n.doLogin()
                    }
                },
                "#local-login-button": {
                    "ev_button_click": function() {
                        n.doLogin()
                    }
                },
                ".local-login-switch-to-tp": {
                    "click": function(e) {
                        r.login.goToChildModule("tpLogin")
                    }
                },
                ".local-login-forget-pwd": {
                    "click": function() {
                        e.localLoginView.forgetPasswordMsg.show()
                    }
                }
            })
        }
    }, function(l, c, i, e, a, n) {
        return {
            encryptKey: null,
            doLogin: function() {
                if (i.localLoginControl.password.validate()) {
                    var r = i.localLoginControl.password.getValue();
                    c.localLoginView.loginBtn.disable(),
                    a.main.getTmpKey().then(function(e) {
                        return l.enableGDPR(e)
                    }).then(function(e) {
                        var n = s.Deferred();
                        return l.getGDPRKey().then(function() {
                            n.resolve(e)
                        }, function() {}),
                        n
                    }).then(function(e) {
                        var n = s.su.encrypt(e[3], s.encrypt.MD5(r), e[4])
                          , o = s.Deferred()
                          , t = s.encrypt.encryptManager.getEncryptor();
                        return localStorage.setItem("id", n),
                        s.ajax({
                            url: s.su.url("?code=7&asyn=0"),
                            data: t.rsaEncrypt(r)
                        }).then(function(e) {
                            "00000" === e.split("\r\n")[0] ? (i.localLoginControl.password.setValue(null),
                            o.resolve()) : o.reject()
                        }, function(e) {
                            var n = e.responseText.split("\r\n");
                            switch (parseInt(n[1])) {
                            case 0:
                                i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginFull);
                                break;
                            case 1:
                                i.localLoginControl.password.disable(),
                                c.localLoginView.leftAttemptsMsgContent.setText(s.su.CHAR.LOGIN.loginLock),
                                c.localLoginView.leftAttemptsMsg.show();
                                break;
                            case 2:
                                i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginTimeout);
                                break;
                            case 3:
                            case 5:
                                var t = n[2] == undefined ? 10 : 10 - parseInt(n[2]);
                                if (10 === t)
                                    break;
                                l.disableButton(c.localLoginView.loginBtn),
                                t <= 5 ? (c.localLoginView.leftAttemptsMsgContent.setText(s.su.CHAR.LOGIN.loginErrorTipH.replace("%s", t)),
                                c.localLoginView.leftAttemptsMsg.show()) : i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginPwdErr);
                                break;
                            case 6:
                                l.disableButton(c.localLoginView.loginBtn),
                                i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginPwdErr)
                            }
                            o.reject()
                        }),
                        o
                    }).then(function() {
                        return l.setGDPRKey()
                    }).then(function() {
                        a.main.loadBasicModule("index")
                    }).always(function() {
                        0 < l.countDownSec || c.localLoginView.loginBtn.enable()
                    })
                }
            },
            loginSuccessDealer: function(e, n) {
                var t, o, r, l = e.stok || (t = "12345",
                o = top.location.href,
                0 <= (r = o.indexOf("stok=")) && (t = o.substring(r + 5)),
                t);
                localStorage && (a.main.setToken(l),
                a.main.reload())
            },
            loginFailDealer: function(e, n) {
                var t = e.result;
                switch (c.localLoginView.loginBtn.enable(),
                n) {
                case -5002:
                    var o, r = t.failureCount, l = t.attemptsAllowed;
                    if (r < 7) {
                        var a = String(-5002).replace(/^-/, "E");
                        i.localLoginControl.password.setError(s.su.CHAR.ERROR[a])
                    } else
                        o = s.su.CHAR.LOGIN.LOGIN_FAILED_COUNT.replace("%num1", r).replace("%num2", l),
                        c.localLoginView.leftAttemptsMsgContent.setText(o),
                        c.localLoginView.leftAttemptsMsg.show();
                    break;
                case -5003:
                    c.localLoginView.maxAttemptsMsgContent.setText(s.su.CHAR.ERROR["00000089"]),
                    c.localLoginView.maxAttemptsMsg.show()
                }
            },
            queryRecoveryPasswordMethod: function() {
                var e = s.Deferred();
                return e.resolve(!0),
                e.promise()
            },
            enableGDPR: function(n) {
                var t = s.Deferred();
                return s.ajax({
                    url: s.su.url("?code=16&asyn=0"),
                    data: "enable"
                }).then(function(e) {
                    "00000" === e.split("\r\n")[0] ? t.resolve(n) : t.reject()
                }, function() {
                    t.reject()
                }),
                t
            },
            getGDPRKey: function() {
                var n = s.Deferred()
                  , t = s.encrypt.encryptManager.getEncryptor() || s.encrypt.encryptManager.genEncryptor();
                return s.ajax({
                    url: s.su.url("/?code=16&asyn=0"),
                    data: "get"
                }).then(function(e) {
                    data = e.split("\r\n"),
                    "00000" === data[0] ? (t.setRSAKey(data[2], data[1]),
                    t.setSeq(data[3]),
                    t.genAESKey(),
                    s.encrypt.encryptManager.recordEncryptor(),
                    n.resolve()) : n.reject()
                }, function() {
                    n.reject()
                }),
                n
            },
            setGDPRKey: function() {
                var n = s.Deferred()
                  , e = s.encrypt.encryptManager.getEncryptor();
                return s.ajax({
                    url: s.su.url("/?code=16&asyn=0"),
                    data: "set " + e.getEncodeAESKey()
                }).then(function(e) {
                    "00000" === e.split("\r\n")[0] ? n.resolve() : n.reject()
                }, function() {
                    n.reject()
                }),
                n
            },
            disableButton: function(e) {
                l.countDownSec = 30,
                e.setText(s.su.CHAR.LOGIN.LOGIN_BUTTON_COUNTDOWN.replace("%num%", l.countDownSec)),
                l.intervalId = setInterval(function n() {
                    l.countDownSec--,
                    0 < l.countDownSec ? e.setText(s.su.CHAR.LOGIN.LOGIN_BUTTON_COUNTDOWN.replace("%num%", l.countDownSec)) : (clearInterval(l.intervalId),
                    e.enable(),
                    e.setText(s.su.CHAR.LOGIN.LOGIN_BUTTON_TEXT))
                }, 1e3),
                e.disable()
            }
        }
    })
}(jQuery);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

6 participants