Codecov Report

Patch coverage is 31.06% of modified lines.

📢 Thoughts on this report? Let us know!.

Looks like you beat me to this by about a week! Thanks for getting this updated, and apologies my PR got a little lost (#267).

Just did a test and it worked with the sockets and setup I have here! Thank you for the additional work you put into this too.

C

I have updated this to include the 9999 port configurable changes from #471 (still only works with the old protocol atm) and reverted my change of the "target" variable name so it doesn't break homeassistant. I've now tested this with homeassistant and have some changes shortly to be in a home assistant fork which will enable support for both protocols and unauthenticated devices.

Hi, I've updated this PR so it passes all the CI tests including pypy-3.8 windows-latest which was intermittently problematic. There's also a fix for an issue introduced yesterday by a Cython release which affected the pypy-3.8 ubuntu-latest and I think will affect all CI runs regardless of my changes. @rytilahti I know it's a chunky PR but do you have a feel for when we might be able to start reviewing?

Hi @rytilahti, thanks for taking the time to review. Some comments inline

Hi @sdb9696 and thanks for the updated PR!

I am currently not able to go through all of this PR given its size and as I have no way to test it myself properly, but I did a partial (i.e., without klapprotocol.py) review to give some feedback and improvement ideas.

Some thoughts without specific order of importance:

• Please keep this PR as simple as possible. This means that avoid introducing any unrelated changes and create separate PRs for those, I added some comments inline.

Understood, will do.

• The existing functionality shall not be broken. We want to keep the existing APIs functioning without any changes necessary for downstreams who are already happy.

Understood, will do, although I'd appreciate it if you could reconsider adding "port" to all the device classes that was introduced in PR #471 as it's quite contrary to the idea of having different protocols. Once the protocol is also added to the device constructor the only purpose of having the port is to display it in the only added in order to display the port in the state() function in cli.py. I'd be happy to do a separate PR simply to replace port with protocol for the device classes for the next release.

Update - For now the port remains

• Avoid any type of protocol-specific handling in the main classes (this includes things like conditionals, checking for instance types/ports) wherever possible. Looks like the TAPO devices are using the same discovery protocol, but other parts are different. Keeping things uncoupled makes it easier to introduce support for those devices if wanted.

I tried to take on board your comments from the original PR for this around moving conditional logic out of discovery and into the protocol classes but it's difficult to completely avoid the authenticated/unauthenticated flow here. The reason is there are different flows due to the way it's been implemented by tp-link and the very real possibility of not being able to authenticate. Broadcast discover() results in some basic info being returned, you then have to do a separate get_info call which could succeed or fail. Then there is discover_single which is simple if authentication succeeds but if it fails a subsequent broadcast is needed to get that basic info. I'll try to think of how I can further extract this away to reduce the impact on the discovery class but hopefully you can see it's quite intrinsic.

Update - I think this is now a lot simpler, please let me know

• Please rebase your changes on top of the previously existing PR, as this will keep the commit history intact and give credit also to those who worked on this earlier.

I will try to do this

Update - now done

My suggestion is the following:

• Extract the discovery (and other unrelated changes as commented inline) from this PR into their own so we can review and merge them separately. Starting with discovery feels like a good place to start as having available as a first step will let us inform users about unsupported devices and avoid issue reports on them.

Ok will do

Update - separate PRs now awaiting review

• The discovery should be unauthenticated at first, we can revisit this when we find a proper level of abstraction. Instead of having a separate "unauthenticateddevice", maybe we can find a solution where trying to access authenticated features would raise an exception and allow authentication?

Not sure how to this easily for a few following reasons: We don't know what Device type to create if we can't authenticate. We could try to derive it from the discovery info payload but that would make Discovery protocol aware and you're then left with a device that errors all the time. In my homeassistant fork I create a device with no entities when encountering this UnauthenticatedDevice which works well as there's no functionality to be expected but if it's treated as a normal device homeassistant would produce errors. It is actually common for my devices to suddenly be unable to authenticate as the kasa app seems to update the device owner when it changes state on a schedule and you need to manually switch the light on and off via the app to get the right owner back.

Update - This is now removed and the discovery makes a call to get_sys_info. Let me know what you think

• I will see if I find some time to look into the APIs at some point, maybe I will try integrating the tapo protocol for getting a feeling on how the relevant architectural parts should be done, but I will make no promises on that.

P.S. Are devices using this protocol still being produced and available? That is, could I go and buy myself one for testing (e.g. https://www.amazon.co.uk/TP-Link-Monitoring-SmartThings-Wireless-KP115/dp/B08LZWBTR6)? I am just trying to get an idea if it would be a better idea to have a separate fork/project for this product line instead of trying to shoehorn support into this project? AFAIK, this protocol has never been seen in the wild in EU and US versions of kasa devices, and probably never will considering that plenty of new devices are using tapo branding now...

Good question, I would say it's hard to get hold of these devices new now but I know there are a lot of people who have these and really want to the get them working with homeasistant. Also a big driver for me creating a base class of TPLinkAuthenticationProtocol was this comment by TP-link https://community.tp-link.com/en/smart-home/forum/topic/239364?page=1 that they would in the future be updating all kasa devices to use a more secure local protocol. The trouble with a separate fork is that it would be great to get this working out of the box with homeassistant. Let me know what you want to do though before I put in all the effort above :)

Update - I put the effort in anyway :)

Update: The same command works again, after waiting ~2 hours. Could it be their server limiting my requests?

Hello, I'm currently using your commit "Remove whitespace changes and re-fix codeql" (commit 99073f7). I used the following command to toggle my HS100 V4.1 firmware V1.1.0 on and off:

kasa --enable_klap --username [email protected] --password XXXXXX --host 192.168.1.111 off
where I have set a static IP 192.168.1.111 for the HS100.

It worked fine for a 5-6 hours in a script that runs every few minutes, and suddenly it threw an error
Got error: SmartDeviceAuthenticationException("Server response doesn't match our challenge on ip 192.168.1.111")
which I'm unable to resolve. The app works fine as usual.

Any advice would be helpful, thanks!

I ordered a box of KP125M's this week and quickly ended up here when I couldn't get them into HomeAssistant. Appreciate you all working on the updates here! 🎉

@shuanang I'm getting a similar result with my KP125M on the same commit. KP125M, firmware 1.1.0

➜  python-kasa git:(master) poetry run kasa --enable_klap --username [email protected] --password XXX --host 192.168.0.63            
No --type defined, discovering..
Got error: SmartDeviceAuthenticationException("Server response doesn't match our challenge on ip 192.168.0.63")

If I pass the --type plug, it seems to try to connect over 9999, which if I understand correctly, is the old/unsupported mechanism.

➜  python-kasa git:(master) poetry run kasa --enable_klap --username [email protected] --password XXX --host 192.168.0.63 --type plug
Got error: SmartDeviceException("Unable to connect to the device: 192.168.0.63:9999:  Connect call failed ('192.168.0.63', 9999)")

EDIT: with the -d debug option:

DEBUG:kasa.klapprotocol:[KLAP] Created KLAP object for 192.168.0.63
DEBUG:kasa.klapprotocol:[KLAP] Starting handshake with 192.168.0.63
DEBUG:kasa.klapprotocol:Handshake1 posted at 2023-08-07 15:16:27.958500.  Host is 192.168.0.63, Session cookie is Set-Cookie: TP_SESSIONID=None, Response status is 200, Request was ba7a24929f4b1873d9a8c68704464772
DEBUG:kasa.klapprotocol:Server remote_seed is: 3c68746d6c3e3c626f64793e3c63656e, server hash is: 7465723e323030204f4b3c2f63656e7465723e3c2f626f64793e3c2f68746d6c3e
DEBUG:kasa.klapprotocol:Expected 00a8400e199d05687c7b7350f4312d596bae163e339e811c0e71915e6bc9d1a2 got 7465723e323030204f4b3c2f63656e7465723e3c2f626f64793e3c2f68746d6c3e in handshake1.  Checking if blank auth is a match
DEBUG:kasa.klapprotocol:Server response doesn't match our challenge on ip 192.168.0.63
DEBUG:kasa.klapprotocol:Unable to complete handshake for device 192.168.0.63, authentication failed
DEBUG:kasa.klapprotocol:Unable to authenticate with 192.168.0.63, not retrying
Got error: SmartDeviceAuthenticationException("Server response doesn't match our challenge on ip 192.168.0.63")

The Expected {XXX} changes each time, but the got 7465723e323030204f4b3c2f63656e7465723e3c2f626f64793e3c2f68746d6c3e is constant.

EDIT 2: Adding some debug printing code, it looks like the response_data is a HTML message, not the expected handshake parameters?

DEBUG:kasa.klapprotocol:remote_seed: b'<html><body><center>200 OK</center></body></html>'

Update: The same command works again, after waiting ~2 hours. Could it be their server limiting my requests?

Hello, I'm currently using your commit "Remove whitespace changes and re-fix codeql" (commit 99073f7). I used the following command to toggle my HS100 V4.1 firmware V1.1.0 on and off:

kasa --enable_klap --username [email protected] --password XXXXXX --host 192.168.1.111 off where I have set a static IP 192.168.1.111 for the HS100.

It worked fine for a 5-6 hours in a script that runs every few minutes, and suddenly it threw an error Got error: SmartDeviceAuthenticationException("Server response doesn't match our challenge on ip 192.168.1.111") which I'm unable to resolve. The app works fine as usual.

Any advice would be helpful, thanks!

Hi, apologies for the delay in replying I have been away. I think the issue you have been encountering is the same one I have been getting and it seems to be related to the device randomly changing owner. It seems to be something to do with when the device operates on it's internal scheduler and it changes the owner to something that produced this md5 hash: 994661e5222b8e5e3e1d90e73a322315.

I was wrestling with this before I went away but I've now been able to identify the new owner from the kasa apk source code as [email protected]. I've updated the PR to fall back and try these credentials if the correct username and password does not work. This seems to be working fine for me now. It would great if you could try out with my latest commit and let me know.

@shuanang I'm getting a similar result with my KP125M on the same commit. KP125M, firmware 1.1.0

I don't think your issue is the same as @shuanang. Did this ever work for you? (@shuanang had it working then it stopped). afaik this fix will not work with the KP125M as it only affects HS100 with the new klap protocol. It looks to me like your KP125M might be doing something with the TAPO protocol.

@mthole feel free to give #499 (it implements tapo support using an external lib) a try and let us know if that works on that KP125M. Alternatively, you could also try #488 to see and let us know how the discovery response looks for this device so we can get an idea if we need to accommodate those separately.

Let's keep the discussion about KP125M in one of those issues & in #450.

Update: The same command works again, after waiting ~2 hours. Could it be their server limiting my requests?
Hello, I'm currently using your commit "Remove whitespace changes and re-fix codeql" (commit 99073f7). I used the following command to toggle my HS100 V4.1 firmware V1.1.0 on and off:
kasa --enable_klap --username [email protected] --password XXXXXX --host 192.168.1.111 off where I have set a static IP 192.168.1.111 for the HS100.
It worked fine for a 5-6 hours in a script that runs every few minutes, and suddenly it threw an error Got error: SmartDeviceAuthenticationException("Server response doesn't match our challenge on ip 192.168.1.111") which I'm unable to resolve. The app works fine as usual.
Any advice would be helpful, thanks!

Hi, apologies for the delay in replying I have been away. I think the issue you have been encountering is the same one I have been getting and it seems to be related to the device randomly changing owner. It seems to be something to do with when the device operates on it's internal scheduler and it changes the owner to something that produced this md5 hash: 994661e5222b8e5e3e1d90e73a322315.

I was wrestling with this before I went away but I've now been able to identify the new owner from the kasa apk source code as [email protected]. I've updated the PR to fall back and try these credentials if the correct username and password does not work. This seems to be working fine for me now. It would great if you could try out with my latest commit and let me know.

Hey, thanks for getting back! I'd like to update that I've tried your latest commit cce1f6e and my script has been running without issues for almost a day now. Thanks for the hard work and for detailing the issue! :-)

Authentication failed for device: is print by python-kasa by launching discovery using the code of this PR

Got it thanks

EDIT: you can found my implementation (inspired by this PR) here: https://github.com/petretiandrea/plugp100/blob/main/plugp100/protocol/klap_protocol.py. This library is used to integrate Tapo devices with home assistant. The chiper is the same implemented in this PR

That’s really good and glad to see klapprotocol.py in use. You turned around this fix really fast so I’m sure your users are v.grateful. At some point we should compare notes on how this new protocol behaves in the real world.

Hi @rytilahti, I've rebased this PR from master following your merge of the discovery changes in #488. I've also incorporated the comments from #488 (comment) with regard to the protocol not being in the constructors. Let me know what you think so far.

See some comments inline, I didn't yet take a look into the klapprotocol implementation but I think the best steps forward would be to split this PR even more:

1. A PR to add support for credentials (incl. the new exception), this will unblock me so I can experiment with the tapo support.

   • This involves adding authentication to the ctor of SmartDevice derived classes.
   • SmartDevice.__init__ can store this info, which can then be used wherever needed.

Done in PR #507 but only as far as discovery as I still think the credentials belong to the protocol rather than the device. Perhaps only the TAPO shim can add it to it's own __init__?

2. Refactor the discovery support to make it easier to write tests. Instead of having everything contained in the datagram received, let's split it up:

   • _get_device_instance_legacy(info) is responsible for constructing a device instance for the old 9999/udp responses.

   • _get_device_instance(info, credentials) which handles the 20002/udp initialization. I'm not sure if this should already do the update try, but rather leave it for the first time the downstream calls update() (which would then raise the exception if auth fails).

     • This method is also responsible for setting the protocol class for klap devices.
     • This would be the place where I could hook the tapo plug draft nicely, and in case tplink decides to introduce klap for future tapo devices, this library could be extended to support those directly.

Ok will do

I think there are still some open questions on how to approach integrating this with homeassistant, but I think we are getting there :-)

Open questions include things like:

• What if discovery was done using invalid creds or no creds at all? I suppose that could be a fairly normal downstream use-case, so I'm somehow imagining that initialization of device instances should work without valid creds, update() raising the exception if necessary. For example, homeassistant can deliver discovery also using mac addresses from DHCP. In such cases, the device could be added to homeassistant even when getting them to function will require user action to input the creds.

Ok I will try this

• On that topic, maybe Discover.discover should not have a separate "auth failed", but deliver those devices just as-is, and leave it to the downstream user to 1) set new Credentials/Authenticaton object and 2) retry update()? I think that will also handle the use case where the user changes the password even when the device has already been added to home assistant: the "repair" item will appear and ask for the creds.

Ok I will try this

Btw, do you have a homeassistant integration fork using this PR for testing? I could experiment with that and the tapo plug to get a better feeling about how it would work with homeassistant, at the moment I'm reading the creds from the envvars in that tapo draft PR.

I do but it's not great right now as I was trying to store the credentials globally in a hacky way. I'm planning to update ti store the credentials in an OptionFlow config and then update all the configentries whenever one of the OptionsFlow changes. Unless you have a better suggestion. Once you've merged the credentials PR I'll update it work with the new classname and point you to it

Done in PR #507 but only as far as discovery as I still think the credentials belong to the protocol rather than the device. Perhaps only the TAPO shim can add it to it's own __init__?

The thing is (as mentioned in a comment above) is that it is better to have a common, stable interface for downstream users no matter what device they may have.

I do but it's not great right now as I was trying to store the credentials globally in a hacky way. I'm planning to update ti store the credentials in an OptionFlow config and then update all the configentries whenever one of the OptionsFlow changes. Unless you have a better suggestion. Once you've merged the credentials PR I'll update it work with the new classname and point you to it

I don't know if there's a nicer way to store the creds for the integration, but I hope so. Updating config entries by force has a couple of problems:

• What if the user has different devices using different creds?
• What if some of the devices are blocked from the internet and the account password gets changed, i.e., the device internal secrets are only updated to some of the devices?

Btw, do you have a homeassistant integration fork using this PR for testing? I could experiment with that and the tapo plug to >>> get a better feeling about how it would work with homeassistant, at the moment I'm reading the creds from the envvars in that tapo draft PR.

I do but it's not great right now as I was trying to store the credentials globally in a hacky way. I'm planning to update ti store the credentials in an OptionFlow config and then update all the configentries whenever one of the OptionsFlow changes. Unless you have a better suggestion. Once you've merged the credentials PR I'll update it work with the new classname and point you to it
I don't know if there's a nicer way to store the creds for the integration, but I hope so. Updating config entries by force has a couple of problems:

What if the user has different devices using different creds?
What if some of the devices are blocked from the internet and the account password gets changed, i.e., the device internal >secrets are only updated to some of the devices?

Have a look at PR sdb9696/core#1 for a working ha fork with this PR. I am using the options flow now as I think that is the best way to go and the user can configure whether they want to sync credentials across other config entries or not

Hi @rytilahti, looking forwards to getting this one closed out now. Two things:

• There is quite a long conversation on this PR, a lot of which is now addressed in the other PRs we finished. Let me know if you want me to create a fresh PR so we can have a smaller conversation
• The codeQL check for weak hashing due to the md5 hash is constantly failing on this PR. I think if you dismiss the alert in github it will stop failing the check and we won't have to configure CodeQL to exclude it or do any post processing

Thanks!

Hi @sdb9696 and thanks for your efforts to get this properly integrated, and opening the potential for future extensions! :-)

There is quite a long conversation on this PR, a lot of which is now addressed in the other PRs we finished. Let me know if you want me to create a fresh PR so we can have a smaller conversation

Your call, it probably makes sense to start with a clean PR at this point if it's not too complicated while still keeping the git history intact?

The codeQL check for weak hashing due to the md5 hash is constantly failing on this PR. I think if you dismiss the alert in github it will stop failing the check and we won't have to configure CodeQL to exclude it or do any post processing

I can definitely do that. And looks like it's working as expected with manual dismissal as it's green again.

edit: I just noticed your hass fork, I will find some time to give it a whirl during the weekend. Tbh, I'm starting to have second thoughts if it makes sense to complicate the code to support the corner-cases I mentioned.. I doubt there is a way to globally store the credentials in homeassistant, but at least the self.hass[DOMAIN] can be used for that so that user has to input them only once per restart.

Closing this PR as it was broken into multiple PRs and the conversation is now very long and relating other PRs (#488 & #507). PR #509 is opened to replace it.