Refactoring: split up core.py #575

tringenbach · 2025-01-01T16:56:33Z

This PR refactors core.py by splitting it up into about a dozen smaller files. It puts it in a core/ directory, so existing code that imports from tinytuya.core should hopefully be unaffected.

I've been a software engineer for a while now. I'm less experienced in Python though compared to some other ecosystems, and I've never messed with this project before.

You didn't ask me to do this refactor, and I didn't even reach out first, so no hard feelings if you reject this outright. I understand that my idea of better organized code might not be yours.

My motivation is that I was looking into what it would take to add an asyncio mode. Splitting the code up into separate files allowed me to look at the resulting import lines to understand which pieces of the code directly do IO. (Though of course, making anything async has a viral affect.) If nothing else, this exercise helped me learn, or begin to learn, the codebase.

I used a combination of VSCode's "move to file" feature, manual copy/pasting, and cleaning up of the imports.

I ended up duplicating IS_PY2 and version into two files, but nothing else should be duplicated. I suspect version being used in XenonDevice is unintentional on your part.

If nothing else, please take a look at my comment # FIXME: check if we really meant the global version here or not in XenonDevice.py. I think that may be a real bug. I did not attempt to fix it, as I wanted this PR to have no functional or API changes.

In header.py, I used the syntax from . import command_types as CT, because I liked having the CT. prefix to group the constants. I was tempted to use that in the other places command_types and header were imported, but I couldn't find a way to get VSCode to do that for me, and didn't want to make a lot of manual changes. Besides, I don't know if you would like that or not.

Thanks!

This makes importing from tinytuya.core continue to work.

Looks like I no longer depend on a lot of the new files directly.

tinytuya/core/crypto_helper.py

jasonacox · 2025-01-01T20:46:34Z

Thanks @tringenbach ! I definitely welcome the help , code cleanup and prep for async support.

@uzlonewolf if you have time, would love your review (added).

jasonacox · 2025-01-01T21:57:29Z

tinytuya/core/XenonDevice.py

+# FIXME: The place where this is used looks like it might have meant `self.version`, but I'm not really sure yet
+#        this is copy/pasted instead of moved because it feels like a bug to me
+version_tuple = (1, 15, 1)
+version = __version__ = "%d.%d.%d" % version_tuple


This global should not be in this file. It refers to the library version and exists in core.py. The 'self.version' or local version parameters in the functions refer to the Tuya device versions.

@jasonacox

So later in that file (321 in my PR, 948 of core.py in master), there is:

# FIXME: check if we really meant the global `version` here or not self.version_str = "v" + str(version) self.version_bytes = str(version).encode('latin1') self.version_header = self.version_bytes + PROTOCOL_3x_HEADER self.payload_dict = None

Which I assume meant to say new_version or self.version (which are the same at that point), but accidentally uses the global version instead.

Actually, it might have been copied from set_version, where version is an argument.

jasonacox · 2025-01-01T22:00:31Z

While there should be no functional difference, due to the significant code changes, I propose we set this as v0.16.0.

We can remove python 2.7 support - that legacy code is not needed now that we have officially moved on to python 3 (Remove Python 2.7 Support #376)
On the "command_types" and "header" import shortcuts, I'm in favor of that (CT and H) but recommend we make it consistent.
I fixed the duplicate global scoped library version that was showing up in XenonDevice

Also use the `as H` / `as DT` aliases in more places.

tringenbach · 2025-01-02T00:29:33Z

While there should be no functional difference, due to the significant code changes, I propose we set this as v0.16.0.

We can remove python 2.7 support - that legacy code is not needed now that we have officially moved on to python 3 (Remove Python 2.7 Support #376)

On the "command_types" and "header" import shortcuts, I'm in favor of that (CT and H) but recommend we make it consistent.

I fixed the duplicate global scoped library version that was showing up in XenonDevice

I went ahead and changed version to self.version in that one place, and also initialized the three related variables to None (it was showing a warning about that in my IDE).

And I am now using the as DT and as H import aliases more consistently.

jasonacox · 2025-01-02T01:18:20Z

I went ahead and changed version to self.version in that one place, and also initialized the three related variables to None (it was showing a warning about that in my IDE).

That works!

And I am now using the as DT and as H import aliases more consistently.

Ugh, it just occurs to me that there there are examples that take advantage of these constants and are likley used by others who use this library. We will need to still support these:

import tinytuya

d = tinytuya.OutletDevice('DEVICEID', 'Auto', 'DEVICEKEY', persist=True)
payload = d.generate_payload(tinytuya.UPDATEDPS)
d.send(payload)

So it probalby makes more sense to just use something like

from .command_types import *

and use these constants as we have been.

Also, on line 30, the from . import scanner is not working and would need to be from .. import scanner since scanner didn't move into core.

tringenbach · 2025-01-02T14:36:39Z

And I am now using the as DT and as H import aliases more consistently.

Ugh, it just occurs to me that there there are examples that take advantage of these constants and are likley used by others who use this library. We will need to still support these:
import tinytuya

d = tinytuya.OutletDevice('DEVICEID', 'Auto', 'DEVICEKEY', persist=True)
payload = d.generate_payload(tinytuya.UPDATEDPS)
d.send(payload)
So it probalby makes more sense to just use something like
from .command_types import *
and use these constants as we have been.

In core/__init__.py I am doing

from .command_types import *
from .header import *

I see in __main__.py that pylint is unhappy with that:

But it still works, it prints out 18 and VSCode / pylance understands it in VSCode. So I am confused about that.

Also, on line 30, the from . import scanner is not working and would need to be from .. import scanner since scanner didn't move into core.

Ah, good catch. Is there a test or something that runs that part? I've mainly been running python -m tinytuya scan to make sure I didn't break anything, so I was surprised I broke scanner related code. But I guess it doesn't go through that code path for the cli scanner.

uzlonewolf · 2025-01-02T16:23:22Z

Is there a test or something that runs that part?

Device Auto-IP (d = tinytuya.Device('...', address='Auto')) uses it to find the device IP. It looks like it's called in a few other places as well since the original core.py file has from . import scanner in 3 places.

jasonacox · 2025-01-02T16:31:21Z

In core/init.py
...

Perfect! I see that it does indeed work for the possible existing use case if we address those imports (once I make the scanner import change):

import tinytuya

d = tinytuya.OutletDevice(device_id, 'Auto', key, persist=True)
d.status()
payload = d.generate_payload(tinytuya.UPDATEDPS)
d.send(payload)
d.status()

I see in main.py that pylint is unhappy with that

I think we can stay with your approach (CT and H). I don't see any pylint warnings with that.

Is there a test or something that runs that part?
...
Device Auto-IP (d = tinytuya.Device('...', address='Auto')) uses it to find the device IP.

As @uzlonewolf says, the 'Auto' part initiates find_device() function which is where the from . import scanner was found. It works if we change it to ... This was placed in the function itself to avoid a circular reference if placed in the global scope. The better practice may be to break up scanner.py similar to what you did to core.py as it contains many different classes. How far do we want to go down this rabbit hole? 😁

I've mainly been running python -m tinytuya scan to make sure I didn't break anything

That's usually what I do as well, but it does not test all the module functions. The github workflow for pylint and test is also very basic and in need of improvement too.

Since I moved core.py into a subdirectory, it needs to import scanner from `..`. Since I added a subdirectory, pylint needs to be recursive. That also lints contrib, so I fixed a typo and disabled use-before-assign a logging line.

tringenbach · 2025-01-04T18:33:59Z

Is there a test or something that runs that part?
...
Device Auto-IP (d = tinytuya.Device('...', address='Auto')) uses it to find the device IP.

As @uzlonewolf says, the 'Auto' part initiates find_device() function which is where the from . import scanner was found. It works if we change it to ... This was placed in the function itself to avoid a circular reference if placed in the global scope. The better practice may be to break up scanner.py similar to what you did to core.py as it contains many different classes. How far do we want to go down this rabbit hole? 😁

I've mainly been running python -m tinytuya scan to make sure I didn't break anything

That's usually what I do as well, but it does not test all the module functions. The github workflow for pylint and test is also very basic and in need of improvement too.

I pushed up your suggested fix to the import scanner issue. I also noticed that the pylint command in the pipeline wasn't linting my new subdirectory, so I added the recursive option. That also made it lint contrib, so I fixed a typo in one file, and disabled a rule on a single line in another, so that it passes.

I couldn't get the tests to work locally though.

.github/workflows/test.yml says to run python -m test.py but that just gives me Usage info. Does it do that in github too, or are there env vars set or something to make it work?

I also tried running tests.py (with an "s"), but all of those failed for me. I dug into it a little bit, but I think they hadn't been run a while and that the mocking they do doesn't exactly match implementation anymore. First it was mad it was getting a keyword arg it didn't expect, and then when I added that to the mock function, it was blowing up because it was expecting a bytes but getting a tuple. (I think the tuple is a MessagePayload instance).

jasonacox · 2025-01-04T20:21:44Z

I couldn't get the tests to work locally though.

The pylint test?

.github/workflows/test.yml says to run python -m test.py but that just gives me Usage info. Does it do that in github too, or are there env vars set or something to make it work?

That's about right. We don't have good tests. This only verifies that the module can be imported. I don' t know if UDP packets would be allowed, but it would be ideal to have a Tuya device simulator and have the test run a scan, at the minimum, if not have a slate of critical function tests (e.g. get status, set values).

tringenbach · 2025-01-05T03:02:11Z

The pylint test?

pylint works fine locally. I'm running either pylint -E tinytuya/*.py on master (which I got from the .github/workflow/pylint.yml, or now pylint --recursive y -E tinytuya on this branch.

I meant that python -m test gives me usage info and python -m tests was failing, and when I wrote that comment, I wasn't yet able to get them to pass.

Since then, as you've already seen, I got tests.py to pass and opened #576

jasonacox · 2025-01-05T03:20:22Z

tinytuya/Contrib/PresenceDetectorDevice.py

        """
        status = self.status()
        if "Error" in status:
-            return satus


jasonacox

Thanks for the great work on this @tringenbach . My tests look good and based on summary ❤️ feedback from @uzlonewolf, I'm comfortable merging this and releasing as 1.16.0.

jasonacox · 2025-01-06T06:47:31Z

Merge and released as https://pypi.org/project/tinytuya/1.16.0/

# upgrade library
pip install tinytuya --upgrade

# upgrade CLI 
pipx upgrade tinytuya

Docker for server: jasonacox/tinytuya:1.16.0p13

Please report any bugs/issues.

Thank @tringenbach ! 🙏

uzlonewolf · 2025-01-06T15:48:26Z

but it would be ideal to have a Tuya device simulator and have the test run a scan

Yet something else I intended to do but never found the time to implement T_T I did add the start of a simulator that currently only does v3.5 and status() a while back though https://github.com/jasonacox/tinytuya/blob/master/tools/fake-v35-device.py

tringenbach added 9 commits December 31, 2024 14:55

refactor some code into its own file

6b1eba0

more refactoring

2dfe7c1

revert unintended changes

aa54133

refactor: move XenonDevice and friends to own file

6cedd1f

refactor udp decryption to own file

ff4b2f2

refactor: move core.py and new files to core/ folder

b5d0b24

This makes importing from tinytuya.core continue to work.

clean up new imports in core.py

b46865c

Looks like I no longer depend on a lot of the new files directly.

refactor: move protocol headers and command types consts to own files

6115dd0

fix import

be63f9b

github-advanced-security bot found potential problems Jan 1, 2025

View reviewed changes

tinytuya/core/crypto_helper.py Dismissed Show dismissed Hide dismissed

tinytuya/core/crypto_helper.py Dismissed Show dismissed Hide dismissed

tinytuya/core/crypto_helper.py Dismissed Show dismissed Hide dismissed

jasonacox requested a review from uzlonewolf January 1, 2025 20:42

v1.16.0 prep

fece5c3

jasonacox reviewed Jan 1, 2025

View reviewed changes

Remove unused library version variables

7fa261b

fix: change version to self.version

3d47dc7

Also use the `as H` / `as DT` aliases in more places.

fix import of scanner, fix pylint

b901724

Since I moved core.py into a subdirectory, it needs to import scanner from `..`. Since I added a subdirectory, pylint needs to be recursive. That also lints contrib, so I fixed a typo and disabled use-before-assign a logging line.

jasonacox reviewed Jan 5, 2025

View reviewed changes

tinytuya/Contrib/PresenceDetectorDevice.py

"""

status = self.status()

if "Error" in status:

return satus

Copy link

Owner

jasonacox Jan 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good fix.

jasonacox approved these changes Jan 5, 2025

View reviewed changes

jasonacox removed the request for review from uzlonewolf January 6, 2025 05:24

jasonacox merged commit fc832ce into jasonacox:master Jan 6, 2025

3735943886 mentioned this pull request Sep 6, 2025

Async support #643

Closed

Refactoring: split up core.py #575

Refactoring: split up core.py #575

Uh oh!

Conversation

tringenbach commented Jan 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasonacox commented Jan 1, 2025

Uh oh!

jasonacox Jan 1, 2025

Choose a reason for hiding this comment

Uh oh!

tringenbach Jan 1, 2025

Choose a reason for hiding this comment

Uh oh!

jasonacox commented Jan 1, 2025

Uh oh!

tringenbach commented Jan 2, 2025

Uh oh!

jasonacox commented Jan 2, 2025

Uh oh!

tringenbach commented Jan 2, 2025

Uh oh!

uzlonewolf commented Jan 2, 2025

Uh oh!

jasonacox commented Jan 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tringenbach commented Jan 4, 2025

Uh oh!

jasonacox commented Jan 4, 2025

Uh oh!

tringenbach commented Jan 5, 2025

Uh oh!

jasonacox Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

jasonacox left a comment

Choose a reason for hiding this comment

Uh oh!

jasonacox commented Jan 6, 2025

Uh oh!

uzlonewolf commented Jan 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jasonacox commented Jan 2, 2025 •

edited

Loading