Thanks to visit codestin.com
Credit goes to github.com

Skip to content

refactor: use functools.wrap in func_metadata decorator#347

Merged
eakmanrq merged 3 commits into
eakmanrq:mainfrom
themattmorris:metadata-signature
Mar 28, 2025
Merged

refactor: use functools.wrap in func_metadata decorator#347
eakmanrq merged 3 commits into
eakmanrq:mainfrom
themattmorris:metadata-signature

Conversation

@themattmorris

Copy link
Copy Markdown
Contributor

fixes #342

I didn't see where CALLING_CLASS was being used and assumed perhaps it was something that was previously used? I also removed import from __future__ import annotations because I didn't see where it was necessary in this module, but let me know if you disagree with either of these changes.

Thanks for considering this submission!

@themattmorris themattmorris requested a review from eakmanrq as a code owner March 24, 2025 15:58
@themattmorris

Copy link
Copy Markdown
Contributor Author

Well that's a new error.

I do see the line in pyspark that the error is referring to, and they have since updated this. But this is not new to pyspark, and version 3.5.5 has been able to be successfully installed in test runs as recent as 2 days prior to this one.

I believe this is occurring because of this update to setuptools that was released. I don't know how you want to handle that one, if setuptools will need to be pinned for the time being.

There is this PR out there for pyspark. I'm not sure how this will affect 3.5 versions, but perhaps waiting for that could fix the problem here.

@themattmorris

Copy link
Copy Markdown
Contributor Author

The pyspark fix has been made that was causing the installation failure. Would you be able to rerun the checks?

@eakmanrq

Copy link
Copy Markdown
Owner

@themattmorris seems like some issues that are specific to your branch

…func_metadata having narrower types in signature

Signed-off-by: themattmorris <[email protected]>
…w that their source code has changed since being wrapped

Signed-off-by: themattmorris <[email protected]>
@themattmorris

Copy link
Copy Markdown
Contributor Author

So the scope of changes in this PR certainly ended up being more than I expected, but that was a side effect of having the function signatures being updated for functions decorated with func_metadata.

Some of the changes were tests where mypy was complaining about types being passed to these functions since the signature no longer accepts anything. However, this change was due to the actual source code in the wrapped function being different. For functions ascii and current_schema, "invoke_anonymous_function" is being called, whereas this was not previously the case from the main branch:

import inspect

from sqlframe.standalone import functions as SF

print(inspect.getsource(SF.ascii))
        def wrapper(*args, **kwargs):
            funcs_to_not_auto_alias = [
                "posexplode",
                "explode_outer",
                "json_tuple",
                "posexplode_outer",
                "stack",
                "inline",
                "inline_outer",
                "window",
                "session_window",
                "window_time",
            ]

            result = func(*args, **kwargs)
            if (
                isinstance(result, Column)
                and isinstance(result.column_expression, exp.Func)
                and not isinstance(result.expression, exp.Alias)
                and func.__name__ not in funcs_to_not_auto_alias
            ):
                col_name = result.column_expression.find(exp.Identifier)
                if col_name:
                    col_name = col_name.name
                else:
                    col_name = result.column_expression.find(exp.Literal)
                    if col_name:
                        col_name = col_name.this
                alias_name = f"{func.__name__}__{col_name or ''}__"
                # BigQuery has restrictions on alias names so we constrain it to alphanumeric characters and underscores
                return result.alias(re.sub(r"\W", "_", alias_name))  # type: ignore
            return result

(clearly this is the source code of the inner function inside the decorator)

@eakmanrq eakmanrq left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @themattmorris!

@eakmanrq eakmanrq merged commit 223ee4a into eakmanrq:main Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrap signature of methods decorated with sqlframe.base.decorators.func_metadata

2 participants