Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Core: Add type hints to aws/core.py #12617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -164,15 +164,16 @@ jobs:
steps:
- checkout
- restore_cache:
key: python-requirements-{{ checksum "requirements-dev.txt" }}
key: python-requirements-{{ checksum "requirements-typehint.txt" }}
- run:
name: Setup environment
command: |
make install-dev-types
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@k-a-il @silv-io Now that the CircleCI pipeline is being migrated to GitHub Actions I think this would mean that we also have to change the make target here, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, would be great to add this in this PR.

On the other hand I think we should also be cautious with this switch because the typehint dependencies take waaaay longer to install in my experience (except when using uv :D)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, is this a blocker then? What does "waaaay longer" mean here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@silv-io Could you please give some guidance here?

  • What does it mean to update the other action?
  • Where do we need to adjust the dependency installation for the type checking?
  • How can we properly verify that this PR covers this aspect, since it's already fully green?
  • What are the timing implications of this change you mentioned and what is acceptable for this PR to be merged?

Copy link
Member

@silv-io silv-io May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes :)

  • I'll add a suggestion in my review
  • After doing the suggested changes, the AWS / Build, Test, Push pipeline will run on this PR and we'll see if it works there as well. (We've still got some unrelated test failures around the Docker SDK there. These will not be blocking this PR)
  • I've checked the CircleCI Pipeline and it looks like it will add 30s of installation time. Seems like caching works a bit better here than on-device, so the impact is not as big. The trade-off is fine in this case.

make install
mkdir -p target/reports
mkdir -p target/coverage
- save_cache:
key: python-requirements-{{ checksum "requirements-dev.txt" }}
key: python-requirements-{{ checksum "requirements-typehint.txt" }}
paths:
- "~/.cache/pip"
- persist_to_workspace:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ runs:
with:
python-version-file: '.python-version'
cache: 'pip'
cache-dependency-path: 'requirements-dev.txt'
cache-dependency-path: 'requirements-typehint.txt'

- name: Install docker helper dependencies
shell: bash
Expand Down
4 changes: 2 additions & 2 deletions .github/actions/setup-tests-env/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ runs:
with:
python-version-file: '.python-version'
cache: 'pip'
cache-dependency-path: 'requirements-dev.txt'
cache-dependency-path: 'requirements-typehint.txt'

- name: Install Community Dependencies
shell: bash
run: make install-dev
run: make install-dev-types

- name: Setup environment
shell: bash
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/aws-main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ jobs:
fetch-depth: 0

- name: Load Localstack ${{ env.PLATFORM_NAME_AMD64 }} Docker Image
uses: localstack/localstack/.github/actions/load-localstack-docker-from-artifacts@master
uses: ./.github/actions/load-localstack-docker-from-artifacts
with:
platform: ${{ env.PLATFORM_NAME_AMD64 }}

Expand Down Expand Up @@ -213,7 +213,7 @@ jobs:
TARGET_IMAGE_NAME="public.ecr.aws/localstack/localstack" ./bin/docker-helper.sh push

- name: Load Localstack ${{ env.PLATFORM_NAME_ARM64 }} Docker Image
uses: localstack/localstack/.github/actions/load-localstack-docker-from-artifacts@master
uses: ./.github/actions/load-localstack-docker-from-artifacts
with:
platform: ${{ env.PLATFORM_NAME_ARM64 }}

Expand Down
20 changes: 10 additions & 10 deletions .github/workflows/aws-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ jobs:
fetch-depth: 0

- name: Prepare Local Test Environment
uses: localstack/localstack/.github/actions/setup-tests-env@master
uses: ./.github/actions/setup-tests-env

- name: Linting
run: make lint
Expand Down Expand Up @@ -309,7 +309,7 @@ jobs:
tests/aws/services/lambda_/functions/common

- name: Load Localstack Docker Image
uses: localstack/localstack/.github/actions/load-localstack-docker-from-artifacts@master
uses: ./.github/actions/load-localstack-docker-from-artifacts
with:
platform: "${{ env.PLATFORM }}"

Expand Down Expand Up @@ -375,10 +375,10 @@ jobs:
fetch-depth: 0

- name: Prepare Local Test Environment
uses: localstack/localstack/.github/actions/setup-tests-env@master
uses: ./.github/actions/setup-tests-env

- name: Load Localstack Docker Image
uses: localstack/localstack/.github/actions/load-localstack-docker-from-artifacts@master
uses: ./.github/actions/load-localstack-docker-from-artifacts
with:
platform: "${{ env.PLATFORM }}"

Expand Down Expand Up @@ -449,7 +449,7 @@ jobs:
fetch-depth: 0

- name: Load Localstack Docker Image
uses: localstack/localstack/.github/actions/load-localstack-docker-from-artifacts@master
uses: ./.github/actions/load-localstack-docker-from-artifacts
with:
platform: "${{ env.PLATFORM }}"

Expand Down Expand Up @@ -496,7 +496,7 @@ jobs:
uses: actions/checkout@v4

- name: Prepare Local Test Environment
uses: localstack/localstack/.github/actions/setup-tests-env@master
uses: ./.github/actions/setup-tests-env

- name: Run Cloudwatch v1 Provider Tests
timeout-minutes: 30
Expand Down Expand Up @@ -539,7 +539,7 @@ jobs:
uses: actions/checkout@v4

- name: Prepare Local Test Environment
uses: localstack/localstack/.github/actions/setup-tests-env@master
uses: ./.github/actions/setup-tests-env

- name: Download Test Selection
if: ${{ env.TESTSELECTION_PYTEST_ARGS }}
Expand Down Expand Up @@ -588,7 +588,7 @@ jobs:
uses: actions/checkout@v4

- name: Prepare Local Test Environment
uses: localstack/localstack/.github/actions/setup-tests-env@master
uses: ./.github/actions/setup-tests-env

- name: Download Test Selection
if: ${{ env.TESTSELECTION_PYTEST_ARGS }}
Expand Down Expand Up @@ -639,7 +639,7 @@ jobs:
uses: actions/checkout@v4

- name: Prepare Local Test Environment
uses: localstack/localstack/.github/actions/setup-tests-env@master
uses: ./.github/actions/setup-tests-env

- name: Download Test Selection
if: ${{ env.TESTSELECTION_PYTEST_ARGS }}
Expand Down Expand Up @@ -691,7 +691,7 @@ jobs:
fetch-depth: 0

- name: Load Localstack Docker Image
uses: localstack/localstack/.github/actions/load-localstack-docker-from-artifacts@master
uses: ./.github/actions/load-localstack-docker-from-artifacts
with:
platform: "${{ env.PLATFORM }}"

Expand Down
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ repos:
hooks:
- id: mypy
entry: bash -c 'cd localstack-core && mypy --install-types --non-interactive'
additional_dependencies: ['botocore-stubs', 'rolo']

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
Expand Down
65 changes: 40 additions & 25 deletions localstack-core/localstack/aws/api/core.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
import functools
from typing import Any, NamedTuple, Optional, Protocol, Type, TypedDict, Union
from typing import (
Any,
Callable,
NamedTuple,
ParamSpec,
Protocol,
Type,
TypedDict,
TypeVar,
)

from botocore.model import OperationModel, ServiceModel
from rolo.gateway import RequestContext as RoloRequestContext
Expand All @@ -13,6 +22,10 @@ class ServiceRequest(TypedDict):
pass


P = ParamSpec("P")
T = TypeVar("T")


ServiceResponse = Any


Expand All @@ -28,7 +41,7 @@ class ServiceException(Exception):
sender_fault: bool
message: str

def __init__(self, *args, **kwargs):
def __init__(self, *args: Any, **kwargs: Any):
super(ServiceException, self).__init__(*args)

if len(args) >= 1:
Expand Down Expand Up @@ -72,38 +85,38 @@ class RequestContext(RoloRequestContext):
context, so it can be used for logging or modification before going to the serializer.
"""

request: Optional[Request]
request: Request
"""The underlying incoming HTTP request."""
service: Optional[ServiceModel]
service: ServiceModel | None
"""The botocore ServiceModel of the service the request is made to."""
operation: Optional[OperationModel]
operation: OperationModel | None
"""The botocore OperationModel of the AWS operation being invoked."""
region: Optional[str]
region: str
"""The region the request is made to."""
partition: str
"""The partition the request is made to."""
account_id: Optional[str]
account_id: str
"""The account the request is made from."""
request_id: Optional[str]
request_id: str | None
"""The autogenerated AWS request ID identifying the original request"""
service_request: Optional[ServiceRequest]
service_request: ServiceRequest | None
"""The AWS operation parameters."""
service_response: Optional[ServiceResponse]
service_response: ServiceResponse | None
"""The response from the AWS emulator backend."""
service_exception: Optional[ServiceException]
service_exception: ServiceException | None
"""The exception the AWS emulator backend may have raised."""
internal_request_params: Optional[InternalRequestParameters]
internal_request_params: InternalRequestParameters | None
"""Data sent by client-side LocalStack during internal calls."""
trace_context: dict
trace_context: dict[str, Any]
"""Tracing metadata such as X-Ray trace headers"""

def __init__(self, request=None) -> None:
def __init__(self, request: Request):
super().__init__(request)
self.service = None
self.operation = None
self.region = None
self.region = None # type: ignore[assignment] # type=str, because we know it will always be set downstream
self.partition = "aws" # Sensible default - will be overwritten by region-handler
self.account_id = None
self.account_id = None # type: ignore[assignment] # type=str, because we know it will always be set downstream
self.request_id = long_uid()
self.service_request = None
self.service_response = None
Expand All @@ -119,7 +132,7 @@ def is_internal_call(self) -> bool:
return self.internal_request_params is not None

@property
def service_operation(self) -> Optional[ServiceOperation]:
def service_operation(self) -> ServiceOperation | None:
"""
If both the service model and the operation model are set, this returns a tuple of the service name and
operation name.
Expand All @@ -130,7 +143,7 @@ def service_operation(self) -> Optional[ServiceOperation]:
return None
return ServiceOperation(self.service.service_name, self.operation.name)

def __repr__(self):
def __repr__(self) -> str:
return f"<RequestContext {self.service=}, {self.operation=}, {self.region=}, {self.account_id=}, {self.request=}>"


Expand All @@ -141,7 +154,7 @@ class ServiceRequestHandler(Protocol):

def __call__(
self, context: RequestContext, request: ServiceRequest
) -> Optional[Union[ServiceResponse, Response]]:
) -> ServiceResponse | Response | None:
"""
Handle the given request.

Expand All @@ -152,19 +165,21 @@ def __call__(
raise NotImplementedError


def handler(operation: str = None, context: bool = True, expand: bool = True):
def handler(
operation: str | None = None, context: bool = True, expand: bool = True
) -> Callable[[Callable[P, T]], Callable[P, T]]:
"""
Decorator that indicates that the given function is a handler
"""

def wrapper(fn):
def wrapper(fn: Callable[P, T]) -> Callable[P, T]:
@functools.wraps(fn)
def operation_marker(*args, **kwargs):
def operation_marker(*args: P.args, **kwargs: P.kwargs) -> T:
return fn(*args, **kwargs)

operation_marker.operation = operation
operation_marker.expand_parameters = expand
operation_marker.pass_context = context
operation_marker.operation = operation # type: ignore[attr-defined]
operation_marker.expand_parameters = expand # type: ignore[attr-defined]
operation_marker.pass_context = context # type: ignore[attr-defined]

return operation_marker

Expand Down
3 changes: 1 addition & 2 deletions localstack-core/localstack/aws/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -395,8 +395,7 @@ def __call__(
operation: OperationModel = request.operation_model

# create request
context = RequestContext()
context.request = create_http_request(request)
context = RequestContext(request=create_http_request(request))

# TODO: just a hacky thing to unblock the service model being set to `sqs-query` blocking for now
# this is using the same services as `localstack.aws.protocol.service_router.resolve_conflicts`, maybe
Expand Down
3 changes: 1 addition & 2 deletions localstack-core/localstack/aws/forwarder.py
Original file line number Diff line number Diff line change
Expand Up @@ -262,11 +262,10 @@ def create_aws_request_context(
)

aws_request: AWSPreparedRequest = client._endpoint.create_request(request_dict, operation)
context = RequestContext()
context = RequestContext(request=create_http_request(aws_request))
context.service = service
context.operation = operation
context.region = region
context.request = create_http_request(aws_request)
context.service_request = parameters

return context
9 changes: 5 additions & 4 deletions localstack-core/localstack/testing/aws/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,13 +108,14 @@ def create_client_with_keys(
def create_request_context(
service_name: str, operation_name: str, region: str, aws_request: AWSPreparedRequest
) -> RequestContext:
context = RequestContext()
if hasattr(aws_request.body, "read"):
aws_request.body = aws_request.body.read()
request = create_http_request(aws_request)

context = RequestContext(request=request)
context.service = load_service(service_name)
context.operation = context.service.operation_model(operation_name=operation_name)
context.region = region
if hasattr(aws_request.body, "read"):
aws_request.body = aws_request.body.read()
context.request = create_http_request(aws_request)
parser = create_parser(context.service)
_, instance = parser.parse(context.request)
context.service_request = instance
Expand Down
2 changes: 1 addition & 1 deletion localstack-core/mypy.ini
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[mypy]
explicit_package_bases = true
mypy_path=localstack-core
files=localstack/packages,localstack/services/kinesis/packages.py
files=localstack/aws/api/core.py,localstack/packages,localstack/services/kinesis/packages.py
ignore_missing_imports = False
follow_imports = silent
ignore_errors = False
Expand Down
3 changes: 1 addition & 2 deletions tests/aws/test_moto.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,11 +246,10 @@ def test_call_with_sns_with_full_uri():
headers={"Content-Type": "application/x-www-form-urlencoded; charset=utf-8"},
)
sns_service = load_service("sns")
context = RequestContext()
context = RequestContext(sns_request)
context.account = "test"
context.region = "us-west-1"
context.service = sns_service
context.request = sns_request
context.operation = sns_service.operation_model("CreateTopic")

create_topic_response = moto.call_moto(context)
Expand Down
4 changes: 2 additions & 2 deletions tests/integration/test_forwarder.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ def test_request_forwarder(_, __) -> ServiceResponse:

# invoke the function and expect the result from the fallback function
dispatcher = ForwardingFallbackDispatcher(test_provider, test_request_forwarder)
assert dispatcher["TestOperation"](RequestContext(), ServiceRequest()) == "fallback-result"
assert dispatcher["TestOperation"](RequestContext(None), ServiceRequest()) == "fallback-result"


def test_forwarding_fallback_dispatcher_avoid_fallback():
Expand All @@ -44,4 +44,4 @@ def test_request_forwarder(_, __) -> ServiceResponse:
# expect a NotImplementedError exception (and not the ServiceException from the fallthrough)
dispatcher = ForwardingFallbackDispatcher(test_provider, test_request_forwarder)
with pytest.raises(NotImplementedError):
dispatcher["TestOperation"](RequestContext(), ServiceRequest())
dispatcher["TestOperation"](RequestContext(None), ServiceRequest())
2 changes: 1 addition & 1 deletion tests/unit/aws/handlers/analytics.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def test_ignores_requests_without_service(self):
counter = ServiceRequestCounter(service_request_aggregator=aggregator)

chain = HandlerChain([counter])
chain.handle(RequestContext(), Response())
chain.handle(RequestContext(None), Response())

aggregator.start.assert_not_called()
aggregator.add_request.assert_not_called()
Expand Down
3 changes: 1 addition & 2 deletions tests/unit/aws/handlers/service.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,7 @@ def test_sets_exception_from_error_response(self, service_response_handler_chain
assert context.service_response is None

def test_nothing_set_does_nothing(self, service_response_handler_chain):
context = RequestContext()
context.request = Request("GET", "/_localstack/health")
context = RequestContext(request=Request("GET", "/_localstack/health"))

service_response_handler_chain.handle(context, Response("ok", 200))

Expand Down
Loading
Loading