Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Context local worker_ctx#438

Closed
defat wants to merge 8 commits into
nameko:masterfrom
fraglab:master
Closed

Context local worker_ctx#438
defat wants to merge 8 commits into
nameko:masterfrom
fraglab:master

Conversation

@defat

@defat defat commented May 17, 2017

Copy link
Copy Markdown

global.worker_ctx context local variable added (request in flask style).
It is pushed in entrypoints and in standalone proxy creation.
It can be used in the scope of request handling e.g. in log formatters for logging call_ids, etc.
We write logs in json, tag them with call_id_stack[0] and store them in elasticsearch. With such approach we are able to retrieve logs from all microservices for specified request.

@mattbennett

Copy link
Copy Markdown
Member

Nameko makes a conscious choice to use dependency injection rather than global or thread-local variables. Dependency injection is chosen because it makes testing easier, and many of the bundled test helpers rely on this pattern.

The "nameko way" of achieving your use-case would be to define a DependencyProvider that performed the logging. For an example, see this mailing list thread.

It's also viable to expose different parts of the worker_ctx directly to workers if you need to (as in contextdata.py), but it should always be done via a DependencyProvider.

@defat

defat commented May 18, 2017

Copy link
Copy Markdown
Author

Implementing this in a DependencyProvider instead of global variable seems extremely reasonable for me.
However there is only one way to append correlation_id or call_id to every log record including logs from other libraries - to implement a filter object (not formatter as I mentioned earlier, sorry for this). Implementation should be smth like:

class CorrelationIdFilter(logging.Filter):
    def filter(self, record):
        if global_worker_ctx:
            record.correlation_id = global_worker_ctx.call_id_ctack[0]
        return True

With logging cfg smth like:

LOGGING:
  ...
  formatters:
    ....
      format: '(asctime)(msecs)(levelname)(process)(module)(lineno)(args)(msg)(correlation_id)'
      class: pythonjsonlogger.jsonlogger.JsonFormatter
  filters:
    request_id_filter:
      (): my_module.CorrelationIdFilter
...

Such approach gives ability to tag and retrieve all log records made for particular request even made by other libraries.
Looks like implementing DependencyProvider storing needed information from worker_ctx in a global variable that will be used in log filter is the right way.

@mattbennett

Copy link
Copy Markdown
Member

I understand the desire now. In this case, I would implement a DependencyProvider that pulled the current call_id out and stashed it in a thread-local variable during worker_setup. Your logging filter could then just read the call id from there, and there'd be no need for internal changes to Nameko.

Also, note we use a "correlation_id" as an internal part of the RPC protocol. You might want to change your naming convention to reflect that what you're capturing is really the Nameko call_id.

@defat

defat commented May 18, 2017

Copy link
Copy Markdown
Author

Yeah, I've used correlation_id just for example.
Thanks for so fast feedback and recommendations!
Pull request should be closed.

@defat defat closed this May 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants