Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit cf35585

Browse files
Bill Prindhermes
Bill Prin
authored and
dhermes
committed
Add Async Background Thread transport
Refactors handlers into separate package Adds background threaded transport Adds fix to Batch commit to properly set log name
1 parent 943b273 commit cf35585

20 files changed

+768
-47
lines changed

‎docs/index.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,9 @@
112112
logging-metric
113113
logging-sink
114114
logging-handlers
115+
logging-transports-sync
116+
logging-transports-thread
117+
logging-transports-base
115118

116119
.. toctree::
117120
:maxdepth: 0

‎docs/logging-handlers.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
Python Logging Module Handler
22
==============================
33

4-
.. automodule:: gcloud.logging.handlers
4+
.. automodule:: gcloud.logging.handlers.handlers
55
:members:
66
:show-inheritance:
77

‎docs/logging-transports-base.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Python Logging Handler Sync Transport
2+
======================================
3+
4+
.. automodule:: gcloud.logging.handlers.transports.base
5+
:members:
6+
:show-inheritance:
7+

‎docs/logging-transports-sync.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Python Logging Handler Sync Transport
2+
======================================
3+
4+
.. automodule:: gcloud.logging.handlers.transports.sync
5+
:members:
6+
:show-inheritance:
7+

‎docs/logging-transports-thread.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
Python Logging Handler Threaded Transport
2+
=========================================
3+
4+
5+
.. automodule:: gcloud.logging.handlers.transports.background_thread
6+
:members:
7+
:show-inheritance:
8+
9+

‎docs/logging-usage.rst‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -402,12 +402,21 @@ Logging client.
402402
>>> cloud_logger = logging.getLogger('cloudLogger')
403403
>>> cloud_logger.setLevel(logging.INFO) # defaults to WARN
404404
>>> cloud_logger.addHandler(handler)
405-
>>> cloud_logger.error('bad news') # API call
405+
>>> cloud_logger.error('bad news')
406406

407407
.. note::
408408

409-
This handler currently only supports a synchronous API call, which means each logging statement
410-
that uses this handler will require an API call.
409+
This handler by default uses an asynchronous transport that sends log entries on a background
410+
thread. However, the API call will still be made in the same process. For other transport
411+
options, see the transports section.
412+
413+
All logs will go to a single custom log, which defaults to "python". The name of the Python
414+
logger will be included in the structured log entry under the "python_logger" field. You can
415+
change it by providing a name to the handler:
416+
417+
.. doctest::
418+
419+
>>> handler = CloudLoggingHandler(client, name="mycustomlog")
411420

412421
It is also possible to attach the handler to the root Python logger, so that for example a plain
413422
`logging.warn` call would be sent to Cloud Logging, as well as any other loggers created. However,
@@ -424,4 +433,25 @@ this automatically:
424433
>>> handler = CloudLoggingHandler(client)
425434
>>> logging.getLogger().setLevel(logging.INFO) # defaults to WARN
426435
>>> setup_logging(handler)
427-
>>> logging.error('bad news') # API call
436+
>>> logging.error('bad news')
437+
438+
You can also exclude certain loggers:
439+
440+
.. doctest::
441+
442+
>>> setup_logging(handler, excluded_loggers=('werkzeug',)))
443+
444+
445+
446+
Python logging handler transports
447+
==================================
448+
449+
The Python logging handler can use different transports. The default is
450+
:class:`gcloud.logging.handlers.BackgroundThreadTransport`.
451+
452+
1. :class:`gcloud.logging.handlers.BackgroundThreadTransport` this is the default. It writes
453+
entries on a background :class:`python.threading.Thread`.
454+
455+
1. :class:`gcloud.logging.handlers.SyncTransport` this handler does a direct API call on each
456+
logging statement to write the entry.
457+

‎gcloud/logging/handlers/__init__.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Copyright 2016 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Python :mod:`logging` handlers for Google Cloud Logging."""
16+
17+
from gcloud.logging.handlers.handlers import CloudLoggingHandler, setup_logging

gcloud/logging/handlers.py renamed to gcloud/logging/handlers/handlers.py

Lines changed: 25 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,15 @@
1616

1717
import logging
1818

19+
from gcloud.logging.handlers.transports import BackgroundThreadTransport
20+
1921
EXCLUDE_LOGGER_DEFAULTS = (
2022
'gcloud',
21-
'oauth2client.client'
23+
'oauth2client'
2224
)
2325

26+
DEFAULT_LOGGER_NAME = "python"
27+
2428

2529
class CloudLoggingHandler(logging.StreamHandler, object):
2630
"""Python standard logging handler to log messages to the Google Cloud
@@ -36,6 +40,17 @@ class CloudLoggingHandler(logging.StreamHandler, object):
3640
:type client: :class:`gcloud.logging.client`
3741
:param client: the authenticated gcloud logging client for this handler
3842
to use
43+
:type name: str
44+
:param name: the name of the custom log in Stackdriver Logging. Defaults
45+
to "python". The name of the Python logger will be represented
46+
in the "python_logger" field.
47+
48+
:type transport: :class:`gcloud.logging.handlers.transports.Transport`
49+
:param transport: the class object to instantiate. It should extend from
50+
the base Transport type and implement
51+
:meth`gcloud.logging.handlers.transports.base.Transport.send`
52+
Defaults to BackgroundThreadTransport. The other
53+
option is SyncTransport.
3954
4055
Example:
4156
@@ -55,9 +70,13 @@ class CloudLoggingHandler(logging.StreamHandler, object):
5570
5671
"""
5772

58-
def __init__(self, client):
73+
def __init__(self, client,
74+
name=DEFAULT_LOGGER_NAME,
75+
transport=BackgroundThreadTransport):
5976
super(CloudLoggingHandler, self).__init__()
77+
self.name = name
6078
self.client = client
79+
self.transport = transport(client, name)
6180

6281
def emit(self, record):
6382
"""
@@ -66,13 +85,11 @@ def emit(self, record):
6685
See: https://docs.python.org/2/library/logging.html#handler-objects
6786
"""
6887
message = super(CloudLoggingHandler, self).format(record)
69-
logger = self.client.logger(record.name)
70-
logger.log_struct({"message": message},
71-
severity=record.levelname)
88+
self.transport.send(record, message)
7289

7390

7491
def setup_logging(handler, excluded_loggers=EXCLUDE_LOGGER_DEFAULTS):
75-
"""Helper function to attach the CloudLoggingAPI handler to the Python
92+
"""Helper function to attach the CloudLogging handler to the Python
7693
root logger, while excluding loggers this library itself uses to avoid
7794
infinite recursion
7895
@@ -90,11 +107,11 @@ def setup_logging(handler, excluded_loggers=EXCLUDE_LOGGER_DEFAULTS):
90107
91108
import logging
92109
import gcloud.logging
93-
from gcloud.logging.handlers import CloudLoggingAPIHandler
110+
from gcloud.logging.handlers import CloudLoggingHandler
94111
95112
client = gcloud.logging.Client()
96113
handler = CloudLoggingHandler(client)
97-
setup_logging(handler)
114+
gcloud.logging.setup_logging(handler)
98115
logging.getLogger().setLevel(logging.DEBUG)
99116
100117
logging.error("bad news") # API call

gcloud/logging/test_handlers.py renamed to gcloud/logging/handlers/test_handlers.py

Lines changed: 15 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
#!/usr/bin/env python
21
# Copyright 2016 Google Inc. All Rights Reserved.
32
#
43
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -23,32 +22,32 @@ class TestCloudLoggingHandler(unittest2.TestCase):
2322
PROJECT = 'PROJECT'
2423

2524
def _getTargetClass(self):
26-
from gcloud.logging.handlers import CloudLoggingHandler
25+
from gcloud.logging.handlers.handlers import CloudLoggingHandler
2726
return CloudLoggingHandler
2827

2928
def _makeOne(self, *args, **kw):
3029
return self._getTargetClass()(*args, **kw)
3130

3231
def test_ctor(self):
3332
client = _Client(self.PROJECT)
34-
handler = self._makeOne(client)
33+
handler = self._makeOne(client, transport=_Transport)
3534
self.assertEqual(handler.client, client)
3635

3736
def test_emit(self):
3837
client = _Client(self.PROJECT)
39-
handler = self._makeOne(client)
38+
handler = self._makeOne(client, transport=_Transport)
4039
LOGNAME = 'loggername'
4140
MESSAGE = 'hello world'
4241
record = _Record(LOGNAME, logging.INFO, MESSAGE)
4342
handler.emit(record)
44-
self.assertEqual(client.logger(LOGNAME).log_struct_called_with,
45-
({'message': MESSAGE}, logging.INFO))
43+
44+
self.assertEqual(handler.transport.send_called_with, (record, MESSAGE))
4645

4746

4847
class TestSetupLogging(unittest2.TestCase):
4948

5049
def _callFUT(self, handler, excludes=None):
51-
from gcloud.logging.handlers import setup_logging
50+
from gcloud.logging.handlers.handlers import setup_logging
5251
if excludes:
5352
return setup_logging(handler, excluded_loggers=excludes)
5453
else:
@@ -95,20 +94,10 @@ def release(self):
9594
pass # pragma: NO COVER
9695

9796

98-
class _Logger(object):
99-
100-
def log_struct(self, message, severity=None):
101-
self.log_struct_called_with = (message, severity)
102-
103-
10497
class _Client(object):
10598

10699
def __init__(self, project):
107100
self.project = project
108-
self.logger_ = _Logger()
109-
110-
def logger(self, _): # pylint: disable=unused-argument
111-
return self.logger_
112101

113102

114103
class _Record(object):
@@ -123,3 +112,12 @@ def __init__(self, name, level, message):
123112

124113
def getMessage(self):
125114
return self.message
115+
116+
117+
class _Transport(object):
118+
119+
def __init__(self, client, name):
120+
pass
121+
122+
def send(self, record, message):
123+
self.send_called_with = (record, message)

‎gcloud/logging/handlers/transports/__init__.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright 2016 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Transport classes for Python logging integration
16+
17+
Currently two options are provided, a synchronous transport that makes
18+
an API call for each log statement, and an asynchronous handler that
19+
sends the API using a :class:`gcloud.logging.Batch` object in the background.
20+
"""
21+
22+
from gcloud.logging.handlers.transports.base import Transport
23+
from gcloud.logging.handlers.transports.sync import SyncTransport
24+
from gcloud.logging.handlers.transports.background_thread import (
25+
BackgroundThreadTransport)

0 commit comments

Comments
 (0)