Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 6b9eec8

Browse files
plamutplamut
authored
chore: transition the library to the new microgenerator (#158)
* chore: regenerate the library with microgenerator * Update setup.py for Python 3.6+ * Fix GAPIC types imports * Inject SERVICE_ADDRESS and _DEFAULT_SCOPES to generated classes * Cleanup types.py * Adjust determining message size to new protobufs * Adjust Message.publish_time property to new protobuf * Remove client_config argument to publisher client The argument is not supported anymore in the new generated client. * Remove subscriber client config This configuration is not anymore supported by the generated subscriber client. * Fix method patch in gapic instance method test * Fix transport-related test failures * Adjust calls to changed client.publish() signature * Remove obsolete replacement rules from synth.py * Fix streaming pull monkeypatch (no prefetch) * Update supported Python versions in README * Adjust system tests to new method signatures * Adjust method calls in samples to new client * Fix tests creating subscription with custom settings * Regenerate library with latest synthtool * Adjust IAM tests to new IAM method signatures * Fix docstring format in generated *_iam_policy() methods * Iterate directly over topic subscriptions response A recent update in the code generator made list topic subscriptions responses iterable again. * Copy default publish Retry to a class variable * Adjust retry deadline if publish with ordering key * Add suport for custom publish retry settings * Add UPGRADING guide to docs * Mention default retry settings in sample comments * Remove synth URL changes from templates * Adjust the ordering key sample to new code * Adjust client to changes in generated code * Regenerate the code to pick the latest changes * Bump dependencies to latest versions * Optimize creating and accesing pubsub messages Profiling shows that the speed of creating a new pubsub message and the speed of accessing the message's attributes significantly affects the throughput of publisher and subscriber. This commit makes everything faster by circumventing the wrapper class around the raw protobuf pubsub messages where possible. * Regenerate the code with the latest changes
1 parent 2631980 commit 6b9eec8

File tree

95 files changed

+21791
-13247
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

95 files changed

+21791
-13247
lines changed

google/cloud/pubsub_v1/gapic/__init__.py renamed to .github/snippet-bot.yml

File renamed without changes.

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ pip-log.txt
4646
# Built documentation
4747
docs/_build
4848
bigquery/docs/generated
49+
docs.metadata
4950

5051
# Virtual environment
5152
env/
@@ -57,4 +58,4 @@ system_tests/local_test_setup
5758

5859
# Make sure a generated file isn't accidentally committed.
5960
pylintrc
60-
pylintrc.test
61+
pylintrc.test

‎.kokoro/build.sh‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,10 @@ python3.6 -m pip uninstall --yes --quiet nox-automation
3636
python3.6 -m pip install --upgrade --quiet nox
3737
python3.6 -m nox --version
3838

39-
python3.6 -m nox
39+
# If NOX_SESSION is set, it only runs the specified session,
40+
# otherwise run all the sessions.
41+
if [[ -n "${NOX_SESSION:-}" ]]; then
42+
python3.6 -m nox -s "${NOX_SESSION:-}"
43+
else
44+
python3.6 -m nox
45+
fi

‎.kokoro/docker/docs/Dockerfile‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Copyright 2020 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
from ubuntu:20.04
16+
17+
ENV DEBIAN_FRONTEND noninteractive
18+
19+
# Ensure local Python is preferred over distribution Python.
20+
ENV PATH /usr/local/bin:$PATH
21+
22+
# Install dependencies.
23+
RUN apt-get update \
24+
&& apt-get install -y --no-install-recommends \
25+
apt-transport-https \
26+
build-essential \
27+
ca-certificates \
28+
curl \
29+
dirmngr \
30+
git \
31+
gpg-agent \
32+
graphviz \
33+
libbz2-dev \
34+
libdb5.3-dev \
35+
libexpat1-dev \
36+
libffi-dev \
37+
liblzma-dev \
38+
libreadline-dev \
39+
libsnappy-dev \
40+
libssl-dev \
41+
libsqlite3-dev \
42+
portaudio19-dev \
43+
redis-server \
44+
software-properties-common \
45+
ssh \
46+
sudo \
47+
tcl \
48+
tcl-dev \
49+
tk \
50+
tk-dev \
51+
uuid-dev \
52+
wget \
53+
zlib1g-dev \
54+
&& add-apt-repository universe \
55+
&& apt-get update \
56+
&& apt-get -y install jq \
57+
&& apt-get clean autoclean \
58+
&& apt-get autoremove -y \
59+
&& rm -rf /var/lib/apt/lists/* \
60+
&& rm -f /var/cache/apt/archives/*.deb
61+
62+
63+
COPY fetch_gpg_keys.sh /tmp
64+
# Install the desired versions of Python.
65+
RUN set -ex \
66+
&& export GNUPGHOME="$(mktemp -d)" \
67+
&& echo "disable-ipv6" >> "${GNUPGHOME}/dirmngr.conf" \
68+
&& /tmp/fetch_gpg_keys.sh \
69+
&& for PYTHON_VERSION in 3.7.8 3.8.5; do \
70+
wget --no-check-certificate -O python-${PYTHON_VERSION}.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \
71+
&& wget --no-check-certificate -O python-${PYTHON_VERSION}.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \
72+
&& gpg --batch --verify python-${PYTHON_VERSION}.tar.xz.asc python-${PYTHON_VERSION}.tar.xz \
73+
&& rm -r python-${PYTHON_VERSION}.tar.xz.asc \
74+
&& mkdir -p /usr/src/python-${PYTHON_VERSION} \
75+
&& tar -xJC /usr/src/python-${PYTHON_VERSION} --strip-components=1 -f python-${PYTHON_VERSION}.tar.xz \
76+
&& rm python-${PYTHON_VERSION}.tar.xz \
77+
&& cd /usr/src/python-${PYTHON_VERSION} \
78+
&& ./configure \
79+
--enable-shared \
80+
# This works only on Python 2.7 and throws a warning on every other
81+
# version, but seems otherwise harmless.
82+
--enable-unicode=ucs4 \
83+
--with-system-ffi \
84+
--without-ensurepip \
85+
&& make -j$(nproc) \
86+
&& make install \
87+
&& ldconfig \
88+
; done \
89+
&& rm -rf "${GNUPGHOME}" \
90+
&& rm -rf /usr/src/python* \
91+
&& rm -rf ~/.cache/
92+
93+
RUN wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \
94+
&& python3.7 /tmp/get-pip.py \
95+
&& python3.8 /tmp/get-pip.py \
96+
&& rm /tmp/get-pip.py
97+
98+
CMD ["python3.7"]

‎.kokoro/docker/docs/fetch_gpg_keys.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/bash
2+
# Copyright 2020 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# A script to fetch gpg keys with retry.
17+
# Avoid jinja parsing the file.
18+
#
19+
20+
function retry {
21+
if [[ "${#}" -le 1 ]]; then
22+
echo "Usage: ${0} retry_count commands.."
23+
exit 1
24+
fi
25+
local retries=${1}
26+
local command="${@:2}"
27+
until [[ "${retries}" -le 0 ]]; do
28+
$command && return 0
29+
if [[ $? -ne 0 ]]; then
30+
echo "command failed, retrying"
31+
((retries--))
32+
fi
33+
done
34+
return 1
35+
}
36+
37+
# 3.6.9, 3.7.5 (Ned Deily)
38+
retry 3 gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \
39+
0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D
40+
41+
# 3.8.0 (Łukasz Langa)
42+
retry 3 gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \
43+
E3FF2839C048B25C084DEBE9B26995E310250568
44+
45+
#

‎.kokoro/docs/common.cfg‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ action {
1111
gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
1212

1313
# Use the trampoline script to run in docker.
14-
build_file: "python-pubsub/.kokoro/trampoline.sh"
14+
build_file: "python-pubsub/.kokoro/trampoline_v2.sh"
1515

1616
# Configure the docker image for kokoro-trampoline.
1717
env_vars: {
1818
key: "TRAMPOLINE_IMAGE"
19-
value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
19+
value: "gcr.io/cloud-devrel-kokoro-resources/python-lib-docs"
2020
}
2121
env_vars: {
2222
key: "TRAMPOLINE_BUILD_FILE"
@@ -28,6 +28,23 @@ env_vars: {
2828
value: "docs-staging"
2929
}
3030

31+
env_vars: {
32+
key: "V2_STAGING_BUCKET"
33+
value: "docs-staging-v2-staging"
34+
}
35+
36+
# It will upload the docker image after successful builds.
37+
env_vars: {
38+
key: "TRAMPOLINE_IMAGE_UPLOAD"
39+
value: "true"
40+
}
41+
42+
# It will always build the docker image.
43+
env_vars: {
44+
key: "TRAMPOLINE_DOCKERFILE"
45+
value: ".kokoro/docker/docs/Dockerfile"
46+
}
47+
3148
# Fetch the token needed for reporting release status to GitHub
3249
before_action {
3350
fetch_keystore {

‎.kokoro/docs/docs-presubmit.cfg‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Format: //devtools/kokoro/config/proto/build.proto
2+
3+
env_vars: {
4+
key: "STAGING_BUCKET"
5+
value: "gcloud-python-test"
6+
}
7+
8+
env_vars: {
9+
key: "V2_STAGING_BUCKET"
10+
value: "gcloud-python-test"
11+
}
12+
13+
# We only upload the image in the main `docs` build.
14+
env_vars: {
15+
key: "TRAMPOLINE_IMAGE_UPLOAD"
16+
value: "false"
17+
}

‎.kokoro/publish-docs.sh‎

Lines changed: 24 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,26 +18,16 @@ set -eo pipefail
1818
# Disable buffering, so that the logs stream through.
1919
export PYTHONUNBUFFERED=1
2020

21-
cd github/python-pubsub
22-
23-
# Remove old nox
24-
python3.6 -m pip uninstall --yes --quiet nox-automation
21+
export PATH="${HOME}/.local/bin:${PATH}"
2522

2623
# Install nox
27-
python3.6 -m pip install --upgrade --quiet nox
28-
python3.6 -m nox --version
24+
python3 -m pip install --user --upgrade --quiet nox
25+
python3 -m nox --version
2926

3027
# build docs
3128
nox -s docs
3229

33-
python3 -m pip install gcp-docuploader
34-
35-
# install a json parser
36-
sudo apt-get update
37-
sudo apt-get -y install software-properties-common
38-
sudo add-apt-repository universe
39-
sudo apt-get update
40-
sudo apt-get -y install jq
30+
python3 -m pip install --user gcp-docuploader
4131

4232
# create metadata
4333
python3 -m docuploader create-metadata \
@@ -52,4 +42,23 @@ python3 -m docuploader create-metadata \
5242
cat docs.metadata
5343

5444
# upload docs
55-
python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket docs-staging
45+
python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket "${STAGING_BUCKET}"
46+
47+
48+
# docfx yaml files
49+
nox -s docfx
50+
51+
# create metadata.
52+
python3 -m docuploader create-metadata \
53+
--name=$(jq --raw-output '.name // empty' .repo-metadata.json) \
54+
--version=$(python3 setup.py --version) \
55+
--language=$(jq --raw-output '.language // empty' .repo-metadata.json) \
56+
--distribution-name=$(python3 setup.py --name) \
57+
--product-page=$(jq --raw-output '.product_documentation // empty' .repo-metadata.json) \
58+
--github-repository=$(jq --raw-output '.repo // empty' .repo-metadata.json) \
59+
--issue-tracker=$(jq --raw-output '.issue_tracker // empty' .repo-metadata.json)
60+
61+
cat docs.metadata
62+
63+
# upload docs
64+
python3 -m docuploader upload docs/_build/html/docfx_yaml --metadata-file docs.metadata --destination-prefix docfx --staging-bucket "${V2_STAGING_BUCKET}"

0 commit comments

Comments
 (0)