Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Feb 5, 2025. It is now read-only.

Commit 7dc5d02

Browse files
rahulKQLrahulKQL
authored
chore: adds BigtableHBaseSettings with classic and veneer implementation (googleapis#2393)
* chore: adds BigtableHBaseSettings with classic and veneer implementation This change introduces `BigtableHBaseSettings`, which is then extended for specific implementation(i.e. classic and veneer): - `wrappers.BigtableHBaseSettings`: abstract layer for settings. - `wrappers.classic.BigtableHBaseClassicSettings` - `wrappers.veneer.BigtableHBaseVeneerSettings` Along with those, It includes getters for `BigtablOptions.Builder` and exposes `BigtableExtendedConfiguration` as internal public class for technical purpose. **Assumption:** I think we should keep the BigtableOptionsFactory(this contains configuration keys) in its current state, reason being it is exists in `bigtable-hbase` module and publicly exposed. we would be deprecating & removing the `BigtableOptionsFactory#fromConfiguration()` utility in the future. updated buildCredentials to preserve user provided credentials * chore: addressing feedback comments - Updated RetryCode logic to respect USE_TIMEOUT_KEY - updated the sequence of operatoin - Temporary dependencies included. - code comment for configuration. - merged plaintextConfigure with configureConnection() - Cannot replace buildIdempotentRetrySettings() - Renaming the methods - Moved the configureEmulator() * chore: address feedback comments - Now using BigtableHBaseVersion for user agent - applied check for maxRetryDelay * chore: add getBatchingMaxRequestSize getter This getter is intended to use for HBase's `BufferMutator#getWriteBufferSize`. * added new configuration keys for veneer settings - ENABLE_GRPC_RETRY_DEADLINEEXCEEDED_KEY - if this is explicitly set to false, we should remove it from the retry codes in idempotent settings. If its explicitly set to true , then we should explicitly add DEADLINE_EXCEEDED to the retry codes in idempotent settings - BIGTABLE_USE_BATCH - - useCachedPool is set to true. - data endpoint is set to "batch-bigtable.googleapis.com" - if InitialBackoffMs is not explicitly updated then set it to 5 seconds. - If MaxElapsedBackoff is not explicitly set, then set it to 5 mins. - BIGTABLE_BUFFERED_MUTATOR_ENABLE_THROTTLING throwing an unsupported exception - BIGTABLE_LONG_RPC_TIMEOUT_MS_KEY - fallback to this if BIGTABLE_READ_RPC_TIMEOUT_MS_KEY is unset - BIGTABLE_MUTATE_RPC_TIMEOUT_MS_KEY - set's mutateRowSettings's totalTiemout, fallbacks to BIGTABLE_LONG_RPC_TIMEOUT_MS_KEY. * addressing minor feedback comments - moved BIGTABLE_BUFFERED_MUTATOR_ENABLE_THROTTLING check in configureBulkMutation for grouping purpose. - moved up initialRetryDelay assignment.
1 parent 8da640b commit 7dc5d02

File tree

10 files changed

+1822
-8
lines changed

10 files changed

+1822
-8
lines changed

‎bigtable-client-core-parent/bigtable-client-core/src/main/java/com/google/cloud/bigtable/config/BigtableOptions.java‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,13 @@ public Builder() {
153153
options.useCachedDataPool = false;
154154
options.useGCJClient = false;
155155

156+
options.bulkOptions =
157+
BulkOptions.builder()
158+
.setMaxInflightRpcs(
159+
BulkOptions.BIGTABLE_MAX_INFLIGHT_RPCS_PER_CHANNEL_DEFAULT
160+
* options.dataChannelCount)
161+
.build();
162+
156163
options.retryOptions = new RetryOptions.Builder().build();
157164
options.callOptionsConfig = CallOptionsConfig.builder().build();
158165
// CredentialOptions.defaultCredentials() gets credentials from well known locations, such as
@@ -230,11 +237,19 @@ public Builder setRetryOptions(RetryOptions retryOptions) {
230237
return this;
231238
}
232239

240+
public RetryOptions getRetryOptions() {
241+
return options.retryOptions;
242+
}
243+
233244
public Builder setBulkOptions(BulkOptions bulkOptions) {
234245
options.bulkOptions = bulkOptions;
235246
return this;
236247
}
237248

249+
public BulkOptions getBulkOptions() {
250+
return options.bulkOptions;
251+
}
252+
238253
public Builder setUsePlaintextNegotiation(boolean usePlaintextNegotiation) {
239254
options.usePlaintextNegotiation = usePlaintextNegotiation;
240255
return this;
@@ -258,6 +273,10 @@ public Builder setCallOptionsConfig(CallOptionsConfig callOptionsConfig) {
258273
return this;
259274
}
260275

276+
public CallOptionsConfig getCallOptionsConfig() {
277+
return options.callOptionsConfig;
278+
}
279+
261280
public Builder setUseBatch(boolean useBatch) {
262281
options.useBatch = useBatch;
263282
return this;
@@ -318,11 +337,8 @@ public Builder enableEmulator(String host, int port) {
318337
}
319338

320339
public BigtableOptions build() {
321-
if (options.bulkOptions == null) {
322-
int maxInflightRpcs =
323-
BulkOptions.BIGTABLE_MAX_INFLIGHT_RPCS_PER_CHANNEL_DEFAULT * options.dataChannelCount;
324-
options.bulkOptions = BulkOptions.builder().setMaxInflightRpcs(maxInflightRpcs).build();
325-
} else if (options.bulkOptions.getMaxInflightRpcs() <= 0) {
340+
Preconditions.checkNotNull(options.bulkOptions, "bulk options cannot be null");
341+
if (options.bulkOptions.getMaxInflightRpcs() <= 0) {
326342
int maxInflightRpcs =
327343
BulkOptions.BIGTABLE_MAX_INFLIGHT_RPCS_PER_CHANNEL_DEFAULT * options.dataChannelCount;
328344
options.bulkOptions =

‎bigtable-client-core-parent/bigtable-client-core/src/test/java/com/google/cloud/bigtable/config/TestBigtableOptions.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,20 @@ public void testUseBatch_withOverrides() {
132132
Assert.assertEquals(300_000, options.getRetryOptions().getMaxElapsedBackoffMillis());
133133
}
134134

135+
@Test
136+
public void testBuilder_getters() {
137+
BigtableOptions.Builder optionsBuilder = BigtableOptions.builder();
138+
Assert.assertEquals(
139+
BulkOptions.builder()
140+
.setMaxInflightRpcs(
141+
BulkOptions.BIGTABLE_MAX_INFLIGHT_RPCS_PER_CHANNEL_DEFAULT
142+
* optionsBuilder.getDataChannelCount())
143+
.build(),
144+
optionsBuilder.getBulkOptions());
145+
Assert.assertEquals(CallOptionsConfig.builder().build(), optionsBuilder.getCallOptionsConfig());
146+
Assert.assertEquals(RetryOptions.builder().build(), optionsBuilder.getRetryOptions());
147+
}
148+
135149
/**
136150
* This is a dirty way to override the environment that is accessible to a test. It only modifies
137151
* the JVM's view of the environment, not the environment itself. From:

‎bigtable-client-core-parent/bigtable-hbase/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,20 @@ limitations under the License.
123123
<version>${commons-logging.version}</version>
124124
</dependency>
125125

126+
<!-- For veneer client settings configurations -->
127+
<dependency>
128+
<groupId>io.grpc</groupId>
129+
<artifactId>grpc-core</artifactId>
130+
</dependency>
131+
<dependency>
132+
<groupId>com.google.api</groupId>
133+
<artifactId>gax-grpc</artifactId>
134+
</dependency>
135+
<dependency>
136+
<groupId>com.google.auth</groupId>
137+
<artifactId>google-auth-library-oauth2-http</artifactId>
138+
</dependency>
139+
126140
<!-- TODO: remove this is direct dep, its only used for its Clock interface -->
127141
<dependency>
128142
<groupId>com.google.http-client</groupId>

‎bigtable-client-core-parent/bigtable-hbase/src/main/java/com/google/cloud/bigtable/hbase/BigtableExtendedConfiguration.java‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,17 +15,19 @@
1515
*/
1616
package com.google.cloud.bigtable.hbase;
1717

18-
import com.google.api.core.InternalExtensionOnly;
18+
import com.google.api.core.InternalApi;
1919
import com.google.auth.Credentials;
2020
import org.apache.hadoop.conf.Configuration;
2121

2222
/**
2323
* Allows users to set an explicit {@link Credentials} object.
2424
*
25+
* <p>For internal use only - public for technical reasons.
26+
*
2527
* @see BigtableConfiguration#withCredentials(Configuration, Credentials).
2628
*/
27-
@InternalExtensionOnly
28-
class BigtableExtendedConfiguration extends Configuration {
29+
@InternalApi("For internal usage only")
30+
public class BigtableExtendedConfiguration extends Configuration {
2931
private Credentials credentials;
3032

3133
BigtableExtendedConfiguration(Configuration conf, Credentials credentials) {

‎bigtable-client-core-parent/bigtable-hbase/src/main/java/com/google/cloud/bigtable/hbase/wrappers/BigtableHBaseSettings.java‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
/*
2+
* Copyright 2020 Google LLC.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
package com.google.cloud.bigtable.hbase.wrappers;
17+
18+
import static com.google.cloud.bigtable.hbase.BigtableOptionsFactory.BIGTABLE_USE_GCJ_CLIENT;
19+
import static com.google.cloud.bigtable.hbase.BigtableOptionsFactory.INSTANCE_ID_KEY;
20+
import static com.google.cloud.bigtable.hbase.BigtableOptionsFactory.PROJECT_ID_KEY;
21+
import static com.google.common.base.Strings.isNullOrEmpty;
22+
23+
import com.google.api.core.InternalApi;
24+
import com.google.cloud.bigtable.hbase.BigtableOptionsFactory;
25+
import com.google.cloud.bigtable.hbase.util.Logger;
26+
import com.google.cloud.bigtable.hbase.wrappers.classic.BigtableHBaseClassicSettings;
27+
import com.google.cloud.bigtable.hbase.wrappers.veneer.BigtableHBaseVeneerSettings;
28+
import com.google.common.base.Preconditions;
29+
import java.io.IOException;
30+
import org.apache.hadoop.conf.Configuration;
31+
32+
/** For internal use only - public for technical reasons. */
33+
@InternalApi("For internal usage only")
34+
public abstract class BigtableHBaseSettings {
35+
36+
protected static final Logger LOG = new Logger(BigtableOptionsFactory.class);
37+
38+
private final Configuration configuration;
39+
private final String projectId;
40+
private final String instanceId;
41+
42+
public static BigtableHBaseSettings create(Configuration configuration) throws IOException {
43+
if (configuration.getBoolean(BIGTABLE_USE_GCJ_CLIENT, false)) {
44+
return new BigtableHBaseVeneerSettings(configuration);
45+
} else {
46+
return new BigtableHBaseClassicSettings(configuration);
47+
}
48+
}
49+
50+
public BigtableHBaseSettings(Configuration configuration) {
51+
this.configuration = new Configuration(configuration);
52+
this.projectId = getRequiredValue(PROJECT_ID_KEY, "Project ID");
53+
this.instanceId = getRequiredValue(INSTANCE_ID_KEY, "Instance ID");
54+
}
55+
56+
public Configuration getConfiguration() {
57+
return configuration;
58+
}
59+
60+
public String getProjectId() {
61+
return projectId;
62+
}
63+
64+
public String getInstanceId() {
65+
return instanceId;
66+
}
67+
68+
public abstract String getDataHost();
69+
70+
public abstract String getAdminHost();
71+
72+
public abstract int getPort();
73+
74+
public abstract int getBulkMaxRowCount();
75+
76+
public abstract long getBatchingMaxRequestSize();
77+
78+
protected String getRequiredValue(String key, String displayName) {
79+
String value = configuration.get(key);
80+
Preconditions.checkArgument(
81+
!isNullOrEmpty(value), String.format("%s must be supplied via %s", displayName, key));
82+
return value;
83+
}
84+
}

0 commit comments

Comments
 (0)