Added support to connect and perform CRUD operations with couchbase #4

giriraj-singh-couchbase · 2025-09-10T12:53:12Z

This pull request introduces Couchbase Binary Protocol support to bRPC, enabling communication with Couchbase Server and Capella deployments. It adds a new protocol handler, request/response builders, comprehensive example clients, performance benchmarking tools, and detailed documentation. The changes are grouped below by theme:

Couchbase Binary Protocol Integration

Protocol Layer (src/brpc/policy/):

Added couchbase_protocol.h and couchbase_protocol.cpp implementing the complete binary protocol framing, parsing, with support for all standard Couchbase commands (GET, SET, ADD, REPLACE, DELETE, etc.)

High-Level API (src/brpc/):

Implemented CouchbaseOperations class in src/brpc/couchbase.h and src/brpc/couchbase.cpp (~3,500 lines), providing simplified APIs for:
- Authentication: authenticate() and authenticateSSL() for both non-SSL and SSL connections (required for Couchbase Capella)
- Bucket management: selectBucket() for bucket selection
- CRUD operations: get(), upsert(), add(), replace(), append(), prepend(), delete_() with Result struct for clean error handling
- Pipeline batching: beginPipeline(), pipelineRequest(), executePipeline(), clearPipeline() for batching multiple operations in a single network call.
Implemented CouchbaseRequest and CouchbaseResponse classes (inherit from NonreflectableMessage) providing low-level request construction and response parsing
Implemented CouchbaseManifestManager singleton for thread-safe collection manifest management with automatic caching and refresh

Example Clients

Basic Client (example/couchbase_c++/couchbase_client.cpp - 462 lines):

Demonstrates authentication (both SSL and non-SSL), bucket selection, basic CRUD operations, collection-scoped operations, and error handling patterns

Multithreaded Client (example/couchbase_c++/multithreaded_couchbase_client.cpp - 375 lines):

Demonstrates two threading patterns:
- Multiple threads sharing a single CouchbaseOperations instance (same bucket, shared connection)
- Multiple threads with dedicated CouchbaseOperations instances (different buckets, isolated connections)

Documentation

Comprehensive Guide (docs/en/couchbase_example.md):

Key Features

SSL/TLS Support: Built-in encryption for Couchbase Capella and secure on-premises deployments
Collection Management: Native Couchbase 7.0+ collections with automatic ID resolution, manifest caching, and scope support
Pipeline Batching: Batch multiple operations.
Thread Safety: Reader-writer locks for manifest caching, support for shared or isolated connection patterns
Connection Pooling: Intelligent connection sharing per server+bucket combination
Error Handling: Clean Result struct with success flag, error messages, status codes, and values

What problem does this PR solve?

Added native support to interact with Couchbase servers using the binary protocol, enabling efficient CRUD operations, SSL/TLS encrypted connections and collection management.

Side effects:

Performance effects: None
Breaking backward compatibility: NO

Check List:

Please make sure your changes are compilable.
When providing us with a new feature, it is best to add related tests.
Please follow Contributor Covenant Code of Conduct.

github-actions

license-eye has totally checked 1732 files.

Valid	Invalid	Ignored	Fixed
869	3	860	0

Click to see the invalid file list

example/couchbase_c++/couchbase_client.cpp
src/brpc/couchbase.h
src/brpc/policy/couchbase_protocol.h

github-actions · 2025-09-10T12:54:14Z

example/couchbase_c++/couchbase_client.cpp

@@ -0,0 +1,718 @@
+#include <gflags/gflags.h>


Suggested change

#include <gflags/gflags.h>

/*

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing,

* software distributed under the License is distributed on an

* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

* KIND, either express or implied. See the License for the

* specific language governing permissions and limitations

* under the License.

*/

#include <gflags/gflags.h>

github-actions · 2025-09-10T12:54:15Z

src/brpc/couchbase.h

@@ -0,0 +1,229 @@
+#ifndef BRPC_COUCHBASE_H


Suggested change

#ifndef BRPC_COUCHBASE_H

/*

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing,

* software distributed under the License is distributed on an

* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

* KIND, either express or implied. See the License for the

* specific language governing permissions and limitations

* under the License.

*/

#ifndef BRPC_COUCHBASE_H

github-actions · 2025-09-10T12:54:15Z

src/brpc/policy/couchbase_protocol.h

@@ -0,0 +1,167 @@
+


Suggested change

/*

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing,

* software distributed under the License is distributed on an

* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

* KIND, either express or implied. See the License for the

* specific language governing permissions and limitations

* under the License.

*/

ingenthr

Only a few comments on the docs are most important, but there are some questions around the scope/collection management. If you want to proceed with those as is and let the user manage it or improve it in the future, I'd be fine with that too.

docs/en/couchbase_example.md

example/couchbase_c++/couchbase_client.cpp

src/brpc/couchbase.h

example/couchbase_c++/couchbase_client.cpp

docs/en/couchbase_example.md

Merging from source

…ouchbaselabs/cb_brpc into couchbase_binary_protocol_brpc t :wq

…oogle format

giriraj-singh-couchbase · 2025-10-06T03:05:55Z

src/brpc/couchbase.h

+    // the status codes are from Couchbase Binary Protocol documentation,
+    // for original reference of status codes visit
+    // https://github.com/couchbase/kv_engine/blob/master/include/mcbp/protocol/status.h
+    enum Status {


error_status code has been referenced.

giriraj-singh-couchbase · 2025-10-06T05:39:28Z

src/brpc/couchbase.cpp

+
+// collectionID fetching either from the metadata cache or if doesn't exist then
+// fetch from the server.
+bool CouchbaseOperations::CouchbaseRequest::getCachedOrFetchCollectionId(


this method checks the global cache, and upon a miss the manifest is fetched from server using refreshCollectionManifest(). It shall be noted that upon a hit in global cache, local cache is also updated.

giriraj-singh-couchbase · 2025-10-06T05:45:38Z

src/brpc/couchbase.cpp

+
+  brpc::CouchbaseMetadataTracking::CollectionManifest manifest;
+  // check if the server/bucket exists in the cached collection manifest
+  if (!metadata_tracking->getBucketToCollectionManifest(server, selected_bucket,


probes server->bucket if a miss then the manifest for that bucket is fetched.

giriraj-singh-couchbase · 2025-10-06T05:46:42Z

src/brpc/couchbase.cpp

+                << selected_bucket << " on server " << server
+                << ", fetching from server");
+    // No cached manifest found, fetch from server
+    if (!refreshCollectionManifest(channel, server, selected_bucket)) {


refreshCollectionManifest for the server using which channel/connection is established, and the bucket which is currently selected.

giriraj-singh-couchbase · 2025-10-06T05:53:48Z

src/brpc/couchbase.cpp

+  return true;
+}
+
+bool CouchbaseOperations::CouchbaseRequest::refreshCollectionManifest(


this method refreshes the manifest for a particular sever/bucket and updates the caches.

giriraj-singh-couchbase · 2025-10-06T06:13:47Z

src/brpc/couchbase.cpp

+  return result;
+}
+
+CouchbaseOperations::Result CouchbaseOperations::selectBucket(


This method selects a bucket for the connection and can be used to select different buckets.
As a bucket is selected the collection manifest is fetched for that bucket.
In case of failure of fetching the collection manifest, it will be retried later when a collection operation is operated on this bucket.

Using connection_groups to differentiate between connections across CouchbaseOperations instances to different buckets. Renamed CollectionManifestTracker class to CollectionManifestManager and all the related functionality inside it as before refreshing method was outside this class Added two different authenticate method authenticate(not secure) and authenticateSSL(secure)

…ouchbaselabs/cb_brpc into couchbase_binary_protocol_brpc merging as the branch is not synced with the local-code base.

giriraj-singh-couchbase · 2025-10-16T06:24:32Z

src/brpc/policy/couchbase_protocol.cpp

-      msg->pi = pi;
-      return MakeMessage(msg);
-    } else {
+    // if (header->command == CB_BINARY_SASL_AUTH) {


fixed the logic where authentication throws an error during parsing itself if the credentials are wrong.

giriraj-singh-couchbase · 2025-10-16T06:25:19Z

src/brpc/couchbase.cpp


 #include <zlib.h>  //for crc32 Vbucket_id

+#define CB_ADD(a,b) (a+b)


this is not needed anywhere.

giriraj-singh-couchbase · 2025-10-16T06:26:29Z

src/brpc/couchbase.cpp

+
 // Debug flag for enabling debug statements
-static bool DBUG = true;  // Set to true to enable debug logs
+static bool DBUG = false;  // Set to true to enable debug logs


added a debug flag which can be set as true/false and is intended to be used for debugging purposes.

Added an example where a single instance is being shared across the threads when operating on single bucket.

updated the documentation on thread safe operations and fixed small small discrepancies.

ingenthr

Some small comments that you can use to improve this if you'd like, but otherwise offering approval. I'm sure you want to wrap this one up.

docs/en/couchbase_example.md

src/brpc/policy/couchbase_protocol.cpp

…ownload certificate

giriraj-singh-couchbase added 9 commits July 30, 2025 07:02

Implemented Couchbase binary protocol support

0c758fa

added support for single connection type for couchbase

aa034c7

removed unnecessary cout statements

9d81669

added protocol code for helo packet

7946cb4

fixed vbucketID code for identification, fixed add and get functions

d2ca0d4

Added test cases for threaded get and add functions

d1497c1

Added Error Handling code and made upsert and delete examples

aae6a3d

Merge branch 'apache:master' into couchbase_binary_protocol_brpc

95ba8c0

added makefile for example/couchbase_c++

2ff17ff

github-actions bot reviewed Sep 10, 2025

View reviewed changes

giriraj-singh-couchbase added 2 commits September 19, 2025 15:16

fixed bugs in couchbase header files

10964aa

Added License and formatted to google c++ norms

8bc4c59

giriraj-singh-couchbase changed the title ~~Couchbase binary protocol brpc~~ Added support to connect and perform CRUD operations with couchbase Sep 19, 2025

giriraj-singh-couchbase added 2 commits September 20, 2025 06:11

fixed bugs, added support for collections and added couchbase_client.md

b38bb2d

fixed license issue

07d1b0c

giriraj-singh-couchbase self-assigned this Sep 20, 2025

giriraj-singh-couchbase requested a review from ingenthr September 20, 2025 01:14

ingenthr requested changes Sep 23, 2025

View reviewed changes

giriraj-singh-couchbase added 12 commits September 30, 2025 14:38

added custom logic for caching collectionIDs

f8defb1

added caching of collection manifests

6a8200d

Added example code for multithreaded demonstration

db1afc1

updated CMake

753e662

Abstracted CRUD operations

4bdb6a4

Added pipeline/batching support

80713be

commented unused variables

e77e3cd

Merge pull request #5 from apache/master

792a004

Merging from source

Updated support for C++17

1ce8f28

Merge branch 'couchbase_binary_protocol_brpc' of https://github.com/c…

d229fc2

…ouchbaselabs/cb_brpc into couchbase_binary_protocol_brpc t :wq

fixed some issue.

f03e0f6

Using Mutex instead of shared lock to support c++11

68b4a51

giriraj-singh-couchbase added 3 commits October 5, 2025 22:02

Fixed bugs/repetitive calls to refreshing manifest on server

844d536

Formatted function/variable naming scheme and formatted code in c++ g…

fe51d10

…oogle format

removed unnecessary code

5a55549

giriraj-singh-couchbase commented Oct 6, 2025

View reviewed changes

giriraj-singh-couchbase added 2 commits October 6, 2025 11:27

updated comments

0d8b7b2

updated comments

4d4d181

giriraj-singh-couchbase commented Oct 6, 2025

View reviewed changes

giriraj-singh-couchbase added 9 commits October 6, 2025 12:05

updated documentation

80da22c

updated documentation

ec37f12

updated documentation

f469609

updated documentation

6988d33

Updated documentation

2912a63

Updated documentation

2e13a7e

Update documentation

592db28

Merge branch 'couchbase_binary_protocol_brpc' of https://github.com/c…

442a3c3

…ouchbaselabs/cb_brpc into couchbase_binary_protocol_brpc merging as the branch is not synced with the local-code base.

giriraj-singh-couchbase commented Oct 16, 2025

View reviewed changes

giriraj-singh-couchbase added 2 commits October 17, 2025 01:58

Updated multithreaded and single threaded code.

8cab3d1

Added an example where a single instance is being shared across the threads when operating on single bucket.

updated documentation

1639f5d

updated the documentation on thread safe operations and fixed small small discrepancies.

giriraj-singh-couchbase requested a review from ingenthr October 17, 2025 06:55

ingenthr approved these changes Nov 5, 2025

View reviewed changes

docs/en/couchbase_example.md Outdated Show resolved Hide resolved

src/brpc/policy/couchbase_protocol.cpp Outdated Show resolved Hide resolved

giriraj-singh-couchbase added 2 commits November 5, 2025 10:56

removed commented code and updated readme to have links for cluster d…

e4242b9

…ownload certificate

removed unused code.

90d5522

giriraj-singh-couchbase merged commit cf7ce31 into master Nov 5, 2025
16 checks passed

-#include <gflags/gflags.h>
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <gflags/gflags.h>


		#include <zlib.h> //for crc32 Vbucket_id

		#define CB_ADD(a,b) (a+b)

Added support to connect and perform CRUD operations with couchbase #4

Added support to connect and perform CRUD operations with couchbase #4

Uh oh!

Conversation

giriraj-singh-couchbase commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Couchbase Binary Protocol Integration

Example Clients

Documentation

Key Features

What problem does this PR solve?

Check List:

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

ingenthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giriraj-singh-couchbase Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ingenthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

giriraj-singh-couchbase commented Sep 10, 2025 •

edited

Loading

giriraj-singh-couchbase Oct 6, 2025 •

edited

Loading