Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ Choose the deployment mode that fits your needs.
- ✅ Edge deployments without network
- ✅ No server setup required

**Embedded Concurrent Mode (NEW in v0.4.8):**
- ✅ Web applications (Flask, FastAPI, Django)
- ✅ Multi-process workers (Gunicorn, uWSGI)
- ✅ Hot reloading development servers
- ✅ Multi-reader, single-writer architecture
- ✅ Lock-free reads (~100ns latency)

**Server Mode (gRPC):**
- ✅ Production deployments
- ✅ Multi-language teams (Python, Node.js, Go)
Expand All @@ -47,6 +54,59 @@ Choose the deployment mode that fits your needs.

---

## Concurrent Embedded Mode (v0.4.8+)

For web applications that need multiple processes to access the same database:

```python
from sochdb import Database

# Open in concurrent mode - multiple processes can access simultaneously
db = Database.open_concurrent("./app_data")

# Reads are lock-free and can run in parallel (~100ns)
value = db.get(b"user:123")

# Writes are automatically coordinated (~60µs amortized)
db.put(b"user:123", b'{"name": "Alice"}')

# Check if concurrent mode is active
print(f"Concurrent mode: {db.is_concurrent}") # True
```

### Flask Example

```python
from flask import Flask
from sochdb import Database

app = Flask(__name__)
db = Database.open_concurrent("./flask_db")

@app.route("/user/<user_id>")
def get_user(user_id):
# Multiple concurrent requests can read simultaneously
data = db.get(f"user:{user_id}".encode())
return data or "Not found"

@app.route("/user/<user_id>", methods=["POST"])
def update_user(user_id):
# Writes are serialized automatically
db.put(f"user:{user_id}".encode(), request.data)
return "OK"
```

### Performance

| Operation | Standard Mode | Concurrent Mode |
|-----------|---------------|-----------------|
| Read (single process) | ~100ns | ~100ns |
| Read (multi-process) | **Blocked** ❌ | ~100ns ✅ |
| Write | ~5ms (fsync) | ~60µs (amortized) |
| Max concurrent readers | 1 | 1024 |

---

## Installation

```bash
Expand Down
8 changes: 4 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"

[project]
name = "sochdb"
version = "0.4.7"
version = "0.4.8"
description = "SochDB is an AI-native database with token-optimized output, O(|path|) lookups, built-in vector search, and durable transactions."
readme = "README.md"
license = {text = "Apache-2.0"}
license = {text = "AGPL-3.0-or-later"}
authors = [
{name = "Sushanth", email = "[email protected]"}
{name = "Sushanth Reddy Vanagala", email = "[email protected]"}
]
maintainers = [
{name = "Sushanth", email = "[email protected]"}
Expand All @@ -18,7 +18,7 @@ keywords = ["database", "llm", "ai", "vector-search", "embedded", "key-value", "
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
Expand Down
116 changes: 114 additions & 2 deletions src/sochdb/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,21 @@ def _setup_bindings(cls):
lib.sochdb_open_with_config.argtypes = [ctypes.c_char_p, C_DatabaseConfig]
lib.sochdb_open_with_config.restype = ctypes.c_void_p

# sochdb_open_concurrent(path: *const c_char) -> *mut DatabasePtr
# Concurrent mode: multi-reader, single-writer for web apps
try:
lib.sochdb_open_concurrent.argtypes = [ctypes.c_char_p]
lib.sochdb_open_concurrent.restype = ctypes.c_void_p
except (AttributeError, OSError):
pass # Not available in older library versions

# sochdb_is_concurrent(ptr: *mut DatabasePtr) -> c_int
try:
lib.sochdb_is_concurrent.argtypes = [ctypes.c_void_p]
lib.sochdb_is_concurrent.restype = ctypes.c_int
except (AttributeError, OSError):
pass

# sochdb_close(ptr: *mut DatabasePtr)
lib.sochdb_close.argtypes = [ctypes.c_void_p]
lib.sochdb_close.restype = None
Expand Down Expand Up @@ -909,27 +924,124 @@ class Database:
Provides direct access to a SochDB database file.
This is the recommended mode for single-process applications.

For web applications or multi-process scenarios, use ``Database.open_concurrent()``
instead, which allows multiple processes to access the database simultaneously.

Example:
# Standard mode (single process)
db = Database.open("./my_database")
db.put(b"key", b"value")
value = db.get(b"key")
db.close()

# Concurrent mode (multiple processes, e.g., Flask/FastAPI)
db = Database.open_concurrent("./my_database")
# Multiple workers can now access the database

Or with context manager:
with Database.open("./my_database") as db:
db.put(b"key", b"value")
"""

def __init__(self, path: str, _handle):
def __init__(self, path: str, _handle, _is_concurrent: bool = False):
"""
Initialize a database connection.

Use Database.open() to create instances.
Use Database.open() or Database.open_concurrent() to create instances.
"""
self._path = path
self._handle = _handle
self._closed = False
self._lib = _FFI.get_lib()
self._is_concurrent = _is_concurrent

@property
def is_concurrent(self) -> bool:
"""
Check if database is in concurrent mode.

Concurrent mode allows multiple processes to access the database
simultaneously with lock-free reads and single-writer coordination.

Returns:
True if database is in concurrent mode.
"""
return self._is_concurrent

@classmethod
def open_concurrent(cls, path: str) -> "Database":
"""
Open database in concurrent mode (multi-reader, single-writer).

This mode allows multiple processes to access the database simultaneously:

- **Readers**: Lock-free, concurrent access via MVCC snapshots (~100ns latency)
- **Writers**: Single-writer coordination through atomic locks (~60µs amortized)

Use this for:

- Web applications (Flask, FastAPI, Django, Gunicorn, uWSGI)
- Hot reloading development servers
- Multi-process worker pools (multiprocessing, Celery)
- Any scenario with concurrent read access

Args:
path: Path to the database directory.

Returns:
Database instance in concurrent mode.

Example:
# Flask application with multiple workers
from flask import Flask
from sochdb import Database

app = Flask(__name__)
db = Database.open_concurrent("./app_data")

@app.route("/user/<user_id>")
def get_user(user_id):
# Multiple requests can read simultaneously
data = db.get(f"user:{user_id}".encode())
return data or "Not found"

@app.route("/user/<user_id>", methods=["POST"])
def update_user(user_id):
# Writes are serialized automatically
db.put(f"user:{user_id}".encode(), request.data)
return "OK"

Performance:
- Read latency: ~100ns (lock-free atomic operations)
- Write latency: ~60µs amortized (with group commit)
- Concurrent readers: Up to 1024 per database
"""
lib = _FFI.get_lib()
path_bytes = path.encode("utf-8")

# Check if sochdb_open_concurrent is available
if not hasattr(lib, 'sochdb_open_concurrent'):
raise DatabaseError(
"Concurrent mode requires a newer version of the native library. "
"Please update sochdb to the latest version."
)

handle = lib.sochdb_open_concurrent(path_bytes)

if not handle:
raise DatabaseError(
f"Failed to open database at {path} in concurrent mode. "
"Check if the path exists and is accessible."
)

# Track database open event
try:
from .analytics import track_database_open
track_database_open(path, mode="embedded-concurrent")
except Exception:
pass

return cls(path, handle, _is_concurrent=True)

@classmethod
def open(cls, path: str, config: Optional[dict] = None) -> "Database":
Expand Down