fix(telemetry-py): reliable event delivery and per-install identity#292
Conversation
Product analytics under-reported vs downloads because buffered PostHog
events (flushAt=20, flushInterval=10s) only drained on an explicit
shutdown() that short-lived consumers never call, and distinct_id was
read from the target Valkey ({name}:__instance_id), collapsing every
client sharing one Valkey into a single "person".
- Flush the *_init event immediately on start (asyncio.to_thread) so the
install/active signal lands independent of any exit hook.
- Register a best-effort atexit flush on the enabled path only; opt-out
still returns NOOP and registers nothing.
- distinct_id is now a per-install UUID persisted at ~/.betterdb/instance_id
(honors XDG_STATE_HOME, override BETTERDB_INSTANCE_ID); the Valkey-scoped
id is demoted to a deployment_id property for fleet roll-up.
- agent-memory: add memory_session aggregate (integer counts only) at
close/exit and a memory_consolidated event.
Applies to agent-memory-py, retrieval-py, agent-cache-py, semantic-cache-py.
TS/npm packages still pending.
When the install-id file can't be read or written (read-only fs, ephemeral container without a writable home), _get_install_id minted a fresh UUID on every call, fragmenting one install into many distinct_ids. Memoize the id in a module-global on the write-failure path so a single process reports one stable ephemeral identity. Successful file writes are unaffected. Addresses Cursor Bugbot finding on #292.
Rely solely on the build-time baked PostHog key, matching the other three Python packages. The BETTERDB_POSTHOG_API_KEY/HOST runtime overrides were unused and made key resolution inconsistent across packages.
- _capture_session: only arm the one-shot flag once counts are non-zero, so an early atexit with zero activity no longer blocks a later close() from emitting memory_session. - close(): flush the analytics queue after capturing the session summary so it lands even if shutdown()'s posthog close swallows an error, matching the atexit backstop. - semantic-cache-py CHANGELOG: document the baked-only telemetry key change, superseding the stale env-override note in the 0.1.0 entry.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5af5dc2. Configure here.
| # Flush the start event immediately so it lands even for processes that | ||
| # exit before the flush interval or any exit hook fires. Off-loaded to a | ||
| # thread so the network round-trip doesn't stall the event loop. | ||
| await asyncio.to_thread(self.flush) |
There was a problem hiding this comment.
Init failure orphans PostHog client
Medium Severity
_PostHogAnalytics registers an atexit flush in __init__, before init finishes. If init raises (for example when resolving the install id path calls Path.home()), MemoryStore._ensure_analytics_started swaps _analytics to NOOP_ANALYTICS without shutting down the constructed client. The orphaned instance still flushes at exit, while the store skips session atexit registration yet keeps incrementing _session_counts, so memory_session summaries can be lost.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 5af5dc2. Configure here.
There was a problem hiding this comment.
Acknowledged. This only triggers on a published wheel with telemetry enabled and init() actually raising — and nearly everything in init() is internally guarded, leaving only Path.home() failing (no resolvable HOME). Even then the impact is just a leaked idle consumer thread flushing an empty queue; no crash and no wrong events. Far too rare to block this PR — will be folded into the broader telemetry cleanup.
| # Persistence failed — hold the id for the rest of this process so | ||
| # repeated calls return a stable ephemeral identity. | ||
| _ephemeral_install_id = new_id | ||
| return new_id |
There was a problem hiding this comment.
Ephemeral id not process-wide
Medium Severity
When on-disk install id persistence fails, _get_install_id stores the fallback UUID in a module-level _ephemeral_install_id. Each BetterDB Python package defines its own copy of that global, so one process loading both betterdb_agent_memory and betterdb_agent_cache (or retrieval/semantic-cache) can emit PostHog events under multiple distinct_id values despite the docstring promising a single ephemeral per-process identity.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 5af5dc2. Configure here.
There was a problem hiding this comment.
Acknowledged, and intentionally left as-is. Divergence only happens with a read-only home dir AND multiple betterdb packages in one process AND no BETTERDB_INSTANCE_ID pin. Different ephemeral ids per package on the same machine are harmless for our install/active metric, and BETTERDB_INSTANCE_ID already pins it when exactness matters. Too narrow to warrant cross-package env mutation here — will revisit in further telemetry improvements.
Summary
Product analytics under-reported vs downloads because buffered PostHog events (flushAt=20, flushInterval=10s) only drained on an explicit shutdown() that short-lived consumers never call, and distinct_id was read from the target Valkey ({name}:__instance_id), collapsing every client sharing one Valkey into a single "person".
Changes
Applies to agent-memory-py, retrieval-py, agent-cache-py, semantic-cache-py. TS/npm packages still pending.
Checklist
roborev review --branchor/roborev-review-branchin Claude Code (internal)Note
Medium Risk
Changes how installs are counted in analytics and removes semantic-cache PostHog env overrides; writes a local instance id file unless overridden—operators relying on runtime API keys need to adjust.
Overview
Improves product telemetry accuracy and delivery across the Python BetterDB packages (
agent-cache,agent-memory,retrieval,semantic-cache).Identity: PostHog
distinct_idis now a per-machine install UUID (file under~/.betterdb/instance_id,XDG_STATE_HOME, orBETTERDB_INSTANCE_ID), so many processes sharing one Valkey no longer collapse to one “user.” The former Valkey{name}:__instance_idvalue is kept as adeployment_idevent property for fleet roll-up.Delivery: Init events are flushed immediately after capture (
asyncio.to_thread), and enabled clients register anatexitflush so short-lived scripts that never callshutdown()/close()still drain PostHog’s buffer.agent-memory: Adds in-process integer-only session counters emitted once as
memory_sessiononclose()or process exit, plus amemory_consolidatedevent; tests cover install id and deployment id behavior.semantic-cache: Drops runtime
BETTERDB_POSTHOG_API_KEY/BETTERDB_POSTHOG_HOSToverrides (baked wheel key only), documented inCHANGELOG.md.Reviewed by Cursor Bugbot for commit 5af5dc2. Bugbot is set up for automated code reviews on this repo. Configure here.