DM-42005: HTTP-based Worker Replication server#883
Open
iagaponenko wants to merge 11 commits into
Open
Conversation
91b3c7e to
b5b2f7f
Compare
e90255c to
ea7962d
Compare
013e4f5 to
021a07a
Compare
c8f58ae to
9973d08
Compare
cae38b9 to
da02c98
Compare
bf0a008 to
24c5495
Compare
154426a to
91c03f7
Compare
b581b6a to
124acba
Compare
2e85099 to
d7b40a9
Compare
5485b61 to
1af3b73
Compare
6833b15 to
2575b5b
Compare
1aab3fd to
ae1a11c
Compare
5548365 to
dc93085
Compare
…uests This function that was provisioned in the original design of the Replication System has shown no benefits. Besides, the "duplicate" requests can't be reliably detected since the detection algorithm is time dependent.
…ation classes The class is now obsolete.
…service Extended Replication Worker's config page on the Web Dashbnoard to display connection parameters of the HTTP-based worker management services.
Added specialized method for parsing values of specific types.
The "none" option allows to send the data AS-IS where it's possible (all bytes are characters which can be passedwiting a valid JSON string w/io any additional encoding).
There was a problem hiding this comment.
Pull request overview
This PR introduces an HTTP frontend for the Replication Worker service (DM-42005) and updates the replication REST API to version 56, including new JSON protocol utilities and worker-side HTTP request processing infrastructure.
Changes:
- Adds a new HTTP-based Replication Worker server with endpoints for replica create/delete/find, director-index extraction/deletion, SQL, request tracking, and service management.
- Introduces/extends JSON-focused protocol helpers (request parameters parsing, performance JSON, SQL metadata/rows JSON) and updates API versioning across JS/Python/C++ meta modules.
- Refactors Controller-side request classes to remove
RequestMessenger/duplicate-request tracking and updates config schema/test data/docs accordingly.
Reviewed changes
Copilot reviewed 95 out of 95 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/www/qserv/js/ReplicationConfigWorkers.js | Shows HTTP worker replication endpoint info in the dashboard. |
| src/www/qserv/js/Common.js | Bumps web UI REST API version to 56. |
| src/replica/worker/WorkerSqlHttpRequest.h | Declares HTTP SQL request implementation for worker DB queries. |
| src/replica/worker/WorkerProcessor.cc | Removes prior duplicate-request scope checks for replication/delete enqueue. |
| src/replica/worker/WorkerHttpSvcMod.h | Declares HTTP module routing worker replication REST submodules. |
| src/replica/worker/WorkerHttpSvcMod.cc | Implements HTTP submodule dispatch and handlers (echo/replica/index/sql/track/etc.). |
| src/replica/worker/WorkerHttpSvc.h | Declares the HTTP frontend service for worker replication. |
| src/replica/worker/WorkerHttpSvc.cc | Registers HTTP routes and mounts static director-index content. |
| src/replica/worker/WorkerHttpRequest.h | Adds base class for worker HTTP-queued requests (lifecycle/cancel/expire/toJson). |
| src/replica/worker/WorkerHttpRequest.cc | Implements request lifecycle, expiration timer, and JSON serialization. |
| src/replica/worker/WorkerHttpProcessorThread.h | Declares thread worker for processing HTTP-queued requests. |
| src/replica/worker/WorkerHttpProcessorThread.cc | Implements thread loop fetching/executing HTTP requests. |
| src/replica/worker/WorkerFindReplicaHttpRequest.h | Declares HTTP request for finding a replica (optionally checksums). |
| src/replica/worker/WorkerFindReplicaHttpRequest.cc | Implements replica lookup and optional incremental checksum computation. |
| src/replica/worker/WorkerFindAllReplicasHttpRequest.h | Declares HTTP request for finding all replicas in a DB. |
| src/replica/worker/WorkerFindAllReplicasHttpRequest.cc | Implements scanning data dir and reporting replica completeness. |
| src/replica/worker/WorkerEchoHttpRequest.h | Declares HTTP echo test request. |
| src/replica/worker/WorkerEchoHttpRequest.cc | Implements echo request with delay/cancellation support. |
| src/replica/worker/WorkerDirectorIndexHttpRequest.h | Declares HTTP request to dump director index data to CSV for download. |
| src/replica/worker/WorkerDirectorIndexHttpRequest.cc | Implements director index extraction and returns download URL/size. |
| src/replica/worker/WorkerDeleteReplicaHttpRequest.h | Declares HTTP request to delete a replica’s files from disk. |
| src/replica/worker/WorkerDeleteReplicaHttpRequest.cc | Implements replica deletion with filesystem locking and status. |
| src/replica/worker/WorkerCreateReplicaHttpRequest.h | Declares HTTP request to copy a replica from another worker. |
| src/replica/worker/CMakeLists.txt | Adds new HTTP worker sources to the build. |
| src/replica/util/Performance.h | Adds JSON helpers and makes worker perf counters atomic. |
| src/replica/util/Performance.cc | Implements perf JSON export and atomic timestamp updates. |
| src/replica/util/Common.h | Adds JSON parsing helpers for SQL column/index definitions. |
| src/replica/util/Common.cc | Implements SQL column/index parsing from JSON. |
| src/replica/tests/testProtocolRequestParams.cc | Adds unit tests for JSON request parameter parsing. |
| src/replica/tests/testConfiguration.cc | Extends configuration tests for new HTTP worker parameters. |
| src/replica/requests/StopRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/StopRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/StatusRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/StatusRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/SqlRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/SqlRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/ServiceManagementRequestBase.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/ServiceManagementRequestBase.cc | Updates constructor base class wiring. |
| src/replica/requests/RequestMessenger.h | Removes obsolete RequestMessenger class (header). |
| src/replica/requests/RequestMessenger.cc | Removes obsolete RequestMessenger class (impl). |
| src/replica/requests/Request.h | Refactors Request API (remove duplicate tracking; move dispose logic into Request). |
| src/replica/requests/Request.cc | Implements disposal behavior in Request and removes remoteId/duplicate paths. |
| src/replica/requests/ReplicationRequest.h | Updates API to remove allow-duplicate plumbing. |
| src/replica/requests/ReplicationRequest.cc | Removes duplicate-request follow logic and uses request id for tracking. |
| src/replica/requests/FindRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/FindRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/FindAllRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/FindAllRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/EchoRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/EchoRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/DisposeRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/DisposeRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/DirectorIndexRequest.h | Switches from RequestMessenger to Request base. |
| src/replica/requests/DirectorIndexRequest.cc | Updates constructor base class wiring. |
| src/replica/requests/DeleteRequest.h | Updates API to remove allow-duplicate plumbing. |
| src/replica/requests/DeleteRequest.cc | Removes duplicate-request follow logic and uses request id for tracking. |
| src/replica/requests/CMakeLists.txt | Drops RequestMessenger.cc from build. |
| src/replica/registry/Registry.cc | Adds http-svc-port/host handling for worker discovery/registration. |
| src/replica/proto/protocol.proto | Removes DUPLICATE status_ext and duplicate_request_id fields in responses. |
| src/replica/proto/Protocol.h | Adds JSON-oriented protocol types and RequestParams parser. |
| src/replica/proto/Protocol.cc | Implements protocol string conversions and RequestParams parsing. |
| src/replica/proto/CMakeLists.txt | Adds Protocol.cc to build. |
| src/replica/mysql/DatabaseMySQLRow.h | Adds JSON conversion for SQL rows with binary encoding support. |
| src/replica/mysql/DatabaseMySQLRow.cc | Implements row-to-JSON conversion with binary encoding modes. |
| src/replica/mysql/DatabaseMySQL.h | Adds JSON export for field metadata. |
| src/replica/mysql/DatabaseMySQL.cc | Implements fields-to-JSON conversion. |
| src/replica/jobs/FixUpJob.cc | Updates ReplicationRequest invocation (remove allowDuplicate arg). |
| src/replica/jobs/DeleteReplicaJob.cc | Updates DeleteRequest invocation (remove allowDuplicate arg). |
| src/replica/jobs/CreateReplicaJob.cc | Updates ReplicationRequest invocation (remove allowDuplicate arg). |
| src/replica/ingest/IngestDataHttpSvcMod.cc | Treats BinaryEncodingMode::NONE explicitly as unsupported (fallthrough). |
| src/replica/config/ConfigWorker.h | Adds worker httpSvcHost/httpSvcPort configuration. |
| src/replica/config/ConfigWorker.cc | Parses/serializes http-svc host/port and includes it in equality. |
| src/replica/config/ConfigurationSchema.cc | Adds worker HTTP service config params and updates descriptions. |
| src/replica/config/ConfigTestData.cc | Extends test config data with HTTP worker service settings. |
| src/replica/CMakeLists.txt | Registers new unit test target for protocol request params. |
| src/replica/apps/WorkerApp.cc | Launches the new WorkerHttpSvc alongside existing worker services. |
| src/replica/apps/ControllerApp.h | Removes CLI flag/state for allowing duplicate requests. |
| src/replica/apps/ControllerApp.cc | Removes allow-duplicates CLI option and parameter propagation. |
| src/replica/apps/ConfigAppBase.cc | Adds HTTP replication service endpoint column to worker table output. |
| src/http/RequestBodyJSON.h | Improves JSON param helpers (optional(), typed string/bool getters, messages). |
| src/http/RequestBodyJSON.cc | Implements typed string/bool getters and improves integer parsing errors. |
| src/http/MetaModule.cc | Bumps REST API version constant to 56. |
| src/http/ChttpMetaModule.cc | Bumps REST API version constant to 56. |
| src/http/BinaryEncoding.h | Adds none binary encoding mode to allowed list and enum. |
| src/http/BinaryEncoding.cc | Parses/serializes none binary encoding mode. |
| src/czar/HttpCzarQueryModule.cc | Supports BinaryEncodingMode::NONE when encoding result rows. |
| python/lsst/qserv/admin/replication_interface.py | Bumps replication API version to 56. |
| doc/ingest/api/reference/rest/general.rst | Documents binary_encoding=none and its caveats. |
| doc/dev/api/TODO.rst | Adds developer TODO checklist for the new worker HTTP replication work. |
| doc/dev/api/index.rst | Updates dev API docs version reference and adds new sections. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+57
to
+60
| bool WorkerHttpProcessorThread::isRunning() const { return _thread != nullptr; } | ||
|
|
||
| void WorkerHttpProcessorThread::run() { | ||
| if (isRunning()) return; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.