Summary
Proposing a new connector for Apache Doris — a publisher to read from Doris and a subscriber to write to Doris.
Per assets/docs/CONTRIBUTING.md, opening this issue first to coordinate before sending a PR (will also reach out to [email protected]).
Why Doris fits well
Apache Doris exposes two endpoints that map cleanly onto the Tabsdata connector model:
- FE query port (9030) speaks the MySQL wire protocol — reads are plain SQL over a MySQL driver, no Doris-specific client needed.
- FE HTTP port (8030) serves Stream Load, the recommended bulk-ingestion API (same path used by the official Spark/Flink Doris connectors).
Proposed design
A standalone connector package connectors/python/tabsdata-doris/, mirroring the tabsdata-mssql layout:
DorisSrc (publisher) — runs SQL over the MySQL-protocol port via SQLAlchemy + PyMySQL, streaming results to TableFrames in bounded chunks.
DorisDest (subscriber) — writes via the Stream Load HTTP API; supports append/replace, JSON/CSV wire formats.
DorisConn — manages both endpoints.
The package is additive and needs no changes to Tabsdata core. An optional core-factory integration path (routing the source through the shared SQL helper, like MSSQLSrc) is documented as a follow-up choice.
Status
A working implementation with unit tests is ready on a branch. Happy to open a PR once the team confirms interest and any preferences (package naming, the tabsdata[doris] extra wiring, integration-test setup against a Dockerized apache/doris).
Questions for maintainers
- Is a standalone
tabsdata-doris package the preferred shape (vs. a built-in _io/ connector)?
- Preference between the self-contained source vs. the core-factory integration path?
- Anything specific you'd like for the
tabsdata[doris] pip extra wiring on the core side?
Summary
Proposing a new connector for Apache Doris — a publisher to read from Doris and a subscriber to write to Doris.
Per
assets/docs/CONTRIBUTING.md, opening this issue first to coordinate before sending a PR (will also reach out to [email protected]).Why Doris fits well
Apache Doris exposes two endpoints that map cleanly onto the Tabsdata connector model:
Proposed design
A standalone connector package
connectors/python/tabsdata-doris/, mirroring thetabsdata-mssqllayout:DorisSrc(publisher) — runs SQL over the MySQL-protocol port via SQLAlchemy + PyMySQL, streaming results to TableFrames in bounded chunks.DorisDest(subscriber) — writes via the Stream Load HTTP API; supportsappend/replace, JSON/CSV wire formats.DorisConn— manages both endpoints.The package is additive and needs no changes to Tabsdata core. An optional core-factory integration path (routing the source through the shared SQL helper, like
MSSQLSrc) is documented as a follow-up choice.Status
A working implementation with unit tests is ready on a branch. Happy to open a PR once the team confirms interest and any preferences (package naming, the
tabsdata[doris]extra wiring, integration-test setup against a Dockerizedapache/doris).Questions for maintainers
tabsdata-dorispackage the preferred shape (vs. a built-in_io/connector)?tabsdata[doris]pip extra wiring on the core side?