Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add Apache Doris connector #14

Description

@boluor

Summary

Proposing a new connector for Apache Doris — a publisher to read from Doris and a subscriber to write to Doris.

Per assets/docs/CONTRIBUTING.md, opening this issue first to coordinate before sending a PR (will also reach out to [email protected]).

Why Doris fits well

Apache Doris exposes two endpoints that map cleanly onto the Tabsdata connector model:

  • FE query port (9030) speaks the MySQL wire protocol — reads are plain SQL over a MySQL driver, no Doris-specific client needed.
  • FE HTTP port (8030) serves Stream Load, the recommended bulk-ingestion API (same path used by the official Spark/Flink Doris connectors).

Proposed design

A standalone connector package connectors/python/tabsdata-doris/, mirroring the tabsdata-mssql layout:

  • DorisSrc (publisher) — runs SQL over the MySQL-protocol port via SQLAlchemy + PyMySQL, streaming results to TableFrames in bounded chunks.
  • DorisDest (subscriber) — writes via the Stream Load HTTP API; supports append/replace, JSON/CSV wire formats.
  • DorisConn — manages both endpoints.

The package is additive and needs no changes to Tabsdata core. An optional core-factory integration path (routing the source through the shared SQL helper, like MSSQLSrc) is documented as a follow-up choice.

Status

A working implementation with unit tests is ready on a branch. Happy to open a PR once the team confirms interest and any preferences (package naming, the tabsdata[doris] extra wiring, integration-test setup against a Dockerized apache/doris).

Questions for maintainers

  1. Is a standalone tabsdata-doris package the preferred shape (vs. a built-in _io/ connector)?
  2. Preference between the self-contained source vs. the core-factory integration path?
  3. Anything specific you'd like for the tabsdata[doris] pip extra wiring on the core side?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions