Doris Pipeline Connector

This article introduces of Doris Pipeline Connector

Example

source:
   type: values
   name:ValuesSource

sink:
   type: doris
   name: Doris Sink
   fenodes: 127.0.0.1:8030
   username: root
   password: ""
   table.create.properties.replication_num: 1

pipeline:
   parallelism: 1

Pipeline options

Option Required Default Type Description
type required (none) String Specify the Sink to use, here is 'doris'.
name optional (none) String Name of PipeLine
fenodes required (none) String Http address of Doris cluster FE, such as 127.0.0.1:8030
benodes optional (none) String Http address of Doris cluster BE, such as 127.0.0.1:8040
jdbc-url optional (none) String JDBC address of Doris cluster, for example: jdbc:mysql://127.0.0.1:9030/db
username required (none) String Username of Doris cluster
password optional (none) String Password for Doris cluster
auto-redirect optional false String Whether to write through FE redirection and directly connect to BE to write
sink.enable.batch-mode optional true Boolean Whether to use the batch method to write to Doris
sink.flush.queue-size optional 2 Integer Queue size for batch writing
sink.buffer-flush.max-rows optional 50000 Integer Maximum number of Flush records in a single batch
sink.buffer-flush.max-bytes optional 10485760(10MB) Integer Maximum number of bytes flushed in a single batch
sink.buffer-flush.interval optional 10s String Flush interval duration. If this time is exceeded, the data will be flushed asynchronously
sink.properties. optional (none) String Parameters of StreamLoad. For example: sink.properties.strict_mode: true. See more about StreamLoad Properties properties
table.create.properties.* optional (none) String Create the Properties configuration of the table. For example: table.create.properties.replication_num: 1. See more about Doris Table Properties properties
## Data Type Mapping
CDC type Doris type NOTE
TINYINT TINYINT
SMALLINT SMALLINT
INT INT
BIGINT BIGINT
DECIMAL DECIMAL
FLOAT FLOAT
DOUBLE DOUBLE
BOOLEAN BOOLEAN
DATE DATE
TIMESTAMP [(p)] DATETIME [(p)]
TIMESTAMP_LTZ [(p)] DATETIME [(p)]
CHAR(n) CHAR(n*3) In Doris, strings are stored in UTF-8 encoding, so English characters occupy 1 byte and Chinese characters occupy 3 bytes. The length here is multiplied by 3. The maximum length of CHAR is 255. Once exceeded, it will automatically be converted to VARCHAR type.
VARCHAR(n) VARCHAR(n*3) Same as above. The length here is multiplied by 3. The maximum length of VARCHAR is 65533. Once exceeded, it will automatically be converted to STRING type.
BINARY(n) STRING
VARBINARY(N) STRING
STRING STRING

FAQ