Greenplum Sink

1、Introduce

Greenplum sink

2、Version Support

Greenplum 5 and above

3、Connector name

Type	Name
Sync	greenplumsink、greenplumwriter
---	---
SQL	greenplum-x

4、Parameter description

1、Sync

• connection

  • Description:param for Database connection,including jdbcUrl、schema、table and so on
  • Required:required
  • Type:List
  • Default:none

        "connection": [{
         "jdbcUrl": ["jdbc:pivotal:greenplum://localhost:5432;DatabaseName=postgres"],
         "table": ["table"],
         "schema":"public"
        }]

    
    

• jdbcUrl

  • Description：jdbc connection string for relational database,jdbcUrl reference：greenplum
  • Required: required
  • Type: string
  • Default: none
    

• schema

  • Description:Database schema
  • Required:optional
  • Type:string
  • Default:none
    

• table

  • Description：The table name of the target table. Currently only single table is supported, and multiple tables will be supported in the future
  • Required:required
  • Type:List
  • Default:none
    

• username

  • Description:user name
  • Required:required
  • Type:String
  • Default:none
    

• password

  • Description:password
  • Required:required
  • Type:String
  • Default:none
    

• column

  • Description：The fields that the destination table needs to write data into, and the fields are separated by commas.for example: "column": ["id","name","age"]
  • Required:required
  • Type:List
  • Default:none
    

• fullcolumn

  • Description:All fields in the destination table ,which is separated by English commas.for example: "column": ["id","name","age","hobby"],if not configured, it will be obtained in the system table
  • Required:optional
  • Type:List
  • Default:none
    

• preSql

  • Description:the sql executed before writing data into the destination table
  • Required:optional
  • Type:List
  • Default:none
    

• postSql

  • Description:the sql executed after writing data into the destination table
  • Required:optional
  • Type:List
  • Default:none
    

• writeMode

  • Description：Controls writing data to the target table using insert into or replace into or ON DUPLICATE KEY UPDATE
  • Required:required
  • All options:insert/replace/update
  • Type:String
  • Default:insert
    

• batchSize

  • Description:The number of records submitted in batch at one time. This value can greatly reduce the number of network interactions between flinkx and the database and improve the overall throughput,Setting this value too large may cause the flinkx process to run oom
  • Required:optional
  • Type:int
  • Default:1024
    

• updateKey

  • Description:When the write mode is update, you need to specify the value of this parameter as the unique index field
  • attention:
    • If this parameter is empty and the write mode is update, the application will automatically obtain the unique index in the database;
    • If the data table does not have a unique index, but the required write mode is configured as update and, the application will write data in the way of insert;
  • Required:optional
  • Type:Map<String,List>
    • for example:"updateKey": {"key": ["id"]}
  • Default:none
    

• semantic

  • Description:sink operator support phase two commit
  • attention: -If this parameter is blank, phase two commit is not enabled by default,which means sink operators do not support exactly-once semantics -Currently only supported exactly-once and at-least-once
  • Required:optional
  • Type:String
    • for example:"semantic": "exactly-once"
  • Default:at-least-once
    

2、SQL

• connector

  • Description:greenplum-x
  • Required:required
  • Type:String
  • Default:none
    

• url

  • Description：jdbc:pivotal:greenplum://localhost:5432;DatabaseName=postgres
  • Required:required
  • Type:String
  • Default:none
    

• schema

  • Description:Database schema
  • Required:optional
  • Type:string
  • Default:none
    

• table-name

  • Description: table name
  • Required:required
  • Type:String
  • Default:none
    

• username

  • Description:username
  • Required:required
  • Type:String
  • Default:none
    

• password

  • Description:password
  • Required:required
  • Type:String
  • Default:none
    

• sink.buffer-flush.max-rows

  • Description:Number of data pieces written in batch,Unit:piece.
  • Required:optional
  • Type:String
  • Default:1024
    

• sink.buffer-flush.interval

  • Description:Batch write interval,Unit: ms
  • Required:optional
  • Type:String
  • Default:10000
    

• sink.all-replace

  • Description：whether to replace all data in the database(If the original value in the database is not null, the new value is null, if it is true, it will be replaced with null)
  • Required:optional
  • Type:String
  • Default:false
    

• sink.parallelism

  • Description:the parallelism of sink operator
  • Required:optional
  • Type:String
  • Default:none
    

• sink.semantic

  • Description:sink operator support phase two commit
  • attention: -If this parameter is blank, phase two commit is not enabled by default,which means sink operators do not support exactly-once semantics; -Currently only supported exactly-once and at-least-once
  • Required:optional
  • Type:String
    • for example:"semantic": "exactly-once"
  • Default:at-least-once
    

五、数据类型

Whether to support	Data Type
Supported	SMALLINT、SMALLSERIAL、INT2、INT、INTEGER、SERIAL、INT4、BIGINT、BIGSERIAL、OID、INT8、REAL、FLOAT4、FLOAT、DOUBLE PRECISION、FLOAT8、DECIMAL、NUMERIC、 CHARACTER VARYING、VARCHAR、CHARACTER、CHAR、TEXT、NAME、BPCHAR、BYTEA、TIMESTAMP、TIMESTAMPTZ、DATE、TIME、TIMETZ、 BOOLEAN、BOOL
Unsupported	ARRAY etc.

6. Example

The details are in flinkx-examples dir.