Skip to content

Java library provides Apache Flink connector sink for JDBC database that can be used with Flink 1.8 runtime version.

License

Notifications You must be signed in to change notification settings

klarna-incubator/flink-connector-jdbc-1.8

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

flink-connector-jdbc

Java library provides Apache Flink connector sink for JDBC database that can be used with Flink 1.8 runtime version. Connector code is backported from the latest Flink version (1.11) in order to be used in Amazon Kinesis Data Analytics applications.

Build Status License Developed at Klarna

At Klarna we use streaming applications extensively. Amazon Kinesis Data Analytics with Flink 1.8 is starting to be one of the choices for the development of new streaming analytics applications at Klarna. Unfortunately, some of the latest features developed in the Apache Flink project version after 1.8 are not available yet in Amazon Kinesis Data Analytics.

flink-connector-jdbc-1.8 is a Java library that contains code backported from the latest Flink version (1.11) flink-connector-jdbc library that can be used in Amazon Kinesis Data Analytics / Flink 1.8.

Usage example

import com.klarna.org.apache.flink.api.java.io.jdbc.JDBCOptions;
import com.klarna.org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat;
import com.klarna.org.apache.flink.api.java.io.jdbc.JDBCUpsertSinkFunction;

...
env.addSource(createConsumer())
   .addSink(new JDBCUpsertSinkFunction(JDBCUpsertOutputFormat.builder()
            .setFieldNames(new String[]{
                    "event_id",
                    "created_at"
            })
            .setFieldTypes(new int[]{
                    Types.VARCHAR,
                    Types.TIMESTAMP
            })
            .setFlushIntervalMills(10000)
            .setFlushMaxSize(5000)
            .setKeyFields(new String[]{ "event_id" })
            .setMaxRetryTimes(3)
            .setOptions(JDBCOptions.builder()
                    .setDBUrl(dbUrl)
                    .setDriverName(Driver.class.getName())
                    .setUsername(dbUsername)
                    .setPassword(dbPassword)
                    .setTableName(tableName)
                    .build())
            .build()));

Development setup

This project uses Maven to set up the development environment. The recommended workflow to build and install the library is the following.

mvn clean install

How to contribute

See our guide on contributing.

Release History

See our changelog.

License

Copyright © 2020 Klarna Bank AB

For license details, see the LICENSE file in the root of this project.

About

Java library provides Apache Flink connector sink for JDBC database that can be used with Flink 1.8 runtime version.

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Packages

 
 
 

Languages