Siddhi IO CDC
The siddhi-io-cdc extension is an extension to Siddhi that captures change data from databases such as MySQL, MS SQL, PostgreSQL, H2 and Oracle.
- Versions 3.x and above with group id
- Versions 2.x and lower with group id
Latest API Docs
Latest API Docs is 2.0.3.
- cdc (Source)
The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.
There are two modes you could perform CDC: Listening mode and Polling mode.
In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
In polling mode, you can only capture INSERT and UPDATE changes.
On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.
The key values of the map of a CDC change event are as follows.
For 'listening' mode:
For insert: Keys are specified as columns of the table.
For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
For 'polling' mode: Keys are specified as the columns of the table.
See parameter: mode for supported databases and change events.
JDBC connector jar should be added to the runtime. Download the JDBC connector jar based on the database type that is being used.
For MySQL, use connector version 5.1.xx.
In addition to that, there are some prerequisites that need to be met based on the CDC mode used. Please find them below.
Default mode (Listening mode):
Currently MySQL, PostgreSQL and SQLServer are supported in Listening Mode. To capture the change events, databases have to be configured as shown below.
- MySQL - https://debezium.io/docs/connectors/mysql/#setting-up-mysql
- PostgreSQL - https://debezium.io/docs/connectors/postgresql/#setting-up-PostgreSQL
- SQLServer - https://debezium.io/docs/connectors/sqlserver/#setting-up-sqlserver
- Change data capturing table should be have a polling column. Auto Incremental column or Timestamp can be used.
Please see API docs for more details about change data capturing modes.
For installing this extension on various siddhi execution environments refer Siddhi documentation section on adding extensions.
Running Integration tests in docker containers(Optional)
The CDC functionality are tested with the docker base integration test framework. The test framework initialize a docker container with required configuration before execute the test suit.
Start integration tests
Install and run docker
To run the integration tests, navigate to the siddhi-io-cdc/ directory and issue the following commands.
mvn clean install
mvn verify -P local-mysql -Dskip.surefire.test=true
mvn verify -P local-postgres -Dskip.surefire.test=true
mvn verify -P local-mssql -Dskip.surefire.test=true
mvn verify -P local-oracle -Dskip.surefire.test=true
Support and Contribution
We encourage users to ask questions and get support via StackOverflow, make sure to add the
siddhitag to the issue for better response.
If you find any issues related to the extension please report them on the issue tracker.
For production support and other contribution related information refer Siddhi Community documentation.