Siddhi Store RDBMS
The siddhi-store-rdbms extension is an extension to Siddhi that persist and retrieve events to/from RDBMS databases such as MySQL, MS SQL, PostgreSQL, H2 and Oracle.
For information on Siddhi and it's features refer Siddhi Documentation.
Download
- Versions 7.x & 6.x and above with group id
io.siddhi.extension.*from here. - Versions 5.x and lower with group id
org.wso2.extension.siddhi.*from here.
Latest API Docs
Latest API Docs is 7.0.17.
Features
- cud (Stream Processor)
This function performs SQL CUD (INSERT, UPDATE, DELETE) queries on data sources.
Note: This function to work data sources should be set at the Siddhi Manager level. - procedure (Stream Processor)
This function execute stored procedure and retrieve data to siddhi .
Note: This function to work data sources should be set at the Siddhi Manager level. - query (Stream Processor)
This function performs SQL retrieval queries on data sources.
Note: This function to work data sources should be set at the Siddhi Manager level. - rdbms (Store)
This extension assigns data sources and connection instructions to event tables. It also implements read-write operations on connected data sources. A new improvement is added when running with SI / SI Tooling 1.1.0 or higher product pack, where an external configuration file can be provided to read supported RDBMS databases. Prerequisites - Configuration file needed to be added to [Product_Home]/conf/siddhi/rdbms path with the configuration file name as rdbms-table-config.xml , <database name=”[Database_Name]”> for each database name should be the equivalent database product name returned from java sql Connection.getMetaData().getDatabaseProductName() as shown in API documentation https://docs.oracle.com/javase/7/docs/api/java/sql/DatabaseMetaData.html#getDatabaseProductName()).Sample Configuration for one of the databases can be as follows,<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<rdbms-table-configuration>
<database name="Teradata">
<tableCreateQuery>CREATE TABLE {{TABLE_NAME}} ({{COLUMNS, PRIMARY_KEYS}})</tableCreateQuery>
<tableCheckQuery>SELECT 1 FROM {{TABLE_NAME}} SAMPLE 1</tableCheckQuery>
<indexCreateQuery>CREATE INDEX {{TABLE_NAME}}INDEX{{INDEX_NUM}} ({{INDEX_COLUMNS}}) ON {{TABLE_NAME}}
</indexCreateQuery>
<recordExistsQuery>SELECT 1 FROM {{TABLE_NAME}} {{CONDITION}} SAMPLE 1</recordExistsQuery>
<recordSelectQuery>SELECT * FROM {{TABLE_NAME}} {{CONDITION}}</recordSelectQuery>
<recordInsertQuery>INSERT INTO {{TABLE_NAME}} ({{COLUMNS}}) VALUES ({{Q}})</recordInsertQuery>
<recordUpdateQuery>UPDATE {{TABLE_NAME}} SET {{COLUMNS_AND_VALUES}} {{CONDITION}}</recordUpdateQuery>
<recordDeleteQuery>DELETE FROM {{TABLE_NAME}} {{CONDITION}}</recordDeleteQuery>
<recordContainsCondition>({{COLUMNS}} LIKE {{VALUES}})</recordContainsCondition>
<selectQueryTemplate>
<selectClause>SELECT {{SELECTORS}} FROM {{TABLE_NAME}}</selectClause>
<selectQueryWithSubSelect>SELECT {{SELECTORS}} FROM {{TABLE_NAME}}, ( {{INNER_QUERY}} ) AS t2
</selectQueryWithSubSelect>
<whereClause>WHERE {{CONDITION}}</whereClause>
<groupByClause>GROUP BY {{COLUMNS}}</groupByClause>
<havingClause>HAVING {{CONDITION}}</havingClause>
<orderByClause>ORDER BY {{COLUMNS}}</orderByClause>
<limitClause>SAMPLE {{Q}}</limitClause>
</selectQueryTemplate>
<stringSize>254</stringSize>
<batchEnable>true</batchEnable>
<batchSize>1000</batchSize>
<typeMapping>
<binaryType>
<typeName>BLOB</typeName>
<typeValue>2004</typeValue>
</binaryType>
<booleanType>
<typeName>SMALLINT</typeName>
<typeValue>5</typeValue>
</booleanType>
<doubleType>
<typeName>FLOAT</typeName>
<typeValue>8</typeValue>
</doubleType>
<floatType>
<typeName>FLOAT</typeName>
<typeValue>6</typeValue>
</floatType>
<integerType>
<typeName>INTEGER</typeName>
<typeValue>4</typeValue>
</integerType>
<longType>
<typeName>BIGINT</typeName>
<typeValue>-5</typeValue>
</longType>
<stringType>
<typeName>VARCHAR</typeName>
<typeValue>12</typeValue>
</stringType>
</typeMapping>
</database>
</rdbms-table-configuration>
Dependencies
JDBC connector jar should be added to the runtime. Download the JDBC connector jar based on the RDBMS type that is connected through the Siddhi store.
Installation
For installing this extension on various siddhi execution environments refer Siddhi documentation section on adding extensions.
Running Integration tests in docker containers(Optional)
The RDBMS functionality are tested with the docker base integration test framework, except the H2 default embedded database is not uses external docker container. The test framework initialize the docker container for each database according to the given profile before execute the test suit.
Start integration tests
-
Install and run docker in daemon mode.
-
Installing docker on Linux,
Note:
These commands retrieve content from get.docker.com web in a quiet output-document mode and install.wget -qO- https://get.docker.com/ | sh -
On installing docker on Mac, see Get started with Docker for Mac
-
On installing docker on Windows, see Get started with Docker for Windows
-
-
To run the integration test, navigate to the siddhi-store-rdbms/ directory and issue the following commands.
-
H2 default:
mvn clean installNote : h2 is the default activated profile as it is not used docker.
-
MySQL 5.7:
mvn verify -P local-mysql -f /component/pom.xml -Dskip.surefire.test=true -Ddocker.removeVolumes=true -
Postgres 9.6:
mvn verify -P local-postgres -f /component/pom.xml -Dskip.surefire.test=true -Ddocker.removeVolumes=true -
MSSQL CTP 2.0:
mvn verify -P local-mssql -f /component/pom.xml -Dskip.surefire.test=true -Ddocker.removeVolumes=true -
Oracle 11.2.0.2 Express Edition:
mvn verify -P local-oracle -f component/pom.xml -Dskip.surefire.test=true -Ddocker.removeVolumes=true -
Oracle 12.1.0.2 Standard Edition:
-
Download Oracle driver version 12.1.0.2 ojdbc7.jar
-
To install the JAR file as a Maven plugin, issue the following command:
mvn install:install-file -Dfile=/tmp/ojdbc7.jar -DgroupId=com.oracle -DartifactId=ojdbc7 -Dversion=12.1.0.2 -Dpackaging=jar -
Uncomment the following test dependency in the /component/pom.xml file as shown below:
<dependency> <groupId>com.oracle</groupId> <artifactId>ojdbc7</artifactId> <scope>test</scope> <version>12.1.0.2</version> </dependency> -
To run the integration test, navigate to siddhi-store-rdbms/ directory and issue the following commands:
mvn verify -P local-oracle12 -f component/pom.xml -Dskip.surefire.test=true -Ddocker.removeVolumes=true
-
-
DB2
-
Download DB2 driver version 4.19.26 db2jcc4.jar
-
To install the JAR file as a Maven plugin, issue the following command:
mvn install:install-file -Dfile=/tmp/db2jcc4.jar -DgroupId=com.ibm.db2 -DartifactId=db2jcc -Dversion=4.19.26 -Dpackaging=jar -
Uncomment the following test dependency in the /component/pom.xml file as shown below:
<dependency> <groupId>com.ibm.db2</groupId> <artifactId>db2jcc</artifactId> <scope>test</scope> <version>4.19.26</version> </dependency> -
To run the integration test, navigate to siddhi-store-rdbms/ directory and issue the following commands:
mvn verify -P local-db2 -f /component/pom.xml -Dskip.surefire.test=true -Ddocker.removeVolumes=true
-
-
Start integration tests in debug mode
mvn -P local-mysql -Dmaven.failsafe.debug verify
Note: local-mysql is the profile. Use other profiles accordingly.
Support and Contribution
-
We encourage users to ask questions and get support via StackOverflow, make sure to add the
siddhitag to the issue for better response. -
If you find any issues related to the extension please report them on the issue tracker.
-
For production support and other contribution related information refer Siddhi Community documentation.