

API Docs - v2.0.9¶

Tested Siddhi Core version: 5.1.5

It could also support other Siddhi Core minor versions.

File¶

isDirectory (Function)¶

This function checks for a given file path points to a directory

Syntax

<BOOL> file:isDirectory(<STRING> uri)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	The path to be checked for a directory.		STRING	No	Yes

Examples EXAMPLE 1

file:isDirectory(filePath) as isDirectory

Checks whether the given path is a directory. Result will be returned as an boolean.

isExist (Function)¶

This function checks whether a file or a folder exists in a given path

Syntax

<BOOL> file:isExist(<STRING> uri)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	File path to check for existence.		STRING	No	Yes

Examples EXAMPLE 1

file:isExist('/User/wso2/source/test.txt') as exists

Checks existence of a file in the given path. Result will be returned as an boolean .

EXAMPLE 2

file:isExist('/User/wso2/source/') as exists

Checks existence of a folder in the given path. Result will be returned as an boolean .

isFile (Function)¶

This function checks for a given file path points to a file

Syntax

<BOOL> file:isFile(<STRING> file.path)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
file.path	The path to be checked for a file.		STRING	No	Yes

Examples EXAMPLE 1

file:isFile(filePath) as isFile

Checks whether the given path is a file. Result will be returned as an boolean.

lastModifiedTime (Function)¶

Checks for the last modified time for a given file path

Syntax

<STRING> file:lastModifiedTime(<STRING> uri)
<STRING> file:lastModifiedTime(<STRING> uri, <STRING> datetime.format)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	File path to be checked for te last modified time.		STRING	No	Yes
datetime.format	Format of the last modified datetime to be returned.	MM/dd/yyyy HH:mm:ss	STRING	Yes	No

Examples EXAMPLE 1

file:lastModifiedTime(filePath) as lastModifiedTime

Last modified datetime of a file will be returned as an string in MM/dd/yyyy HH:mm:ss.

EXAMPLE 2

file:lastModifiedTime(filePath, dd/MM/yyyy HH:mm:ss) as lastModifiedTime

Last modified datetime of a file will be returned as an string in 'dd/MM/yyyy HH:mm:ss' format.

size (Function)¶

This function checks for a given file's size

Syntax

<LONG> file:size(<STRING> uri)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute path to the file or directory to be checked for the size.		STRING	No	Yes

Examples EXAMPLE 1

file:size('/User/wso2/source/test.txt') as fileSize

Size of a file in a given path will be returned.

archive (Stream Function)¶

Archives files and folders as a zip or in tar format that are available in the given file uri.

Syntax

file:archive(<STRING> uri, <STRING> destination.dir.uri)
file:archive(<STRING> uri, <STRING> destination.dir.uri, <STRING> archive.type)
file:archive(<STRING> uri, <STRING> destination.dir.uri, <STRING> archive.type, <STRING> include.by.regexp)
file:archive(<STRING> uri, <STRING> destination.dir.uri, <STRING> archive.type, <STRING> include.by.regexp, <BOOL> exclude.subdirectories)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute path of the file or the directory		STRING	No	Yes
destination.dir.uri	Absolute directory path of the the archived file.		STRING	No	Yes
archive.type	Archive type can be zip or tar	zip	STRING	Yes	No
include.by.regexp	Only the files matching the patterns will be archived. Note: Add an empty string to match all files		STRING	Yes	No
exclude.subdirectories	This flag is used to exclude the subdirectories and its files without archiving.	false	BOOL	Yes	No

Examples EXAMPLE 1

InputStream#file:archive('/User/wso2/to_be_archived', '/User/wso2/archive_destination/file.zip')

Archives to_be_archived folder in zip format and stores archive_destination folder as file.zip.

EXAMPLE 2

InputStream#file:archive('/User/wso2/to_be_archived', '/User/wso2/archive_destination/file', 'tar')

Archives to_be_archived folder in tar format and stores in archive_destination folder as file.tar.

EXAMPLE 3

InputStream#file:archive('/User/wso2/to_be_archived', '/User/wso2/archive_destination/file', 'tar', '.*test3.txt$')

Archives files which adheres to '.*test3.txt$' regex in to_be_archived folder in tar format and stores in archive_destination folder as file.tar.

EXAMPLE 4

InputStream#file:archive('/User/wso2/to_be_archived', '/User/wso2/archive_destination/file', '', '', 'false')

Archives to_be_archived folder excluding the sub-folders in zip format and stores in archive_destination folder as file.tar.

copy (Stream Function)¶

This function performs copying file from one directory to another.

Syntax

file:copy(<STRING> uri, <STRING> destination.dir.uri)
file:copy(<STRING> uri, <STRING> destination.dir.uri, <STRING> include.by.regexp)
file:copy(<STRING> uri, <STRING> destination.dir.uri, <STRING> include.by.regexp, <BOOL> exclude.root.dir)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute path of the File or the directory.		STRING	No	Yes
destination.dir.uri	Absolute path of the destination directory. Note: Parent folder structure will be created if it does not exist.		STRING	No	Yes
include.by.regexp	Only the files matching the patterns will be copied. Note: Add an empty string to match all files		STRING	Yes	No
exclude.root.dir	This flag is used to exclude parent folder when copying the content.	false	BOOL	Yes	No

Extra Return Attributes

Name	Description	Possible Types
isSuccess	Status of the file copying operation (true if success)	BOOL

Examples EXAMPLE 1

InputStream#file:copy('/User/wso2/source/test.txt', 'User/wso2/destination/')

Copies 'test.txt' in 'source' folder to the 'destination' folder.

EXAMPLE 2

InputStream#file:copy('/User/wso2/source/', 'User/wso2/destination/')

Copies 'source' folder to the 'destination' folder with all its content

EXAMPLE 3

InputStream#file:copy('/User/wso2/source/', 'User/wso2/destination/', '.*test3.txt$')

Copies 'source' folder to the 'destination' folder ignoring files doesnt adhere to the given regex.

EXAMPLE 4

InputStream#file:copy('/User/wso2/source/', 'User/wso2/destination/', '', true)

Copies only the files resides in 'source' folder to 'destination' folder.

create (Stream Function)¶

Create a file or a folder in the given location

Syntax

file:create(<STRING> uri, <STRING> is.directory)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute file path which needs to be created.		STRING	No	Yes
is.directory	This flag is used when creating file path is a directory		STRING	No	Yes

Examples EXAMPLE 1

from CreateFileStream#file:create('/User/wso2/source/test.txt', false)

Creates a file in the given path with the name of 'test.txt'.

EXAMPLE 2

from CreateFileStream#file:create('/User/wso2/source/', true)

Creates a folder in the given path with the name of 'source'.

move (Stream Function)¶

This function performs copying file from one directory to another.

Syntax

file:move(<STRING> uri, <STRING> destination.dir.uri)
file:move(<STRING> uri, <STRING> destination.dir.uri, <STRING> include.by.regexp)
file:move(<STRING> uri, <STRING> destination.dir.uri, <STRING> include.by.regexp, <BOOL> exclude.root.dir)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute file or directory path.		STRING	No	Yes
destination.dir.uri	Absolute file path to the destination directory. Note: Parent folder structure will be created if it does not exist.		STRING	No	Yes
include.by.regexp	Only the files matching the patterns will be moved. Note: Add an empty string to match all files		STRING	Yes	No
exclude.root.dir	Exclude parent folder when moving the content.	false	BOOL	Yes	No

Extra Return Attributes

Name	Description	Possible Types
isSuccess	Status of the file moving operation (true if success)	BOOL

Examples EXAMPLE 1

InputStream#file:move('/User/wso2/source/test.txt', 'User/wso2/destination/')

Moves 'test.txt' in 'source' folder to the 'destination' folder.

EXAMPLE 2

InputStream#file:move('/User/wso2/source/', 'User/wso2/destination/')

Moves 'source' folder to the 'destination' folder with all its content

EXAMPLE 3

InputStream#file:move('/User/wso2/source/', 'User/wso2/destination/', '.*test3.txt$')

Moves 'source' folder to the 'destination' folder excluding files doesnt adhere to the given regex.

EXAMPLE 4

InputStream#file:move('/User/wso2/source/', 'User/wso2/destination/', '', true)

Moves only the files resides in 'source' folder to 'destination' folder.

search (Stream Function)¶

Searches files in a given folder and lists.

Syntax

file:search(<STRING> uri)
file:search(<STRING> uri, <STRING> include.by.regexp)
file:search(<STRING> uri, <STRING> include.by.regexp, <BOOL> exclude.subdirectories)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute file path of the directory.		STRING	No	Yes
include.by.regexp	Only the files matching the patterns will be searched. Note: Add an empty string to match all files		STRING	Yes	Yes
exclude.subdirectories	This flag is used to exclude the files un subdirectories when listing.	false	BOOL	Yes	No

Extra Return Attributes

Name	Description	Possible Types
fileNameList	The lit file name matches in the directory.	OBJECT

Examples EXAMPLE 1

ListFileStream#file:search(filePath)

This will list all the files (also in sub-folders) in a given path.

EXAMPLE 2

ListFileStream#file:search(filePath, '.*test3.txt$')

This will list all the files (also in sub-folders) which adheres to a given regex file pattern in a given path.

EXAMPLE 3

ListFileStream#file:search(filePath, '.*test3.txt$', true)

This will list all the files excluding the files in sub-folders which adheres to a given regex file pattern in a given path.

searchInArchive (Stream Function)¶

This.

Syntax

file:searchInArchive(<STRING> uri)
file:searchInArchive(<STRING> uri, <STRING> include.by.regexp)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute file path of the zip or tar file.		STRING	No	Yes
include.by.regexp	Only the files matching the patterns will be searched. Note: Add an empty string to match all files		STRING	Yes	No

Extra Return Attributes

Name	Description	Possible Types
fileNameList	The list file names in the archived file.	OBJECT

Examples EXAMPLE 1

ListArchivedFileStream#file:listFilesInArchive(filePath)

Lists the files inside the compressed file in the given path.

EXAMPLE 2

ListArchivedFileStream#file:listFilesInArchive(filePath, '.*test3.txt$')

Filters file names adheres to the given regex and lists the files inside the compressed file in the given path.

unarchive (Stream Function)¶

This function decompresses a given file

Syntax

file:unarchive(<STRING> uri, <STRING> destination.dir.uri)
file:unarchive(<STRING> uri, <STRING> destination.dir.uri, <BOOL> exclude.root.dir)

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
uri	Absolute path of the file to be decompressed in the format of zip or tar.		STRING	No	Yes
destination.dir.uri	Absolute path of the destination directory. Note: If the folder structure does not exist, it will be created.		STRING	No	Yes
exclude.root.dir	This flag excludes parent folder when extracting the content.	false	BOOL	Yes	No

Examples EXAMPLE 1

file:unarchive('/User/wso2/source/test.zip', '/User/wso2/destination')

Unarchive a zip file in a given path to a given destination.

EXAMPLE 2

file:unarchive('/User/wso2/source/test.tar', '/User/wso2/destination')

Unarchive a tar file in a given path to a given destination.

EXAMPLE 3

file:unarchive('/User/wso2/source/test.tar', '/User/wso2/destination', true)

Unarchive a tar file in a given path to a given destination excluding the root folder.

Sink¶

file (Sink)¶

File Sink can be used to publish (write) event data which is processed within siddhi to files.
Siddhi-io-file sink provides support to write both textual and binary data into files

Syntax

@sink(type="file", file.uri="<STRING>", append="<BOOL>", add.line.separator="<BOOL>", @map(...)))

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
file.uri	Used to specify the file for data to be written.		STRING	No	Yes
append	This parameter is used to specify whether the data should be append to the file or not. If append = 'true', data will be write at the end of the file without changing the existing content. If file does not exist, a new fill will be crated and then data will be written. If append append = 'false', If given file exists, existing content will be deleted and then data will be written back to the file. If given file does not exist, a new file will be created and then data will be written on it.	true	BOOL	Yes	No
add.line.separator	This parameter is used to specify whether events added to the file should be separated by a newline. If add.event.separator= 'true',then a newline will be added after data is added to the file.	true. (However, if csv mapper is used, it is false)	BOOL	Yes	No

Examples EXAMPLE 1

@sink(type='file', @map(type='json'), append='false', file.uri='/abc/{{symbol}}.txt') define stream BarStream (symbol string, price float, volume long);

Under above configuration, for each event, a file will be generated if there's no such a file,and then data will be written to that file as json messagesoutput will looks like below.
{
    "event":{
        "symbol":"WSO2",
        "price":55.6,
        "volume":100
    }
}

Source¶

file (Source)¶

File Source provides the functionality for user to feed data to siddhi from files. Both text and binary files are supported by file source.

Syntax

@source(type="file", dir.uri="<STRING>", file.uri="<STRING>", mode="<STRING>", tailing="<BOOL>", action.after.process="<STRING>", action.after.failure="<STRING>", move.after.process="<STRING>", move.after.failure="<STRING>", begin.regex="<STRING>", end.regex="<STRING>", file.polling.interval="<STRING>", dir.polling.interval="<STRING>", timeout="<STRING>", file.read.wait.timeout="<STRING>", header.present="<BOOL>", read.only.header="<BOOL>", @map(...)))

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
dir.uri	Used to specify a directory to be processed. All the files inside this directory will be processed. Only one of 'dir.uri' and 'file.uri' should be provided. This uri MUST have the respective protocol specified.	file:/var/tmp	STRING	Yes	No
file.uri	Used to specify a file to be processed. Only one of 'dir.uri' and 'file.uri' should be provided. This uri MUST have the respective protocol specified.	file:/var/temp/tmp.text	STRING	Yes	No
mode	This parameter is used to specify how files in given directory should.Possible values for this parameter are, 1. TEXT.FULL : to read a text file completely at once. 2. BINARY.FULL : to read a binary file completely at once. 3. LINE : to read a text file line by line. 4. REGEX : to read a text file and extract data using a regex.	line	STRING	Yes	No
tailing	This can either have value true or false. By default it will be true. This attribute allows user to specify whether the file should be tailed or not. If tailing is enabled, the first file of the directory will be tailed. Also tailing should not be enabled in 'binary.full' or 'text.full' modes.	true	BOOL	Yes	No
action.after.process	This parameter is used to specify the action which should be carried out after processing a file in the given directory. It can be either DELETE or MOVE and default value will be 'DELETE'. If the action.after.process is MOVE, user must specify the location to move consumed files using 'move.after.process' parameter.	delete	STRING	Yes	No
action.after.failure	This parameter is used to specify the action which should be carried out if a failure occurred during the process. It can be either DELETE or MOVE and default value will be 'DELETE'. If the action.after.failure is MOVE, user must specify the location to move consumed files using 'move.after.failure' parameter.	delete	STRING	Yes	No
move.after.process	If action.after.process is MOVE, user must specify the location to move consumed files using 'move.after.process' parameter. This should be the absolute path of the file that going to be created after moving is done. This uri MUST have the respective protocol specified.		STRING	Yes	No
move.after.failure	If action.after.failure is MOVE, user must specify the location to move consumed files using 'move.after.failure' parameter. This should be the absolute path of the file that going to be created after moving is done. This uri MUST have the respective protocol specified.		STRING	Yes	No
begin.regex	This will define the regex to be matched at the beginning of the retrieved content.	None	STRING	Yes	No
end.regex	This will define the regex to be matched at the end of the retrieved content.	None	STRING	Yes	No
file.polling.interval	This parameter is used to specify the time period (in milliseconds) of a polling cycle for a file.	1000	STRING	Yes	No
dir.polling.interval	This parameter is used to specify the time period (in milliseconds) of a polling cycle for a directory.	1000	STRING	Yes	No
timeout	This parameter is used to specify the maximum time period (in milliseconds) for waiting until a file is processed.	5000	STRING	Yes	No
file.read.wait.timeout	This parameter is used to specify the maximum time period (in milliseconds) till it waits before retrying to read the full file content.	1000	STRING	Yes	No
header.present	This parameter used to specify a particular text file (eg: CSV) contains a header line or not. This can either have value true or false. If it's set to `true` then it indicates a file contains a header line, and it will not process.	false	BOOL	Yes	No
read.only.header	This parameter used to read only the header or the first line of a particular text file (eg: CSV). This is only applicable if the mode is LINE. If it's set to false, the full file content will be read line by line.	false	BOOL	Yes	No

Examples EXAMPLE 1

@source(type='file',
mode='text.full',
tailing='false'
 dir.uri='file://abc/xyz',
action.after.process='delete',
@map(type='json')) 
define stream FooStream (symbol string, price float, volume long);

Under above configuration, all the files in directory will be picked and read one by one.
In this case, it's assumed that all the files contains json valid json strings with keys 'symbol','price' and 'volume'.
Once a file is read, its content will be converted to an event using siddhi-map-json extension and then, that event will be received to the FooStream.
Finally, after reading is finished, the file will be deleted.

EXAMPLE 2

@source(type='file',
mode='files.repo.line',
tailing='true',
dir.uri='file://abc/xyz',
@map(type='json')) 
define stream FooStream (symbol string, price float, volume long);

Under above configuration, the first file in directory '/abc/xyz' will be picked and read line by line.
In this case, it is assumed that the file contains lines json strings.
For each line, line content will be converted to an event using siddhi-map-json extension and then, that event will be received to the FooStream.
Once file content is completely read, it will keep checking whether a new entry is added to the file or not.
If such entry is added, it will be immediately picked up and processed.

EXAMPLE 3

@source(type='file',
mode='text.full',
tailing='false'
 dir.uri='file://abc/xyz',
action.after.process='delete',
@map(type='csv' @attributes(eof = 'trp:eof', fp = 'trp:file.path'))) 
define stream FooStream (symbol string, price float, volume long);

Under above configuration, all the files in directory will be picked and read one by one.
In this case, it's assumed that all the files contains valid json strings with keys 'symbol' and 'price'.
Once a file is read, its content will be converted to an event using siddhi-map-json with additional eof attribute and then, that event will be received to the FooStream.
Finally, after reading is finished, the file will be deleted.

fileeventlistener (Source)¶

Fileeventlistener provides the functionality for user to get the details of files which have been created or modified or deleted in the execution time.

Syntax

@source(type="fileeventlistener", dir.uri="<STRING>", monitoring.interval="<STRING>", file.name.list="<STRING>", @map(...)))

QUERY PARAMETERS

Name	Description	Default Value	Possible Data Types	Optional	Dynamic
dir.uri	This parameter is used to specify a folder to be processed. All the files inside this directory will be processed. This uri MUST have the respective protocol specified.		STRING	No	No
monitoring.interval	This parameter is used to specify the time interval (in milliseconds) that the process monitor the changes for.	100	STRING	Yes	No
file.name.list	This parameter is used to filter the files to be monitored in the given directory uri (dir.uri). The files should be entered comma separated eg: 'abc.txt,xyz.csv'		STRING	Yes	No

Examples EXAMPLE 1

@source(type='fileeventlistener', dir.uri='file://abc/xyz, file.name.list = 'xyz.txt, test') 
define stream FileListenerStream (filepath string, filename string, status string);
@sink(type='log')
define stream FooStream (filepath string, filename string, status string); 
from FileListenerStream
select *
insert into FooStream;

Under above configuration, An event is triggered if the files in the file.name.list gets created, modified or deleted.
An event is created with the filepath, filename and status of the file. Then that will be received by the FooStream.

EXAMPLE 2

@source(type='fileeventlistener',dir.uri='file://abc/xyz') 
define stream FileListenerStream (filepath string, filename string, status string);
@sink(type='log')
define stream FooStream (filepath string, filename string, status string); 
from FileListenerStream
select *
insert into FooStream;

Under above configuration, An event is triggered if any file under the given directory uri gets created, modified or deleted in the execution time. An event is created with the filepath, filename and status of the file.Then that will be received by the FooStream.

EXAMPLE 3

@source(type='fileeventlistener',dir.uri='file://abc/xyz', monitoring.interval='200')
define stream FileListenerStream (filepath string, filename string, status string);
@sink(type='log')
define stream FooStream (filepath string, filename string, status string);
from FileListenerStream
select *
insert into FooStream;

Under above configuration, An event is triggered if any file under the given directory uri gets created, modified or deleted in the execution time. An event is created with the filepath, filename and status of the file. Then that will be received by the FooStream.
If there are any changes a new event will be generated in every 200 milliseconds.

API Docs - v2.0.9¶

File¶

isDirectory (Function)¶

isExist (Function)¶

isFile (Function)¶

lastModifiedTime (Function)¶

size (Function)¶

archive (Stream Function)¶

copy (Stream Function)¶

create (Stream Function)¶

delete (Stream Function)¶

move (Stream Function)¶

search (Stream Function)¶

searchInArchive (Stream Function)¶

unarchive (Stream Function)¶

Sink¶

file (Sink)¶

Source¶

file (Source)¶

fileeventlistener (Source)¶