Sink configuration

Sink configuration is also Json map string whose key is the id of the sink and the value is sink configuration.

Local file sink

Local file sink is writing message set to the local file system.

Argument	Description
outputDir	directory path where files are stored
writer	Its type can be 'text' or 'sequence'. 'text' means text file and 'sequence' means Hadoop sequence file. It is getting codec argument denoting Hadoop codec class.
maxFileSize	when the file size reaches to this value, local file sink will rotate the file - close current file, create a new file and write message set to the new file.
rotationPeriod	local file sink will rotate the file with this period. It can
minPercentFreeDisk	when the disk is filled and the free disk space percentage is getting bigger than this threshold value, local file sink will abort the operation and it will make Suro server stop taking the traffic.
notify	It describes Notify interface for how to notify its progress.

S3 file sink

S3 file sink is embedding local file sink and uploading the file created from its local file sink.

Argument	Description
localFileSink	local file sink configuration
bucket	S3 bucket name
s3Endpoint	S3 bucket end point
maxPartSize	S3 multiple upload max partition size
concurrentUpload	number of uploading threads
notify	It describes Notify interface for how to notify its progress.
prefixFormatter	It describes RemotePrefixFormatter for how to create a file name.