-
Notifications
You must be signed in to change notification settings - Fork 170
Sink configuration
Jae Hyeon Bae edited this page May 9, 2013
·
10 revisions
Sink configuration is also Json map string whose key is the id of the sink and the value is sink configuration.
Local file sink is writing message set to the local file system.
| Argument | type | Description |
|---|---|---|
| outputDir | String | directory path where files are stored |
| writer | FileWriter | Its type can be 'text' or 'sequence'. 'text' means text file and 'sequence' means Hadoop sequence file. It is getting codec argument denoting Hadoop codec class. |
| maxFileSize | long | when the file size reaches to this value, local file sink will rotate the file - close current file, create a new file and write message set to the new file. Its Json type is long. |
| rotationPeriod | String | local file sink will rotate the file with this period. It can be ISO period string such as PT10m or milliseconds value. |
| minPercentFreeDisk | int | when the disk is filled and the free disk space percentage is getting bigger than this threshold value, local file sink will abort the operation and it will make Suro server stop taking the traffic. |
| notify | Notify | It describes Notify interface for how to notify its progress. |
The following is the example local file sink configuration
S3 file sink is embedding local file sink and uploading the file created from its local file sink.
| Argument | Description |
|---|---|
| localFileSink | local file sink configuration |
| bucket | S3 bucket name |
| s3Endpoint | S3 bucket end point |
| maxPartSize | S3 multiple upload max partition size |
| concurrentUpload | number of uploading threads |
| notify | It describes Notify interface for how to notify its progress. |
| prefixFormatter | It describes RemotePrefixFormatter for how to create a file name. |