-
Notifications
You must be signed in to change notification settings - Fork 67
Conversation
Caused by appending error log in action result log, which is used by compressionfileinfo.
0816267
to
fd01240
Compare
@PHILO-HE This PR has passed all building checks and UTs. Pls review it ASAP, such that we can merge it with trunk. |
SSM docs should be updated, e.g., updating web help page for action, adding a design doc under SSM/doc. |
public static final String SMART_COMPRESSION_BUFFER_SIZE = "smart.compression.buffer.size"; | ||
public static final int SMART_COMPRESSION_BUFFER_SIZE_DEFAULT = 262144; | ||
public static final String SMART_COMPRESSION_MAX_SPLIT = "smart.compression.max.split"; | ||
public static final int SMART_COMPRESSION_MAX_SPLIT_DEFAULT = 1000; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The config file smart-default.xml should be updated accordingly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added smart.compression.impl into smart-default.
if (srcFile.getLen() == 0) { | ||
compressionFileInfo = new CompressionFileInfo(false, compressionFileState); | ||
} else { | ||
String tempPath = "/tmp/ssm" + filePath + "." + System.currentTimeMillis() + ".ssm_compression"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To keep consistent with EC, we can set the temp dir as /system/ssm/compress_tmp in the scheduler. Then in the scheduler instead of action, the tempPath is set as TEMP_DIR/ + fileName + "" + "aid" + action.getActionId() + "" + System.currentTimeMillis(). Using filePath is not proper. Adding aid is friendly to debug.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Refined this part according to comments.
|
||
public void setDfsClient(DFSClient dfsClient) { | ||
this.dfsClient = dfsClient; | ||
} | ||
|
||
public void setDefaultDfsClient(DFSClient dfsClient) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems that creating defaultDfsClient is unnecessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed.
gson.toJson(compressionInfo.getCompressedPos())); | ||
} | ||
|
||
public void deleteByName(String fileName) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deleteByPath(String filePath)
new CompressFileRowMapper()); | ||
} | ||
|
||
public CompressionFileState getInfoByName(String fileName) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getInfoByPath(String filePath)
+ " type tinyint(4) NOT NULL,\n" | ||
+ " stage tinyint(4) NOT NULL\n" | ||
+ ");", | ||
"CREATE TABLE compression_file (\n" | ||
+ " file_name varchar(512) PRIMARY KEY,\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
file_name -> path
@@ -179,6 +187,16 @@ public FileStatus getFileStatus(Path f) throws IOException { | |||
oldStatus.getOwner(), oldStatus.getGroup(), | |||
oldStatus.isSymlink() ? oldStatus.getSymlink() : null, oldStatus.getPath()); | |||
} | |||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if oldStatus=null? This else code block will throw NPE when oldStatus=null.
@@ -171,6 +185,16 @@ public FileStatus getFileStatus(Path f) throws IOException { | |||
oldStatus.getOwner(), oldStatus.getGroup(), | |||
oldStatus.isSymlink() ? oldStatus.getSymlink() : null, oldStatus.getPath()); | |||
} | |||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if oldStatus=null? This else code block will throw NPE when oldStatus=null.
@@ -171,6 +190,16 @@ public FileStatus getFileStatus(Path f) throws IOException { | |||
oldStatus.getOwner(), oldStatus.getGroup(), | |||
oldStatus.isSymlink() ? oldStatus.getSymlink() : null, oldStatus.getPath()); | |||
} | |||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if oldStatus=null? This else code block will throw NPE when oldStatus=null.
Please check how compression action react for a directory arg. |
No description provided.