Merge SSM Compression to trunk #1873

qiyuangong · 2018-07-27T07:40:36Z

No description provided.

…mpression

Caused by appending error log in action result log, which is used by compressionfileinfo.

qiyuangong · 2019-01-22T00:15:28Z

@PHILO-HE This PR has passed all building checks and UTs. Pls review it ASAP, such that we can merge it with trunk.

PHILO-HE · 2019-01-24T03:36:22Z

SSM docs should be updated, e.g., updating web help page for action, adding a design doc under SSM/doc.

PHILO-HE · 2019-01-24T03:01:15Z

smart-common/src/main/java/org/smartdata/conf/SmartConfKeys.java

+  public static final String SMART_COMPRESSION_BUFFER_SIZE = "smart.compression.buffer.size";
+  public static final int SMART_COMPRESSION_BUFFER_SIZE_DEFAULT = 262144;
+  public static final String SMART_COMPRESSION_MAX_SPLIT = "smart.compression.max.split";
+  public static final int SMART_COMPRESSION_MAX_SPLIT_DEFAULT = 1000;


The config file smart-default.xml should be updated accordingly.

Added smart.compression.impl into smart-default.

PHILO-HE · 2019-01-24T03:22:05Z

...t-hadoop-support/smart-hadoop/src/main/java/org/smartdata/hdfs/action/CompressionAction.java

+    if (srcFile.getLen() == 0) {
+      compressionFileInfo = new CompressionFileInfo(false, compressionFileState);
+    } else {
+      String tempPath = "/tmp/ssm" + filePath + "." + System.currentTimeMillis() + ".ssm_compression";


To keep consistent with EC, we can set the temp dir as /system/ssm/compress_tmp in the scheduler. Then in the scheduler instead of action, the tempPath is set as TEMP_DIR/ + fileName + "" + "aid" + action.getActionId() + "" + System.currentTimeMillis(). Using filePath is not proper. Adding aid is friendly to debug.

Refined this part according to comments.

PHILO-HE · 2019-01-24T03:25:52Z

...t-hadoop-support/smart-hadoop-common/src/main/java/org/smartdata/hdfs/action/HdfsAction.java


  public void setDfsClient(DFSClient dfsClient) {
    this.dfsClient = dfsClient;
  }

+  public void setDefaultDfsClient(DFSClient dfsClient) {


It seems that creating defaultDfsClient is unnecessary.

PHILO-HE · 2019-01-24T07:25:56Z

smart-metastore/src/main/java/org/smartdata/metastore/dao/CompressionFileDao.java

+        gson.toJson(compressionInfo.getCompressedPos()));
+  }
+
+  public void deleteByName(String fileName) {


deleteByPath(String filePath)

PHILO-HE · 2019-01-24T07:27:09Z

smart-metastore/src/main/java/org/smartdata/metastore/dao/CompressionFileDao.java

+        new CompressFileRowMapper());
+  }
+
+  public CompressionFileState getInfoByName(String fileName) {


getInfoByPath(String filePath)

PHILO-HE · 2019-01-24T07:27:35Z

smart-metastore/src/main/java/org/smartdata/metastore/utils/MetaStoreUtils.java

              + " type tinyint(4) NOT NULL,\n"
              + " stage tinyint(4) NOT NULL\n"
              + ");",
+          "CREATE TABLE compression_file (\n"
+              + " file_name varchar(512) PRIMARY KEY,\n"


file_name -> path

PHILO-HE · 2019-01-24T08:08:14Z

...art-hadoop-client-cdh-2.6/src/main/java/org/smartdata/hadoop/filesystem/SmartFileSystem.java

@@ -179,6 +187,16 @@ public FileStatus getFileStatus(Path f) throws IOException {
            oldStatus.getOwner(), oldStatus.getGroup(),
            oldStatus.isSymlink() ? oldStatus.getSymlink() : null, oldStatus.getPath());
      }
+    } else {


What if oldStatus=null? This else code block will throw NPE when oldStatus=null.

PHILO-HE · 2019-01-24T08:17:21Z

...t/smart-hadoop-client-3.1/src/main/java/org/smartdata/hadoop/filesystem/SmartFileSystem.java

@@ -171,6 +185,16 @@ public FileStatus getFileStatus(Path f) throws IOException {
            oldStatus.getOwner(), oldStatus.getGroup(),
            oldStatus.isSymlink() ? oldStatus.getSymlink() : null, oldStatus.getPath());
      }
+    } else {


What if oldStatus=null? This else code block will throw NPE when oldStatus=null.

PHILO-HE · 2019-01-24T08:18:03Z

...t/smart-hadoop-client-2.7/src/main/java/org/smartdata/hadoop/filesystem/SmartFileSystem.java

@@ -171,6 +190,16 @@ public FileStatus getFileStatus(Path f) throws IOException {
            oldStatus.getOwner(), oldStatus.getGroup(),
            oldStatus.isSymlink() ? oldStatus.getSymlink() : null, oldStatus.getPath());
      }
+    } else {


What if oldStatus=null? This else code block will throw NPE when oldStatus=null.

PHILO-HE · 2019-01-25T02:05:51Z

Please check how compression action react for a directory arg.

timmyyao and others added 30 commits October 30, 2017 15:09

init version

57a0591

SmartDFSOutputStream not yet complete

09e5198

revert SmartDFSOutputStream to original version to pass UT

8b7a0ca

CompressionAction

a64c2d1

Fix bug in CompressionAction and refactor SmartFileCompressionInfo

4448237

UT for CompressionAction

d14c620

CompressionScheduler

6dce6ac

Merge branch 'compression' of https://github.com/timmyyao/SSM into co…

1abd3e8

…mpression

init SmartDFSInputStream

f7e1e25

refomat TestCompressionAction.java

87bab0f

checkstyle

ba8a96a

Refactor MiniClusterHarness and init version of TestCompressionScheduler

43be281

determine whether a file needs to be compressed by the path of file

926ea4f

Merge branch 'compression' of https://github.com/timmyyao/SSM into co…

0b60685

…mpression

UT TestCompressionScheduler

3648e3e

Complete SmartDFSInputStream

10aa7d2

RPC and UT for SmartDFSInputStream

e48f002

Add fileLock in CompressionScheduler

0c4f250

Fix replication and block size after compression

427333b

metastore add two new fields originalPos and compressedPos

2428480

UT for CompressionFileDao

8135907

UT for CompressionFileDao

74f00f1

Random read for compressed file

afc3995

init version of SmartFileSystem for compressed file split

d583084

Refactor TestCompressionAction

8c51d90

Refactor postion mapping in SmartFileCompressionInfo

89c4a7e

CompressionTrunk and implement methods in SmartFileCompressionInfo

9e14756

Complete SmartFileSystem.listLocatedStatus

15960fc

testListLocatedStatus

f8f1b86

change the tmp generated compressionfile path

a381a1a

qiyuangong added 6 commits September 11, 2018 14:27

Add several keys into SmartConfKeys

81ca366

Fix buff Size problem in compressFileState

e38c6d2

Fix a bug in compresssionscheduler

9d14e74

Caused by appending error log in action result log, which is used by compressionfileinfo.

Modify log

a13ae80

Fix snappy buffer problem

17e4b86

Fix length problem in dfs command

fd01240

PHILO-HE force-pushed the trunk branch from 7fa9466 to 58123de Compare December 19, 2018 02:36

PHILO-HE force-pushed the compression-rebase branch 2 times, most recently from 0816267 to fd01240 Compare December 19, 2018 03:17

qiyuangong added 3 commits January 15, 2019 22:04

Merge branch 'trunk' (b2c293a) into compression-rebase

daad2e7

Fix building issue in 2.7

d35325a

Fix building issue with Hadoop 3.1

f69a99b

qiyuangong changed the title ~~[Do not merge] Test merge Compression to trunk~~ Merge SSM Compression to trunk Jan 16, 2019

qiyuangong added 7 commits January 16, 2019 13:14

Fix nullPoint in TestCompression and fix CDH support

d1543f4

Fix code style

cce231f

Fix CDH 2.6 and Hadoop 3.1 UT

0deec10

Fix code style and add comments

a7dbdaa

Add 3.1 support

5d5ef07

Add CDH-2.6 support

97fefcc

Fix open in CDH-2.6

2dc3d75

qiyuangong requested a review from PHILO-HE January 22, 2019 08:24

PHILO-HE reviewed Jan 24, 2019

View reviewed changes

qiyuangong added 2 commits January 28, 2019 11:26

Refine compression based on Feilong's comments

f18dfee

Fix UT fails caused by defaultDfsClient

d217f7e

qiyuangong merged commit 85db2e5 into trunk Jan 28, 2019

qiyuangong deleted the compression-rebase branch January 28, 2019 14:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge SSM Compression to trunk #1873

Merge SSM Compression to trunk #1873

qiyuangong commented Jul 27, 2018

qiyuangong commented Jan 22, 2019

PHILO-HE commented Jan 24, 2019

PHILO-HE Jan 24, 2019

qiyuangong Jan 28, 2019

PHILO-HE Jan 24, 2019

qiyuangong Jan 28, 2019

PHILO-HE Jan 24, 2019

qiyuangong Jan 28, 2019

PHILO-HE Jan 24, 2019

PHILO-HE Jan 24, 2019

PHILO-HE Jan 24, 2019

PHILO-HE Jan 24, 2019

PHILO-HE Jan 24, 2019

PHILO-HE Jan 24, 2019

PHILO-HE commented Jan 25, 2019

Merge SSM Compression to trunk #1873

Merge SSM Compression to trunk #1873

Conversation

qiyuangong commented Jul 27, 2018

qiyuangong commented Jan 22, 2019

PHILO-HE commented Jan 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PHILO-HE commented Jan 25, 2019