Add ComputeSHA1 method; resolves #363.

[ruebot]

GitHub issue(s): #363

What does this Pull Request do?

Add ComputeSHA1 method.

How should this be tested?

• Unit tests, and TravisCI should take care of it.

Additional Notes:

Do we want to add this method to all our DataFrame methods? Basically add a sha1 column, along with the md5 column? If so, something like this here:

    /* Extract PDF bytes and PDF metadata. */
    def extractPDFDetailsDF(): DataFrame = {
      val records = rdd
        .map(r =>
            (r, (DetectMimeTypeTika(r.getBinaryBytes)))
            )
        .filter(r => r._2 == "application/pdf")
        .map(r => {
          val bytes = r._1.getBinaryBytes
          val md5Hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
          val sha1Hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
          val encodedBytes = Base64.getEncoder.encodeToString(bytes)
          val url = new URL(r._1.getUrl)
          val filename = FilenameUtils.getName(url.getPath())
          val extension = GetExtensionMime(url.getPath(), r._2)
          (r._1.getUrl, filename, extension, r._1.getMimeType,
            DetectMimeTypeTika(r._1.getBinaryBytes), md5Hash, sha1Hash, encodedBytes)
        })
        .map(t => Row(t._1, t._2, t._3, t._4, t._5, t._6, t._7, t._8))

      val schema = new StructType()
        .add(StructField("url", StringType, true))
        .add(StructField("filename", StringType, true))
        .add(StructField("extension", StringType, true))
        .add(StructField("mime_type_web_server", StringType, true))
        .add(StructField("mime_type_tika", StringType, true))
        .add(StructField("md5", StringType, true))
        .add(StructField("sha1", StringType, true))
        .add(StructField("bytes", StringType, true))

      val sqlContext = SparkSession.builder();
      sqlContext.getOrCreate().createDataFrame(records, schema)
    }

[codecov]

Codecov Report

Merging #364 into master will increase coverage by 0.3%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #364     +/-   ##
=========================================
+ Coverage   75.93%   76.24%   +0.3%     
=========================================
  Files          39       40      +1     
  Lines        1392     1410     +18     
  Branches      267      267             
=========================================
+ Hits         1057     1075     +18     
  Misses        218      218             
  Partials      117      117

[ianmilligan1]

Builds nicely locally.

Re:

Do we want to add this method to all our DataFrame methods? Basically add a sha1 column, along with the md5 column? If so, something like this here:

That makes sense to me?

[ruebot]

I'll update the bleeding edge documentation once we're good to go here.

[ianmilligan1]

New commit looks great! Runs nicely with this script:

import io.archivesunleashed._
import io.archivesunleashed.df._

val df = RecordLoader.loadArchives("example.arc.gz", sc).extractImageDetailsDF();
df.select($"url", $"filename", $"extension", $"mime_type_web_server", $"mime_type_tika", $"width", $"height", $"md5", $"sha1", $"bytes").orderBy(desc("md5")).show()

[ruebot]

Updated! https://github.com/archivesunleashed/aut/wiki/Bleeding-Edge/_compare/3512e6e842bce50e1959fd35e739c2f02cf772b6