Add additional filters for fextFiles; resolves #362.

[ruebot]

GitHub issue(s):

• DataFrame error with text files: java.net.MalformedURLException: unknown protocol: filedesc #362

What does this Pull Request do?

Add additional filters for fextFiles; resolves #362.

• Add filedesc, and dns filter (arc files)
• Add test case

You can see filedesc and dns in the ARC test fixtures:

import io.archivesunleashed._
import io.archivesunleashed.df._

val df = RecordLoader.loadArchives("/home/nruest/Projects/au/aut/src/test/resources/arc/*gz",sc)
df.all().select("url").show(5, false)

+-------------------------------------------------+
|url                                              |
+-------------------------------------------------+
|filedesc://IAH-20080430204825-00000-blackbook.arc|
|dns:www.archive.org                              |
|http://www.archive.org/robots.txt                |
|http://www.archive.org/                          |
|http://www.archive.org/index.php                 |
+-------------------------------------------------+
only showing top 5 rows

How should this be tested?

The updated test catches the above examples examples.

I'm doing a more robust test on the BANQ collection in question on #362 now. I'll move this out of draft if it is successful.

[ruebot]

Testing with:

import io.archivesunleashed._
import io.archivesunleashed.df._

val df = RecordLoader.loadArchives("/data/banq-datathon/PQ/warcs/*gz", sc).textFiles();

df.select($"url", $"filename", $"extension", $"mime_type_web_server", $"mime_type_tika", $"md5").orderBy(desc("md5")).write.parquet("/data/banq-datathon/PQ/derivatives/parquet/text")

df.select($"bytes", $"extension").saveToDisk("bytes", "/data/banq-datathon/PQ/derivatives/binaries/text/pq-2012-text", "extension")

sys.exit

[codecov]

Codecov Report

Merging #393 into master will decrease coverage by 0.03%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master     #393      +/-   ##
==========================================
- Coverage   77.15%   77.11%   -0.04%     
==========================================
  Files          40       40              
  Lines        1484     1486       +2     
  Branches      278      280       +2     
==========================================
+ Hits         1145     1146       +1     
  Misses        217      217              
- Partials      122      123       +1

[ruebot]

Successfully ran the job on the BANQ dataset twice (with both commits) without issue.

[ianmilligan1]

Tested locally and looks great!