More Serializable APIs for DataFrames

[SinghGursimran]

More Serializable APIs for DataFrames

#223

For Testing:

keepHttpStatusDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.all()
			.select($"url",$"crawl_date",$"HttpStatus")
			.keepHttpStatusDF(Set("200"))
			.show(10,false)

keepDateDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webpages()
			.select($"url",$"crawl_date")
			.keepDateDF(List("04"),"MM")
			.show(10,false)

keepUrlsDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.all()
			.select($"url",$"crawl_date",$"HttpStatus")
			.keepUrlsDF(Set("http://www.archive.org/","http://www.archive.org/robots.txt"))
			.show(10,false)

keepDomainsDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.all()
			.select($"url",$"crawl_date",$"HttpStatus")
			.keepDomainsDF(Set("www.archive.org"))
			.show(10,false)

[codecov]

Codecov Report

Merging #396 into master will increase coverage by 0.07%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #396      +/-   ##
==========================================
+ Coverage   77.11%   77.19%   +0.07%     
==========================================
  Files          40       40              
  Lines        1486     1495       +9     
  Branches      280      280              
==========================================
+ Hits         1146     1154       +8     
- Misses        217      218       +1     
  Partials      123      123