Redesign of PySpark DataFrame interface for filtering

[lintool]

Currently, the PySpark DataFrame interface is something like:

	path = "/Users/Prince/Projects/pyaut/aut/example.arc.gz"
	
	spark = SparkSession.builder.appName("filterByDate").getOrCreate()
	sc = spark.sparkContext

	df = RecordLoader.loadArchivesAsDF(path, sc, spark)
	filtered_df = keepDate(df, "2008", DateComponent.YYYY).filter(df['url'].like("%archive%"))
	rdd = filtered_df.rdd
	rdd.map(lambda r: (r.crawlDate, r.domain, r.url, RemoveHTML(r.contentString))) \
	   .saveAsTextFile("out/")

Above snipped from https://github.com/MapleOx/aut/blob/bfb2678e8f88c994f94cc0919f352303f2f1d412/src/main/python/scripts/filterByDateScript.py

We should redesign to use standard NOUN.VERB pattern, something like:

.filter(df['url'].like("%archive%"))
.filter(df['date'].in(DateComponent.YYYY(2008))
.filter(df['date'].in(DateRange.YYYY(2007, 2008)))
.filter(df['date'].in(DateRange.YYYYMM(2007, 1, 2008, 2)))
.filter(df['date'].in(DateRange.YYYYMMDD(2007, 1, 2, 2008, 2, 2)))

[ianmilligan1]

FYI we are writing the scripts over here on a branch of the website (documentation): https://github.com/archivesunleashed/archivesunleashed.org/blob/pyspark/content/aut/pyspark.md.

[ianmilligan1]

@dhop is looking into PySpark!

[ruebot]

Resolved with 505c47a