Finding Hyperlinks within Collection on Pages with Certain Keyword

[SinghGursimran]

Extract hyperlinks within a collection filtered on pages containing a particular keyword (case insensitive) using df.

import io.archivesunleashed._
import io.archivesunleashed.df._

val result = udf((vs: Seq[Any]) => vs(0).toString.split(",")(1))

val df= RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
.select(     RemovePrefixWWW(ExtractDomain($"url")).as("Domain"),
  	        $"url".as("url"),
  	        $"crawl_date",
  	        explode_outer(ExtractLinks($"url",$"content")).as("link")
  	  )
.filter(lower($"content").contains("internet")) //filtered on keyword "internet"

df.select($"url",$"Domain",$"crawl_date",result(array($"link")).as("destination_page"))
.write
.option("header","true")
.csv("filtered_results/")

#238

Returns a csv file with URL, Domain, crawl_date, and destination_page.

[codecov]

Codecov Report

Merging #377 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #377      +/-   ##
==========================================
+ Coverage   76.36%   76.37%   +0.01%     
==========================================
  Files          40       40              
  Lines        1413     1414       +1     
  Branches      268      268              
==========================================
+ Hits         1079     1080       +1     
  Misses        217      217              
  Partials      117      117

[ruebot]

Looks good!

import io.archivesunleashed._
import io.archivesunleashed.df._

val result = udf((vs: Seq[Any]) => vs(0).toString.split(",")(1))

val df= RecordLoader.loadArchives("/home/nruest/Projects/au/aut-resources/Sample-Data/*gz", sc).extractValidPagesDF()
.select(RemovePrefixWWW(ExtractDomain($"url")).as("Domain"),$"url".as("url"),$"crawl_date",explode_outer(ExtractLinks($"url",$"content")).as("link"))
.filter($"content".contains("keystone"))

df.select($"url",$"Domain",$"crawl_date",result(array($"link")).as("destination_page"))
.show()

// Exiting paste mode, now interpreting.

+--------------------+---------------+----------+--------------------+          
|                 url|         Domain|crawl_date|    destination_page|
+--------------------+---------------+----------+--------------------+
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
|http://www.davids...|davidsuzuki.org|  20091219|http://www.davids...|
+--------------------+---------------+----------+--------------------+
only showing top 20 rows

import io.archivesunleashed._
import io.archivesunleashed.df._
result: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,StringType,None)
df: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [Domain: string, url: string ... 2 more fields]

^^^ @ianmilligan1 that what you're looking for? If so, I'll squash and merge, and add this to the cookbook section if that makes sense.

[lintool]

hey @ruebot is this something that should be encoded in a test case while we're at it?

[ruebot]

@lintool we already have a test case for ExtractLinks, unless you're thinking of something else.

[lintool]

@ruebot add a separate test case that explicitly include filtering? Maybe not. I dunno.

[ruebot]

Yeah, we could add a new test that add filter or just add a filter to https://github.com/archivesunleashed/aut/blob/e32ae17c55740c6c6adc177d42690b39fa6321dd/src/test/scala/io/archivesunleashed/matchbox/ExtractLinksTest.scala if that's what you're thinking.

...and if that's the case, @SinghGursimran, want to update the PR with an updated test?

[ianmilligan1]

@ruebot Looks perfect to me - and I love the sample results with "keystone" leading to David Suzuki. Thanks @SinghGursimran, great work.

[SinghGursimran]

Yeah, we could add a new test that add filter or just add a filter to https://github.com/archivesunleashed/aut/blob/e32ae17c55740c6c6adc177d42690b39fa6321dd/src/test/scala/io/archivesunleashed/matchbox/ExtractLinksTest.scala if that's what you're thinking.

...and if that's the case, @SinghGursimran, want to update the PR with an updated test?

Should I add a separate test for ExtractLink udf with a filter?

[ruebot]

@SinghGursimran yeah, why not. Let's go with that.