Discussion: Should we align our Named Entity Recognition output with WANE format?

[ruebot]

Adapting our example NER script:

import io.archivesunleashed._
import io.archivesunleashed.app._
import io.archivesunleashed.matchbox._

ExtractEntities.extractFromRecords("/home/ruestn/english.all.3class.distsim.crf.ser.gz", "/tuna1/scratch/nruest/geocites/warcs/1/GEOCITIES-20090723023506-00000-crawling08.us.archive.org.warc.gz", "/tuna1/scratch/nruest/geocites/ner/", sc)

Produces the following example output:

(20090723,http://uk.geocities.com/pendock@btinternet.com/index.htm,{"PERSON":["Frampton","Hardwicke","Hardwicke","Martin","Hardwicke","Hardwicke","Hardwicke","Hutchings","Hopkins","Saunders","Butler","Jones","Frampton","Frampton","Hardwicke","Mark Chapple","Mark Medland","Glos"],"ORGANIZATION":["Hardwicke Cricket Club","Hardwicke Cricket Club","Stroud District Cricket Association","EJ Taylor & Sons Eric Vick Transport Club"],"LOCATION":["China","Gloucester","Ireland","Gloucester"]})

This is very similar to WANE output. Is it worth normalizing the output ExtractEntities produces to the documented WANE output?

JSON object per line: URL ("url"), timestamp ("timestamp"), content digest ("digest") and the named entities ("named_entities") containing data arrays of "persons", "organizations", and "locations".

[ianmilligan1]

I think this is a good idea – to my mind, there aren't any standardized named-entity formats out there, so if there's a format we might as well try to encourage some standardization around tools down the road?

[ianmilligan1]

Seeing no objections, let's do it – I like the idea of saying we produce "WANE" files which we can the point at the Archive-It page. Maybe we can start a trend towards a standardized NER format. 😉

[ruebot]

@ianmilligan1 @lintool

aut/src/main/scala/io/archivesunleashed/app/ExtractEntities.scala

Lines 45 to 63 in 9b3e025

	/** Extracts named entities from tuple-formatted derivatives scraped from a website.
	*
	* @param iNerClassifierFile path of classifier file
	* @param inputFile path of file containing tuples (date: String, url: String, content: String)
	* from which to extract entities
	* @param outputFile path of output directory
	* @return an rdd with classification entities.
	*/
	def extractFromScrapeText(iNerClassifierFile: String, inputFile: String, outputFile: String, sc: SparkContext): RDD[(String, String, String)] = {
	val rdd = sc.textFile(inputFile)
	.map(line => {
	val ind1 = line.indexOf(",")
	val ind2 = line.indexOf(",", ind1 + 1)
	(line.substring(1, ind1),
	line.substring(ind1 + 1, ind2),
	line.substring(ind2 + 1, line.length - 1))
	})
	extractAndOutput(iNerClassifierFile, rdd, outputFile)
	}

Can I remove that function? The function above it solves the use case of parsing ARCs/WARCs, not sure about parsing a relatively unknown, and under-documented file format. Seems a bit out-of-scope for the toolkit, especially since we removed the Twitter analysis/parsing.

[ruebot]

After that commit, I just need to sort out PERSON -> persons, etc., and it should finally be done.

[ianmilligan1]

Can I remove that function? The function above it solves the use case of parsing ARCs/WARCs, not sure about parsing a relatively unknown, and under-documented file format. Seems a bit out-of-scope for the toolkit, especially since we removed the Twitter analysis/parsing.

I'd completely forgotten about this! My vote would be to remove this in the PR. It's an outlier in that it takes derivative files as an input and processes them. I think it makes sense to stick to ARC and WARC files as inputs only, and put the emphasis on users to either use notebooks or their own solutions to work with the derivative files.

[ruebot]

Mostly resolved with 379cc68.

Still need to do: PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations (involves writing a new class or overriding NER output 🤢

Helper note from @helgeho for when I (or somebody else) loops back around to this in the future:

we're not exactly overriding it, because the corenlp output is not json, we simply take the PERSON class and put it under a key that we call persons

[ruebot]

@SinghGursimran this one one last item to get to before it's done. If you're interested, or see any easy path, let me know.

[SinghGursimran]

@ruebot May I simply replace keywords PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations in the NER output String. That will conform to WANE output.
(Overriding class would be a cleaner solution but I guess we won't be requiring that class at any other place)

[ruebot]

@SinghGursimran sure, I'm good with that. Curious what you come up with.

[ianmilligan1]

Fantastic! Great work @SinghGursimran and @ruebot!