dealing with delimiters.txt

At times, we might have to deal with text data where line delimiter is different than new line character. like comments with multiple new line characters.
In this case we need to use HDFS APIs to read data from files with custom line delimiter into RDD and process further (either using transformations/actions or data frame operations)

path = "/public/yelp-dataset/yelp_review.csv"

yelpReview = sc.newAPIHadoopFile(path, 
  'org.apache.hadoop.mapreduce.lib.input.TextInputFormat', 
  'org.apache.hadoop.io.LongWritable', 
  'org.apache.hadoop.io.Text', 
  conf={'textinputformat.record.delimiter' : '\r'})
  
yelpReview.count()

for i in yelpReview.map(lambda r: str(r[1])).take(10): print(i)

for i in yelpReview. \
  map(lambda r: (len(str(r[1]).split('","')), 1)). \
  reduceByKey(lambda x, y: x + y). \
  collect():
  print(i)
  
  
When it comes to field separator, we should use ascii null character '\00', as its very uncommon to have null as a field value.