SPARKNLP-1094 Adding Support to Read Word Files

[danilojsl]

Description

This pull request introduces a new feature that enables reading and parsing Microsoft Word files (.docx and doc) into a structured Spark DataFrame. Leveraging this functionality allows for efficient processing and analysis of word documents content, seamlessly integrating with Spark NLP for enhanced downstream natural language processing tasks.

Key Changes

Added sparknlp.read().email() Method: This method accepts file paths or a file path to parse Email content into a Spark DataFrame.
Support for Varied Sources: The method is designed to handle both local directories, distributed file systems containing email

Motivation and Context

• Structured Data Representation: By transforming word documents content into a well-defined DataFrame structure, we enable seamless integration with Spark’s powerful analytical and data processing capabilities.
• Scalability: Leveraging Spark’s distributed architecture, this feature supports the efficient processing of large volumes of Word documents data, which is critical for big data projects.
• Simplified Data Manipulation: Working with a structured DataFrame simplifies data manipulation tasks, such as filtering, aggregating, and transforming Word documents content, reducing complexity and improving productivity.
• Enhanced Context for LLM Tasks: By organizing Word documents data into structured formats, we can curate and provide more specific, context-rich content for large language model (LLM) prompts. This improves the quality of LLM-generated outputs by allowing for more targeted and relevant contextual information, which is essential for applications like NLU and content generation.

How Has This Been Tested?

Screenshots (if appropriate):

• Local Tests
• Google Colab notebook
• Databricks notebooks

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.