-
Notifications
You must be signed in to change notification settings - Fork 25
/
ngrams_with_pyspark.py
224 lines (143 loc) · 659 KB
/
ngrams_with_pyspark.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
# This file was generated from ngrams_with_pyspark.ipynb with nbconvert
# Source: https://github.com/groda/big_data
#!/usr/bin/env python
# coding: utf-8
# <a href="https://colab.research.google.com/github/groda/big_data/blob/master/ngrams_with_pyspark.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
# <a href="https://github.com/groda/big_data"><div><img src="https://github.com/groda/big_data/blob/master/logo_bdb.png?raw=true" align=right width="90" alt="Logo Big Data for Beginners"></div></a>
# # N-grams with PySpark
#
#
# In this notebook we are going to extract _n-grams_ from a text file using PySpark.
#
# N-grams in Natural Language Processing refer to sequences of tokens (or words) used to calculate the probability of a word given its preceding context. In a bigram model, the prediction for the next word relies on the single preceding word, while in a trigram model, it considers the two preceding words, and so forth.
#
# We will utilize the PySpark distribution provided by `pip` with its integrated Spark engine running as a single Java virtual machine in _pseudo-distributed mode_.
#
# More questions on PySpark are answered in [PySpark on Google Colab](https://github.com/groda/big_data/blob/master/PySpark_On_Google_Colab.ipynb).
# # Install PySpark
# In[1]:
get_ipython().system('pip install pyspark')
# # Download a text file
# In[2]:
FILENAME = 'don_quixote.txt'
get_ipython().system('wget --no-clobber https://www.gutenberg.org/cache/epub/996/pg996.txt -O {FILENAME}')
# # Spark Web UI
#
# The Spark Web UI
#
# > “*provides a suite of web user interfaces (UIs) that you can use to monitor the status and resource consumption of your Spark cluster.*” (https://spark.apache.org/docs/3.5.0/web-ui.html)
#
# See more in the official [Spark Documentation](https://spark.apache.org/docs/3.5.0/web-ui.html).
#
# ## Set `SPARK_HOME`
# In[3]:
import subprocess
import os
# Run the script and capture its output
result = subprocess.run(["find_spark_home.py"], capture_output=True, text=True)
# Print or use the captured output
print("Output of find_spark_home.py:", result.stdout)
# set SPARK_HOME environment variable
os.environ['SPARK_HOME'] = result.stdout.strip()
# ## Import `output` library if in Colab
#
# The Web UI can be viewed in a separate windows or tab thanks to Colab's `output` library.
#
# In[4]:
# true if running on Google Colab
import sys
IN_COLAB = 'google.colab' in sys.modules
if IN_COLAB:
from google.colab import output
# ### Spark UI
# In[5]:
if IN_COLAB:
output.serve_kernel_port_as_window(4040, path='/jobs/index.html')
# # Create a Spark context
# In[6]:
from pyspark import SparkContext
sc = SparkContext(
appName = "Ngrams with pyspark " + FILENAME
)
# ## View Spark context
# In[7]:
sc
# # Create RDD from file
#
# The second parameter ($80$) indicates the desired number of partitions.
# In[8]:
textFile = sc.textFile(FILENAME, 80)
print("textFile is of type: {}
Number of partitions: {}".format(type(textFile), textFile.getNumPartitions()))
# # Generate trigrams
#
# Change `n` to generate n-grams.
# In[9]:
from operator import add
n = 3
ngrams = textFile \
.flatMap(lambda x: [x.split()]) \
.flatMap(lambda x: [tuple(y) for y in zip(*[x[i:] for i in range(n)])]) \
.map(lambda x: (x, 1)) \
.reduceByKey(add) \
.sortBy(lambda x: x[1], ascending=False)
# In[10]:
for (ngram, count) in ngrams.take(10):
print("%s: %i" % (ngram, count))
# # View job in the Web UI
#
# Here's the link:
# In[11]:
if IN_COLAB:
output.serve_kernel_port_as_window(4040, path='/jobs/index.html')
# 
# # Stop context
#
# Stop the current Spark context to free resources. You also need to stop the current context to be able to create a new one.
# In[12]:
sc.stop()
# # Appendix: ChatGPT on n-grams
#
# My prompt was: "who better than you to answer the following question? Where can I locate n-grams in the timeline of nlp/language models?"
# In[13]:
# @title
from IPython.core.display import HTML
HTML("""
<div style="background-color:rgb(16, 163, 127,.2);border:2px solid rgb(16, 163, 127,.3);padding:3px;">
<svg fill="none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 320 320" style="width:32px;height:32px;">
<g fill="currentColor">
<path d="m297.06 130.97c7.26-21.79 4.76-45.66-6.85-65.48-17.46-30.4-52.56-46.04-86.84-38.68-15.25-17.18-37.16-26.95-60.13-26.81-35.04-.08-66.13 22.48-76.91 55.82-22.51 4.61-41.94 18.7-53.31 38.67-17.59 30.32-13.58 68.54 9.92 94.54-7.26 21.79-4.76 45.66 6.85 65.48 17.46 30.4 52.56 46.04 86.84 38.68 15.24 17.18 37.16 26.95 60.13 26.8 35.06.09 66.16-22.49 76.94-55.86 22.51-4.61 41.94-18.7 53.31-38.67 17.57-30.32 13.55-68.51-9.94-94.51zm-120.28 168.11c-14.03.02-27.62-4.89-38.39-13.88.49-.26 1.34-.73 1.89-1.07l63.72-36.8c3.26-1.85 5.26-5.32 5.24-9.07v-89.83l26.93 15.55c.29.14.48.42.52.74v74.39c-.04 33.08-26.83 59.9-59.91 59.97zm-128.84-55.03c-7.03-12.14-9.56-26.37-7.15-40.18.47.28 1.3.79 1.89 1.13l63.72 36.8c3.23 1.89 7.23 1.89 10.47 0l77.79-44.92v31.1c.02.32-.13.63-.38.83l-64.41 37.19c-28.69 16.52-65.33 6.7-81.92-21.95zm-16.77-139.09c7-12.16 18.05-21.46 31.21-26.29 0 .55-.03 1.52-.03 2.2v73.61c-.02 3.74 1.98 7.21 5.23 9.06l77.79 44.91-26.93 15.55c-.27.18-.61.21-.91.08l-64.42-37.22c-28.63-16.58-38.45-53.21-21.95-81.89zm221.26 51.49-77.79-44.92 26.93-15.54c.27-.18.61-.21.91-.08l64.42 37.19c28.68 16.57 38.51 53.26 21.94 81.94-7.01 12.14-18.05 21.44-31.2 26.28v-75.81c.03-3.74-1.96-7.2-5.2-9.06zm26.8-40.34c-.47-.29-1.3-.79-1.89-1.13l-63.72-36.8c-3.23-1.89-7.23-1.89-10.47 0l-77.79 44.92v-31.1c-.02-.32.13-.63.38-.83l64.41-37.16c28.69-16.55 65.37-6.7 81.91 22 6.99 12.12 9.52 26.31 7.15 40.1zm-168.51 55.43-26.94-15.55c-.29-.14-.48-.42-.52-.74v-74.39c.02-33.12 26.89-59.96 60.01-59.94 14.01 0 27.57 4.92 38.34 13.88-.49.26-1.33.73-1.89 1.07l-63.72 36.8c-3.26 1.85-5.26 5.31-5.24 9.06l-.04 89.79zm14.63-31.54 34.65-20.01 34.65 20v40.01l-34.65 20-34.65-20z"></path>
</svg>
N-grams have been a fundamental concept in natural language processing (NLP) and language modeling for quite some time. Here's a brief overview of their timeline:
<ol>
<li><h3>Early Days (1950s-1960s):</h3></li> The concept of n-grams was introduced in the early days of computational linguistics and information theory. Researchers like Claude Shannon laid the groundwork for the statistical analysis of language.
<li><h3>Traditional NLP (1970s-1990s):</h3></li> N-grams became popular in traditional NLP during this period. Researchers used n-grams as a basic unit for language modeling and to capture local context in text. Simple n-gram models were widely used in tasks like machine translation and speech recognition.
<li><h3>Statistical Language Models (1990s-early 2000s):</h3></li> N-grams played a crucial role in statistical language modeling. Techniques like N-gram models and Hidden Markov Models were prevalent. These models formed the backbone of various NLP applications.
<li><h3>Advent of Neural Networks (2010s):</h3></li> With the resurgence of neural networks, traditional n-gram models faced competition from more sophisticated approaches. Neural network-based language models, such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), started to outperform traditional n-gram models by capturing long-range dependencies in language.
<li><h3>Deep Learning and Sequence Models (2010s-2020s):</h3></li> Transformer-based architectures, like OpenAI's GPT series, further revolutionized NLP. These models, based on self-attention mechanisms, are capable of capturing contextual information effectively, making them more powerful than n-gram models. However, n-grams are still used in certain contexts and as components of more advanced models.
<li><h3>Current State (2020s and beyond):</h3></li> State-of-the-art language models, such as GPT-3 and its successors, leverage large-scale transformer architectures and pre-training on massive datasets. These models have largely surpassed the need for explicit n-gram modeling, as they can capture complex language patterns and dependencies across long sequences.
While n-grams were crucial in the early development of language models, the field has progressed significantly, and contemporary models have evolved beyond simple n-gram approaches. Nonetheless, understanding n-grams remains essential for foundational knowledge in NLP and language modeling.
</div>
""")
# ## What is the difference between n-grams and modern language models?
#
# The distinction between n-grams and more modern models like transformers lies in their underlying modeling paradigms. N-grams rely on traditional statistical methods, specifically frequency counts, for language modeling. In contrast, transformers leverage deep learning techniques, which include complex neural network architectures, embeddings, and attention mechanisms. While both approaches involve statistical concepts, transformers harness the power of deep learning to capture intricate patterns and dependencies within data, allowing for more sophisticated language understanding and generation
#
# ## Comparing model sizes: a rough estimate of N-gram versus Transformer dimensions
#
# An interesting point to note is that n-gram models have an order of magnitude smaller storage footprint compared to transformer models, especially in terms of model size. This discrepancy is not surprising given the fact that transformers learn more intricate patterns, relationships, and context-aware representations, resulting in significantly larger model sizes.
#
# Here are some rough estimates of model sizes just to get an idea:
#
#
#
# * **N-gram Model:** With a trigram model and a vocabulary of $50,000$ words, the model size might be in the megabyte range.
# This is because the model primarily needs to store the counts for each trigram in the vocabulary, which is relatively straightforward.
# * **Transformer Model:**
# Even for smaller transformer models with a vocabulary of $50,000$ words, the model size can be in the gigabyte range.
# Transformers have a large number of parameters, and the model size increases significantly with model depth, attention heads, and the embedding dimension.
# For instance, GPT-3, a large transformer model, has 175 billion parameters and a model size that's several gigabytes.
#
#
#
#