In this project, a Support Vector Classifier was built and tuned to identify mushrooms edibility. A mushroom is classified as edible or poisonous with given color, habitat, class, and others. The final classifier performed quite well on unseen test data, with a final overall accuracy of 0.99 and $F_{\beta}$ score with $\beta = 2$ of 0.99. Furthermore, we use confusion matrix to show the accuracy of classification poisonous or edible mushroom. The model makes 12174 correct predictions out of 12214 test observations. 17 mistakes were predicting a poisonous mushroom as edible (false negative), while 23 mistakes were predicting a edible mushroom as poisonous (false positive). The model’s performance shows promise for implementation, prioritizing safety by minimizing false negatives that could result in consuming poisonous mushrooms. While false positives may lead to unnecessarily discarding safe mushrooms, they pose no safety risk. Further development is needed to make this model useful. Research should focus on improving performance and analyzing cases of incorrect predictions.
Mushrooms are the most common food which is rich in vitamins and minerals. However, not all mushrooms can be consumed directly, most of them are poisonous and identifying edible or poisonous mushroom through the naked eye is quite difficult. Our aim is to using machine learning to identify mushrooms edibility. In this project, three methods are used to detect the edibility of mushrooms: Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Logistic Regression.
The dataset used in this project is the Secondary Mushroom Dataset created by Wagner, D., Heider, D., & Hattab, G. from UCI Machine Learning Repository. This dataset contains 61069 hypothetical mushrooms with caps based on 173 species (353 mushrooms per species). Each mushroom is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended (the latter class was combined with the poisonous class).
The mushroom dataset is balanced with 56% of poisonous mushroom and 44% of edible mushroom. All variables were standardized and variables with more than 15% missing values are dropped, because imputing a variable that has a significant proportion of missing data might introduce too much noise or bias, making it unreliable. Data was splitted with 80% being partitioned into the training set and 20% being partitioned into the test set. Three classification models including Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Logistic Regression are used to predict whether a mushroom is edible or poisonous. The fine tuned Support Vector Classifier has the best overall performance. The hyperparameter was chosen using 5-fold cross validation with $F_{\beta}$ score as the classification metric. $\beta$ was chosen to be set to 2 for the $F_{\beta}$ score to increase the weight on recall during fitting because predicting a mushroom to be edible when it is in fact poisonous could have severe health consequences. Therefore the goal is to prioritize the minimization of false negatives. The Python programming language (Van Rossum and Drake 2009) and the following Python packages were used to perform the analysis: Matplotlib (Hunter, 2007), Pandas (McKinney, 2010), Scikit-learn (Pedregosa et al., 2011), NumPy (Harris et al., 2020), SciPy (Virtanen et al., 2020), UCIMLRepo.
The EDA shows that all numeric columns in the mushroom dataset are nearly normal with some skewness. A robust preprocessing scheme QuantileTransformer is used because it can transform skewed data or heavy-tailed distributions into a more Gaussian-like shape and reduce the impact of outliers.
+OneHotEncoder is applied for categorical features in the mushroom dataset, because each feature does not contains much categories and they are not ordered. It is critical to keep all important information in the features. Since ring type feature has many missing values, it was filled in with a "Missing" class. Treating missing values as a distinct category provides a way to model the absence of data directly. This can be valuable because missingness itself might carry information.
# fetch dataset as pandas DataFrames
+secondary_mushroom=fetch_ucirepo(id=848)
+X=secondary_mushroom.data.features
+y=secondary_mushroom.data.targets
+
+
+
+
+
+
+
+
+
+
+
+
+
Before splitting the data into test and training sets, we want to check for missing values in each column to determine whether they can be used in our model.
+
+
+
+
+
+
+
+
+
In [3]:
+
+
+
# Check the missing values
+missing_values=X.isnull().sum().reset_index()
+missing_values.columns=['Column','Missing Count']
+
+# Highlight values with a gradient
+styled_missing=missing_values.style.format(
+ precision=0
+).background_gradient(
+ subset=['Missing Count'],
+ cmap='YlOrRd'
+).set_caption("Missing Values by Column")
+
+# Display the styled DataFrame
+display(styled_missing)
+
After examining the data set, we decided to drop columns with a high proportion of missing values (over 15%), which include cap-surface, gill-attachment, gill-spacing, stem-root, stem-surface, veil-type, veil-color, and spore-print-color.
+
+
+
+
+
+
+
+
+
In [5]:
+
+
+
# Split the data test and training set
+
+X_train,X_test,y_train,y_test=train_test_split(
+ X,y,test_size=0.2,random_state=123
+)
+
----------------------------------------
+
+Frequency and Percentage for 'cap-color':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
cap-color
+
+
+
+
+
+
+
n
+
19399
+
39.71
+
+
+
y
+
6841
+
14.00
+
+
+
w
+
6149
+
12.59
+
+
+
g
+
3490
+
7.14
+
+
+
e
+
3225
+
6.60
+
+
+
o
+
2943
+
6.02
+
+
+
r
+
1429
+
2.92
+
+
+
p
+
1391
+
2.85
+
+
+
u
+
1370
+
2.80
+
+
+
k
+
999
+
2.04
+
+
+
b
+
969
+
1.98
+
+
+
l
+
650
+
1.33
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+Frequency and Percentage for 'does-bruise-or-bleed':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
does-bruise-or-bleed
+
+
+
+
+
+
+
f
+
40424
+
82.74
+
+
+
t
+
8431
+
17.26
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+Frequency and Percentage for 'gill-color':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
gill-color
+
+
+
+
+
+
+
w
+
14878
+
30.45
+
+
+
n
+
7716
+
15.79
+
+
+
y
+
7655
+
15.67
+
+
+
p
+
4738
+
9.70
+
+
+
g
+
3310
+
6.78
+
+
+
f
+
2825
+
5.78
+
+
+
o
+
2312
+
4.73
+
+
+
k
+
1909
+
3.91
+
+
+
r
+
1134
+
2.32
+
+
+
e
+
825
+
1.69
+
+
+
u
+
807
+
1.65
+
+
+
b
+
746
+
1.53
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+Frequency and Percentage for 'stem-color':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
stem-color
+
+
+
+
+
+
+
w
+
18445
+
37.75
+
+
+
n
+
14410
+
29.50
+
+
+
y
+
6283
+
12.86
+
+
+
g
+
2071
+
4.24
+
+
+
o
+
1762
+
3.61
+
+
+
e
+
1603
+
3.28
+
+
+
u
+
1167
+
2.39
+
+
+
f
+
849
+
1.74
+
+
+
p
+
837
+
1.71
+
+
+
k
+
676
+
1.38
+
+
+
r
+
430
+
0.88
+
+
+
l
+
182
+
0.37
+
+
+
b
+
140
+
0.29
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+Frequency and Percentage for 'has-ring':
+
+
+
+
+
Out[8]:
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
has-ring
+
+
+
+
+
+
+
f
+
36564
+
74.84
+
+
+
t
+
12291
+
25.16
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The initial X_train assessment has demonstrated no missing values within remaining features except for the ring-type (1,998). However, the proportion of missing values in this feature is reasonable, and simply dropping this column could result in loss of potentially valuable information, introduction of biases etc., which might reduce the overall accuracy of the classifier. Therefore, we decided to retain this column and perform imputation on ring-type in the data preprocessing phase.
To understand the numeric features in the data set, we plotted histograms for each numeric column in X_train, which helps identify the distribution patterns as well as detecting any skewness or outliers. The numeric columns being plotted are cap-diameter, stem-height, and stem-width.
----------------------------------------
+
+Frequency and Percentage for 'ring-type':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
ring-type
+
+
+
+
+
+
+
f
+
38562
+
82.30
+
+
+
e
+
1968
+
4.20
+
+
+
z
+
1735
+
3.70
+
+
+
r
+
1145
+
2.44
+
+
+
l
+
1127
+
2.41
+
+
+
p
+
1031
+
2.20
+
+
+
g
+
1003
+
2.14
+
+
+
m
+
286
+
0.61
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+Frequency and Percentage for 'habitat':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
habitat
+
+
+
+
+
+
+
d
+
35401
+
72.46
+
+
+
g
+
6327
+
12.95
+
+
+
l
+
2508
+
5.13
+
+
+
m
+
2346
+
4.80
+
+
+
h
+
1611
+
3.30
+
+
+
w
+
290
+
0.59
+
+
+
p
+
279
+
0.57
+
+
+
u
+
93
+
0.19
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+Frequency and Percentage for 'season':
+
+
+
+
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
season
+
+
+
+
+
+
+
a
+
24116
+
49.36
+
+
+
u
+
18322
+
37.50
+
+
+
w
+
4255
+
8.71
+
+
+
s
+
2162
+
4.43
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Based on the histograms, here are our findings for each feature being plotted.
+
+
cap-diameter: The distribution is highly skewed to the right, with most values concentrated between 0 and 10 cm. There are also some outliers sitting at around 40 to 60 cm.
+
+
stem-height: Slightly right-skewed distribution. The majority of mushrooms have stem heights between 4 and 10 cm, with few having stem heights over 20 cm.
+
+
stem-width: Another heavily right-skewed distribution, with the majority of mushrooms having stem width below 20 cm, and a some rare cases exceeding 50 cm.
+
+
+
The skewness observed across the 3 numeric features will be addressed in the preprocessing phase with QuantileTransformer from sklearn.preprocessing which maps data to a normal distribution while retaining the relative rank of values, making them more suitable for models sensitive to feature distributions, such as SVC and LogisticRegression.
To understand the categorical features in the data set, we analyzed their frequency and percentage distributions, providing insights into the variability and class imbalance that might occur for each feature.
+
+
+
+
+
+
+
+
+
In [8]:
+
+
+
categorical_columns=X_train.select_dtypes(include='object')# Select only categorical columns
+
+# Calculate frequency and percentage for each categorical features
+forcolumnincategorical_columns.columns:
+ print(f"Frequency and Percentage for '{column}':")
+
+ # Frequency
+ frequency=X_train[column].value_counts()
+ # Percentage
+ percentage=round(X_train[column].value_counts(normalize=True)*100,2)
+
+ # Combine into one DataFrame
+ freq_percent_df=pd.DataFrame({
+ "Frequency":frequency,
+ "Percentage":percentage
+ })
+
+ # Highlight values with a gradient
+ styled_df=freq_percent_df.style.format(
+ precision=2
+ ).background_gradient(
+ subset=['Percentage'],
+ cmap='YlOrRd'
+ )
+
+ # Display the styled DataFrame
+ display(styled_df)
+ print("-"*40,'\n')
+
+
+
+
+
+
+
+
+
+
+
+
+
----------------------------------------
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Based on the Frequency and Percentage distributions, here are our findings:
+
+
cap-shape: The most common cap shape is x (convex), comprising 43.97% of the data. Other shapes like f (flat) and s (sunken) are also prevalent, while c (conical) is the least common with 2.95% appearance.
+
+
cap-color: The most frequently appeared color is n (brown), with 39.71% of the data. Other colors like y (yellow), w (white), and g (gray) are also well-represented, while rare colors like b (buff) and l (blue) appear in less than 2% of the data.
+
+
does-bruise-or-bleed: The majority of the mushrooms are f (do not bruise or bleed), while their counterpart make up 17.26% of the data.
+
+
gill-color: The most common gill color is w (white), with 30.45% of the data. Other colors such as n (brown) and y (yellow) are also frequent, while rare gill colors like e (red), b (buff) and u (purple) appear in less than 2% of the data.
+
+
stem-color: w (white) and n (brown) are the dominating stem colors, accounting for 37.75% and 29.5% of the data, respectively. Other colors like r (green), l (blue) and b (buff) are less frequent, appearing in less than 1% of the observations.
+
+
has-ring: Most mushrooms are f (do not have a ring), with 74.84% observations. The remaining 25.16% mushrooms are t (have a ring).
+
+
ring-type: f (none) is the most common ring type, accounting for 82.3% of the data. Other types like e (evanescent) and z (zone) are less frequent, while rare types like m (movable) occur in less than 1% of the data.
+
+
habitat: The predominant habitat is d (woods), with 72.46% appearance. Other habitats such as g (grasses) and l (leaves) are less common, while w (waste), p (paths), and u (urban) only make up less than 1% of the data individually.
+
+
season: Most mushrooms grow in a (autumn), comprising 49.36% of the data, followed by u (summer) at 37.5%. The other two seasons w (winter) and s (spring) are less frequent.
+
+
+
Categorical features will be encoded into binary format in the following preprocessing phase with OneHotEncoder. Since we are dealing with a mix of binary and non-binary categorical features, for features like does-bruise-or-bleed and has-ring that have two unique values, they will be handled with drop='if_binary' argument to reduce redundancy while still capturing the information.
The target variable class represents whether a mushroom is p (poisonous) or e (edible). Understanding the distribution of the target helps assessing class balance, which might have impact on models' performance.
+
+
+
+
+
+
+
+
+
In [9]:
+
+
+
# Frequency
+frequency=y_train.value_counts()
+# Percentage
+percentage=round(y_train.value_counts(normalize=True)*100,2)
+
+# Combine into one DataFrame
+freq_percent_df=pd.DataFrame({
+ "Frequency":frequency,
+ "Percentage":percentage
+})
+freq_percent_df
+
+
+
+
+
+
+
+
+
+
+
Out[9]:
+
+
+
+
+
+
+
+
Frequency
+
Percentage
+
+
+
class
+
+
+
+
+
+
+
p
+
27143
+
55.56
+
+
+
e
+
21712
+
44.44
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Based on the Frequency and Percentage distribution, here are our findings:
+
+
p (Poisonous): There are 27,143 instances of poisonous mushrooms, accounting for 55.56% of the data.
+
+
e (Edible): There are 21,712 instances of edible mushrooms, constituting 44.44% of the data.
+
+
+
Using $F_{\beta}$, precision, recall, or confusion matrix to evaluate the model's performance is advisable in the following procedure.
Three classification models including Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Logistic Regression are used to predict whether a mushroom is edible or poisonous. Predicting a mushroom to be edible when it is in fact poisonous could have severe health consequences. Therefore the best model should prioritize the minimization of this error. To do this, we can evaluate models on an $F_{\beta}$ score with $\beta = 2$.
+
+
+
+
+
+
+
+
+
In [10]:
+
+
+
# loading in some models
+fromsklearn.neighborsimportKNeighborsClassifier
+fromsklearn.svmimportSVC
+fromsklearn.linear_modelimportLogisticRegression
+
+
+
+
+
+
+
+
+
+
+
In [11]:
+
+
+
# importing required preprocessors, pipelines, etc.
+fromsklearn.imputeimportSimpleImputer
+fromsklearn.preprocessingimportQuantileTransformer,OneHotEncoder
+fromsklearn.composeimportmake_column_transformer
+fromsklearn.pipelineimportmake_pipeline
+
+# converting targets to Series objects to avoid warnings
+y_train=y_train.squeeze()
+y_test=y_test.squeeze()
+
+# random state for reproducability
+SEED=123
+
+# feature sets for each transformation
+numeric_cols=['cap-diameter','stem-height','stem-width']
+categorical_cols=['does-bruise-or-bleed','has-ring','cap-shape','cap-color','gill-color','stem-color','habitat','season']
+impute_cols=['ring-type']
+
+# creating transformers
+numeric_transformer=QuantileTransformer(output_distribution='normal',random_state=SEED)
+categorical_transformer=OneHotEncoder(drop='if_binary',handle_unknown='ignore',sparse_output=False)
+impute_transformer=make_pipeline(
+ SimpleImputer(strategy='constant',fill_value='missing'),
+ categorical_transformer
+)
+
+# final preprocessor
+preprocessor=make_column_transformer(
+ (numeric_transformer,numeric_cols),
+ (impute_transformer,impute_cols),
+ (categorical_transformer,categorical_cols)
+)
+preprocessor
+
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
# compilng hyperparameters and scores of best models into one dataframe
+cols=['params','mean_fit_time','mean_test_accuracy','std_test_accuracy','mean_test_f2_score','std_test_f2_score']
+final_results=pd.concat(
+ [pd.DataFrame(result.cv_results_).query('rank_test_f2_score == 1')[cols]for_,resultincv_results.items()]
+)
+final_results.index=['Logisic Regression','KNN','SVC']
+final_results
+
+
+
+
+
+
+
+
+
+
+
Out[19]:
+
+
+
+
+
+
+
+
params
+
mean_fit_time
+
mean_test_accuracy
+
std_test_accuracy
+
mean_test_f2_score
+
std_test_f2_score
+
+
+
+
+
Logisic Regression
+
{'logisticregression__C': 0.05784745785308777}
+
0.353165
+
0.747313
+
0.002863
+
0.780611
+
0.003412
+
+
+
KNN
+
{'svc__C': 20.74024196289186, 'svc__gamma': 0....
+
41.681297
+
0.996582
+
0.000226
+
0.997112
+
0.000218
+
+
+
SVC
+
{'kneighborsclassifier__n_neighbors': 327}
+
0.128557
+
0.932310
+
0.001731
+
0.937919
+
0.001838
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
After tuning the hyperparameter, the Logistic Regression model has the mean accuracy of 0.75 and mean $F_{\beta}$ score of 0.78 on the validation set. The KNN model has the mean accuracy of 0.74 and mean $F_{\beta}$ score of 0.75. The SVC outperforms both Logistic Regression and KNN significantly in both accuracy of 0.99 and $F_{\beta}$ score of 0.99. Thus, SVC is the ideal choice to identify edible or poisonous mushroom (recall is the highest priority).
+
+
+
+
+
+
+
+
+
In [20]:
+
+
+
best_model=cv_results['svc'].best_estimator_
+best_model.fit(X_train,y_train)
+
+# confusion matrix of test results
+ConfusionMatrixDisplay.from_estimator(
+ best_model,
+ X_train,
+ y_train
+)
+
+
+
+
+
+
+
+
+
+
+
Out[20]:
+
+
<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x30728d220>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
In [21]:
+
+
+
# Finally, report the test score and confusion matrix
+y_test_predict=best_model.predict(X_test)
+
+test_f2_score=fbeta_score(y_test,y_test_predict,beta=2,pos_label='p')
+test_accuracy=accuracy_score(y_test,y_test_predict)
+print(f'Test F2-Score: {test_f2_score}\nTest Accuracy: {test_accuracy}')
+
+
+
+
+
+
+
+
+
+
+
+
+
Test F2-Score: 0.9973021849337405
+Test Accuracy: 0.9967250695922711
+
+
+
+
+
+
+
+
+
+
+
In [22]:
+
+
+
# Plotting confusion matrix for test set
+ConfusionMatrixDisplay.from_predictions(
+ y_test,
+ y_test_predict
+)
+
+
+
+
+
+
+
+
+
+
+
Out[22]:
+
+
<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x309e63dd0>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The prediction model performed quite well on test data, with a final overall accuracy of 0.99 and $F_{\beta}$ score of 0.99. The model only makes 40 mistakes out of 12214 test samples. 17 mistakes were predicting a poisonous mushroom as edible (false negative), while 23 mistakes were predicting a edible mushroom as poisonous (false positive). The model’s performance is promising for implementation, as false negatives represent potential safety risks and these errors could lead to consuming poisonous mushrooms, it is minimized to protect users. On the other hand, false positives are less harmful, they may lead to discarding safe mushrooms unnecessarily but do not endanger safety.
+
+
+
+
+
+
+
+
+
+
+
While the overall performance of the SVC model are impressive, efforts could focus on further reducing false negatives to enhance the safety of predictions. It might be important to take a closer look at the 40 misclassified observations to identify specific features contributing to these misclassifications. Implementing feature engineering on those features such as encoding rare categories differently can enhance the model’s power and reduce the misclassification cases. Additionally, trying other classifiers like Decision Tree and Random Forest which are less sensitive to scaling or irrelevant features might improve the prediction.
Hunter, J. D. (2007). Matplotlib: A 2D Graphics Environment. Computing in Science & Engineering, 9(3), 90–95.
+
McKinney, W. (2010). Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference, 51–56.
+
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., … Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
+
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., … Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825), 357–362.
+
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., … van der Walt, S. J. (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17, 261–272.
+
+
+
+
+
+
+
diff --git a/notebooks/.ipynb_checkpoints/Load_Data_and_EDA-checkpoint.ipynb b/notebooks/.ipynb_checkpoints/Load_Data_and_EDA-checkpoint.ipynb
new file mode 100644
index 0000000..c1c34a3
--- /dev/null
+++ b/notebooks/.ipynb_checkpoints/Load_Data_and_EDA-checkpoint.ipynb
@@ -0,0 +1,3289 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "1b40693f-6d55-4951-96a5-48a10ccb6773",
+ "metadata": {},
+ "source": [
+ "# Mushroom Edibility Classification Using Feature-Based Machine Learning Approach"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "18590e2e-138a-4ed1-821b-8a1850fdce9b",
+ "metadata": {},
+ "source": [
+ "by Benjamin Frizzell, Hankun Xiao, Essie Zhang, Mason Zhang 2024/11/23"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "81a65442-e81c-4e9d-9755-885bb2aebac9",
+ "metadata": {},
+ "source": [
+ "#### Import Library"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "0d2500b5-022e-4ff0-818c-ad1013efb69d",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "from ucimlrepo import fetch_ucirepo \n",
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import pandera as pa\n",
+ "from pandera import Check\n",
+ "from deepchecks import Dataset\n",
+ "import json\n",
+ "import logging\n",
+ "import matplotlib.pyplot as plt\n",
+ "from sklearn.model_selection import train_test_split"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f49175e2-5eab-4b03-816c-f20995c50c96",
+ "metadata": {},
+ "source": [
+ "## Summary"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1f39a05-24c5-4a5e-a34c-830e8efeee78",
+ "metadata": {},
+ "source": [
+ "In this project, a Support Vector Classifier was built and tuned to identify mushrooms edibility. A mushroom is classified as edible or poisonous with given color, habitat, class, and others. The final classifier performed quite well on unseen test data, with a final overall accuracy of 0.99 and $F_{\\beta}$ score with $\\beta = 2$ of 0.99. Furthermore, we use confusion matrix to show the accuracy of classification poisonous or edible mushroom. The model makes 12174 correct predictions out of 12214 test observations. 17 mistakes were predicting a poisonous mushroom as edible (false negative), while 23 mistakes were predicting a edible mushroom as poisonous (false positive). The model’s performance shows promise for implementation, prioritizing safety by minimizing false negatives that could result in consuming poisonous mushrooms. While false positives may lead to unnecessarily discarding safe mushrooms, they pose no safety risk. Further development is needed to make this model useful. Research should focus on improving performance and analyzing cases of incorrect predictions."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5626172b-fdbc-486d-ba3d-550432375290",
+ "metadata": {},
+ "source": [
+ "## Introduction"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d6ca1f13-3cd2-4bc6-bfae-4d0d6440e1b7",
+ "metadata": {},
+ "source": [
+ "Mushrooms are the most common food which is rich in vitamins and minerals. However, not all mushrooms can be consumed directly, most of them are poisonous and identifying edible or poisonous mushroom through the naked eye is quite difficult. Our aim is to using machine learning to identify mushrooms edibility. In this project, three methods are used to detect the edibility of mushrooms: Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Logistic Regression. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "921597ef-c12e-4c4c-b8bf-b1eb20e90814",
+ "metadata": {},
+ "source": [
+ "## Methods"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a0920cdf-10c1-4151-bbf3-a689486257dd",
+ "metadata": {},
+ "source": [
+ "### Data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "549db552-150b-4744-ba90-497604b5b601",
+ "metadata": {},
+ "source": [
+ "The dataset used in this project is the Secondary Mushroom Dataset created by Wagner, D., Heider, D., & Hattab, G. from UCI Machine Learning Repository. This dataset contains 61069 hypothetical mushrooms with caps based on 173 species (353 mushrooms per species). Each mushroom is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended (the latter class was combined with the poisonous class)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "809c0908-7030-437e-bb4c-56bdf0066119",
+ "metadata": {},
+ "source": [
+ "### Analysis"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "614cbeca-8401-49b3-8eed-cce9dc41d292",
+ "metadata": {},
+ "source": [
+ "The mushroom dataset is balanced with 56% of poisonous mushroom and 44% of edible mushroom. All variables were standardized and variables with more than 15% missing values are dropped, because imputing a variable that has a significant proportion of missing data might introduce too much noise or bias, making it unreliable. Data was splitted with 80% being partitioned into the training set and 20% being partitioned into the test set. Three classification models including Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Logistic Regression are used to predict whether a mushroom is edible or poisonous. The fine tuned Support Vector Classifier has the best overall performance. The hyperparameter was chosen using 5-fold cross validation with $F_{\\beta}$ score as the classification metric. $\\beta$ was chosen to be set to 2 for the $F_{\\beta}$ score to increase the weight on recall during fitting because predicting a mushroom to be edible when it is in fact poisonous could have severe health consequences. Therefore the goal is to prioritize the minimization of false negatives. The Python programming language (Van Rossum and Drake 2009) and the following Python packages were used to perform the analysis: Matplotlib (Hunter, 2007), Pandas (McKinney, 2010), Scikit-learn (Pedregosa et al., 2011), NumPy (Harris et al., 2020), SciPy (Virtanen et al., 2020), UCIMLRepo."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "58647071-18ff-44cb-9f2d-fd243555cff0",
+ "metadata": {},
+ "source": [
+ "## Results & Discussion"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f94469b3-143e-4a67-8c22-5bfa73baeccc",
+ "metadata": {},
+ "source": [
+ "The EDA shows that all numeric columns in the mushroom dataset are nearly normal with some skewness. A robust preprocessing scheme `QuantileTransformer` is used because it can transform skewed data or heavy-tailed distributions into a more Gaussian-like shape and reduce the impact of outliers.\n",
+ "`OneHotEncoder` is applied for categorical features in the mushroom dataset, because each feature does not contains much categories and they are not ordered. It is critical to keep all important information in the features. Since ring type feature has many missing values, it was filled in with a \"Missing\" class. Treating missing values as a distinct category provides a way to model the absence of data directly. This can be valuable because missingness itself might carry information."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7558f2ed-854e-492b-8a71-7e37cdecf1f3",
+ "metadata": {},
+ "source": [
+ "#### Load Data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "20a3d74b-174b-420e-9745-6d68f9d7da5f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# fetch dataset as pandas DataFrames\n",
+ "secondary_mushroom = fetch_ucirepo(id=848) \n",
+ "X = secondary_mushroom.data.features \n",
+ "y = secondary_mushroom.data.targets "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e4fa6da5-7876-43ba-b5ff-e12a66c78c75",
+ "metadata": {},
+ "source": [
+ "##### Before splitting the data into test and training sets, we want to check for missing values in each column to determine whether they can be used in our model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "eeee4dd9-6fa9-47d3-b86f-e128e791a96e",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# Check the missing values\n",
+ "missing_values = X.isnull().sum().reset_index()\n",
+ "missing_values.columns = ['Column', 'Missing Count']\n",
+ "\n",
+ "# Highlight values with a gradient\n",
+ "styled_missing = missing_values.style.format(\n",
+ " precision=0\n",
+ ").background_gradient(\n",
+ " subset=['Missing Count'],\n",
+ " cmap='YlOrRd'\n",
+ ").set_caption(\"Missing Values by Column\")\n",
+ "\n",
+ "# Display the styled DataFrame\n",
+ "display(styled_missing)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "302deb34-f082-41b2-bd10-7b20ba0b3dbd",
+ "metadata": {},
+ "source": [
+ "The initial `X_train` assessment has demonstrated no missing values within remaining features except for the `ring-type` . However, the proportion of missing values in this feature is reasonable, and simply dropping this column could result in loss of potentially valuable information, introduction of biases etc., which might reduce the overall accuracy of the classifier. Therefore, we decided to retain this column and perform imputation on `ring-type` in the data preprocessing phase. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9dd6a42f-b507-4d81-aa68-1274d20872c1",
+ "metadata": {},
+ "source": [
+ "##### Part 2: The distribution of numeric features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dc6a5a2b-84f8-402a-ac8c-0bb727ec5c13",
+ "metadata": {},
+ "source": [
+ "To understand the numeric features in the data set, we plotted histograms for each numeric column in `X_train`, which helps identify the distribution patterns as well as detecting any skewness or outliers. The numeric columns being plotted are `cap-diameter`, `stem-height`, and `stem-width`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "c7c4cb2e-0f89-44fd-8faa-11088dd290e2",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAHUCAYAAAAN/ZAQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKMUlEQVR4nO3deVhV1f4G8PfIcBiEI4OHw1FErgMOoKIUgpY4oShyE7tqGE6EmVNeIQu7XrFSTHMozSEzHKCwcsgycdYyRXHAnCJNRFQQUmb1gLB+f3TdP4+AIkKb4f08z37yrPXd66wF6NseOFshhBAgIiKiv10DuSdARERUXzGEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISpRlq7di0UCgWOHz9eZr+fnx+aN2+u19a8eXOMHj36qd7n8OHDiIiIQHZ2duUmWg9t3LgR7du3h6mpKRQKBRITE+We0jM5cOAAFAoFDhw4ILWNHj261M9XTXPjxg1ERETU+q9/fccQpjpjy5YtmDlz5lPtc/jwYcyePZshXEGZmZkICgpCixYtEBcXhyNHjqB169ZyT6vKzZw5E1u2bJF7Go9148YNzJ49myFcyxnKPQGiquLm5ib3FJ5aUVERFAoFDA1rx1/F33//HUVFRXj11VfRo0cPuadTbVq0aCH3FGRz9+5dmJiYQKFQyD2VeoFHwlRnPHo6uqSkBB988AGcnZ1hamqKRo0aoUOHDvj4448BABEREXjrrbcAAE5OTlAoFHqnJUtKSjB//ny0adMGSqUSarUaI0eOxLVr1/TeVwiBuXPnwtHRESYmJnB3d8fu3bvh7e0Nb29vqe7Bac8NGzYgNDQUTZo0gVKpxKVLl5CZmYkJEyagXbt2aNiwIdRqNXr16oWff/5Z772uXLkChUKBBQsW4MMPP0Tz5s1hamoKb29vKSDfeecdaLVaqFQqDB48GBkZGRX6+m3btg2enp4wMzODhYUF+vbtiyNHjkj9o0ePRvfu3QEAw4YNg0Kh0FtfWa5fv45x48bBwcEBxsbG0Gq1ePnll3Hz5k0AwL179xAaGopOnTpBpVLB2toanp6e+O6770qNpVAoMGnSJKxatQqtW7eGUqlEu3btEBsbW6H1AcBvv/2G/v37w8zMDLa2thg/fjzy8vJK1ZV1OvrTTz/Fiy++CLVaDXNzc7i6umL+/PkoKirSq/P29oaLiwuOHDkCLy8vmJqaonnz5oiKigIAbN++HZ07d4aZmRlcXV0RFxdX6v0vXryIwMBAqNVqKJVKtG3bFp9++qnUf+DAATz33HMAgDFjxkg/uxEREVLN8ePH4e/vD2tra5iYmMDNzQ1ff/213vs8uOyza9cujB07Fo0bN4aZmRl0Ol2Fv6b0bGrH/35TvVVcXIz79++Xaq/Iw7/mz5+PiIgI/Oc//8GLL76IoqIi/Pbbb9Kp59deew23b9/G0qVLsXnzZtjb2wMA2rVrBwB444038Nlnn2HSpEnw8/PDlStXMHPmTBw4cAAnT56Era0tAODdd99FZGQkxo0bh4CAAKSmpuK1115DUVFRmadqw8PD4enpiZUrV6JBgwZQq9XIzMwEAMyaNQsajQb5+fnYsmULvL29sXfv3lJh9+mnn6JDhw749NNPkZ2djdDQUAwaNAgeHh4wMjLCF198gZSUFISFheG1117Dtm3bHvu1+vLLLzFixAj4+Pjgq6++gk6nw/z586X37969O2bOnInnn38eEydOxNy5c9GzZ09YWlqWO+b169fx3HPPoaioCDNmzECHDh1w69Yt7Ny5E1lZWbCzs4NOp8Pt27cRFhaGJk2aoLCwEHv27EFAQACioqIwcuRIvTG3bduG/fv347333oO5uTmWL1+OV155BYaGhnj55Zcfu8abN2+iR48eMDIywvLly2FnZ4eYmBhMmjTpsfs98McffyAwMBBOTk4wNjbG6dOnMWfOHPz222/44osv9GrT09MxZswYTJ8+HU2bNsXSpUsxduxYpKam4ttvv8WMGTOgUqnw3nvv4aWXXsLly5eh1WoBAOfPn4eXlxeaNWuGhQsXQqPRYOfOnZgyZQr+/PNPzJo1C507d0ZUVBTGjBmD//znPxg4cCAAoGnTpgCA/fv3o3///vDw8MDKlSuhUqkQGxuLYcOG4c6dO6XunRg7diwGDhyIDRs2oKCgAEZGRhX6mlAVEEQ1UFRUlADw2M3R0VFvH0dHRzFq1CjptZ+fn+jUqdNj32fBggUCgEhOTtZrv3DhggAgJkyYoNd+9OhRAUDMmDFDCCHE7du3hVKpFMOGDdOrO3LkiAAgevToIbXt379fABAvvvjiE9d///59UVRUJHr37i0GDx4stScnJwsAomPHjqK4uFhqX7JkiQAg/P399caZOnWqACBycnLKfa/i4mKh1WqFq6ur3ph5eXlCrVYLLy+vUmv45ptvnriGsWPHCiMjI3H+/Pkn1j7wYN3BwcHCzc1Nrw+AMDU1Fenp6Xr1bdq0ES1btnzi2G+//bZQKBQiMTFRr71v374CgNi/f7/UNmrUqFI/Xw8rLi4WRUVFYv369cLAwEDcvn1b6uvRo4cAII4fPy613bp1SxgYGAhTU1Nx/fp1qT0xMVEAEJ988onU1q9fP9G0adNS37NJkyYJExMT6b0SEhIEABEVFVVqfm3atBFubm6iqKhIr93Pz0/Y29tL3+cHf89GjhxZ7lqpevF0NNVo69evR0JCQqntwWnRx3n++edx+vRpTJgwATt37kRubm6F33f//v0AUOqI4fnnn0fbtm2xd+9eAEB8fDx0Oh2GDh2qV9e1a9dy764dMmRIme0rV65E586dYWJiAkNDQxgZGWHv3r24cOFCqdoBAwagQYP//+vbtm1bAJCOiB5tv3r1ajkrBZKSknDjxg0EBQXpjdmwYUMMGTIE8fHxuHPnTrn7l2fHjh3o2bOnNIfyfPPNN+jWrRsaNmworXvNmjVlrrt3796ws7OTXhsYGGDYsGG4dOmSdJng/v37epv431mT/fv3o3379ujYsaPemIGBgRVaz6lTp+Dv7w8bGxsYGBjAyMgII0eORHFxMX7//Xe9Wnt7e3Tp0kV6bW1tDbVajU6dOklHvMD/f39SUlIA/HV6fu/evRg8eDDMzMz01jFgwADcu3cP8fHxj53npUuX8Ntvv2HEiBGlvh4DBgxAWloakpKS9PYp72eSqh9DmGq0tm3bwt3dvdSmUqmeuG94eDg++ugjxMfHw9fXFzY2Nujdu3e5v/b0sFu3bgGAdIr6YVqtVup/8N+Hg+GBstrKG3PRokV444034OHhgU2bNiE+Ph4JCQno378/7t69W6re2tpa77WxsfFj2+/du1fmXB5eQ3lrLSkpQVZWVrn7lyczM1M6PVqezZs3Y+jQoWjSpAmio6Nx5MgRJCQkYOzYsWXOWaPRlNv2YB1GRkZ627p166T+x+3/OFevXsULL7yA69ev4+OPP8bPP/+MhIQE6Trto9+jR78PwF/fiyd9f27duoX79+9j6dKlpdYxYMAAAMCff/752Lk+uN4eFhZWaowJEyaUOUZZ33v6e/CaMNVZhoaGmDZtGqZNm4bs7Gzs2bMHM2bMQL9+/ZCamgozM7Ny97WxsQEApKWllQqSGzduSNeDH9Q9+IfvYenp6WUeDZd112l0dDS8vb2xYsUKvfaybhqqag+v9VE3btxAgwYNYGVl9dTjNm7cuNRNbI+Kjo6Gk5MTNm7cqPd1Ke/GoPT09HLbHqwjISFBr9/JyUnqf9z+j7N161YUFBRg8+bNcHR0lNqr+teDrKysYGBggKCgIEycOLHMmgfrKc+Dn83w8HAEBASUWePs7Kz3mndCy4chTPVCo0aN8PLLL+P69euYOnUqrly5gnbt2kGpVAIofSTTq1cvAH+FxIO7UIG//oG/cOEC3n33XQCAh4cHlEolNm7cqPcPXnx8PFJSUir8gQ8KhUKaywO//vorjhw5AgcHh6de79NwdnZGkyZN8OWXXyIsLEz6B7mgoACbNm2S7ph+Wr6+vtiwYQOSkpJK/aP/gEKhgLGxsV4IpKenl3l3NADs3bsXN2/elM4yFBcXY+PGjWjRooX0P0vu7u5l7tuzZ0/Mnz8fp0+f1jsl/eWXXz5xLQ/m9/D3SAiB1atXP3Hfp2FmZoaePXvi1KlT6NChg3SkXJbyfnadnZ3RqlUrnD59GnPnzq3S+VHVYwhTnTVo0CC4uLjA3d0djRs3RkpKCpYsWQJHR0e0atUKAODq6goA+PjjjzFq1CgYGRnB2dkZzs7OGDduHJYuXYoGDRrA19dXujvawcEB//73vwH8ddpx2rRpiIyMhJWVFQYPHoxr165h9uzZsLe317vG+jh+fn54//33MWvWLPTo0QNJSUl477334OTkVObd4VWpQYMGmD9/PkaMGAE/Pz+8/vrr0Ol0WLBgAbKzszFv3rxKjfvee+9hx44dePHFFzFjxgy4uroiOzsbcXFxmDZtGtq0aQM/Pz9s3rwZEyZMwMsvv4zU1FS8//77sLe3x8WLF0uNaWtri169emHmzJnS3dG//fZbhX5NaerUqfjiiy8wcOBAfPDBB9Ld0b/99tsT9+3bty+MjY3xyiuvYPr06bh37x5WrFhRqdP0T/Lxxx+je/fueOGFF/DGG2+gefPmyMvLw6VLl/D9999j3759AP76XWZTU1PExMSgbdu2aNiwIbRaLbRaLVatWgVfX1/069cPo0ePRpMmTXD79m1cuHABJ0+exDfffFPl86ZKkvvOMKKyPLhrMyEhocz+gQMHPvHu6IULFwovLy9ha2srjI2NRbNmzURwcLC4cuWK3n7h4eFCq9WKBg0a6N0lW1xcLD788EPRunVrYWRkJGxtbcWrr74qUlNT9fYvKSkRH3zwgWjatKkwNjYWHTp0ED/88IPo2LGj3p3Nj7uzWKfTibCwMNGkSRNhYmIiOnfuLLZu3VrqLt0Hd0cvWLBAb//yxn7S1/FhW7duFR4eHsLExESYm5uL3r17i19++aVC71Oe1NRUMXbsWKHRaISRkZHQarVi6NCh4ubNm1LNvHnzRPPmzYVSqRRt27YVq1evFrNmzRKP/vMEQEycOFEsX75ctGjRQhgZGYk2bdqImJiYCs1FCCHOnz8v+vbtK0xMTIS1tbUIDg4W3333XYXujv7+++9Fx44dhYmJiWjSpIl46623xI4dO0rt26NHD9G+fftS7+3o6CgGDhxYqv3Buh6WnJwsxo4dK5o0aSKMjIxE48aNhZeXl/jggw/06r766ivRpk0bYWRkJACIWbNmSX2nT58WQ4cOFWq1WhgZGQmNRiN69eolVq5cKdU8zc8HVQ+FEBX4hUsieirJyclo06YNZs2ahRkzZsg9nTpBoVBg4sSJWLZsmdxTIaoyPB1N9IxOnz6Nr776Cl5eXrC0tERSUhLmz58PS0tLBAcHyz09IqrBGMJEz8jc3BzHjx/HmjVrkJ2dDZVKBW9vb8yZM6fcX1MiIgIAno4mIiKSCT+sg4iISCYMYSIiIpkwhImIiGTCG7OqUElJCW7cuAELCwt+DBwRUT0lhEBeXh60Wu0TP7CHIVyFbty4Ue0fMUhERLVDamrqEx9iwhCuQhYWFgD++sI/7mHnRERUd+Xm5sLBwUHKhMdhCFehB6egLS0tGcJERPVcRS5L8sYsIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMJHGdZDmZmZyM3NrZaxLS0t0bhx42oZm4iormEI1zOZmZl4dcxruJ13p1rGt7YwQ3TU5wxiIqIKYAjXM7m5ubiddweNPYfA3NquSscuuH0TmUc2ITc3lyFMRFQBDOF6ytzaDpbqplU+bmaVj0hEVHfxxiwiIiKZMISJiIhkwhAmIiKSCUOYiIhIJrwxqwaqzt/jTUlJwf2i+9UyNhERPR2GcA1T3b/He+/uHVy7noZmRUXVMj4REVUcQ7iGqc7f4wWAjD/OIiX1CxTfZwgTEcmNIVxDVdfv8ebfSq/yMYmIqHJ4YxYREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMmEIExERyUTWEP7pp58waNAgaLVaKBQKbN26Va9foVCUuS1YsECq8fb2LtU/fPhwvXGysrIQFBQElUoFlUqFoKAgZGdn69VcvXoVgwYNgrm5OWxtbTFlyhQUFhZW19KJiIjkDeGCggJ07NgRy5YtK7M/LS1Nb/viiy+gUCgwZMgQvbqQkBC9ulWrVun1BwYGIjExEXFxcYiLi0NiYiKCgoKk/uLiYgwcOBAFBQU4dOgQYmNjsWnTJoSGhlb9oomIiP5H1k/M8vX1ha+vb7n9Go1G7/V3332Hnj174h//+Ideu5mZWanaBy5cuIC4uDjEx8fDw8MDALB69Wp4enoiKSkJzs7O2LVrF86fP4/U1FRotVoAwMKFCzF69GjMmTMHlpaWz7JMIiKiMtWaa8I3b97E9u3bERwcXKovJiYGtra2aN++PcLCwpCXlyf1HTlyBCqVSgpgAOjatStUKhUOHz4s1bi4uEgBDAD9+vWDTqfDiRMnyp2TTqdDbm6u3kZERFRRteazo9etWwcLCwsEBATotY8YMQJOTk7QaDQ4e/YswsPDcfr0aezevRsAkJ6eDrVaXWo8tVqN9PR0qcbOTv9hCVZWVjA2NpZqyhIZGYnZs2c/69KIiKieqjUh/MUXX2DEiBEwMTHRaw8JCZH+7OLiglatWsHd3R0nT55E586dAfx1g9ejhBB67RWpeVR4eDimTZsmvc7NzYWDg0PFF0VERPVarTgd/fPPPyMpKQmvvfbaE2s7d+4MIyMjXLx4EcBf15Vv3rxZqi4zM1M6+tVoNKWOeLOyslBUVFTqCPlhSqUSlpaWehsREVFF1YoQXrNmDbp06YKOHTs+sfbcuXMoKiqCvb09AMDT0xM5OTk4duyYVHP06FHk5OTAy8tLqjl79izS0tKkml27dkGpVKJLly5VvBoiIqK/yHo6Oj8/H5cuXZJeJycnIzExEdbW1mjWrBmAv07xfvPNN1i4cGGp/f/44w/ExMRgwIABsLW1xfnz5xEaGgo3Nzd069YNANC2bVv0798fISEh0q8ujRs3Dn5+fnB2dgYA+Pj4oF27dggKCsKCBQtw+/ZthIWFISQkhEe3RERUbWQ9Ej5+/Djc3Nzg5uYGAJg2bRrc3Nzw3//+V6qJjY2FEAKvvPJKqf2NjY2xd+9e9OvXD87OzpgyZQp8fHywZ88eGBgYSHUxMTFwdXWFj48PfHx80KFDB2zYsEHqNzAwwPbt22FiYoJu3bph6NCheOmll/DRRx9V4+qJiKi+k/VI2NvbG0KIx9aMGzcO48aNK7PPwcEBBw8efOL7WFtbIzo6+rE1zZo1ww8//PDEsYiIiKpKrbgmTEREVBcxhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGQiawj/9NNPGDRoELRaLRQKBbZu3arXP3r0aCgUCr2ta9euejU6nQ6TJ0+Gra0tzM3N4e/vj2vXrunVZGVlISgoCCqVCiqVCkFBQcjOztaruXr1KgYNGgRzc3PY2tpiypQpKCwsrI5lExERAZA5hAsKCtCxY0csW7as3Jr+/fsjLS1N2n788Ue9/qlTp2LLli2IjY3FoUOHkJ+fDz8/PxQXF0s1gYGBSExMRFxcHOLi4pCYmIigoCCpv7i4GAMHDkRBQQEOHTqE2NhYbNq0CaGhoVW/aCIiov8xlPPNfX194evr+9gapVIJjUZTZl9OTg7WrFmDDRs2oE+fPgCA6OhoODg4YM+ePejXrx8uXLiAuLg4xMfHw8PDAwCwevVqeHp6IikpCc7Ozti1axfOnz+P1NRUaLVaAMDChQsxevRozJkzB5aWllW4aiIior/U+GvCBw4cgFqtRuvWrRESEoKMjAyp78SJEygqKoKPj4/UptVq4eLigsOHDwMAjhw5ApVKJQUwAHTt2hUqlUqvxsXFRQpgAOjXrx90Oh1OnDhR7tx0Oh1yc3P1NiIiooqq0SHs6+uLmJgY7Nu3DwsXLkRCQgJ69eoFnU4HAEhPT4exsTGsrKz09rOzs0N6erpUo1arS42tVqv1auzs7PT6raysYGxsLNWUJTIyUrrOrFKp4ODg8EzrJSKi+kXW09FPMmzYMOnPLi4ucHd3h6OjI7Zv346AgIBy9xNCQKFQSK8f/vOz1DwqPDwc06ZNk17n5uYyiImIqMJq9JHwo+zt7eHo6IiLFy8CADQaDQoLC5GVlaVXl5GRIR3ZajQa3Lx5s9RYmZmZejWPHvFmZWWhqKio1BHyw5RKJSwtLfU2IiKiiqpVIXzr1i2kpqbC3t4eANClSxcYGRlh9+7dUk1aWhrOnj0LLy8vAICnpydycnJw7Ngxqebo0aPIycnRqzl79izS0tKkml27dkGpVKJLly5/x9KIiKgekvV0dH5+Pi5duiS9Tk5ORmJiIqytrWFtbY2IiAgMGTIE9vb2uHLlCmbMmAFbW1sMHjwYAKBSqRAcHIzQ0FDY2NjA2toaYWFhcHV1le6Wbtu2Lfr374+QkBCsWrUKADBu3Dj4+fnB2dkZAODj44N27dohKCgICxYswO3btxEWFoaQkBAe3RIRUbWRNYSPHz+Onj17Sq8fXF8dNWoUVqxYgTNnzmD9+vXIzs6Gvb09evbsiY0bN8LCwkLaZ/HixTA0NMTQoUNx9+5d9O7dG2vXroWBgYFUExMTgylTpkh3Ufv7++v9brKBgQG2b9+OCRMmoFu3bjA1NUVgYCA++uij6v4SEBFRPSZrCHt7e0MIUW7/zp07nziGiYkJli5diqVLl5ZbY21tjejo6MeO06xZM/zwww9PfD8iIqKqUquuCRMREdUlDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkImsI//TTTxg0aBC0Wi0UCgW2bt0q9RUVFeHtt9+Gq6srzM3NodVqMXLkSNy4cUNvDG9vbygUCr1t+PDhejVZWVkICgqCSqWCSqVCUFAQsrOz9WquXr2KQYMGwdzcHLa2tpgyZQoKCwura+lERETyhnBBQQE6duyIZcuWleq7c+cOTp48iZkzZ+LkyZPYvHkzfv/9d/j7+5eqDQkJQVpamrStWrVKrz8wMBCJiYmIi4tDXFwcEhMTERQUJPUXFxdj4MCBKCgowKFDhxAbG4tNmzYhNDS06hdNRET0P4Zyvrmvry98fX3L7FOpVNi9e7de29KlS/H888/j6tWraNasmdRuZmYGjUZT5jgXLlxAXFwc4uPj4eHhAQBYvXo1PD09kZSUBGdnZ+zatQvnz59HamoqtFotAGDhwoUYPXo05syZA0tLy6pYLhERkZ5adU04JycHCoUCjRo10muPiYmBra0t2rdvj7CwMOTl5Ul9R44cgUqlkgIYALp27QqVSoXDhw9LNS4uLlIAA0C/fv2g0+lw4sSJcuej0+mQm5urtxEREVWUrEfCT+PevXt45513EBgYqHdkOmLECDg5OUGj0eDs2bMIDw/H6dOnpaPo9PR0qNXqUuOp1Wqkp6dLNXZ2dnr9VlZWMDY2lmrKEhkZidmzZ1fF8oiIqB6qFSFcVFSE4cOHo6SkBMuXL9frCwkJkf7s4uKCVq1awd3dHSdPnkTnzp0BAAqFotSYQgi99orUPCo8PBzTpk2TXufm5sLBwaHiCyMionqtxp+OLioqwtChQ5GcnIzdu3c/8fps586dYWRkhIsXLwIANBoNbt68WaouMzNTOvrVaDSljnizsrJQVFRU6gj5YUqlEpaWlnobERFRRdXoEH4QwBcvXsSePXtgY2PzxH3OnTuHoqIi2NvbAwA8PT2Rk5ODY8eOSTVHjx5FTk4OvLy8pJqzZ88iLS1Nqtm1axeUSiW6dOlSxasiIiL6i6yno/Pz83Hp0iXpdXJyMhITE2FtbQ2tVouXX34ZJ0+exA8//IDi4mLpaNXa2hrGxsb4448/EBMTgwEDBsDW1hbnz59HaGgo3Nzc0K1bNwBA27Zt0b9/f4SEhEi/ujRu3Dj4+fnB2dkZAODj44N27dohKCgICxYswO3btxEWFoaQkBAe3RIRUbWR9Uj4+PHjcHNzg5ubGwBg2rRpcHNzw3//+19cu3YN27Ztw7Vr19CpUyfY29tL24O7mo2NjbF3717069cPzs7OmDJlCnx8fLBnzx4YGBhI7xMTEwNXV1f4+PjAx8cHHTp0wIYNG6R+AwMDbN++HSYmJujWrRuGDh2Kl156CR999NHf+wUhIqJ6RdYjYW9vbwghyu1/XB8AODg44ODBg098H2tra0RHRz+2plmzZvjhhx+eOBYREVFVqdHXhImIiOoyhjAREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMmEIExERyYQhTEREJBOGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMKhXCycnJVT0PIiKieqdSIdyyZUv07NkT0dHRuHfvXlXPiYiIqF6oVAifPn0abm5uCA0NhUajweuvv45jx45V9dyIiIjqtEqFsIuLCxYtWoTr168jKioK6enp6N69O9q3b49FixYhMzOzqudJRERU5zzTjVmGhoYYPHgwvv76a3z44Yf4448/EBYWhqZNm2LkyJFIS0urqnkSERHVOc8UwsePH8eECRNgb2+PRYsWISwsDH/88Qf27duH69ev45///GdVzZOIiKjOMazMTosWLUJUVBSSkpIwYMAArF+/HgMGDECDBn9lupOTE1atWoU2bdpU6WSJiIjqkkqF8IoVKzB27FiMGTMGGo2mzJpmzZphzZo1zzQ5IiKiuqxSIXzx4sUn1hgbG2PUqFGVGZ6IiKheqNQ14aioKHzzzTel2r/55husW7fumSdFRERUH1QqhOfNmwdbW9tS7Wq1GnPnzn3mSREREdUHlQrhlJQUODk5lWp3dHTE1atXn3lSRERE9UGlQlitVuPXX38t1X769GnY2Ng886SIiIjqg0qF8PDhwzFlyhTs378fxcXFKC4uxr59+/Dmm29i+PDhVT1HIiKiOqlSd0d/8MEHSElJQe/evWFo+NcQJSUlGDlyJK8JExERVVClQtjY2BgbN27E+++/j9OnT8PU1BSurq5wdHSs6vkRERHVWZUK4Qdat26N1q1bV9VciIiI6pVKhXBxcTHWrl2LvXv3IiMjAyUlJXr9+/btq5LJERER1WWVCuE333wTa9euxcCBA+Hi4gKFQlHV8yIiIqrzKhXCsbGx+PrrrzFgwICqng8REVG9UalfUTI2NkbLli2rei5ERET1SqVCODQ0FB9//DGEEFU9HyIionqjUqejDx06hP3792PHjh1o3749jIyM9Po3b95cJZMjIiKqyyoVwo0aNcLgwYOrei5ERET1SqUfZfi4raJ++uknDBo0CFqtFgqFAlu3btXrF0IgIiICWq0Wpqam8Pb2xrlz5/RqdDodJk+eDFtbW5ibm8Pf3x/Xrl3Tq8nKykJQUBBUKhVUKhWCgoKQnZ2tV3P16lUMGjQI5ubmsLW1xZQpU1BYWPhUXxciIqKnUakQBoD79+9jz549WLVqFfLy8gAAN27cQH5+foXHKCgoQMeOHbFs2bIy++fPn49FixZh2bJlSEhIgEajQd++faX3A4CpU6diy5YtiI2NxaFDh5Cfnw8/Pz8UFxdLNYGBgUhMTERcXBzi4uKQmJiIoKAgqb+4uBgDBw5EQUEBDh06hNjYWGzatAmhoaFP+2UhIiKqsEqdjk5JSUH//v1x9epV6HQ69O3bFxYWFpg/fz7u3buHlStXVmgcX19f+Pr6ltknhMCSJUvw7rvvIiAgAACwbt062NnZ4csvv8Trr7+OnJwcrFmzBhs2bECfPn0AANHR0XBwcMCePXvQr18/XLhwAXFxcYiPj4eHhwcAYPXq1fD09ERSUhKcnZ2xa9cunD9/HqmpqdBqtQCAhQsXYvTo0ZgzZw4sLS0r82UiIiJ6rEodCb/55ptwd3dHVlYWTE1NpfbBgwdj7969VTKx5ORkpKenw8fHR2pTKpXo0aMHDh8+DAA4ceIEioqK9Gq0Wi1cXFykmiNHjkClUkkBDABdu3aFSqXSq3FxcZECGAD69esHnU6HEydOlDtHnU6H3NxcvY2IiKiiKn139C+//AJjY2O9dkdHR1y/fr1KJpaeng4AsLOz02u3s7NDSkqKVGNsbAwrK6tSNQ/2T09Ph1qtLjW+Wq3Wq3n0faysrGBsbCzVlCUyMhKzZ89+ypURERH9pVJHwiUlJXrXXB+4du0aLCwsnnlSD3v0IzGFEE/8mMxHa8qqr0zNo8LDw5GTkyNtqampj50XERHRwyoVwn379sWSJUuk1wqFAvn5+Zg1a1aVfZSlRqMBgFJHohkZGdJRq0ajQWFhIbKysh5bc/PmzVLjZ2Zm6tU8+j5ZWVkoKioqdYT8MKVSCUtLS72NiIiooioVwosXL8bBgwfRrl073Lt3D4GBgWjevDmuX7+ODz/8sEom5uTkBI1Gg927d0tthYWFOHjwILy8vAAAXbp0gZGRkV5NWloazp49K9V4enoiJycHx44dk2qOHj2KnJwcvZqzZ88iLS1Nqtm1axeUSiW6dOlSJeshIiJ6VKWuCWu1WiQmJuKrr77CyZMnUVJSguDgYIwYMULvRq0nyc/Px6VLl6TXycnJSExMhLW1NZo1a4apU6di7ty5aNWqFVq1aoW5c+fCzMwMgYGBAACVSoXg4GCEhobCxsYG1tbWCAsLg6urq3S3dNu2bdG/f3+EhIRg1apVAIBx48bBz88Pzs7OAAAfHx+0a9cOQUFBWLBgAW7fvo2wsDCEhITw6JaIiKpNpUIYAExNTTF27FiMHTu20m9+/Phx9OzZU3o9bdo0AMCoUaOwdu1aTJ8+HXfv3sWECROQlZUFDw8P7Nq1S++68+LFi2FoaIihQ4fi7t276N27N9auXQsDAwOpJiYmBlOmTJHuovb399f73WQDAwNs374dEyZMQLdu3WBqaorAwEB89NFHlV4bERHRk1QqhNevX//Y/pEjR1ZoHG9v78c+BEKhUCAiIgIRERHl1piYmGDp0qVYunRpuTXW1taIjo5+7FyaNWuGH3744YlzJiIiqiqVCuE333xT73VRURHu3LkDY2NjmJmZVTiEiYiI6rNK3ZiVlZWlt+Xn5yMpKQndu3fHV199VdVzJCIiqpMq/dnRj2rVqhXmzZtX6iiZiIiIylZlIQz8dYPTjRs3qnJIIiKiOqtS14S3bdum91oIgbS0NCxbtgzdunWrkokRERHVdZUK4ZdeeknvtUKhQOPGjdGrVy8sXLiwKuZFRERU51UqhEtKSqp6HkRERPVOlV4TJiIiooqr1JHwg0+2qohFixZV5i2IiIjqvEqF8KlTp3Dy5Encv39f+vzl33//HQYGBujcubNU96RHDhIREdVnlQrhQYMGwcLCAuvWrYOVlRWAvz7AY8yYMXjhhRcQGhpapZMkIiKqiyp1TXjhwoWIjIyUAhgArKys8MEHH/DuaCIiogqqVAjn5ubi5s2bpdozMjKQl5f3zJMiIiKqDyoVwoMHD8aYMWPw7bff4tq1a7h27Rq+/fZbBAcHIyAgoKrnSEREVCdV6prwypUrERYWhldffRVFRUV/DWRoiODgYCxYsKBKJ0hERFRXVSqEzczMsHz5cixYsAB//PEHhBBo2bIlzM3Nq3p+REREddYzfVhHWloa0tLS0Lp1a5ibm0MIUVXzIiIiqvMqFcK3bt1C79690bp1awwYMABpaWkAgNdee42/nkRERFRBlQrhf//73zAyMsLVq1dhZmYmtQ8bNgxxcXFVNjkiIqK6rFLXhHft2oWdO3eiadOmeu2tWrVCSkpKlUyMiIiorqvUkXBBQYHeEfADf/75J5RK5TNPioiIqD6oVAi/+OKLWL9+vfRaoVCgpKQECxYsQM+ePatsckRERHVZpU5HL1iwAN7e3jh+/DgKCwsxffp0nDt3Drdv38Yvv/xS1XMkIiKqkyp1JNyuXTv8+uuveP7559G3b18UFBQgICAAp06dQosWLap6jkRERHXSUx8JFxUVwcfHB6tWrcLs2bOrY05ERET1wlMfCRsZGeHs2bN8VjAREdEzqtTp6JEjR2LNmjVVPRciIqJ6pVI3ZhUWFuLzzz/H7t274e7uXuozoxctWlQlkyMiIqrLniqEL1++jObNm+Ps2bPo3LkzAOD333/Xq+FpaiIioop5qhBu1aoV0tLSsH//fgB/fUzlJ598Ajs7u2qZHBERUV32VNeEH31K0o4dO1BQUFClEyIiIqovnulRhnx0IRERUeU9VQgrFIpS13x5DZiIiKhynuqasBACo0ePlh7ScO/ePYwfP77U3dGbN2+uuhlSrVJUWFitT9KytLRE48aNq218IqK/01OF8KhRo/Rev/rqq1U6GarddPk5uJJ8GVNnRFTb07SsLcwQHfU5g5iI6oSnCuGoqKjqmgfVAUW6uyhRGMK2awBstI5VPn7B7ZvIPLIJubm5DGEiqhMq9WEdRI9jZtUYluqm1TJ2ZrWMSkQkj2e6O5qIiIgqr8aHcPPmzaW7sh/eJk6cCAAYPXp0qb6uXbvqjaHT6TB58mTY2trC3Nwc/v7+uHbtml5NVlYWgoKCoFKpoFKpEBQUhOzs7L9rmUREVA/V+BBOSEhAWlqatO3evRsA8K9//Uuq6d+/v17Njz/+qDfG1KlTsWXLFsTGxuLQoUPIz8+Hn58fiouLpZrAwEAkJiYiLi4OcXFxSExMRFBQ0N+zSCIiqpdq/DXhR2/AmTdvHlq0aIEePXpIbUqlEhqNpsz9c3JysGbNGmzYsAF9+vQBAERHR8PBwQF79uxBv379cOHCBcTFxSE+Ph4eHh4AgNWrV8PT0xNJSUlwdnauptUREVF9VuOPhB9WWFiI6OhojB07Vu9DQg4cOAC1Wo3WrVsjJCQEGRkZUt+JEydQVFQEHx8fqU2r1cLFxQWHDx8GABw5cgQqlUoKYADo2rUrVCqVVFMWnU6H3NxcvY2IiKiialUIb926FdnZ2Rg9erTU5uvri5iYGOzbtw8LFy5EQkICevXqBZ1OBwBIT0+HsbExrKys9Mays7NDenq6VKNWq0u9n1qtlmrKEhkZKV1DVqlUcHBwqIJVEhFRfVHjT0c/bM2aNfD19YVWq5Xahg0bJv3ZxcUF7u7ucHR0xPbt2xEQEFDuWEIIvaPpsj5+89GaR4WHh2PatGnS69zcXAYxERFVWK0J4ZSUFOzZs+eJH4lpb28PR0dHXLx4EQCg0WhQWFiIrKwsvaPhjIwMeHl5STU3b94sNVZmZuZjH9OoVCqr7ZOhiIio7qs1p6OjoqKgVqsxcODAx9bdunULqampsLe3BwB06dIFRkZG0l3VAJCWloazZ89KIezp6YmcnBwcO3ZMqjl69ChycnKkGiIioqpWK46ES0pKEBUVhVGjRsHQ8P+nnJ+fj4iICAwZMgT29va4cuUKZsyYAVtbWwwePBgAoFKpEBwcjNDQUNjY2MDa2hphYWFwdXWV7pZu27Yt+vfvj5CQEKxatQoAMG7cOPj5+fHOaCIiqja1IoT37NmDq1evYuzYsXrtBgYGOHPmDNavX4/s7GzY29ujZ8+e2LhxIywsLKS6xYsXw9DQEEOHDsXdu3fRu3dvrF27FgYGBlJNTEwMpkyZIt1F7e/vj2XLlv09CyQionqpVoSwj48PhBCl2k1NTbFz584n7m9iYoKlS5di6dKl5dZYW1sjOjr6meZJRET0NGrNNWEiIqK6hiFMREQkE4YwERGRTBjCREREMmEIExERyYQhTEREJBOGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMmEIExERyYQhTEREJBOGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMmEIExERyYQhTEREJBOGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMGMJEREQyYQgTERHJhCFMREQkkxodwhEREVAoFHqbRqOR+oUQiIiIgFarhampKby9vXHu3Dm9MXQ6HSZPngxbW1uYm5vD398f165d06vJyspCUFAQVCoVVCoVgoKCkJ2d/XcskYiI6rEaHcIA0L59e6SlpUnbmTNnpL758+dj0aJFWLZsGRISEqDRaNC3b1/k5eVJNVOnTsWWLVsQGxuLQ4cOIT8/H35+figuLpZqAgMDkZiYiLi4OMTFxSExMRFBQUF/6zqJiKj+MZR7Ak9iaGiod/T7gBACS5YswbvvvouAgAAAwLp162BnZ4cvv/wSr7/+OnJycrBmzRps2LABffr0AQBER0fDwcEBe/bsQb9+/XDhwgXExcUhPj4eHh4eAIDVq1fD09MTSUlJcHZ2/vsWS0RE9UqNPxK+ePEitFotnJycMHz4cFy+fBkAkJycjPT0dPj4+Ei1SqUSPXr0wOHDhwEAJ06cQFFRkV6NVquFi4uLVHPkyBGoVCopgAGga9euUKlUUk15dDodcnNz9TYiIqKKqtEh7OHhgfXr12Pnzp1YvXo10tPT4eXlhVu3biE9PR0AYGdnp7ePnZ2d1Jeeng5jY2NYWVk9tkatVpd6b7VaLdWUJzIyUrqOrFKp4ODgUOm1EhFR/VOjQ9jX1xdDhgyBq6sr+vTpg+3btwP467TzAwqFQm8fIUSptkc9WlNWfUXGCQ8PR05OjrSlpqY+cU1EREQP1OgQfpS5uTlcXV1x8eJF6Trxo0erGRkZ0tGxRqNBYWEhsrKyHltz8+bNUu+VmZlZ6ij7UUqlEpaWlnobERFRRdWqENbpdLhw4QLs7e3h5OQEjUaD3bt3S/2FhYU4ePAgvLy8AABdunSBkZGRXk1aWhrOnj0r1Xh6eiInJwfHjh2Tao4ePYqcnByphoiIqDrU6Lujw8LCMGjQIDRr1gwZGRn44IMPkJubi1GjRkGhUGDq1KmYO3cuWrVqhVatWmHu3LkwMzNDYGAgAEClUiE4OBihoaGwsbGBtbU1wsLCpNPbANC2bVv0798fISEhWLVqFQBg3Lhx8PPz453RRERUrWp0CF+7dg2vvPIK/vzzTzRu3Bhdu3ZFfHw8HB0dAQDTp0/H3bt3MWHCBGRlZcHDwwO7du2ChYWFNMbixYthaGiIoUOH4u7du+jduzfWrl0LAwMDqSYmJgZTpkyR7qL29/fHsmXL/t7FEhFRvVOjQzg2Nvax/QqFAhEREYiIiCi3xsTEBEuXLsXSpUvLrbG2tkZ0dHRlp0lERFQpteqaMBERUV3CECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmDGEiIiKZMISJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImIiGTCECYiIpIJQ5iIiEgmNTqEIyMj8dxzz8HCwgJqtRovvfQSkpKS9GpGjx4NhUKht3Xt2lWvRqfTYfLkybC1tYW5uTn8/f1x7do1vZqsrCwEBQVBpVJBpVIhKCgI2dnZ1b1EIiKqx2p0CB88eBATJ05EfHw8du/ejfv378PHxwcFBQV6df3790daWpq0/fjjj3r9U6dOxZYtWxAbG4tDhw4hPz8ffn5+KC4ulmoCAwORmJiIuLg4xMXFITExEUFBQX/LOomIqH4ylHsCjxMXF6f3OioqCmq1GidOnMCLL74otSuVSmg0mjLHyMnJwZo1a7Bhwwb06dMHABAdHQ0HBwfs2bMH/fr1w4ULFxAXF4f4+Hh4eHgAAFavXg1PT08kJSXB2dm5zLF1Oh10Op30Ojc395nWS0RE9UuNPhJ+VE5ODgDA2tpar/3AgQNQq9Vo3bo1QkJCkJGRIfWdOHECRUVF8PHxkdq0Wi1cXFxw+PBhAMCRI0egUqmkAAaArl27QqVSSTVliYyMlE5fq1QqODg4VMk6iYiofqg1ISyEwLRp09C9e3e4uLhI7b6+voiJicG+ffuwcOFCJCQkoFevXtIRanp6OoyNjWFlZaU3np2dHdLT06UatVpd6j3VarVUU5bw8HDk5ORIW2pqalUslYiI6okafTr6YZMmTcKvv/6KQ4cO6bUPGzZM+rOLiwvc3d3h6OiI7du3IyAgoNzxhBBQKBTS64f/XF7No5RKJZRK5dMsg55RUWEhUlJSqmVsS0tLNG7cuFrGJiIqS60I4cmTJ2Pbtm346aef0LRp08fW2tvbw9HRERcvXgQAaDQaFBYWIisrS+9oOCMjA15eXlLNzZs3S42VmZkJOzu7KlwJPQtdfg6uJF/G1BkR1fI/P9YWZoiO+pxBTER/mxodwkIITJ48GVu2bMGBAwfg5OT0xH1u3bqF1NRU2NvbAwC6dOkCIyMj7N69G0OHDgUApKWl4ezZs5g/fz4AwNPTEzk5OTh27Bief/55AMDRo0eRk5MjBTXJr0h3FyUKQ9h2DYCN1rFKxy64fROZRzYhNzeXIUxEf5saHcITJ07El19+ie+++w4WFhbS9VmVSgVTU1Pk5+cjIiICQ4YMgb29Pa5cuYIZM2bA1tYWgwcPlmqDg4MRGhoKGxsbWFtbIywsDK6urtLd0m3btkX//v0REhKCVatWAQDGjRsHPz+/cu+MJvmYWTWGpfrxZ0QqI7PKRyQierwaHcIrVqwAAHh7e+u1R0VFYfTo0TAwMMCZM2ewfv16ZGdnw97eHj179sTGjRthYWEh1S9evBiGhoYYOnQo7t69i969e2Pt2rUwMDCQamJiYjBlyhTpLmp/f38sW7as+hdJRET1Vo0OYSHEY/tNTU2xc+fOJ45jYmKCpUuXYunSpeXWWFtbIzo6+qnnSEREVFm15leUiIiI6hqGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMmEIExERyYQhTEREJBOGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMGMJEREQyYQgTERHJxFDuCRDVFEWFhUhJSam28S0tLdG4ceNqG5+Iah+GMBEAXX4OriRfxtQZEVAqldXyHtYWZoiO+pxBTEQShjARgCLdXZQoDGHbNQA2WscqH7/g9k1kHtmE3NxchjARSRjCRA8xs2oMS3XTahk7s1pGJaLajDdmERERyYQhTEREJBOGMBERkUwYwkRERDJhCBMREcmEIUxERCQThjAREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMuEDHIj+JtX5vGI+q5iodmIIE/0Nqvt5xXxWMVHtxBB+xPLly7FgwQKkpaWhffv2WLJkCV544QW5p0W1XHU+r5jPKqbKyMzMRG5ubrWNz7MzFcMQfsjGjRsxdepULF++HN26dcOqVavg6+uL8+fPo1mzZnJPj+qA6npeMZ9VXDdVV1DeunULb/8nAvm6oiof+wGenakYhvBDFi1ahODgYLz22msAgCVLlmDnzp1YsWIFIiMjZZ4dUfmq83ozwKOa8lTn0WR1BuW9u3dw7Xoa3If/G43sqv5/Cgtu38SNg1/hzJkzcHSs2jM/QN36eWQI/09hYSFOnDiBd955R6/dx8cHhw8fLnMfnU4HnU4nvc7JyQGAZ/pLmZeXh+L795GddgVF9+5Uepzy5GZcgygpQW56KgwVtWfs6h6/Ns8968ZlJP9xCZOn/6darjcDQEOlISL+Ew5ra+tqGb82un37NiLmRCL/3v1qGf/evTu4ceMmnHsPg4WVTZWOXXIjGfdTrkF3p6Ba/p0pyM6s1p/J6v55bNSo0TON/SADhBBPLhYkhBDi+vXrAoD45Zdf9NrnzJkjWrduXeY+s2bNEgC4cePGjRu3UltqauoTs4dHwo9QKPQPU4QQpdoeCA8Px7Rp06TXJSUluH37NmxsbMrdpzy5ublwcHBAamoqLC0tn37itQDXWDdwjbVfXV8fIO8ahRDIy8uDVqt9Yi1D+H9sbW1hYGCA9PR0vfaMjAzY2dmVuY9SqSx1qqVRo0bPNA9LS8s6+5fiAa6xbuAaa7+6vj5AvjWqVKoK1fETs/7H2NgYXbp0we7du/Xad+/eDS8vL5lmRUREdRmPhB8ybdo0BAUFwd3dHZ6envjss89w9epVjB8/Xu6pERFRHcQQfsiwYcNw69YtvPfee0hLS4OLiwt+/PHHarnF/lFKpRKzZs2qtrtbawKusW7gGmu/ur4+oPasUSFERe6hJiIioqrGa8JEREQyYQgTERHJhCFMREQkE4YwERGRTBjCNcTy5cvh5OQEExMTdOnSBT///LPcU6q0n376CYMGDYJWq4VCocDWrVv1+oUQiIiIgFarhampKby9vXHu3Dl5JlsJkZGReO6552BhYQG1Wo2XXnoJSUlJejW1fY0rVqxAhw4dpA868PT0xI4dO6T+2r6+R0VGRkKhUGDq1KlSW11YY0REBBQKhd6m0Wik/rqwxuvXr+PVV1+FjY0NzMzM0KlTJ5w4cULqr+lrZAjXAA8eofjuu+/i1KlTeOGFF+Dr64urV6/KPbVKKSgoQMeOHbFs2bIy++fPn49FixZh2bJlSEhIgEajQd++fZGXl/c3z7RyDh48iIkTJyI+Ph67d+/G/fv34ePjg4KCAqmmtq+xadOmmDdvHo4fP47jx4+jV69e+Oc//yn941Xb1/ewhIQEfPbZZ+jQoYNee11ZY/v27ZGWliZtZ86ckfpq+xqzsrLQrVs3GBkZYceOHTh//jwWLlyo98mFNX6Nz/DMA6oizz//vBg/frxeW5s2bcQ777wj04yqDgCxZcsW6XVJSYnQaDRi3rx5Utu9e/eESqUSK1eulGGGzy4jI0MAEAcPHhRC1M01CiGElZWV+Pzzz+vU+vLy8kSrVq3E7t27RY8ePcSbb74phKg738NZs2aJjh07ltlXF9b49ttvi+7du5fbXxvWyCNhmT14hKKPj49e++MeoVibJScnIz09XW+9SqUSPXr0qLXrffAIywePPqtraywuLkZsbCwKCgrg6elZp9Y3ceJEDBw4EH369NFrr0trvHjxIrRaLZycnDB8+HBcvnwZQN1Y47Zt2+Du7o5//etfUKvVcHNzw+rVq6X+2rBGhrDM/vzzTxQXF5d6SISdnV2ph0nUBQ/WVFfWK4TAtGnT0L17d7i4uACoO2s8c+YMGjZsCKVSifHjx2PLli1o165dnVlfbGwsTp48icjIyFJ9dWWNHh4eWL9+PXbu3InVq1cjPT0dXl5euHXrVp1Y4+XLl7FixQq0atUKO3fuxPjx4zFlyhSsX78eQO34PvJjK2uIp3mEYl1QV9Y7adIk/Prrrzh06FCpvtq+RmdnZyQmJiI7OxubNm3CqFGjcPDgQam/Nq8vNTUVb775Jnbt2gUTE5Ny62rzGgHA19dX+rOrqys8PT3RokULrFu3Dl27dgVQu9dYUlICd3d3zJ07FwDg5uaGc+fOYcWKFRg5cqRUV5PXyCNhmVXmEYq12YM7M+vCeidPnoxt27Zh//79aNq0qdReV9ZobGyMli1bwt3dHZGRkejYsSM+/vjjOrG+EydOICMjA126dIGhoSEMDQ1x8OBBfPLJJzA0NJTWUZvXWBZzc3O4urri4sWLdeL7aG9vj3bt2um1tW3bVrqptTaskSEss/r2CEUnJydoNBq99RYWFuLgwYO1Zr1CCEyaNAmbN2/Gvn374OTkpNdfF9ZYFiEEdDpdnVhf7969cebMGSQmJkqbu7s7RowYgcTERPzjH/+o9Wssi06nw4ULF2Bvb18nvo/dunUr9euBv//+u/TQnVqxRrnuCKP/FxsbK4yMjMSaNWvE+fPnxdSpU4W5ubm4cuWK3FOrlLy8PHHq1Clx6tQpAUAsWrRInDp1SqSkpAghhJg3b55QqVRi8+bN4syZM+KVV14R9vb2Ijc3V+aZV8wbb7whVCqVOHDggEhLS5O2O3fuSDW1fY3h4eHip59+EsnJyeLXX38VM2bMEA0aNBC7du0SQtT+9ZXl4bujhagbawwNDRUHDhwQly9fFvHx8cLPz09YWFhI/7bU9jUeO3ZMGBoaijlz5oiLFy+KmJgYYWZmJqKjo6Wamr5GhnAN8emnnwpHR0dhbGwsOnfuLP26S220f/9+AaDUNmrUKCHEX782MGvWLKHRaIRSqRQvvviiOHPmjLyTfgplrQ2AiIqKkmpq+xrHjh0r/Tw2btxY9O7dWwpgIWr/+sryaAjXhTUOGzZM2NvbCyMjI6HVakVAQIA4d+6c1F8X1vj9998LFxcXoVQqRZs2bcRnn32m11/T18hHGRIREcmE14SJiIhkwhAmIiKSCUOYiIhIJgxhIiIimTCEiYiIZMIQJiIikglDmIiISCYMYSIiIpkwhImoQry9vTF16lTpdfPmzbFkyRLZ5kNUFzCEiahSEhISMG7cOLmngbVr16JRo0ZyT4OoUvg8YSKqlMaNG8s9hSpVXFwMhUKBBg14bEJ/H/60EdVwJSUl+PDDD9GyZUsolUo0a9YMc+bMAQC8/fbbaN26NczMzPCPf/wDM2fORFFRkbRvREQEOnXqhFWrVsHBwQFmZmb417/+hezs7Me+Z0FBAUaOHImGDRvC3t4eCxcuLFXz6OnoRYsWwdXVFebm5nBwcMCECROQn58v9T84Yv3hhx/g7OwMMzMzvPzyyygoKMC6devQvHlzWFlZYfLkySguLpb2KywsxPTp09GkSROYm5vDw8MDBw4cAAAcOHAAY8aMQU5ODhQKBRQKBSIiIp6436PzadeuHZRKJVJSUir4XSGqGjwSJqrhwsPDsXr1aixevBjdu3dHWloafvvtNwCAhYUF1q5dC61WizNnziAkJAQWFhaYPn26tP+lS5fw9ddf4/vvv0dubi6Cg4MxceJExMTElPueb731Fvbv348tW7ZAo9FgxowZOHHiBDp16lTuPg0aNMAnn3yC5s2bIzk5GRMmTMD06dOxfPlyqebOnTv45JNPEBsbi7y8PAQEBCAgIACNGjXCjz/+iMuXL2PIkCHo3r07hg0bBgAYM2YMrly5gtjYWGi1WmzZsgX9+/fHmTNn4OXlhSVLluC///2v9FzZhg0bPnG/Vq1aSfOJjIzE559/DhsbG6jV6sp9k4gqS+7HOBFR+XJzc4VSqRSrV6+uUP38+fNFly5dpNezZs0SBgYGIjU1VWrbsWOHaNCggUhLSytzjLy8PGFsbCxiY2Oltlu3bglTU1O9R/05OjqKxYsXlzuXr7/+WtjY2Eivo6KiBABx6dIlqe31118XZmZmIi8vT2rr16+feP3114UQQly6dEkoFApx/fp1vbF79+4twsPDpXFVKpVef0X3AyASExPLXQNRdeORMFENduHCBeh0OvTu3bvM/m+//RZLlizBpUuXkJ+fj/v378PS0lKvplmzZmjatKn02tPTEyUlJUhKSsLFixfh6+sr9a1atQouLi4oLCyEp6en1G5tbQ1nZ+fHznX//v2YO3cuzp8/j9zcXNy/fx/37t1DQUEBzM3NAQBmZmZo0aKFtI+dnR2aN28uHb0+aMvIyAAAnDx5EkIItG7dWu+9dDodbGxsyp1LRfczNjZGhw4dHrsuourEECaqwUxNTcvti4+Px/DhwzF79mz069cPKpUKsbGxZV6/fZhCoZD+6+7ujsTERKnPzs4Of/zxx1PPMyUlBQMGDMD48ePx/vvvw9raGocOHUJwcLDeNWojI6NScymrraSkBMBf18MNDAxw4sQJGBgY6NU9HNyPquh+pqam0teDSA4MYaIarFWrVjA1NcXevXvx2muv6fX98ssvcHR0xLvvviu1lXVj0dWrV3Hjxg1otVoAwJEjR9CgQQO0bt0apqamaNmypV59y5YtYWRkhPj4eDRr1gwAkJWVhd9//x09evQoc57Hjx/H/fv3sXDhQunu4q+//rryC/8fNzc3FBcXIyMjAy+88EKZNcbGxno3clV0P6KagCFMVIOZmJjg7bffxvTp02FsbIxu3bohMzMT586dQ8uWLXH16lXExsbiueeew/bt27Fly5Yyxxg1ahQ++ugj5ObmYsqUKRg6dCg0Gk2Z79mwYUMEBwfjrbfego2NDezs7PDuu+8+9ld3WrRogfv372Pp0qUYNGgQfvnlF6xcufKZ19+6dWuMGDECI0eOxMKFC+Hm5oY///wT+/btg6urKwYMGIDmzZsjPz8fe/fuRceOHWFmZlah/YhqAv6KElENN3PmTISGhuK///0v2rZti2HDhiEjIwP//Oc/8e9//xuTJk1Cp06dcPjwYcycObPU/i1btkRAQAAGDBgAHx8fuLi46N2xXJYFCxbgxRdfhL+/P/r06YPu3bujS5cu5dZ36tQJixYtwocffggXFxfExMQgMjLymdcOAFFRURg5ciRCQ0Ph7OwMf39/HD16FA4ODgAALy8vjB8/HsOGDUPjxo0xf/78Cu1HVBMohBBC7kkQUfWIiIjA1q1b9a77ElHNwSNhIiIimTCEiYiIZMLT0URERDLhkTAREZFMGMJEREQyYQgTERHJhCFMREQkE4YwERGRTBjCREREMmEIExERyYQhTEREJJP/A5v9fmNJM+jiAAAAAElFTkSuQmCC",
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "numeric_columns = X_train.select_dtypes(include='number') # Select only numeric columns\n",
+ "\n",
+ "for column in numeric_columns.columns:\n",
+ " plt.figure(figsize=(5,5))\n",
+ " plt.hist(X_train[column], bins=15, edgecolor='black', alpha=0.7)\n",
+ " plt.title(f'Histogram of {column}')\n",
+ " plt.xlabel(column)\n",
+ " plt.ylabel('Frequency')\n",
+ " plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "717bbe9d-18ee-4463-9a52-9128566851d5",
+ "metadata": {},
+ "source": [
+ "Based on the histograms, here are our findings for each feature being plotted.\n",
+ "\n",
+ "1. `cap-diameter`: The distribution is highly skewed to the right, with most values concentrated between 0 and 10 cm. There are also some outliers sitting at around 40 to 60 cm. \n",
+ "\n",
+ "2. `stem-height`: Slightly right-skewed distribution. The majority of mushrooms have stem heights between 4 and 10 cm, with few having stem heights over 20 cm.\n",
+ "\n",
+ "3. `stem-width`: Another heavily right-skewed distribution, with the majority of mushrooms having stem width below 20 cm, and a some rare cases exceeding 50 cm.\n",
+ "\n",
+ "The skewness observed across the 3 numeric features will be addressed in the preprocessing phase with `QuantileTransformer` from `sklearn.preprocessing` which maps data to a normal distribution while retaining the relative rank of values, making them more suitable for models sensitive to feature distributions, such as `SVC` and `LogisticRegression`. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1b80b242-d2c4-48ba-9124-0f1a75233cf7",
+ "metadata": {},
+ "source": [
+ "##### Part 3: The distribution of categorical features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fc0e83a8-e064-416d-9043-c9a340d24f18",
+ "metadata": {},
+ "source": [
+ "To understand the categorical features in the data set, we analyzed their frequency and percentage distributions, providing insights into the variability and class imbalance that might occur for each feature. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "c5d9010f-313a-4abf-a50c-e55c3b64b186",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Frequency and Percentage for 'cap-shape':\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "---------------------------------------- \n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "categorical_columns = X_train.select_dtypes(include='object') # Select only categorical columns\n",
+ "\n",
+ "# Calculate frequency and percentage for each categorical features\n",
+ "for column in categorical_columns.columns:\n",
+ " print(f\"Frequency and Percentage for '{column}':\")\n",
+ " \n",
+ " # Frequency\n",
+ " frequency = X_train[column].value_counts()\n",
+ " # Percentage\n",
+ " percentage = round(X_train[column].value_counts(normalize=True) * 100, 2)\n",
+ " \n",
+ " # Combine into one DataFrame\n",
+ " freq_percent_df = pd.DataFrame({\n",
+ " \"Frequency\": frequency,\n",
+ " \"Percentage\": percentage\n",
+ " })\n",
+ "\n",
+ " # Highlight values with a gradient\n",
+ " styled_df = freq_percent_df.style.format(\n",
+ " precision=2\n",
+ " ).background_gradient(\n",
+ " subset=['Percentage'],\n",
+ " cmap='YlOrRd'\n",
+ " )\n",
+ "\n",
+ " # Display the styled DataFrame\n",
+ " display(styled_df)\n",
+ " print(\"-\" * 40, '\\n')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "12823bac-ddf4-47d7-bda9-785b424a1837",
+ "metadata": {},
+ "source": [
+ "Based on the Frequency and Percentage distributions, here are our findings:\n",
+ "\n",
+ "1. `cap-shape`: The most common cap shape is `x` (convex), comprising 43.97% of the data. Other shapes like `f` (flat) and `s` (sunken) are also prevalent, while `c` (conical) is the least common with 2.95% appearance.\n",
+ "\n",
+ "2. `cap-color`: The most frequently appeared color is `n` (brown), with 39.71% of the data. Other colors like `y` (yellow), `w` (white), and `g` (gray) are also well-represented, while rare colors like `b` (buff) and `l` (blue) appear in less than 2% of the data.\n",
+ "\n",
+ "3. `does-bruise-or-bleed`: The majority of the mushrooms are `f` (do not bruise or bleed), while their counterpart make up 17.26% of the data.\n",
+ "\n",
+ "4. `gill-color`: The most common gill color is `w` (white), with 30.45% of the data. Other colors such as `n` (brown) and `y` (yellow) are also frequent, while rare gill colors like `e` (red), `b` (buff) and `u` (purple) appear in less than 2% of the data.\n",
+ "\n",
+ "5. `stem-color`: `w` (white) and `n` (brown) are the dominating stem colors, accounting for 37.75% and 29.5% of the data, respectively. Other colors like `r` (green), `l` (blue) and `b` (buff) are less frequent, appearing in less than 1% of the observations.\n",
+ "\n",
+ "6. `has-ring`: Most mushrooms are `f` (do not have a ring), with 74.84% observations. The remaining 25.16% mushrooms are `t` (have a ring).\n",
+ "\n",
+ "7. `ring-type`: `f` (none) is the most common ring type, accounting for 82.3% of the data. Other types like `e` (evanescent) and `z` (zone) are less frequent, while rare types like `m` (movable) occur in less than 1% of the data.\n",
+ "\n",
+ "8. `habitat`: The predominant habitat is `d` (woods), with 72.46% appearance. Other habitats such as `g` (grasses) and `l` (leaves) are less common, while `w` (waste), `p` (paths), and `u` (urban) only make up less than 1% of the data individually.\n",
+ "\n",
+ "9. `season`: Most mushrooms grow in `a` (autumn), comprising 49.36% of the data, followed by `u` (summer) at 37.5%. The other two seasons `w` (winter) and `s` (spring) are less frequent.\n",
+ "\n",
+ "Categorical features will be encoded into binary format in the following preprocessing phase with `OneHotEncoder`. Since we are dealing with a mix of binary and non-binary categorical features, for features like `does-bruise-or-bleed` and `has-ring` that have two unique values, they will be handled with `drop='if_binary'` argument to reduce redundancy while still capturing the information. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a8a13abe-906b-4230-8772-2d799a51a857",
+ "metadata": {},
+ "source": [
+ "##### Part 4: The distribution of the target"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e175de25-1893-40f0-a794-1153470d7230",
+ "metadata": {},
+ "source": [
+ "The target variable `class` represents whether a mushroom is `p` (poisonous) or `e` (edible). Understanding the distribution of the target helps assessing class balance, which might have impact on models' performance."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "9e47fdb0-f94a-4777-a3c1-954e7d62202a",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Frequency
\n",
+ "
Percentage
\n",
+ "
\n",
+ "
\n",
+ "
target
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
p
\n",
+ "
26996
\n",
+ "
55.41
\n",
+ "
\n",
+ "
\n",
+ "
e
\n",
+ "
21726
\n",
+ "
44.59
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Frequency Percentage\n",
+ "target \n",
+ "p 26996 55.41\n",
+ "e 21726 44.59"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ " # Frequency\n",
+ "frequency = y_train.value_counts()\n",
+ "# Percentage\n",
+ "percentage = round(y_train.value_counts(normalize=True) * 100, 2)\n",
+ "\n",
+ "# Combine into one DataFrame\n",
+ "freq_percent_df = pd.DataFrame({\n",
+ " \"Frequency\": frequency,\n",
+ " \"Percentage\": percentage\n",
+ "})\n",
+ "freq_percent_df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "10f25100-5adc-46c3-8503-6faebcc29100",
+ "metadata": {},
+ "source": [
+ "Based on the Frequency and Percentage distribution, here are our findings:\n",
+ "\n",
+ "1. `p` (Poisonous): There are 27,143 instances of poisonous mushrooms, accounting for 55.56% of the data.\n",
+ "\n",
+ "2. `e` (Edible): There are 21,712 instances of edible mushrooms, constituting 44.44% of the data.\n",
+ "\n",
+ "Using $F_{\\beta}$, precision, recall, or confusion matrix to evaluate the model's performance is advisable in the following procedure. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e6000a8d-7e7b-4fd1-aa27-b3c8074e6d91",
+ "metadata": {},
+ "source": [
+ "#### Preprocessing and Model Building\n",
+ "\n",
+ "Three classification models including Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Logistic Regression are used to predict whether a mushroom is edible or poisonous. Predicting a mushroom to be edible when it is in fact poisonous could have severe health consequences. Therefore the best model should prioritize the minimization of this error. To do this, we can evaluate models on an $F_{\\beta}$ score with $\\beta = 2$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "2b26607e-2448-452d-8e0b-e25c56ceba44",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# loading in some models\n",
+ "from sklearn.neighbors import KNeighborsClassifier\n",
+ "from sklearn.svm import SVC\n",
+ "from sklearn.linear_model import LogisticRegression"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "32e69dc9-c9c6-4734-8f01-37a6813ef1d8",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.