12. Data Cleaning/exercise/04. exercise-character-encodings.ipynb

{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Data Cleaning](https://www.kaggle.com/learn/data-cleaning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/character-encodings).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"In this exercise, you'll apply what you learned in the **Character encodings** tutorial.\n\n# Setup\n\nThe questions below will give you feedback on your work. Run the following cell to set up the feedback system.","metadata":{}},{"cell_type":"code","source":"from learntools.core import binder\nbinder.bind(globals())\nfrom learntools.data_cleaning.ex4 import *\nprint(\"Setup Complete\")","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:50:57.927488Z","iopub.execute_input":"2022-07-20T17:50:57.927905Z","iopub.status.idle":"2022-07-20T17:50:58.026556Z","shell.execute_reply.started":"2022-07-20T17:50:57.927817Z","shell.execute_reply":"2022-07-20T17:50:58.025359Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"# Get our environment set up\n\nThe first thing we'll need to do is load in the libraries we'll be using.","metadata":{}},{"cell_type":"code","source":"# modules we'll use\nimport pandas as pd\nimport numpy as np\n\n# helpful character encoding module\nimport chardet\n\n# set seed for reproducibility\nnp.random.seed(0)","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:51:03.233632Z","iopub.execute_input":"2022-07-20T17:51:03.234061Z","iopub.status.idle":"2022-07-20T17:51:03.240550Z","shell.execute_reply.started":"2022-07-20T17:51:03.234012Z","shell.execute_reply":"2022-07-20T17:51:03.239173Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"markdown","source":"# 1) What are encodings?\n\nYou're working with a dataset composed of bytes.  Run the code cell below to print a sample entry.","metadata":{}},{"cell_type":"code","source":"sample_entry = b'\\xa7A\\xa6n'\nprint(sample_entry)\nprint('data type:', type(sample_entry))","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:51:06.207664Z","iopub.execute_input":"2022-07-20T17:51:06.208042Z","iopub.status.idle":"2022-07-20T17:51:06.214696Z","shell.execute_reply.started":"2022-07-20T17:51:06.208012Z","shell.execute_reply":"2022-07-20T17:51:06.213843Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"markdown","source":"You notice that it doesn't use the standard UTF-8 encoding. \n\nUse the next code cell to create a variable `new_entry` that changes the encoding from `\"big5-tw\"` to `\"utf-8\"`.  `new_entry` should have the bytes datatype.","metadata":{}},{"cell_type":"code","source":"before = sample_entry.decode(\"big5-tw\")\nnew_entry = before.encode()\n\n# Check your answer\nq1.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:51:17.285147Z","iopub.execute_input":"2022-07-20T17:51:17.285648Z","iopub.status.idle":"2022-07-20T17:51:17.298909Z","shell.execute_reply.started":"2022-07-20T17:51:17.285607Z","shell.execute_reply":"2022-07-20T17:51:17.297509Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q1.hint()\n#q1.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:51:19.688517Z","iopub.execute_input":"2022-07-20T17:51:19.688979Z","iopub.status.idle":"2022-07-20T17:51:19.695200Z","shell.execute_reply.started":"2022-07-20T17:51:19.688941Z","shell.execute_reply":"2022-07-20T17:51:19.693159Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"markdown","source":"# 2) Reading in files with encoding problems\n\nUse the code cell below to read in this file at path `\"../input/fatal-police-shootings-in-the-us/PoliceKillingsUS.csv\"`.  \n\nFigure out what the correct encoding should be and read in the file to a DataFrame `police_killings`.","metadata":{}},{"cell_type":"code","source":"# TODO: Load in the DataFrame correctly.\npolice_killings = pd.read_csv(\"../input/fatal-police-shootings-in-the-us/PoliceKillingsUS.csv\", encoding='Windows-1252')\n\n# Check your answer\nq2.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:51:49.389547Z","iopub.execute_input":"2022-07-20T17:51:49.390028Z","iopub.status.idle":"2022-07-20T17:51:49.418574Z","shell.execute_reply.started":"2022-07-20T17:51:49.389943Z","shell.execute_reply":"2022-07-20T17:51:49.417443Z"},"trusted":true},"execution_count":11,"outputs":[]},{"cell_type":"markdown","source":"Feel free to use any additional code cells for supplemental work.  To get credit for finishing this question, you'll need to run `q2.check()` and get a result of **Correct**.","metadata":{}},{"cell_type":"code","source":"# (Optional) Use this code cell for any additional work.","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q2.hint()\n#q2.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:51:52.676881Z","iopub.execute_input":"2022-07-20T17:51:52.677259Z","iopub.status.idle":"2022-07-20T17:51:52.682593Z","shell.execute_reply.started":"2022-07-20T17:51:52.677229Z","shell.execute_reply":"2022-07-20T17:51:52.681335Z"},"trusted":true},"execution_count":12,"outputs":[]},{"cell_type":"markdown","source":"# 3) Saving your files with UTF-8 encoding\n\nSave a version of the police killings dataset to CSV with UTF-8 encoding.  Your answer will be marked correct after saving this file.  \n\nNote: When using the `to_csv()` method, supply only the name of the file (e.g., `\"my_file.csv\"`).  This saves the file at the filepath `\"/kaggle/working/my_file.csv\"`.","metadata":{}},{"cell_type":"code","source":"# TODO: Save the police killings dataset to CSV\npolice_killings.to_csv(\"my_file.csv\")\n\n# Check your answer\nq3.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:52:43.454655Z","iopub.execute_input":"2022-07-20T17:52:43.455974Z","iopub.status.idle":"2022-07-20T17:52:43.511636Z","shell.execute_reply.started":"2022-07-20T17:52:43.455922Z","shell.execute_reply":"2022-07-20T17:52:43.510220Z"},"trusted":true},"execution_count":14,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q3.hint()\n#q3.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-20T17:52:48.894388Z","iopub.execute_input":"2022-07-20T17:52:48.894858Z","iopub.status.idle":"2022-07-20T17:52:48.900393Z","shell.execute_reply.started":"2022-07-20T17:52:48.894819Z","shell.execute_reply":"2022-07-20T17:52:48.898632Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":"# (Optional) More practice\n\nCheck out [this dataset of files in different character encodings](https://www.kaggle.com/rtatman/character-encoding-examples). Can you read in all the files with their original encodings and them save them out as UTF-8 files?\n\nIf you have a file that's in UTF-8 but has just a couple of weird-looking characters in it, you can try out the [ftfy module](https://ftfy.readthedocs.io/en/latest/#) and see if it helps. \n\n# Keep going\n\nIn the final lesson, learn how to [**clean up inconsistent text entries**](https://www.kaggle.com/alexisbcook/inconsistent-data-entry) in your dataset.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/data-cleaning/discussion) to chat with other learners.*","metadata":{}}]}