Stack Overflow Annual Developer Survey_2023_ Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Stack Overflow Annual Developer survey 2023 results Analysis. We will answer the next questions:\n",
    "1. How many respondents completed the survey?\n",
    "2. How many respondents answered all mandatory questions?\n",
    "3. What are the median values for the respondents' work experience (WorkExp)?\n",
    "4. How many respondents work remotely?\n",
    "5. What percentage of respondents program in Python?\n",
    "6. How many respondents learned to program through online courses?\n",
    "7. What is the average and median compensation (ConvertedCompYearly) among respondents who program in Python in each country?\n",
    "8. What education levels do the top 5 highest paid respondents have?\n",
    "9. What percentage of respondents program in Python in each age category?\n",
    "10. Which industries are the most prevalent among respondents (who work remotely) in the 75th percentile for average compensation?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "1b1e033a0b531684"
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "survey = pd.read_csv('C:/Users/plish/Desktop/Python_HW/Stack Overflow Developer Survey 2023/survey_results_public.csv') \n",
    "schema = pd.read_csv('C:/Users/plish/Desktop/Python_HW/Stack Overflow Developer Survey 2023/schema.csv')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T14:37:38.757384900Z",
     "start_time": "2023-12-27T14:37:38.748689300Z"
    }
   },
   "id": "c770c24e34eb8f1c"
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [],
   "source": [
    "url = 'https://drive.google.com/file/d/1NjmLdZsLfVf9WyX0S69JLkL9Ff_qVLTJ/view?usp=sharing'\n",
    "url= 'https://drive.google.com/uc?id=' + url.split('/')[-2]\n",
    "survey = pd.read_csv(url)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T14:39:17.327845800Z",
     "start_time": "2023-12-27T14:39:17.034388700Z"
    }
   },
   "id": "bf4cb53f9bb3628f"
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 0 entries\n",
      "Data columns (total 3 columns):\n",
      " #   Column                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             Non-Null Count  Dtype \n",
      "---  ------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             --------------  ----- \n",
      " 0   <!DOCTYPE html><html><head><title>Google Drive - Virus scan warning</title><meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\"/><style nonce=\"Qqyh3eD6bm9D-ThKItESWw\">.goog-link-button{position:relative;color:#15c;text-decoration:underline;cursor:pointer}.goog-link-button-disabled{color:#ccc;text-decoration:none;cursor:default}body{color:#222;font:normal 13px/1.4 arial                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  0 non-null      object\n",
      " 1   sans-serif;margin:0}.grecaptcha-badge{visibility:hidden}.uc-main{padding-top:50px;text-align:center}#uc-dl-icon{display:inline-block;margin-top:16px;padding-right:1em;vertical-align:top}#uc-text{display:inline-block;max-width:68ex;text-align:left}.uc-error-caption                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           0 non-null      object\n",
      " 2   .uc-warning-caption{color:#222;font-size:16px}#uc-download-link{text-decoration:none}.uc-name-size a{color:#15c;text-decoration:none}.uc-name-size a:visited{color:#61c;text-decoration:none}.uc-name-size a:active{color:#d14836;text-decoration:none}.uc-footer{color:#777;font-size:11px;padding-bottom:5ex;padding-top:5ex;text-align:center}.uc-footer a{color:#15c}.uc-footer a:visited{color:#61c}.uc-footer a:active{color:#d14836}.uc-footer-divider{color:#ccc;width:100%}.goog-inline-block{position:relative;display:-moz-inline-box;display:inline-block}* html .goog-inline-block{display:inline}*:first-child+html .goog-inline-block{display:inline}sentinel{}</style><link rel=\"icon\" href=\"//ssl.gstatic.com/docs/doclist/images/drive_2022q3_32dp.png\"/></head><body><div class=\"uc-main\"><div id=\"uc-dl-icon\" class=\"image-container\"><div class=\"drive-sprite-aux-download-file\"></div></div><div id=\"uc-text\"><p class=\"uc-warning-caption\">Google Drive can't scan this file for viruses.</p><p class=\"uc-warning-subcaption\"><span class=\"uc-name-size\"><a href=\"/open?id=1NjmLdZsLfVf9WyX0S69JLkL9Ff_qVLTJ\">survey_results_public.csv</a> (151M)</span> is too large for Google to scan for viruses. Would you still like to download this file?</p><form id=\"download-form\" action=\"https://drive.google.com/uc?id=1NjmLdZsLfVf9WyX0S69JLkL9Ff_qVLTJ&amp;confirm=t&amp;uuid=a1487921-1490-4bd6-83e5-83626cb751a0\" method=\"post\"><input type=\"submit\" id=\"uc-download-link\" class=\"goog-inline-block jfk-button jfk-button-action\" value=\"Download anyway\"/></form></div></div><div class=\"uc-footer\"><hr class=\"uc-footer-divider\"></div></body></html>  0 non-null      object\n",
      "dtypes: object(3)\n",
      "memory usage: 132.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "survey.info() # to check the data structure"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T14:39:19.449071100Z",
     "start_time": "2023-12-27T14:39:19.439238800Z"
    }
   },
   "id": "5fcf8ed81194992c"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 1. How many respondents completed the survey?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e2a81f8d6c719df7"
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "89184 respondents took part in the survey\n"
     ]
    }
   ],
   "source": [
    "# number of respondents compleated the survey\n",
    "Q1 = survey['ResponseId'].nunique()\n",
    "print(f'{Q1} respondents took part in the survey')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:07.903083100Z",
     "start_time": "2023-12-27T13:49:07.891579200Z"
    }
   },
   "id": "4556049f5883e9c7"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 2. How many respondents answered all mandatory questions?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "c640caf10a264a7a"
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2032 respondents have answered all the questions in the survey\n"
     ]
    }
   ],
   "source": [
    "# number of respondents answered all the mandatory questions\n",
    "questions = set(schema.qname.unique()) & set(survey.columns) # connecting survey with survey schema to get all mandatory questions\n",
    "Q2 = survey.dropna(subset = questions).shape[0] # number of respondents who answered all mandatory questions: 2032\n",
    "print(f'{Q2} respondents have answered all the questions in the survey')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.049056100Z",
     "start_time": "2023-12-27T13:49:07.899896800Z"
    }
   },
   "id": "b920391eac644717"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 3. What are the median values for the respondents' work experience (WorkExp)?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b331e702f1a5b2e1"
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0 years is a value of central tendency for the work experience\n"
     ]
    }
   ],
   "source": [
    "Q3 = survey['WorkExp'].mode()[0]\n",
    "print(f'{Q3} years is a value of central tendency for the work experience')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.050164200Z",
     "start_time": "2023-12-27T13:49:08.040209200Z"
    }
   },
   "id": "e58168cc1678d9ef"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 4. How many respondents work remotely?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d8324eca5916f8ba"
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30566 respondents work remote\n"
     ]
    }
   ],
   "source": [
    "# respondents work remote\n",
    "remote = survey.loc[survey['RemoteWork'] == 'Remote']\n",
    "Q4 = remote['ResponseId'].count()\n",
    "print(f'{Q4} respondents work remote')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.092222100Z",
     "start_time": "2023-12-27T13:49:08.076224200Z"
    }
   },
   "id": "249948f5c8585c53"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 5. What percentage of respondents program in Python?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "c0bf8b53093eeefe"
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49.5% of respondents program in Python\n"
     ]
    }
   ],
   "source": [
    "# percentage of respondents who program in Python\n",
    "respondents = survey['LanguageHaveWorkedWith'].count() # total respondents\n",
    "python_use = survey.loc[(survey['LanguageHaveWorkedWith'].str.contains('Python', case=False, na=False))]['ResponseId'].count()\n",
    "Q5 = round((python_use/respondents)*100, 1)\n",
    "print(f'{Q5}% of respondents program in Python')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.192936100Z",
     "start_time": "2023-12-27T13:49:08.082130700Z"
    }
   },
   "id": "f3468575559f2440"
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [],
   "source": [
    "survey['Python'] = survey['LanguageHaveWorkedWith'].str.contains('Python', case=False, na=False) # adding new column 'Python' to ease the next tasks solving"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.272081100Z",
     "start_time": "2023-12-27T13:49:08.207533Z"
    }
   },
   "id": "74a4438d886dd4f7"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 6. How many respondents learned to program through online courses?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e5dc737540114a9f"
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "43201 respondents learned to code via Online Courses\n"
     ]
    }
   ],
   "source": [
    "# number of respondents who learned to code via online courses \n",
    "Q6 = survey.loc[survey['LearnCode'].str.contains('Online Courses', case=False, na=False)]['ResponseId'].count()\n",
    "print(f'{Q6} respondents learned to code via Online Courses')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.409944500Z",
     "start_time": "2023-12-27T13:49:08.273126500Z"
    }
   },
   "id": "722563e7cdee9c44"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 7. What is the average and median compensation (ConvertedCompYearly) among respondents who program in Python in each country?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "a22453efeffb060d"
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average and median compensation among respondents who program in Python in each country: \n",
      "                                          mean   median\n",
      "Country                                               \n",
      "Afghanistan                             665.0     48.0\n",
      "Albania                               28008.6  11844.0\n",
      "Algeria                                8336.3   6586.0\n",
      "Andorra                               32127.0  32127.0\n",
      "Angola                                  662.0    662.0\n",
      "...                                       ...      ...\n",
      "Venezuela, Bolivarian Republic of...  24973.5  12000.0\n",
      "Viet Nam                              20191.9  13401.0\n",
      "Yemen                                  8373.0   9000.0\n",
      "Zambia                                13051.0   9687.0\n",
      "Zimbabwe                               5600.0   6000.0\n",
      "\n",
      "[158 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "# average and median compensation in each country among respondents who program in Python\n",
    "clean = survey.dropna(subset='ConvertedCompYearly') # clean data from NaN values\n",
    "grouped = clean.loc[clean['Python'] == True, ['Country', 'Python', 'ConvertedCompYearly']] # filtering data for Python users\n",
    "Q7 = round(grouped.groupby('Country')['ConvertedCompYearly'].agg(['mean', 'median']), 1)\n",
    "print (f'Average and median compensation among respondents who program in Python in each country: \\n {Q7}')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.478477800Z",
     "start_time": "2023-12-27T13:49:08.406639Z"
    }
   },
   "id": "33a482c4bc1d5aea"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 8. What education levels do the top 5 highest paid respondents have?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "6b0a347d01331578"
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Education levels of Top 5 the highest paid respondents: \n",
      "\n",
      "                                               EdLevel  ConvertedCompYearly\n",
      "53268  Professional degree (JD, MD, Ph.D, Ed.D, etc.)           74351432.0\n",
      "77848  Professional degree (JD, MD, Ph.D, Ed.D, etc.)           73607918.0\n",
      "66223    Bachelor’s degree (B.A., B.S., B.Eng., etc.)           72714292.0\n",
      "28121                       Primary/elementary school           57513831.0\n",
      "19679  Professional degree (JD, MD, Ph.D, Ed.D, etc.)           36573181.0\n"
     ]
    }
   ],
   "source": [
    "# education level of top 5 paid respondents\n",
    "#Q8 = survey.loc[:, ['EdLevel', 'ConvertedCompYearly']].nlargest(5, 'ConvertedCompYearly') # another way to find it\n",
    "Q8 = survey.loc[:, ['EdLevel', 'ConvertedCompYearly']].sort_values(by= 'ConvertedCompYearly', ascending=False).head(5)\n",
    "print('Education levels of Top 5 the highest paid respondents: \\n\\n', Q8)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.487682500Z",
     "start_time": "2023-12-27T13:49:08.474493100Z"
    }
   },
   "id": "520ff9399fd906b"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 9. What percentage of respondents program in Python in each age category?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "de3ed8397632468c"
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of respondents programming in Python in each age group: \n",
      " Age\n",
      "18-24 years old       61.39 %\n",
      "25-34 years old       47.58 %\n",
      "35-44 years old       41.44 %\n",
      "45-54 years old       38.46 %\n",
      "55-64 years old        36.5 %\n",
      "65 years or older     30.91 %\n",
      "Prefer not to say      41.2 %\n",
      "Under 18 years old    68.63 %\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "# percentage of respondents who programs in Python in each age category\n",
    "prog = survey.loc[survey['Python'] == True].groupby('Age')['ResponseId'].count() # programs in Python\n",
    "total = survey.groupby('Age')['Python'].count() # total respondents\n",
    "Q9 = round((prog/total)*100, 2)\n",
    "# Q9 = survey.groupby('Age').apply(lambda x: round((x['Python'].sum() / x['Python'].count()) * 100, 2)) # another way using apply and lambda functions\n",
    "print('Percentage of respondents programming in Python in each age group: \\n', Q9.apply(lambda x : f'{x} %'))"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T13:49:08.546848400Z",
     "start_time": "2023-12-27T13:49:08.490761500Z"
    }
   },
   "id": "11e35355b62e7880"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 10. Which industries are the most prevalent among respondents (who work remotely) in the 75th percentile for average compensation?"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "6f386572e345932b"
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Among respondents in the 75th percentile for compensation who work remotely, the most prevalent industries are: \n",
      "0    Information Services, IT, Software Development...\n",
      "Name: Industry, dtype: object?\n"
     ]
    }
   ],
   "source": [
    "# most prevalent industries among respondents in the 75th percentile for avg compensation\n",
    "short = survey.loc[survey['RemoteWork'] == 'Remote', ['ConvertedCompYearly', 'Industry']] # Remote workers\n",
    "clean2 = short.dropna(subset='ConvertedCompYearly') # clean salary from NaN values\n",
    "percent = clean2['ConvertedCompYearly'].quantile(0.75) # 75 percentile\n",
    "Q10 = clean2.loc[clean2['ConvertedCompYearly'] == percent]['Industry'].mode() # use mode() aggregation to find the most widespread industries 'Industry'\n",
    "print(f'Among respondents in the 75th percentile for compensation who work remotely, the most prevalent industries are: \\n{Q10}?')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-12-27T14:13:42.940140800Z",
     "start_time": "2023-12-27T14:13:42.887556900Z"
    }
   },
   "id": "568192a7ba1e3b3e"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}