2. Network(X) Basics (Student).ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "from datetime import datetime\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Nodes and Edges: How do we represent relationships between individuals using NetworkX?\n",
    "\n",
    "As mentioned earlier, networks, also known as graphs, are comprised of individual entities and their representatives. The technical term for these are nodes and edges, and when we draw them we typically use circles (nodes) and lines (edges). \n",
    "\n",
    "In this notebook, we will work with a synthetic (i.e. simulated) social network, in which nodes are individual people, and edges represent their relationships. If two nodes have an edge between them, then those two individauls know one another. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data Representation\n",
    "\n",
    "In the `networkx` implementation, graph objects store their data in dictionaries. \n",
    "\n",
    "Nodes are part of the attribute `Graph.node`, which is a dictionary where the key is the node ID and the values are a dictionary of attributes. \n",
    "\n",
    "Edges are part of the attribute `Graph.edge`, which is a nested dictionary. Data are accessed as such: `G.edge[node1][node2]['attr_name']`.\n",
    "\n",
    "Because of the dictionary implementation of the graph, any hashable object can be a node. This means strings and tuples, but not lists and sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Synthetic Social Network\n",
    "\n",
    "With this synthetic social network, we will attempt to answer the following basic questions using the NetworkX API:\n",
    "\n",
    "1. How many people are present in the network?\n",
    "2. What is the distribution of attributes of the people in this network?\n",
    "2. How many relationships are represented in the network?\n",
    "3. What is the distribution of the number of friends that each person has?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "First off, let's load up the synthetic social network. This will show you through some of the basics of NetworkX.\n",
    "\n",
    "For those who are interested, I simply created an Erdõs-Rényi graph with `n=30` and `p=0.1`. I used randomized functions that I wrote to generate attributes and append them to each node and edge. I then pickled the graph to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "G = nx.read_gpickle('Synthetic Social Network.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Basic Network Statistics\n",
    "Let's first understand how many people and relationships are represented in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Who are represented in the net work?\n",
    "G.nodes(data=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Can you write a single line of code that returns the number of individuals represented?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's now figure out who is connected to who in the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Who is connected to who in the network?\n",
    "G.edges()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Can you write a single line of code that returns the number of relationships represented?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Since this is a social network of people, there'll be attributes for each individual, such as age, and sex. We can grab that data off from the attributes that are stored with each node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Let's get a list of nodes with their attributes.\n",
    "G.nodes(data=True)\n",
    "\n",
    "# NetworkX will return a list of tuples in the form (node_id, attribute_dictionary) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Can you count how many males and females are represented in the graph?\n",
    "\n",
    "Hint: You may want to use the Counter object from the collections module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Edges can also store attributes in their attribute dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "G.edges(data=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In this synthetic social network, I have stored the date as a datetime object. Datetime objects have attributes, namely `.year`, `.month`, `.day`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Can you figure out the range of dates during which these relationships were forged? Specifically, compute the earliest and last date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Answer\n",
    "dates = [d['_____'] for _, _, d in G.edges(data=True)]\n",
    "mindate = __________\n",
    "maxdate = __________\n",
    "print(mindate, maxdate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "We found out that there are two individuals that we left out of the network, individual no. 31 and 32. They are one male (31) and one female (32), their ages are 22 and 24 respectively, they knew each other on 2010-01-09, and together, they both knew individual 7, on 2009-12-11. Use the functions `G.add_node()` and `G.add_edge()` to introduce this data into the network.\n",
    "\n",
    "If you need more help, check out https://networkx.github.io/documentation/latest/tutorial/tutorial.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Answer\n",
    "G.add_node(31, age=22, sex='Male')\n",
    "# add one line here\n",
    "\n",
    "G.add_edge(31, 32, date=datetime(2010,1,9))\n",
    "# add two lines here \n",
    "\n",
    "\n",
    "\n",
    "G.node[31]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Verify that you have added in the edges and nodes correctly by running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def test_graph_integrity(G):\n",
    "    assert 31 in G.nodes()\n",
    "    assert 32 in G.nodes()\n",
    "    assert G.has_edge(31, 32)\n",
    "    assert G.has_edge(31, 7)\n",
    "    assert G.has_edge(32, 7)\n",
    "    print('All tests passed.')\n",
    "    \n",
    "test_graph_integrity(G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tests\n",
    "\n",
    "A note about the tests: Testing is good practice when writing code. Well-crafted assertion statements help you program defensivel, by forcing you to explicitly state your assumptions about the code. \n",
    "\n",
    "For more references on defensive programming, check out Software Carpentry's website: http://swcarpentry.github.io/python-novice-inflammation/08-defensive.html\n",
    "\n",
    "For more information on writing tests for your data, check out these slides from a lightning talk I gave at Boston Python and SciPy 2015: http://j.mp/data-test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Coding Patterns\n",
    "\n",
    "These are some recommended coding patterns when doing network analysis using NetworkX, which stem from my roughly two years of experience with the package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Iterating using List Comprehensions\n",
    "I would recommend that you use the following for compactness: \n",
    "\n",
    "    [d['attr'] for n, d in G.nodes(data=True)]\n",
    "\n",
    "And if the node is unimportant, you can do:\n",
    "\n",
    "    [d['attr'] for _, d in G.nodes(data=True)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Iterating over Edges using List Comprehensions\n",
    "\n",
    "A similar pattern can be used for edges:\n",
    "\n",
    "    [n2 for n1, n2, d in G.edges(data=True)]\n",
    "\n",
    "or\n",
    "\n",
    "    [n2 for _, n2, d in G.edges(data=True)]\n",
    "\n",
    "If the graph you are constructing is a directed graph, with a \"source\" and \"sink\" available, then I would recommend the following pattern:\n",
    "\n",
    "    [(sc, sk) for sc, sk, d in G.edges(data=True)]\n",
    "\n",
    "or \n",
    "\n",
    "    [d['attr'] for sc, sk, d in G.edges(data=True)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Drawing Graphs\n",
    "\n",
    "As illustrated above, we can draw graphs using the `nx.draw()` function. The most popular format for drawing graphs is the **node-link diagram**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "nx.draw(G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If the network is small enough to visualize, and the node labels are small enough to fit in a circle, then you can use the `with_labels=True` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nx.draw(G, with_labels=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "However, note that if the number of nodes in the graph gets really large, node-link diagrams can begin to look like massive hairballs. This is undesirable for graph visualization.\n",
    "\n",
    "Instead, we can use a **matrix** to represent them. The nodes are on the x- and y- axes, and a filled square represent an edge between the nodes. This is done by using the `nx.to_numpy_matrix(G)` function.\n",
    "\n",
    "We then use `matplotlib`'s `pcolor(numpy_array)` function to plot. Because `pcolor` cannot take in numpy matrices, we will cast the matrix as an array of arrays, and then get `pcolor` to plot it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "matrix = nx.to_numpy_matrix(G)\n",
    "\n",
    "plt.pcolor(np.array(matrix))\n",
    "plt.axes().set_aspect('equal') # set aspect ratio equal to get a square visualization\n",
    "plt.xlim(min(G.nodes()), max(G.nodes())) # set x and y limits to the number of nodes present.\n",
    "plt.ylim(min(G.nodes()), max(G.nodes()))\n",
    "plt.title('Adjacency Matrix')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's try another visualization, the **Circos plot**. We can order the nodes in the Circos plot according to the node ID, but any other ordering is possible as well. Edges are drawn between two nodes.\n",
    "\n",
    "Credit goes to Justin Zabilansky (MIT) for the implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from circos import CircosPlot\n",
    "\n",
    "fig = plt.figure(figsize=(6,6))\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "nodes = sorted(G.nodes())\n",
    "edges = G.edges()\n",
    "\n",
    "c = CircosPlot(nodes, edges, radius=10, ax=ax)\n",
    "c.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "It's pretty obvious in this visualization that there are nodes, such as node 5 and 18, that are not connected to any other node via an edge. There are other nodes, like node number 19, which is highly connected to other nodes.\n",
    "\n",
    "Finally, let's try **hive plots** for the network. Two groups (male and female), and then edges drawn between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from hiveplot import HivePlot\n",
    "\n",
    "nodes = dict()\n",
    "nodes['male'] = [n for n,d in G.nodes(data=True) if d['sex'] == 'Male']\n",
    "nodes['female'] = [n for n,d in G.nodes(data=True) if d['sex'] == 'Female']\n",
    "\n",
    "edges = dict()\n",
    "edges['group1'] = G.edges(data=True)\n",
    "\n",
    "nodes_cmap = dict()\n",
    "nodes_cmap['male'] = 'blue'\n",
    "nodes_cmap['female'] = 'red'\n",
    "\n",
    "edges_cmap = dict()\n",
    "edges_cmap['group1'] = 'black'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "h = HivePlot(nodes, edges, nodes_cmap, edges_cmap)\n",
    "h.draw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}