blog_ds-dataframe_r_and_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The R and Pandas Dataframe \n",
    "Date: 2020-03-02  \n",
    "Author: Jason Beach  \n",
    "Categories: Comparison, Data_Science\n",
    "Tags: python, pandas, r_language, dataframe  \n",
    "<!--eofm-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although Pandas uses the Dataframe as its primary data structure, just as R does, the Pandas syntax and underlying fundamentals can be disorienting for R users.  This post will describe some basic comparisons and inconsistencies between the two languages.  It will also provide some examples of very non-intuitive solutions to common problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In the Datascience R versus Pandas debate, it is really an apples and oranges comparison.  R is a domain specific language in the field of statistics, analytics, and data visualization.  This makes R great for consulting, research, and basic analysis, especially within a careful academic context.  \n",
    "\n",
    "In contrast, Python's statistics packages are woefully inadequate and rarely mention details which are of great importance to statistical practicioners.  An example of this is the use of contrasts in linear models.  The different Types (I-IV) of Analysis Of Variance models use different encodings for data.  Determining their estimators is not trivial.\n",
    "\n",
    "However, if you want tight integration with other applications, the strengths of typical programming languages, and want to 'just get models done', then Python / Pandas is a great solution.  Pandas is quite good at data manipulation.  Python has the very strong NumPy and SciKit Learn module, which are very good for matrix operations and predictive modeling.  And the Python language is a really good general scripting language with strong support for strings and datetime types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "Ensure to import the proper libraries.  Let's compare with R syntax, directly, using `rpy2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext rpy2.ipython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1], dtype=int32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%R require(ggplot2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These Dataframes will be used for the various problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-25 13:30:01.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:01.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.92</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker   price  quantity\n",
       "0 2016-05-25 13:30:01.023   MSFT   51.95        75\n",
       "1 2016-05-25 13:30:01.038   MSFT   51.95       155\n",
       "2 2016-05-25 13:30:03.048   GOOG  720.77       100\n",
       "3 2016-05-25 13:30:03.048   GOOG  720.92       100\n",
       "4 2016-05-25 13:30:03.048   AAPL   98.00       100"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades = pd.DataFrame(\n",
    "    [\n",
    "        [\"2016-05-25 13:30:01.023\", \"MSFT\", 51.95, 75],\n",
    "        [\"2016-05-25 13:30:01.038\", \"MSFT\", 51.95, 155],\n",
    "        [\"2016-05-25 13:30:03.048\", \"GOOG\", 720.77, 100],\n",
    "        [\"2016-05-25 13:30:03.048\", \"GOOG\", 720.92, 100],\n",
    "        [\"2016-05-25 13:30:03.048\", \"AAPL\", 98.00, 100],\n",
    "    ],\n",
    "    columns=[\"timestamp\", \"ticker\", \"price\", \"quantity\"],   #set index during assignment: `, index_col='timestamp'`\n",
    ")\n",
    "trades['timestamp'] = pd.to_datetime(trades['timestamp'])\n",
    "trades.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then set the index to the timestamp.  The index allows selecting rows with the `loc` and `at` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:01.023</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:01.038</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:03.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:03.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.92</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:03.048</th>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        ticker   price  quantity\n",
       "timestamp                                       \n",
       "2016-05-25 13:30:01.023   MSFT   51.95        75\n",
       "2016-05-25 13:30:01.038   MSFT   51.95       155\n",
       "2016-05-25 13:30:03.048   GOOG  720.77       100\n",
       "2016-05-25 13:30:03.048   GOOG  720.92       100\n",
       "2016-05-25 13:30:03.048   AAPL   98.00       100"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx = trades.set_index('timestamp')\n",
    "trades_idx.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Syntax Differences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic selection\n",
    "\n",
    "The primary purpose of the DataFrame indexing operator, `df[ ]` is to select columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    MSFT\n",
       "1    MSFT\n",
       "2    GOOG\n",
       "Name: ticker, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades['ticker'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ticker   price\n",
       "0   MSFT   51.95\n",
       "1   MSFT   51.95\n",
       "2   GOOG  720.77"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades[['ticker','price']].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ticker\n",
      "0   MSFT\n",
      "1   MSFT\n",
      "2   GOOG\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head( trades['ticker'], 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ticker  price\n",
      "0   MSFT  51.95\n",
      "1   MSFT  51.95\n",
      "2   GOOG 720.77\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head( trades[c('ticker','price')], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the indexing operator is passed a string or integer, it attempts to find a column with that particular name and return it as a Series.\n",
    "\n",
    "For example, `df[2]` searches for a column name matching the integer value 2. This column does not exist and a `KeyError` is raised.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "got key error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    trades[2]\n",
    "except:\n",
    "    print(\"got key error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ticker\n",
      "0   MSFT\n",
      "1   MSFT\n",
      "2   GOOG\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head(trades[2], 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"MSFT\" \"MSFT\" \"GOOG\"\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head(trades[,2], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use integer indexing of columns, in R.  Its not necessary to use the `c()` concatenate function to create a vector, but it can often make your code cleaner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ticker  price\n",
      "0   MSFT  51.95\n",
      "1   MSFT  51.95\n",
      "2   GOOG 720.77\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head(trades[,c(2:3)], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the DataFrame indexing operator completely changes behavior to select rows when slice notation is used.  The DataFrame indexing operator selects rows in this manner, and can also do so by integer location, or by index label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-25 13:30:00.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:00.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker   price  quantity\n",
       "0 2016-05-25 13:30:00.023   MSFT   51.95        75\n",
       "1 2016-05-25 13:30:00.038   MSFT   51.95       155\n",
       "2 2016-05-25 13:30:00.048   GOOG  720.77       100"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            timestamp ticker  price quantity\n",
      "0 2016-05-25 13:30:00   MSFT  51.95       75\n",
      "1 2016-05-25 13:30:00   MSFT  51.95      155\n",
      "2 2016-05-25 13:30:00   GOOG 720.77      100\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head(trades[1:3,], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following selects rows beginning at integer location 1, up to but not including 6, by every third row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:00.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker  price  quantity\n",
       "1 2016-05-25 13:30:00.038   MSFT  51.95       155\n",
       "4 2016-05-25 13:30:00.048   AAPL  98.00       100"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades[1:6:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use slices consisting of string labels if your DataFrame index has strings in it.  Here we have new operators, `.iloc` to explicity support only integer indexing, and `.loc` to explicity support only label indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:00.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker  price  quantity\n",
       "1 2016-05-25 13:30:00.038   MSFT  51.95       155\n",
       "4 2016-05-25 13:30:00.048   AAPL  98.00       100"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades.iloc[1:6:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the timestamp index, we can subset using a variety of input string-formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.023</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.038</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        ticker   price  quantity\n",
       "timestamp                                       \n",
       "2016-05-25 13:30:00.023   MSFT   51.95        75\n",
       "2016-05-25 13:30:00.038   MSFT   51.95       155\n",
       "2016-05-25 13:30:00.048   GOOG  720.77       100"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx.loc['2016-05'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.023</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.038</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        ticker   price  quantity\n",
       "timestamp                                       \n",
       "2016-05-25 13:30:00.023   MSFT   51.95        75\n",
       "2016-05-25 13:30:00.038   MSFT   51.95       155\n",
       "2016-05-25 13:30:00.048   GOOG  720.77       100"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx.loc['May 2016'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.loc` / `.iloc` operators can select on both rows and columns, simultaneously, the indexing operator `[ ]` cannot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "timestamp\n",
       "2016-05-25 13:30:00.023    MSFT\n",
       "2016-05-25 13:30:00.038    MSFT\n",
       "2016-05-25 13:30:00.048    GOOG\n",
       "2016-05-25 13:30:00.048    GOOG\n",
       "2016-05-25 13:30:00.048    AAPL\n",
       "Name: ticker, dtype: object"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx.loc['May 2016', 'ticker']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "use R\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    trades[2, 'ticker']\n",
    "except:\n",
    "    print('use R')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This all seems very tideous for R users - just get on with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"MSFT\"\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "head(trades[2,'ticker'], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataframe Index\n",
    "\n",
    "One of the fundamental differences between R and pandas is that pandas leans heavily upon the dataframe index.  For R users, it can seem like an annoyance; however, using it is required because many results of pandas methods return a dataframe with an index.\n",
    "\n",
    "The index is already set for `trades_idx`, but if we want to replace it with a new one, it is easily done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>price</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51.95</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51.95</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>720.77</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>720.92</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98.00</th>\n",
       "      <td>AAPL</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       ticker  quantity\n",
       "price                  \n",
       "51.95    MSFT        75\n",
       "51.95    MSFT       155\n",
       "720.77   GOOG       100\n",
       "720.92   GOOG       100\n",
       "98.00    AAPL       100"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx.set_index('price')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to move the index into a column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.023</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "      <td>2016-05-25 13:30:00.023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.038</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "      <td>2016-05-25 13:30:00.038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.92</td>\n",
       "      <td>100</td>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        ticker   price  quantity               timestamp\n",
       "timestamp                                                               \n",
       "2016-05-25 13:30:00.023   MSFT   51.95        75 2016-05-25 13:30:00.023\n",
       "2016-05-25 13:30:00.038   MSFT   51.95       155 2016-05-25 13:30:00.038\n",
       "2016-05-25 13:30:00.048   GOOG  720.77       100 2016-05-25 13:30:00.048\n",
       "2016-05-25 13:30:00.048   GOOG  720.92       100 2016-05-25 13:30:00.048\n",
       "2016-05-25 13:30:00.048   AAPL   98.00       100 2016-05-25 13:30:00.048"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_trades_idx = trades_idx.copy()\n",
    "new_trades_idx['timestamp'] = trades_idx.index\n",
    "new_trades_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-25 13:30:00.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:00.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.92</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker   price  quantity\n",
       "0 2016-05-25 13:30:00.023   MSFT   51.95        75\n",
       "1 2016-05-25 13:30:00.038   MSFT   51.95       155\n",
       "2 2016-05-25 13:30:00.048   GOOG  720.77       100\n",
       "3 2016-05-25 13:30:00.048   GOOG  720.92       100\n",
       "4 2016-05-25 13:30:00.048   AAPL   98.00       100"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx.reset_index(level=0, inplace=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using a timestamp as an index, default datetime format ISO8601 (“yyyy-mm-dd hh:mm:ss”) is used when selecting data with partial string indexing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot use fractions of seconds\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    trades_idx.loc['2016-05-25 13:30:00.023']\n",
    "except:\n",
    "    print('cannot use fractions of seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:01.023</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:01.038</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        ticker  price  quantity\n",
       "timestamp                                      \n",
       "2016-05-25 13:30:01.023   MSFT  51.95        75\n",
       "2016-05-25 13:30:01.038   MSFT  51.95       155"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades_idx.loc['2016-05-25 13:30:01':'2016-05-25 13:30:02']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the index of a R dataframe is immutable, the index of a pandas dataframe can expand, or even lose arbitrary items.  In this example, some rows are removed, and the index goes with them.  If we don't `reset_index()`, then those rows will always be missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.023</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "      <td>2016-05-25 13:30:00.023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.038</th>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "      <td>2016-05-25 13:30:00.038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.92</td>\n",
       "      <td>100</td>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-25 13:30:00.048</th>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "      <td>2016-05-25 13:30:00.048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        ticker   price  quantity               timestamp\n",
       "timestamp                                                               \n",
       "2016-05-25 13:30:00.023   MSFT   51.95        75 2016-05-25 13:30:00.023\n",
       "2016-05-25 13:30:00.038   MSFT   51.95       155 2016-05-25 13:30:00.038\n",
       "2016-05-25 13:30:00.048   GOOG  720.77       100 2016-05-25 13:30:00.048\n",
       "2016-05-25 13:30:00.048   GOOG  720.92       100 2016-05-25 13:30:00.048\n",
       "2016-05-25 13:30:00.048   AAPL   98.00       100 2016-05-25 13:30:00.048"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-25 13:30:01.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker   price  quantity\n",
       "0 2016-05-25 13:30:01.023   MSFT   51.95        75\n",
       "2 2016-05-25 13:30:03.048   GOOG  720.77       100"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp2 = trades.copy()\n",
    "tmp3 = tmp2[0:4:2]\n",
    "tmp3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2016-05-25 13:30:01.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index               timestamp ticker   price  quantity\n",
       "0      0 2016-05-25 13:30:01.023   MSFT   51.95        75\n",
       "1      2 2016-05-25 13:30:03.048   GOOG  720.77       100"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp3.reset_index(inplace=True)\n",
    "tmp3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forgetting to `reset_index()` can cause serious problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create a multi-index to make things even more confusing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = pd.MultiIndex.from_product([['TX', 'FL', 'CA'], \n",
    "                                    ['North', 'South']], \n",
    "                                   names=['State', 'Direction'])\n",
    "\n",
    "df = pd.DataFrame(index=index, \n",
    "                  data=np.random.randint(0, 10, (6,4)), \n",
    "                  columns=list('abcd'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>State</th>\n",
       "      <th>Direction</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">TX</th>\n",
       "      <th>North</th>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>South</th>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">FL</th>\n",
       "      <th>North</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>South</th>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">CA</th>\n",
       "      <th>North</th>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>South</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 a  b  c  d\n",
       "State Direction            \n",
       "TX    North      6  4  1  5\n",
       "      South      0  9  3  2\n",
       "FL    North      3  2  9  3\n",
       "      South      7  2  3  0\n",
       "CA    North      9  6  3  4\n",
       "      South      5  6  9  0"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subset replacement (the `SettingWithCopy` warning)\n",
    "\n",
    "Conditionally replacing values in R is straightforward because we can use simultaneous selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            timestamp ticker  price quantity\n",
      "0 2016-05-25 13:30:01   MSFT  51.95       75\n",
      "1 2016-05-25 13:30:01   MSFT 100.00      155\n",
      "2 2016-05-25 13:30:03   GOOG 100.00      100\n",
      "3 2016-05-25 13:30:03   GOOG 100.00      100\n",
      "4 2016-05-25 13:30:03   AAPL 100.00      100\n"
     ]
    }
   ],
   "source": [
    "%%R -i trades\n",
    "\n",
    "tmp1 <- trades\n",
    "tmp1[tmp1$quantity > 80, c('price')] <- 100\n",
    "tmp1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that we cannot use simultaneous selection in pandas using the indexing operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TypeError: 'Series' objects are mutable, thus they cannot be hashed\n"
     ]
    }
   ],
   "source": [
    "tmp1 = trades.copy()\n",
    "try:\n",
    "    tmp1[tmp1.quantity > 80, 'price'] = 100\n",
    "except:\n",
    "    print(\"TypeError: 'Series' objects are mutable, thus they cannot be hashed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replacing by index doesn't give us what we want, either>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The direct way fails, too!  Oh, pandas, why must you be so difficult..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-25 13:30:01.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:01.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.77</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>720.92</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>98.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker   price  quantity\n",
       "0 2016-05-25 13:30:01.023   MSFT   51.95        75\n",
       "1 2016-05-25 13:30:01.038   MSFT   51.95       155\n",
       "2 2016-05-25 13:30:03.048   GOOG  720.77       100\n",
       "3 2016-05-25 13:30:03.048   GOOG  720.92       100\n",
       "4 2016-05-25 13:30:03.048   AAPL   98.00       100"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp1 = trades.copy()\n",
    "tmp1[tmp1.quantity > 80]['price'] = 100\n",
    "tmp1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand, we must look under the hood:\n",
    "\n",
    "```\n",
    "df.loc[df.A > 5, 'B'] = 4\n",
    "# becomes\n",
    "df.__setitem__((df.A > 5, 'B'), 4)\n",
    "```\n",
    "\n",
    "With a single `__setitem__` call to df. On the otherhand, consider this code:\n",
    "\n",
    "```\n",
    "df[df.A > 5]['B'] = 4\n",
    "# becomes\n",
    "df.__getitem__(df.A > 5).__setitem__('B\", 4)\n",
    "```\n",
    "\n",
    "Now, depending on whether `__getitem__` returned a view or a copy, the `__setitem__` operation may not work.\n",
    "\n",
    "The documentation addresses these issues, [here](https://pandas-docs.github.io/pandas-docs-travis/user_guide/indexing.html#returning-a-view-versus-a-copy).  But, to summarize:\n",
    "\n",
    "> Whenever an array of labels or a boolean vector are involved in the indexing operation, the result will be a copy.  With single label / scalar indexing and slicing, e.g. df.ix[3:6] or df.ix[:, 'A'], a view will be returned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the `.loc` method gives us what we want because we can use it with boolean indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "3     True\n",
       "4     True\n",
       "Name: quantity, dtype: bool"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp1.quantity > 80"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>ticker</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-25 13:30:01.023</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>51.95</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-25 13:30:01.038</td>\n",
       "      <td>MSFT</td>\n",
       "      <td>100.00</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>100.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>GOOG</td>\n",
       "      <td>100.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-25 13:30:03.048</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>100.00</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp ticker   price  quantity\n",
       "0 2016-05-25 13:30:01.023   MSFT   51.95        75\n",
       "1 2016-05-25 13:30:01.038   MSFT  100.00       155\n",
       "2 2016-05-25 13:30:03.048   GOOG  100.00       100\n",
       "3 2016-05-25 13:30:03.048   GOOG  100.00       100\n",
       "4 2016-05-25 13:30:03.048   AAPL  100.00       100"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp1 = trades.copy()\n",
    "tmp1.loc[tmp1.quantity > 80, 'price'] = 100\n",
    "tmp1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timestamps\n",
    "\n",
    "A pandas object type is used for text or mixed numeric and non-numeric values. To get the correct order of the timestamp column we need to change it to the datetime64 type.\n",
    "\n",
    "Replacing datetime values has its own unique manner, of course.  Don't want to lose this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.timestamp = pd.to_datetime(df.timestamp.str.replace(\"D\", \"T\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merge\n",
    "\n",
    "Merge with an indicator for where the rows came from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_merge = left.merge(right, on='key', how='left', indicator=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `_merge` column is used to check for unexpected rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_merge._merge.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge by the nearest (not exact) timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.merge_asof(trades, quotes, on=\"timestamp\", by='ticker', tolerance=pd.Timedelta('10ms'), direction='backward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing values\n",
    "\n",
    "Because Numpy has no native NA type, pandas uses multiple values denote a missing value:\n",
    "* NaN for numeric/object missing values\n",
    "* NaT for DateTime missing values\n",
    "* None, which comes from Python\n",
    "\n",
    "What surprised me is that None equals None in Python, but nan doesn’t equal nan in numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "None == None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.np.nan == pd.np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is important so that we don’t filter values by None:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.some_column == None]# instead use:\n",
    "df[df.some_column.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Report creation\n",
    "\n",
    "Pandas (with XlsxWriter library) allows us to make an Excel sheet with graphs and other Excel functionality.  We need to define the type of the chart (line chart in our example) and the data series for the chart (the data series needs to be in the Excel spreadsheet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_name = 'example_report.xlsx'\n",
    "sheet_name = 'Sheet1'\n",
    "writer = pd.ExcelWriter(report_name, engine='xlsxwriter')\n",
    "df.to_excel(writer, sheet_name=sheet_name, index=False)\n",
    "# writer.save()\n",
    "\n",
    "# define the workbook\n",
    "workbook = writer.book\n",
    "worksheet = writer.sheets[sheet_name]\n",
    "# create a chart line object\n",
    "chart = workbook.add_chart({'type': 'line'})\n",
    "# configure the series of the chart from the spreadsheet\n",
    "# using a list of values instead of category/value formulas:\n",
    "#     [sheetname, first_row, first_col, last_row, last_col]\n",
    "chart.add_series({\n",
    "    'categories': [sheet_name, 1, 0, 3, 0],\n",
    "    'values':     [sheet_name, 1, 1, 3, 1],\n",
    "})\n",
    "# configure the chart axes\n",
    "chart.set_x_axis({'name': 'Index', 'position_axis': 'on_tick'})\n",
    "chart.set_y_axis({'name': 'Value', 'major_gridlines': {'visible': False}})\n",
    "# place the chart on the worksheet\n",
    "worksheet.insert_chart('E2', chart)\n",
    "# output the excel file\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving objects\n",
    "\n",
    "When we save this file as CSV, it takes almost 300 MB on the hard drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('random_data.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a single argument compression='gzip', we can reduce the file size to 136 MB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "df.to_csv('random_data.gz', compression='gzip', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also easy to read the gzipped data to the DataFrame, so we don’t lose any functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('random_data.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specific examples\n",
    "\n",
    "### Negate a string replacement\n",
    "\n",
    "You can use negative look ahead `(?!)` assertion; `^(?!.*fish).*$` will firstly assert the pattern doesn't contain the word fish and then match every thing till the end of string and replace it with foo:\n",
    "\n",
    "* `^` denotes the beginning of string, combined with `(?!.*fish)`, it asserts at BOS that there is no pattern like `.*fish` in the string;\n",
    "* If the assertion succeeds, it matches everything till the end of string `.*$`, and replaces it with `foo`; If the assertion fails, the pattern doesn't match, nothing would happen;\n",
    "\n",
    "so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.replace(r'^(?!.*fish).*$', 'foo', regex=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Outliers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['size_clip'] = df['size'].clip(df['size'].quantile(0.01),\n",
    "                                  df['size'].quantile(0.99))\n",
    "df.size_clip.plot(kind='box')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bin / group data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['price_discrete_labels'] = pd.cut(df.price, 5, labels=['very low', 'low', 'mid', 'high', 'very high'])\n",
    "df['price_discrete_labels'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equal-sized groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['price_discrete_equal_bins'] = pd.qcut(df.price, 5)\n",
    "df['price_discrete_equal_bins'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "After reviewing these examples, we hope you agree that R is a terse, minimalistic, and powerful language.  You can do so many things with just the dataframe operator that it makes interacting with R a joy.  It is so much of a joy that I typically don't know how the operations are actually being performed - they just work.\n",
    "\n",
    "Pandas is different.  It is bolted-on to Python, and its syntax is either not consistent or is just un-wieldly.  But maybe that is a good thing, because with Python I'm typically automating a process - not performing an interactive investigation.  I want to be forced to understand how the data is actually being worked, and I want to optimize operations for performance because it is a process that will be automated.\n",
    "\n",
    "Let's stop the impassioned vitrioles of one clan against another, and use each of these tools for their respective strengths."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}