{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Gradient_descent_developed_from_scratch.ipynb",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "AgmDHOpwfCNU"
},
"source": [
"# Gradient Descent\n",
"\n",
"Version 1.3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E5Sc55PDfM8B"
},
"source": [
"(C) 2020 - Umberto Michelucci, Michela Sperti\n",
"\n",
"This notebook is part of the book _Applied Deep Learning: a case based approach, **2nd edition**_ from APRESS by [U. Michelucci](mailto:umberto.michelucci@toelt.ai) and [M. Sperti](mailto:michela.sperti@toelt.ai)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NJLkEQ1Flruj"
},
"source": [
"## Notebook Learning Goals"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2ILzEVh_ls61"
},
"source": [
"A the end of this notebook you are going to know how to implement the famous minimization method called **gradient descent** from scratch. This is a useful exercise to prepare yourself to understand more complex neural networks algorithms."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LL3UAPBJnIco"
},
"source": [
"## Theory behind Gradient Descent"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Adc61OGRgGuJ"
},
"source": [
"The objective of gradient descent is to numerically find the minimum of any given function. It is suited for those problems for which an analytical solution is not available, such as in the case of neural networks (big number of weights).\n",
"\n",
"Given a generic function $J(\\mathbf{w})$, where $\\mathbf{w}$ is a vector of weights, the minimum location in weight space (meaning the value for $\\mathbf{w}$ for which $J(\\mathbf{w})$ has a minimum) can be found with an algorithm based on the following steps:\n",
"1. *Iteration 0*: Choose a random initial guess $\\mathbf{w_0}$ \n",
"2. *Iteration $n+1$* (with $n$ starting from 0): The weights at iteration $n+1$ ($\\mathbf{w}_{n+1}$) will be updated from the previous values at iteration $n$ ($\\mathbf{w}_{n}$) using the formula\n",
"\n",
"$$\n",
"\\mathbf{w}_{n+1}=\\mathbf{w}_{n}-\\gamma\\nabla J(\\mathbf{w}_{n}) \\tag{1}\n",
"$$\n",
"\n",
"With $J(\\mathbf{w}_{n})$, we have indicated the gradient of the cost function, which is a vector whose components are the partial derivatives of the cost function with respect to all the components of the weight vector $\\mathbf{w}$, as follows:\n",
"\n",
"$$\n",
"J(\\mathbf{w}_{n})=\\displaystyle\n",
"(\n",
"\\partial J(\\mathbf{w})/\\partial w_1, \n",
"\\dots, \\partial J(\\mathbf{w})/\\partial w_{n_x}) \\tag{2}\n",
"$$\n",
"\n",
"Let's suppose now that we want to fit some data to a generic function $f({\\bf x}^{(i)})$ where ${\\bf x}^{(i)}$ is the $i^th$ observation of an input dataset. In general we will have ${\\bf x}^{(i)}\\in \\mathbb{R}^n$ with $n\\in \\mathbb{N}$, but for this example let's suppose that \n",
"${\\bf x}^{(i)}=x^{(i)}\\in \\mathbb{R}$.\n",
"\n",
"As a cost function, let's choose the **mean squared error** (MSE):\n",
"\n",
"$$\n",
"J(w_o,w_1)=\\frac{1}{m} \\sum_{i=1}^{m} (y_i-f(x^{(i)}))^2 \\tag{3}\n",
"$$\n",
"\n",
"where we have indicated with the subscript $i$ the $i^{th}$ observation. In this example we will try to fit a linear function\n",
"\n",
"$$\n",
"f(x^{(i)}) = w_0 + w_1 x^{(i)} \\tag{4}\n",
"$$\n",
"\n",
"to some artificially generated data. Note that in this case the function $f(x^{(i)})$ will depend on the weights $w_0$ and $w_1$, therefore to make this particularly clear, we will indicate our $f$ function as $f(w_0, w_1, x^{(i)})$."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2gFGsN8MmqRp"
},
"source": [
"## Gradient Descent Implementation with a Simulated Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vPqJX0usndKr"
},
"source": [
"For this example, we will use a simulated dataset made of 1 feature and 30 samples. As cost function we will consider MSE and we will apply a linear regression model as hypothesis."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ItADQQkhf5f1"
},
"source": [
"### Libraries Import"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6AvNt2plf9Pb"
},
"source": [
"This section contains the necessary libraries (such as numpy or matplotlib) you need to import to execute the code."
]
},
{
"cell_type": "code",
"metadata": {
"id": "sjBSp2rmgB76"
},
"source": [
"# general libraries\n",
"import numpy as np\n",
"import random\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.font_manager as fm"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Uwz7ZLF4kn2u"
},
"source": [
"# Referring to the following cell, if you want to re-clone a repository\n",
"# inside the google colab instance, you need to delete it first. \n",
"# You can delete the repositories contained in this instance executing \n",
"# the following two lines of code (deleting the # comment symbol).\n",
"\n",
"# !rm -rf ADL-Book-2nd-Ed "
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Wia4bfsukoVj",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "61fc7606-68cd-421a-ee49-fbef6bc009a7"
},
"source": [
"# This command actually clone the repository of the book in the google colab\n",
"# instance. In this way this notebook will have access to the modules\n",
"# we have written for this book.\n",
"\n",
"# Please note that in case you have already run this cell, and you run it again\n",
"# you may get the error message:\n",
"#\n",
"# fatal: destination path 'ADL-Book-2nd-Ed' already exists and is not an empty directory.\n",
"# \n",
"# In this case you can safely ignore the error message.\n",
"\n",
"!git clone https://github.com/toelt-llc/ADL-Book-2nd-Ed.git"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"fatal: destination path 'ADL-Book-2nd-Ed' already exists and is not an empty directory.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "_sE9TwTrkqZy"
},
"source": [
"# This cell imports some custom written functions that we have created to \n",
"# make the plotting easier. You don't need to undertsand the details and \n",
"# you can simply ignore this cell.\n",
"# Simply run it with CMD+Enter (on Mac) or CTRL+Enter (Windows or Ubuntu) to\n",
"# import the necessary functions.\n",
"\n",
"import sys\n",
"sys.path.append('ADL-Book-2nd-Ed/modules/')\n",
"\n",
"from style_setting import set_style"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "R_rhxdrfOMr_"
},
"source": [
"### Dataset Generation "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mbeYwq5PfQFg"
},
"source": [
"Let’s consider the dataset formed by $m = 30$ observations y generated by the code:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "W1v4TG4KMUZ9"
},
"source": [
"m = 30\n",
"w0 = 2\n",
"w1 = 0.5\n",
"x = np.linspace(-1,1,m)\n",
"y = w0 + w1 * x"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "g8o6XBI1goyu"
},
"source": [
"Note how we are generating our data with Equation (4) that in Python can be written as ```y = w0 + w1 * x```."
]
},
{
"cell_type": "code",
"metadata": {
"id": "leZQbvePnMAB"
},
"source": [
"# The following line contains the path to fonts that are used to plot result in\n",
"# a uniform way.\n",
"\n",
"f = set_style().set_general_style_parameters()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "sO-SI43LnRsO",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 385
},
"outputId": "c882f8d6-7ff8-4907-d27d-ca4324a1e822"
},
"source": [
"# plot the data we are going to use\n",
"\n",
"fig = plt.figure()\n",
"ax = fig.add_subplot(111)\n",
"\n",
"plt.scatter(x, y, marker = 'o', c = 'blue')\n",
"\n",
"plt.ylabel('x', fontproperties = fm.FontProperties(fname = f))\n",
"plt.xlabel('y', fontproperties = fm.FontProperties(fname = f))\n",
"\n",
"plt.ylim(min(y), max(y))\n",
"plt.xlim(min(x), max(x))\n",
"\n",
"plt.axis(True)\n",
"plt.show()"
],
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAAFwCAYAAACRj46qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAMTQAADE0B0s6tTgAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7RkVX3g8e+vXwrdPBQxJN1K0z1qa1QwUREZlUdiRAMSHpNM0GCUODORIAESGRMjusZosiIP0YlJUNugmRjjI2IISgQVmlYMwhoZELQfKCAqIC3dDTbQv/njnILLpapuVd1TVadOfT9r1ap767z2rl2PX529z29HZiJJkqT5WTDuAkiSJDWBQZUkSVIFDKokSZIqYFAlSZJUAYMqSZKkChhUSZIkVcCgSpIkqQKLxl2AOluyZEnuvffe4y6GJEnqYvt22LIleCT15o/J/FmMuhwGVV3svffe3LjhhnEXQ5IkdbF+3UKOeuVSduxoxVErxlIOu/8kSdJEe9GLH2LflTtZtGi8s8QYVEmSpIkWAZ/+3Db2W7WTJUsSGM8cfAZVkiRp4j113+Qb123lcxdvA7bcM44yGFRJkqRGiICDDn4I2LZ9HMd3oLokSRq7TPjaVQvZuHEBq1bt5EUvfogY+fV782NQJUmSxup7twTHHLWUWzYvYPFieOAB2HflTj79uW08dd/xDj7vh91/kiRpbDLhmKOWsmnjAnbsCLZtC3bsCDZtXMCxr146piHngzGokiRJY/O1qxZyy+YFPPjgo/v6Hnww2LxpAV+7auGYStY/gypJkjQ2GzcWXX7tLF5cLJ8Uk1NSSZLUOKtW7eSBB9ove+CBYvmkMKiSJElj0ykb+qJFycr9iqsAJ4VBlSRJGpvZ2dCXLk2WLElWrS6u/puktAqmVJAkSWPVyoZunipJkqR5amVDLzKiTya7/yRJkirgmSpJkjQvTZhipgoGVZIkaWBNmWKmCnb/SZKkgTRpipkqGFRJkqSBNGmKmSoYVEmSpIE0aYqZKkxXbSVJUmWaNMVMFQyqJEnSQJo0xUwVahFURcReEXFSRHwmIr4bEfdFxJaIuDIi3hARA5czIl4TEVneTqqy3JIkTbMmTTFThbqkVDge+GvgB8DlwPeAnwOOAS4AjoiI4zP7u44gIp4CvB/YCiyrtMSSJKkxU8xUoS5B1c3AUcC/ZubDHbAR8VbgauBYigDrU73uMCIC+AhwF/Bp4IwqCyxJkgpNmGKmCrXo/svMyzLzopkBVfn4HcAHy38P6XO3pwCHAb8LbJt3ISVJaqBMWL9uIR+/cDHr1y2cutxSVarLmapuWtcVPNjrBhHxTOA9wHmZ+dWIOGwoJZMkaYKZDb1atThT1UlELAJ+p/z3kj62uZBiXNZbh1Q0SZImmtnQq1froIribNOzgYsz8ws9bvNnwPOA12Xmff0cLCJOi4hbW7etW7f2WVxJkiaD2dCrV9ugKiJOAU4Hvg28tsdtDqQ4O/XezFzf7zEz8+zMXNG6LVvmBYOSpGYyG3r1avmMRcTJwHnADcChmXl3D9ssAv6e4krCtw23hJIkTTazoVevdkFVRJwKnA9cTxFQ3dHjpsuApwPPBO6fkfAzgbeX6/xd+di5lRdckqQJYjb06tXq6r+IeAvFOKrrgF/NzDv72PxnwIc6LPslinFWVwI3AX13DUqS1CStbOizr/5bud90ZkOvQm2Cqoh4G/BO4Brg5d26/CJiMbAaeCAzNwCUg9LbTkMTEWdRBFUfzcwLKi66JEkTyWzo1apFUBURJ1IEVA8BVwCnxGNbdHNmri3/Xg7cCNwCrBxNKSVJah6zoVenFkEVsF95vxA4tcM6XwHWjqQ0kiRNgEw8y1Qj0eccxVNl+fLleeOGG8ZdDEmSHsNs6J3tscuet2XmilEft3ZX/0mSpO7Mhl5PBlWSJE0Ys6HXk0GVJEkTxmzo9eSzLknShDEbej0ZVEmSNGHMhl5PBlWSJE2YVjb0/VbtZMmSZOnSZMmSZNVqs6GPU13yVEmSpD6YDb1+DKokSZpQZkOvF7v/JEmSKuCZKkmSxsApZprHoEqSpBFziplmsvtPkqQRcoqZ5jKokiRphJxiprkMqiRJGiGnmGkuW06SpBFyipnmMqiSJGmEnGKmuQyqJEkaIaeYaS5TKkiSNGJOMdNMBlWSJI2BU8w0j0GVJEl9MBO6OjGokiSpR2ZCVzcOVJckqQdmQtdcDKokSeqBmdA1F4MqSZJ6YCZ0zcVXgCRJPTATuuZiUCVJUg/MhK65GFRJktQDM6FrLqZUkCSpR2ZCVzcGVZIk9cFM6OrE7j9JkqQKeKZKkjQ1nGJGw2RQJUmaCk4xo2Gz+0+S1HhOMaNRMKiSJDWeU8xoFAyqJEmN5xQzGgVfRZKkxnOKGY2CQZUkqfGcYkajYFAlSWo8p5jRKJhSQZI0FZxiRsNmUCVJmhpOMaNhMqiSJE0Es6Gr7gyqJEm1ZzZ0TQIHqkuSas1s6JoUBlWSpFozG7omhUGVJKnWzIauSVGLV2JE7BURJ0XEZyLiuxFxX0RsiYgrI+INEdFTOavajySpPsyGrklRl4HqxwN/DfwAuBz4HvBzwDHABcAREXF85pw951XtR5JUE61s6Js2ProL0GzoqpuoQ3wREYcBS4F/zcydMx7fB7gaeApwXGZ+ahT7aVm+fHneuOGGfqsjSapYu6v/Vu5XXP33lKeO/3tM9bLHLnvelpkrRn3cWpypyszLOjx+R0R8EHgXcAjQNRiqaj+SpHoxG7omQS2Cqjm0etIfrMl+JEljYDZ01V2tB25HxCLgd8p/Lxn3fiRJkjqp+5mq9wDPBi7OzC8Mez8RcRpwWuv/3XfffR6HlCS1OMWMpkEtBqq3ExGnAOcB3wYOzsy7R70fB6pL0vw5xYxGbVwD1WvZ/RcRJ1MEQjcAh84joKpkP5KkwTjFjKZJ7YKqiDgVOB+4niIQumOc+5EkDc4pZjRNahVURcRbgHOA6ygCoR+Ncz+SpPlxihlNk9q8miPibRQDyq8BDs/MO7usuzgi1kTE6vnsR5I0XE4xo2lSi6v/IuJE4J3AQ8AVwCnx2MtCNmfm2vLv5cCNwC3AynnsR5I0RE4xo2lSi6AK2K+8Xwic2mGdrwBrR7QfSVIFIuDTn9vWcYoZ0yqoSWqbUqEOTKkgSdUwT5VGaarn/pMkNZtTzGgaGFRJkrryLJPUG4MqSVJHZkOXeleblAqSpHoxG7rUH4MqSVJbZkOX+mNQJUlqy2zoUn98R0iS2jIbutQfgypJUlutbOiLFj168JTZ0KX2DKokSW21sqHvt2onS5YkS5cmS5Ykq1abDV1qx5QKkqSOnrpv8o3rtpqnSuqBQZUkqSuzoUu9MaiSpAYzG7o0OgZVktRQZkOXRsuB6pLUQGZDl0bPoEqSGshs6NLoGVRJUgOZDV0aPd9VktRAZkOXRs+gSpIayGzo0ugZVElSA5kNXRo9UypIUkOZDV0aLYMqSWows6FLo2P3nyRJUgU8UyVJNeUUM9JkMaiSpBpyihlp8tj9J0k14xQz0mQyqJKkmnGKGWkyGVRJUs04xYw0mXxnSlLNOMWMNJkMqiSpZpxiRppMBlWSVDNOMSNNJlMqSFINOcWMNHkMqiSpppxiRposBlWSNARmQ5emj0GVJFXMbOjSdHKguiRVyGzo0vQyqJKkCpkNXZpeBlWSVCGzoUvTy3e3JFXIbOjS9DKokqQKmQ1dml4GVZJUIbOhS9PLlAqSVDGzoUvTyaBKkobAbOjS9LH7T5IkqQKeqZKkWZxiRtIgDKokaQanmJE0qFp0/0XEXhFxUkR8JiK+GxH3RcSWiLgyIt4QEX2VMyJWRMSHI+L2iPhZRGyOiHMj4gnDqoOkyecUM5LmoxZBFXA88HfAgcDXgXOBTwHPBi4A/imit5PvEbEauAb4XeBq4BxgI/BmYH1E7FV56SU1glPMSJqPfs8APa/H9V7fZzluBo4CVmTmCZn5PzPz9cAa4PvAscAxPe7rfwNPBk7JzKMz88zMPIwiuHoG8K4+yyZpSjjFjKT56PcT4qqI+P1OCyNil4j4e4qzTj3LzMsy86LM3Dnr8TuAD5b/HjLXfsqzVC8HNgMfmLX47cA24LURsbSf8kmaDk4xI2k++g2qfgCcHxGfjIjdZy6IiF8E/gN4DbC+ovIBtD7iHuxh3UPL+y+2CdDuBdYBuwIvqq54kprCKWYkzUe/QdX+wD9RdMd9MyKeDw93932donvtfwEvraJwEbEI+J3y30t62OQZ5f3NHZZ/p7x/+nzKJamZnGJG0nz0lVKhPNvzXyPiUuA84MqIuJLiDNHtwAmZ+dUKy/ceisHqF2fmF3pYf4/yfkuH5a3H95xvwSQ1k1PMSBrUQHmqMvPDEfEgsBY4DLgPODwzO50h6ltEnAKcDnwbeG1V+53jmKcBp7X+33333busLampnGJG0iAGupQlIs6gGIy+A/h3YBfgSxFxSBWFioiTKc6E3QAcmpl397hp60zUHh2Wtx6/p93CzDw7M1e0bsuWLeu5zJLqIRPWr1vIxy9czPp1C80tJWlk+jpTFRFPAv4e+DVgE/CbmXlNRJwIvB+4NCLeDZw1e6B4H8c4lSL9wfUUZ79+1MfmN5X3ncZMPa28r+yMmqT6MBu6pHGK7ONnXETcDuwDfBI4qRxj1Vq2hmIQ+y8C6zKz78HqEfEWinFU1wG/mpl39rn9auC7FCkVVs8M7CJiN4qrFwN4cmZum2t/y5cvzxs33NBPESSNSSa84IBlbNr46OSdixYVA82vvnar46KkKbHHLnvelpkrRn3cfrv/9gT+e2b+5syACiAzvw28EPgQcHC/BYmIt1EEVNdQnKHqGFBFxOKIWFMGUTPLsAH4IrASeNOszd4BLAUu7CWgkjRZzIYuadz6Hah+YGZ+q9PCzLwfeGNEfKmfnZbdh+8EHgKuAE5pMyvN5sxcW/69HLgRuIUigJrp94GrgPdFxOHlegdSXKF4M/An/ZRN0mRoZUPfseOxy1rZ0B14LmmY+k2p0DGgmrXeJ/osx37l/ULg1A7rfIXiasO5jr2hzJ/1TuAVwCspuv3OA96RmT/ps2ySJoDZ0CWNW19jqqaNY6qkyeGYKkktkzKmSpJqyWzoksZtoOSfklRHZkOXNE4GVZIaxWzoksbFoEpSLWTiGSZJE82gStLYmQldUhM4UF3SWGXCMUctZdPGBezYEWzbFuzYEWzauIBjX73UufskTQyDKkljZSZ0SU1hUCVprFqZ0NtpZUKXpEngp5WksTITuqSmMKiSNFYvevFD7LtyJ4sWPXrw1KJFycr9iqsAJWkSGFRJGiszoUtqClMqSBo7M6FLagKDKkm1YCZ0SZPO7j9JkqQKeKZK0rw5xYwkGVRJmienmJGkgt1/kgbmFDOS9AiDKkkDc4oZSXqEQZWkgTnFjCQ9wk88SQNzihlJeoRBlaSBOcWMJD3CoErSwJxiRpIeYUoFSfPiFDOSVDCokjRvTjEjSQZV0tQzG7okVcOgSppiZkOXpOo4UF2aUmZDl6RqGVRJU8ps6JJULYMqaUqZDV2SquWnpjSlzIYuSdUyqJKmlNnQJalaBlXSlDIbuiRVy5QK0hQzG7okVcegSppyZkOXpGrY/SdJklQBz1RJE8wpZiSpPgyqpAnlFDOSVC92/0kTyClmJKl+DKqkCeQUM5JUPwZV0gRyihlJqh8/eaUJ5BQzklQ/BlXSBHKKGUmqH4MqaQI5xYwk1Y8pFaQJ5RQzklQvBlXSBHOKGUmqD4MqaUzMhi5JzVKboCoijgNeBhwA7A/sBnw8M18zwL5eBbwZeBawF/AD4Brg7MxcX1mhpQGZDV2SmqdOA9X/FDiZIqi6bdCdRMRfAJ8Hfgm4BDgP+CbwamBdRPQdpElVMhu6JDVTnYKqPwSeDuwO/I9BdhAR+wBnAD8EnpWZJ2XmmZl5HPBrQADvrKi80kDMhi5JzVSboCozL8/M72TO63f6vhR1+npm/mj2/oF7gb3nsX9p3syGLknN1LRP7+8AO4AXRsSTZi6IiJdSjNP693EUTGoxG7okNVNtBqpXITPvjoi3AGcDN0TEZ4G7gNXAUcClwH8bYxGlh7Ohb9r46C5As6FL0mRr2pkqMvNc4BiKgPH3gDOB44HvA2tndwvOFBGnRcStrdvWrVtHUmZNF7OhS1IzNepMFUBE/DHw58D7gPcDdwBrgHcDH4+IAzLzj9ttm5lnU5zlAmD58uVeh6WhMBu6JDVPo4KqiDgE+AvgM5l52oxF34yI3wBuBk6PiA9m5sZxlFFqMRu6JDVL07r/fr28v3z2gszcDlxNUefnjbJQap5MWL9uIR+/cDHr1y00t5QkqVlnqoDHlfed0ia0Ht8xgrKoocyGLklqZyLPVEXE4ohYExGrZy26orx/Y0Qsn7XNEcDBwP3AVSMophrIbOiSpE5qc6YqIo4Gji7/3ae8Pygi1pZ/35mZZ5R/LwduBG4BVs7YzT9T5KH6FeDGiPgMxUD1Z1J0DQZwZmbeNaRqqOF6yYbuGClJmk61Caoo5vw7cdZjq8obFAHUGXSRmTsj4pXAm4DfAn4D2BW4G7gYeF9mfrHKQmu6tLKh72jTgdzKhm5QJUnTqTZBVWaeBZzV47qbKc46tVv2AHBueZMqZTZ0SVInEzmmShqXVjb0RYsePXjKbOiSJIMqqQ9mQ5ckdVKb7j9pUpgNXZLUjkGVNACzoUuSZrP7T5IkqQKeqdLUycSuO0lS5QyqNFWcYkaSNCx2/2lqOMWMJGmYDKo0NXqZYkaSpEEZVGlqtKaYaac1xYwkSYPyW0RTwylmJEnDZFClqeEUM5KkYTKo0tRwihlJ0jCZUkFTxSlmJEnDYlClqeMUM5KkYTCo0kQxG7okqa4MqjQxzIYuSaozB6prIpgNXZJUdwZVmghmQ5ck1Z1BlSaC2dAlSXXnN5EmgtnQJUl1Z1CliWA2dElS3RlUaSKYDV2SVHemVNDEMBu6JKnODKo0UcyGLkmqK7v/JEmSKuCZKo2MU8xIkprMoEoj4RQzkqSms/tPQ+cUM5KkaWBQpaFzihlJ0jQwqNLQOcWMJGka+G2moXOKGUnSNDCo0tA5xYwkaRoYVGnonGJGkjQNTKmgkXCKGUlS0xlUaWScYkaS1GQGVeqJ2dAlSerOoEpzMhu6JElzc6C6ujIbuiRJvTGoUldmQ5ckqTcGVerKbOiSJPXGb0R1ZTZ0SZJ6Y1ClrsyGLklSbwyq1JXZ0CVJ6o0pFTQns6FLkjS32pypiojjIuL8iLgiIn4aERkRH5vH/g6PiM9ExB0R8bOIuD0ivhARr6yy3NOilQ39hNc+wEEHG1BJkjRbnc5U/SmwP7AVuBVYM+iOIuIvgT8q9/M54E5gb+CXgUOAi+dZVkmSpEepU1D1hxRB0HeBlwGXD7KTiPg9ioDqo8AbM3PHrOUdEgQ0k9PLSJI0GrUJqjLz4SAqBvzWj4jHAe8CvkebgKo8TocEAc3j9DKSJI1ObYKqivwqRTffucDOiHgV8GzgfuDqzFw/zsKN0szpZR58MNhRhpet6WWuvnarZ6wkSapQ04KqF5T39wPXUgRUD4uIrwLHZeaPR12wUetlepmDDjbHlCRJVanN1X8VeXJ5/0dAAi8BdgOeC3wReCnwyfEUbbScXkaSpNFq2jdrqz4PAkdl5pWZuTUzvwX8BsVA+JdFxEHtNo6I0yLi1tZt69atIyp29ZxeRpKk0WpaUHVPeX9tZm6euSAztwNfKP99YbuNM/PszFzRui1btmx4JR0yp5eRJGm0mhZU3VTe39Nh+U/K+11GUJaxcnoZSZJGq2kD1b9EMZbqWRGxIDNn93G1Bq5vGm2xxsPpZSRJGp2JDKrKBJ6rgQcyc0Pr8cy8JSIuAo4C3gycM2OblwO/RnEW65LRlnh8WtPLeKWfJEnDVZugKiKOBo4u/92nvD8oItaWf9+ZmWeUfy8HbgRuAVbO2tWbgOcBZ5d5qq4F9iv3/RBwUmZuGUYdqmY2dEmSJkdtgirgAODEWY+tKm9QBFBnMIfMvDUifhn4M4ozVi8FfgpcBLw7M6+urMRDZDZ0SZImS2T6Bd3J8uXL88YNN4z8uJnwggOWPZwNvWXRomKgudnQJUnqbI9d9rwtM1eM+rhNu/qvEXrJhi5JkurFoKqGzIYuSdLk8du5hsyGLknS5DGoqiGzoUuSNHkMqmrIbOiSJE2eOqVU0AxmQ5ckabIYVNWY2dAlSZocBlVDYjZ0SZKmi0HVEJgNXZKk6eNA9YplwjFHLWXTxgXs2BFs2xbs2BFs2riAY1+9FBPYS5LUTAZVFTMbuiRJ08mgqmJmQ5ckaTr5DV8xs6FLkjSdDKoqZjZ0SZKmk0FVxcyGLknSdDKlwhCYDV2SpOljUDUkZkOXJGm62P0nSZJUAYOqLrZvh/XrFpqwU5IkzcmgqostW4KjXrmUFxywjO/d4oAoSZLUmUFVF5k4xYwkSeqJQVUPnGJGkiTNJdLTLx1FLErYp/wvE7bcA9u2j7VQo7EM2DruQoyB9Z4u1nu6WO/psk9mjjzDgSkVunrotsxbV4y7FKMWEbdmpvWeEtZ7uljv6TLN9R7Hce3+kyRJqoBBlSRJUgUMqro7e9wFGBPrPV2s93Sx3tPFeo+QA9UlSZIq4JkqSZKkChhUSZIkVcCgSpIkqQJTEVRFxOKIeHNEfCQirouIHRGREXHSPPb54oi4OCLujoj7IuL/RsSpEdEx7XpE/HpEfDkitkTE1oj4ekScOGgZhl3eNvs4q3zeut02zNrmkDnWf0/1tX342POuc7mfbuX/WpftJr29l0fEH0TEv0XE5oj4WUTcFRGXRsQxHbYZantHxIqI+HBE3F6WZ3NEnBsRT+hzP08st2vV6/Zyvx3z+VR17EHM99gRsTQiToiIf4iIb0fEtoi4NyL+IyJOj4glHbYb6LVflSqe8/I92K0ej++w3bMi4p8i4kcRcX9E3BQR74iIXaqrYccyz7e953oftm5PmbXd2No7Io6LiPMj4oqI+Gl5zI8NuK++n7+q2ntakn8uBc4t//4hcAfwlM6rdxcRrwY+BdwPfAK4GzgSOAc4GDi+zTYnA+cDdwEfA3YAxwFrI+I5mXnGoOUZRnk7+HKXZUcCvwT8W4flX+mw/ZU9HrsvFda55RZgbZvH2yaYa0h7/wHwFmATcDnF+2Zf4BjgVyLinMw8rcO2lbd3RKwGrgKeDPwL8G3ghcCbgVdExMGZeVcP+9mr3M/TgcuAfwTWAL8LvCoiDsrMjcM49iAqOvZLKF6Hd1O05WeBJwBHAX8FHBMRh2fm/W227eu1X5UhPOfv6PD4g22OfSDFa2Mx8M/A94HDgD8DDi+fq5/1ceyeVVTvzXSu73Mo3sPXZ+b32ywfS3sDfwrsT5H9/VaK92TfBnn+Km3vzGz8DVgCHAH8fPn/WUACJw2wr92BHwE/A54/4/HHlw2ZwG/N2mYlxRfcXcDKGY8/Afhuuc1BQ6p73+Ud4BgLyxdhAs+dteyQ8vGzRtjelda5XP/LfazfiPam+OB9WZvHnwlsKff1y6Nqb+AL5b7/YNbjZ5ePf7DH/fxNuf57Zz1+Svn4JcM69rjqDRwAnAAsmfX4bsA15X5Ob7NdX6/9utW7XP/LQPZx3IXADeUxjprx+AKKL9wEzqx7vbvs//+U+zmlZu19KPA0IGZ8jnxs2M9f1e098ieuDjfmF1S9vtz2o22WHVYu+8qsx99ZPv6OfvZXUV37Lu8Axziy3M/6Nstab46zRti+lda53w+aprd3ua+/pc0X8bDaG1hd7ncTsGDWst0oft1uA5bOsZ9lwPZy/d1mLVtA8Qs/gVVVH3uc9Z7jGL9dHuOiNsvG8iVbZb3pP6jq+D4BVpXLNlOmJKprvTvs/0kUP/i2A3vWpb3blKP1OdJXUDXI81d1e0/FmKqKHVbeX9Jm2VcpXqwvjojH9bjNv81ap2qDlLdfbyzv/7bLOv8pIk6OiLdGxOsj4mnzON5chlHnPctyvzUi3hQRLxrw+E1ob4AHyvvHdJ2Uqm7vQ8v7L2bmzpkLMvNeYB2wK9CtXSiX7wKsK7ebuZ+dFL9yZx6vymMPYhTHnqst+3ntV6XyekfEb0bEmRFxWkQc0eU90PE9lEW38M0U3eCrej12H4bd3icCjwM+mZn3dFhnHO1dlUGev0rb26Cqf88o72+evSAzH6SIkBfx6Abots0PKCLnFRGxa7VFnfPYncrbsygG9h5B0R30iS6rnkAxxuhdwIeAmyPin/sZcNqHYdR5f4pyvwt4P7A+iosentPn8Se6vQEiYnfgWIpfcF/ssFrV7d2xXqXvlPdPH8J+qjr2IEZx7NeX9+0CcejvtV+VYdT7H4F3A+8FLga+FxHHjejYvRr2sX+vvP+bLuuMo72rMvb3t0FV//Yo77d0WN56fM8Bttmjw/L5GKS8/XgDRZ/0xzJze5vlPwbOpBgcuRuwN0UQdi3FF/NFEVH167DqOp9NMcB7b4o6vICir31/4LKIWD7g8SeuvSMigAuAnwP+OjNvnLXKsNq7qnoN8/076Huom2G358nAK4DrgA+3WaXf135Vqqz3v1AMUVhBcZZyDUVwtSfwiYh4xRCP3a+hHTsiXkYRQFyfmVd1WG1c7V2Vsb+/JyaoKi+J7OUS0dZtoEsx66iudS+/HN9Q/tv2l09m/r/M/IvMvD4zt2bmnZl5CUWf+SaKN/CRbfZdmzpn5umZeVVZ9q2Z+R+ZeTzFFXZPAiq9kq9OdW/jvRRXD14BPObKv0HbW6MXRVqMcymu6jw2Mx+Yvc6oX/vDkJnnZObnM/O2zLw/M2/KzLcCp1N8B757zEUclTmHaTShvcdtklIqbKAYYNer24dUjrnOMrQen9lfvYXiBbkHxRVhnbbpFCnPp+6DlLdXR1CkpvhaZn6rnw0z86cR8Q/AnwAvpfg1OVNd6zzTBynOvrx01uONbO+I+EvgDynGZr0q+7ikvIf2nktV9Rr0/VvFsQcxlGNHxNEU3WE/Ag7NWSkketDptV+VUTznF1CkGTkgInabMcauie39RIr2ug+4cIByDbu9qzL29/fEBHoNkA4AAAYTSURBVFWZefi4y1C6CXg+Rf/qNTMXRMQiYD+KAZ8bZ23zpHKb9bO2+XmKPFq3dug+m2/dBylvr1q/fLr1z3fz4/J+6ewFNa7zTJ3K37j2johzgFMpchz9eqeyz6Fje/fgpvK+07iG1kD4TuMi5rOfqo49iMqPHRHHA/9AcYbqsMz8zhybtDOftuzF0J/zzLw/Iu6lSHWyFGgFVY1q71JrgPpHuwxQ72bY7V2Vsb+/J6b7r0YuK+9n98NDEcXvClw161d8t22OmLVO1QYp75wi4heAVzH3APVuWldgzDe4mW0odW6jU/kb095R+ABFQHUpxRmqQQIqmF97X17ev3z2mKyI2I2iW3E7MFfW569R/Fo/uNxu5n4WAC+fdbwqjz2ISo8dESdQ5Cm6nSIH2SABFQzvvdsy9Oc8Ip5BEVDdC9w5Y1HH91BErKL48r2F4dR9WPVuDVDvdoV2N8Nu76oM8vxV296D5JCY9Bs95KmiOOW3hjJh6IzHd6eI2vtJ/rkf400G2W95dy3r/tQu+31bue35cxz/+R0efw2wsyzXyl7qMo46A88FFrc5xnMpPogT+O0mtjdFEr6/K7e5GHh8D8cfWnvTf1K/NcCaNvtpdPLPLvU+EXiI4sth3x6O2/drv271Lt+LT2yz771nvB/+dtaybskgP0nNkn92au8Zy19SbvetOrf3rGMeQpc8VRSZz9cAqyt4/ipt76E/OXW5UVyRtLa8XVc+UetmPHbSrPVfV66zts2+jqboQtlK0S//lxSp8LNshMckCaOY8iPLF+cHKPryW1nI/2rIde+rvDNe0F/usL+ZSRKfM8exN1MEEv9IMR3G+4Gvl9s+ALyuznUuXxt3UUzrcX5Zh8+X+06KX36NbG/g7eXj24E/p/gxMvt29KjamyKx3w/LfX2WYoDxZeX/NwF7zVo/aZP0EdirXD+BL5X7+Wz5/w9p/0Hd17Erbs9515sif89D5bIPdWjLU6t47des3q+j+IHz72V530PR9XlPuf43aJ8E80CK1Cc7yvXfU66bFFMtPa7O9Z61/ELaBBlt1ht3ex/NI9/Hl5TH3DDjsb+ase7Kcvnm+T5/Vbf3UJ6cOt4os+p2ua2dtf7r2j0+Y/nBFL/ef0LRnfAtikG8C7uU4UiKOdHuLRvwG8CJI6p/z+Vl7qDqiHL5YzKot1n3LRTdRt8vj3t/+Ub5CLB/3etcvtE/TREo/LR80/0AuIgZv2qa2N7lB1m390y7981Q25viwoiPlG2wg+K0/LnAE9qs2/HLBngicF65fatNPwysqOLYQ2jPedWbRz7Put02V/Xar1G9n1O+jr9FETA8QDH/4RUUP3yWdDn2syh+hNxJcYb1Zor59Hape71nLHtC+T5sm0G9Tu3NIz1Ic74+6RJU9fv8Vd3eUe5MkiRJ8+BAdUmSpAoYVEmSJFXAoEqSJKkCBlWSJEkVMKiSJEmqgEGVJElSBQyqJEmSKmBQJUmSVAGDKkmSpAoYVEmSJFXAoEqSJKkCBlWSGi8iro+IjIjndli+ICI2RcQ9EbFs1OWT1AwGVZKmwfnl/ckdlh9JMfP9RzJz60hKJKlxIjPHXQZJGqqI2BW4FXgcsCIzfzJr+aXAYcDTM3PDGIooqQE8UyWp8TJzO3ABsCvw+pnLIuIZwOHAxQZUkubDoErStPgA8BDw+xEx87PvZCCA942lVJIaw+4/SVMjIj4FHAMcmZmfLwel3wbclpnPGm/pJE06z1RJmiats1GtAesnArsD7x9PcSQ1iWeqJE2ViLgOeC6wBvgs8AvA8szcNtaCSZp4nqmSNG3eRzGG6mPAM4EPG1BJqoJnqiRNlYh4PPB94EnATuBpmblxvKWS1ASeqZI0VTLzfor0CgD/akAlqSoGVZKm0b7l/fld15KkPhhUSWq8iPjPM/4+EPgvwNWZeen4SiWpaRxTJanRykmUrwU2ALcALykXHZqZ68dWMEmN45kqSU23ieKKv92BFwPfBF5pQCWpap6pkiRJqoBnqiRJkipgUCVJklQBgypJkqQKGFRJkiRVwKBKkiSpAgZVkiRJFTCokiRJqsD/B6f4JJ668pprAAAAAElFTkSuQmCC\n",
"text/plain": [
"