{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Linear_regression_with_one_neuron","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"atdxLNG7Bmfx"},"source":["# Linear Regression with One Neuron\n","\n","Version 1.03"]},{"cell_type":"markdown","metadata":{"id":"jYIdcBEGhc8R"},"source":["(C) 2020 - Umberto Michelucci, Michela Sperti\n","\n","This notebook is part of the book _Applied Deep Learning: a case based approach, **2nd edition**_ from APRESS by [U. Michelucci](mailto:umberto.michelucci@toelt.ai) and [M. Sperti](mailto:michela.sperti@toelt.ai)."]},{"cell_type":"markdown","metadata":{"id":"tVpnrRq9yP7F"},"source":["The purpose of this notebook is to give an example of an application of Linear Regression performed with One Neuron to a dataset taken from real world."]},{"cell_type":"markdown","metadata":{"id":"PGu2kWUryTSC"},"source":["## Notebook Learning Goals"]},{"cell_type":"markdown","metadata":{"id":"VeNwsKOc88sR"},"source":["At the end of the notebook you are going to have a clear idea of what linear regression is, seen through a practical example. Moreover, you will have learnt what is one of the simplest tasks that Neural Networks can solve. You can of course perform linear regression easily by applying traditional math formulas or using dedicated functions such as those found in scikit-learn. However, it is very instructive to follow this example, since it gives a practical grasp of how the building blocks of Deep Learning architectures (i.e. neurons) work."]},{"cell_type":"markdown","metadata":{"id":"sya8rrO3dCR7"},"source":["## Real Case Example: **Radon Contamination**\n","\n"]},{"cell_type":"markdown","metadata":{"id":"2TDH1V4ACBlY"},"source":["### Dataset Overview"]},{"cell_type":"markdown","metadata":{"id":"5_AH8YS-h1Qv"},"source":["Radon is a radioactive gas that enters homes through contact points with the ground. It is a carcinogen that is the primary cause of lung cancer in non-smokers. Radon levels vary greatly from household to household. This dataset contains measured radon levels in U.S. homes by county and state. The *activity* label is the measured radon concentration in *pCi/L*. Important predictors are: \n","- *floor* (the floor of the house in which the measurement was taken), \n","- *county* (the U.S. county in which the house is located), and \n","- *uppm* (a measurement of uranium level of the soil by county).\n"]},{"cell_type":"markdown","metadata":{"id":"wOFzuRWYzyuE"},"source":["This dataset fits well a classical regression problem, since it contains a continuous variable (radon activity) which is interesting to be predicted. The model which will be built is made of one neuron and will fit a linear function to floor, county, uppm and pcterr variables."]},{"cell_type":"markdown","metadata":{"id":"or5Hcrsr95Gg"},"source":["### Libraries and Dataset Import"]},{"cell_type":"markdown","metadata":{"id":"vpeWLaBf1dPc"},"source":["This section contains the necessary libraries (such as tensorflow or pandas) you need to import to run the notebook."]},{"cell_type":"code","metadata":{"id":"rx34o2vMwvL6","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1610562018430,"user_tz":-60,"elapsed":15385,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"f0a17c53-b888-438c-cb5f-4064e3b1715f"},"source":["# This command install code from the tensorflow docs repository.\n","# We need to use tensorflow_docs.modeling function when training our model.\n","# This function will generate a report on the network's perfomances\n","# step by step during the training phase (see Training Phase section of the\n","# notebook). \n","\n","# You can safely ignore this cell if you don't understand what it does.\n","\n","!pip install git+https://github.com/tensorflow/docs"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Collecting git+https://github.com/tensorflow/docs\n"," Cloning https://github.com/tensorflow/docs to /tmp/pip-req-build-i6fpletq\n"," Running command git clone -q https://github.com/tensorflow/docs /tmp/pip-req-build-i6fpletq\n","Requirement already satisfied (use --upgrade to upgrade): tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499- from git+https://github.com/tensorflow/docs in /usr/local/lib/python3.6/dist-packages\n","Requirement already satisfied: astor in /usr/local/lib/python3.6/dist-packages (from tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499-) (0.8.1)\n","Requirement already satisfied: absl-py in /usr/local/lib/python3.6/dist-packages (from tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499-) (0.10.0)\n","Requirement already satisfied: protobuf>=3.14 in /usr/local/lib/python3.6/dist-packages (from tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499-) (3.14.0)\n","Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499-) (3.13)\n","Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499-) (0.8)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from absl-py->tensorflow-docs===0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499-) (1.15.0)\n","Building wheels for collected packages: tensorflow-docs\n"," Building wheel for tensorflow-docs (setup.py) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for tensorflow-docs: filename=tensorflow_docs-0.0.0d7cf3b307cf13e5aadea359db22a77a1cb04f499_-cp36-none-any.whl size=146696 sha256=66a2dc47b5db36fffdcf797cf2ddcac23ff1e0b0f03d721cbed1bbab77e1ca68\n"," Stored in directory: /tmp/pip-ephem-wheel-cache-efk_dabh/wheels/eb/1b/35/fce87697be00d2fc63e0b4b395b0d9c7e391a10e98d9a0d97f\n","Successfully built tensorflow-docs\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"IKQxIMYyeCva"},"source":["# general libraries\n","import os\n","import sys\n","import numpy as np\n","import pandas as pd\n","import matplotlib.pyplot as plt\n","import matplotlib.font_manager as fm\n","\n","# tensorflow libraries\n","import tensorflow as tf\n","from tensorflow import keras\n","from tensorflow.keras import layers\n","import tensorflow_docs as tfdocs\n","import tensorflow_docs.modeling\n","\n","# ignore warnings\n","import warnings\n","warnings.simplefilter('ignore')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"CtgG2-1y_-2o"},"source":["The following cells are needed to **download** the dataset. You don't need to understand all the download and processing steps, since the focus of this section is to apply a simple linear regression model to a real case dataset (therefore you can just execute the following cells, ignoring their content). If you are interested in the details, you can find the complete code in the /modules folder."]},{"cell_type":"markdown","metadata":{"id":"HOfq2TvDbCwr"},"source":["Now we clone the repository for the book to be able to access the modules that we have written for all the juypter notebooks."]},{"cell_type":"code","metadata":{"id":"aD3j0wJoP7hs"},"source":["# Referring to the following cell, if you want to re-clone a repository\n","# inside the google colab instance, you need to delete it first. \n","# You can delete the repositories contained in this instance executing \n","# the following two lines of code (deleting the # comment symbol).\n","\n","# !rm -rf ADL-Book-2nd-Ed \n","# !rm -rf BCCD_Dataset"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"AiWkaBRaYSHH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1610562018432,"user_tz":-60,"elapsed":15374,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"f1a5351f-b158-4954-cd90-b38fbf2b6edc"},"source":["# This command actually clone the repository of the book in the google colab\n","# instance. In this way this notebook will have access to the modules\n","# we have written for this book.\n","\n","# Please note that in case you have already run this cell, and you run it again\n","# you may get the error message:\n","#\n","# fatal: destination path 'ADL-Book-2nd-Ed' already exists and is not an empty directory.\n","# \n","# In this case you can safely ignore the error message.\n","\n","!git clone https://github.com/toelt-llc/ADL-Book-2nd-Ed.git"],"execution_count":null,"outputs":[{"output_type":"stream","text":["fatal: destination path 'ADL-Book-2nd-Ed' already exists and is not an empty directory.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"4FxrxSG771VZ"},"source":["# This cell imports some custom written functions that we have created to \n","# make the loading of the data and the plotting easier. You don't need \n","# to undertsand the details and you can simply ignore this cell.\n","# Simply run it with CMD+Enter (on Mac) or CTRL+Enter (Windows or Ubuntu) to\n","# import the necessary functions.\n","\n","import sys\n","sys.path.append('ADL-Book-2nd-Ed/modules/')\n","\n","from read_radon_dataset import read_data\n","from style_setting import set_style"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Xwr5PiuX5cVS"},"source":["# This cell provides the dataset on which you will implement the linear regression model.\n","# Data are temporarily put into \"tmp\" folder. \"url_base\" contains the link to access the dataset.\n","\n","# After cell's execution, you will have a variable containing features (\"radon_features\"),\n","# a variable containing labels (\"radon_labels\") and an informative variable containing\n","# all countries including in the dataset. \n","\n","# You don't need to understand the implementation's details and you can simply ignore this cell.\n","# Simply run it with CMD+Enter (on Mac) or CTRL+Enter (Windows or Ubuntu) to\n","# import the necessary functions.\n","\n","# inputs to download the dataset\n","CACHE_DIR = os.path.join(os.sep, 'tmp', 'radon')\n","url_base = 'http://www.stat.columbia.edu/~gelman/arm/examples/radon/'\n","# Alternative source:\n","# url_base = ('https://raw.githubusercontent.com/pymc-devs/uq_chapter/master/reference/data/')\n","\n","rd = read_data(CACHE_DIR, url_base)\n","radon_features, radon_labels, county_name = rd.create_dataset()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"P0D4C0bsAWc6"},"source":["Now you have all the necessary elements to successfully implement this tutorial. **Let's have a look at our data**:"]},{"cell_type":"code","metadata":{"id":"UxV7roADedjA","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1610562018822,"user_tz":-60,"elapsed":15751,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"191acb05-8808-4661-e3a9-3f0698f25f9b"},"source":["num_counties = len(county_name)\n","num_observations = len(radon_features)\n","print('Number of counties included in the dataset: ', num_counties)\n","print('Number of total samples: ', num_observations)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Number of counties included in the dataset: 85\n","Number of total samples: 919\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"-rrry8uqeelL","colab":{"base_uri":"https://localhost:8080/","height":198},"executionInfo":{"status":"ok","timestamp":1610562018822,"user_tz":-60,"elapsed":15745,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"a118d83c-92f2-4faf-9112-8c8a0bbcabf0"},"source":["radon_features.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","
\n"," \n","
\n","
\n","
floor
\n","
county
\n","
log_uranium_ppm
\n","
pcterr
\n","
\n"," \n"," \n","
\n","
0
\n","
1
\n","
0
\n","
0.502054
\n","
9.7
\n","
\n","
\n","
1
\n","
0
\n","
0
\n","
0.502054
\n","
14.5
\n","
\n","
\n","
2
\n","
0
\n","
0
\n","
0.502054
\n","
9.6
\n","
\n","
\n","
3
\n","
0
\n","
0
\n","
0.502054
\n","
24.3
\n","
\n","
\n","
4
\n","
0
\n","
1
\n","
0.428565
\n","
13.8
\n","
\n"," \n","
\n","
"],"text/plain":[" floor county log_uranium_ppm pcterr\n","0 1 0 0.502054 9.7\n","1 0 0 0.502054 14.5\n","2 0 0 0.502054 9.6\n","3 0 0 0.502054 24.3\n","4 0 1 0.428565 13.8"]},"metadata":{"tags":[]},"execution_count":26}]},{"cell_type":"markdown","metadata":{"id":"WiFX99w4FhGM"},"source":["The dataset is made of **919 observations**, **1 target** column (`activity`) and **4 features** (`floor`, `county`, `log_uranium_ppm`, `pcterr`). 85 different American counties are included in the dataset."]},{"cell_type":"markdown","metadata":{"id":"IyI4q099I0aD"},"source":["### Dataset Splitting"]},{"cell_type":"markdown","metadata":{"id":"1zMbWztwKkd7"},"source":["*In any machine learning project, it is a good behaviour to split the dataset you have at your disposal in different subsets.* Plenty of theoretical explanations about this need is present in literature. In the [Further Readings](#fr) section of the notebook you will find some advice on useful material about this topic. To simply explain the concept: when you build a machine learning model, you first need to train (i.e. build) the model and then you have to test it (i.e. verify the model's performances on never seen before data). The roughest way to do this is to split the dataset into two subsets: 80% of the original dataset to train the model (the more data you have the better your model will perform) and the remaining 20% to test it."]},{"cell_type":"markdown","metadata":{"id":"iVoeeov7I4DJ"},"source":["Now we build a train and a test set splitting the dataset randomly in two parts with the following proportions: **80%/20%**. "]},{"cell_type":"code","metadata":{"id":"hDfC43LoIy80","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1610562018823,"user_tz":-60,"elapsed":15741,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"909c9dea-acaf-4788-f3bd-b3509f1bdebf"},"source":["np.random.seed(42)\n","rnd = np.random.rand(len(radon_features)) < 0.8\n","\n","train_x = radon_features[rnd] # training dataset (features)\n","train_y = radon_labels[rnd] # training dataset (labels)\n","test_x = radon_features[~rnd] # testing dataset (features)\n","test_y = radon_labels[~rnd] # testing dataset (labels)\n","\n","print('The training dataset dimensions are: ', train_x.shape)\n","print('The testing dataset dimensions are: ', test_x.shape)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["The training dataset dimensions are: (733, 4)\n","The testing dataset dimensions are: (186, 4)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6-ar3G0uoKwH"},"source":["### Linear Regression: the Model"]},{"cell_type":"markdown","metadata":{"id":"6iLEJNtEqAhj"},"source":["From now on, the interesting part begins. Keep in mind that a one neuron model is an overkill for a regression task. We could solve linear regression exactly without the need of using gradient descent or similar optimisation algorithm (employed in the neuron's architecture). You can find an exact regression solution example, implemented with numpy library, in *Linear_regression_with_numpy.ipynb* notebook. \n","\n","To note is (for those with some more experience with neural networks) that we will use **one single neuron** with an **identity function** as activation function. \n","\n","Our dataset can be expressed as a matrix ($X$), where the rows represent the different measurements describing radon activity and the columns the 4 features at our disposal. Then, we have the column containing the label we wants to predict ($y$). When we employ one neuron to perform linear regression, we are simply computing the following equation: \n","\n","\\begin{equation}\n","\\hat{y}=WX+b\n","\\end{equation}\n","\n","that is a linear combination of the input data and the network's weights plus a constat term (bias) $b$. $\\hat{y}$ is the predicted output given a certain input vector ${\\bf x} = ({\\bf x_1}, {\\bf x_2}, ... , {\\bf x_{n_x}})$, where we have indicated the number of observations we have at our disposal with $n_x$."]},{"cell_type":"markdown","metadata":{"id":"J7zwPPV1ujMl"},"source":["### Structure of the Net"]},{"cell_type":"markdown","metadata":{"id":"n_Rju5QcI-0L"},"source":["Despite it is very important to keep in mind the previous considerations, developing this model with Keras is straightforward. The following function builds the one neuron model for linear regression."]},{"cell_type":"code","metadata":{"id":"TLQ3qfSEvZNz"},"source":["def build_model():\n","\n"," # one unit as network's output\n"," # identity function as activation function\n"," # sequential groups a linear stack of layers into a tf.keras.Model\n"," # activation parameter: if you don't specify anything, no activation \n"," # is applied (i.e. \"linear\" activation: a(x) = x).\n"," model = keras.Sequential([ \n"," layers.Dense(1, input_shape = [len(train_x.columns)])\n"," ])\n","\n"," # optimizer that implements the RMSprop algorithm\n"," optimizer = tf.keras.optimizers.RMSprop(learning_rate = 0.001)\n","\n"," # the compile() method takes a metrics argument, which is a list of metrics\n"," # loss = Mean Square Error (mse), metrics = Mean Absolute Error (mae),\n"," # Mean Square Error (mse)\n"," model.compile(loss = 'mse',\n"," optimizer = optimizer,\n"," metrics = ['mse'])\n"," \n"," return model"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"q0ubvr0av4mt"},"source":["model = build_model()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"j1Z6uxSyOzE6"},"source":["Let's have a look at the model summary:"]},{"cell_type":"code","metadata":{"id":"_E0vi5qXwCkH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1610562018824,"user_tz":-60,"elapsed":15727,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"c0b3f4d3-141d-4659-83e3-f75b8cc88cb0"},"source":["model.summary()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"sequential_1\"\n","_________________________________________________________________\n","Layer (type) Output Shape Param # \n","=================================================================\n","dense_1 (Dense) (None, 1) 5 \n","=================================================================\n","Total params: 5\n","Trainable params: 5\n","Non-trainable params: 0\n","_________________________________________________________________\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"SKH9b3SXDXGs"},"source":["**Learning rate** is a very important parameter of the optimizer. In fact, it strongly influences the convergence of the minimization process. It is a common and good behaviour to try different learning rate values and see how the model's convergence changes. You can find further reading advices about this topic in the [Further Reading](#fr) section of this notebook."]},{"cell_type":"markdown","metadata":{"id":"RLFuIbDWO5uP"},"source":["### Training Phase (Model's Learning Phase)"]},{"cell_type":"markdown","metadata":{"id":"4rAnbHnuPOt5"},"source":["Training our neuron means finding the weights and biases that minimize a chosen function (usually called the **cost function** and typically indicated by $J$). The cost function to be minimized in the case of a linear regression task is the **mean square error**. The most famous numerical method to find the minimum of a given function is the **gradient descent** (it is suited for cases in which the solution can not be found analytically, such as all neural network applications). In our example we used the **RMSprop algorithm** as optimizer. \n","\n","The minimization process is iterative, therefore it is necessary to decide when to stop it. The simplest way is to set a number of repetitions (called **epochs**) and to run the algorithm that fixed number of times. Then, results are checked to see if an optimal point has been reached. If not, the number of epochs is increased. \n","\n","We start training our model for **1000** epochs and we look at the summary in terms of performances (MSE). "]},{"cell_type":"code","metadata":{"id":"AJ_MTC-RwV6j","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1610562037798,"user_tz":-60,"elapsed":34696,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"f0636705-088a-47a5-9ad9-795cb879c65e"},"source":["EPOCHS = 1000\n","\n","history = model.fit(\n"," train_x, train_y,\n"," epochs = EPOCHS, verbose = 0,\n"," callbacks = [tfdocs.modeling.EpochDots()])"],"execution_count":null,"outputs":[{"output_type":"stream","text":["\n","Epoch: 0, loss:1784.2560, mse:1784.2560, \n","....................................................................................................\n","Epoch: 100, loss:20.8368, mse:20.8368, \n","....................................................................................................\n","Epoch: 200, loss:16.8424, mse:16.8424, \n","....................................................................................................\n","Epoch: 300, loss:16.1779, mse:16.1779, \n","....................................................................................................\n","Epoch: 400, loss:16.0776, mse:16.0776, \n","....................................................................................................\n","Epoch: 500, loss:16.0403, mse:16.0403, \n","....................................................................................................\n","Epoch: 600, loss:16.0212, mse:16.0212, \n","....................................................................................................\n","Epoch: 700, loss:16.0089, mse:16.0089, \n","....................................................................................................\n","Epoch: 800, loss:15.9947, mse:15.9947, \n","....................................................................................................\n","Epoch: 900, loss:15.9936, mse:15.9936, \n","...................................................................................................."],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"OUB4yKlzxvZV","colab":{"base_uri":"https://localhost:8080/","height":198},"executionInfo":{"status":"ok","timestamp":1610562038096,"user_tz":-60,"elapsed":34987,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"e7b67c70-3068-4539-a471-8007c58864a8"},"source":["hist = pd.DataFrame(history.history)\n","hist['epoch'] = history.epoch\n","hist.tail()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","
\n"," \n","
\n","
\n","
loss
\n","
mse
\n","
epoch
\n","
\n"," \n"," \n","
\n","
995
\n","
15.992101
\n","
15.992101
\n","
995
\n","
\n","
\n","
996
\n","
15.979033
\n","
15.979033
\n","
996
\n","
\n","
\n","
997
\n","
15.987959
\n","
15.987959
\n","
997
\n","
\n","
\n","
998
\n","
15.970425
\n","
15.970425
\n","
998
\n","
\n","
\n","
999
\n","
15.969935
\n","
15.969935
\n","
999
\n","
\n"," \n","
\n","
"],"text/plain":[" loss mse epoch\n","995 15.992101 15.992101 995\n","996 15.979033 15.979033 996\n","997 15.987959 15.987959 997\n","998 15.970425 15.970425 998\n","999 15.969935 15.969935 999"]},"metadata":{"tags":[]},"execution_count":32}]},{"cell_type":"markdown","metadata":{"id":"THjB3LZU5Xhc"},"source":["You can noticed that while the number of epochs increases, the MSE is minimized. But *which is the best number of epochs to set*? A possible hint can be given by the plot of the **cost function vs. number of iterations**. Let's plot it. If you are interested in plotting details you can find the complete code inside the /module folder.\n","\n","The cost function vs. number of iterations plot is also useful to evaluate the model's convergence for different learning rates."]},{"cell_type":"code","metadata":{"id":"_hU-CvZt7d0G"},"source":["# The following line contains the path to fonts that are used to plot result in\n","# a uniform way.\n","\n","f = set_style().set_general_style_parameters()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"hVdybZ2AUxSD","colab":{"base_uri":"https://localhost:8080/","height":389},"executionInfo":{"status":"ok","timestamp":1610562171284,"user_tz":-60,"elapsed":1408,"user":{"displayName":"Michela Sperti","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gh7mD9r-1Xj0Qve63ZPZx9UHRv0PkVhL5ayiHNv=s64","userId":"13210266879998244642"}},"outputId":"d5466982-8d86-45cc-c681-f79dcfed8f94"},"source":["# Cost Function vs. Number of Iterations PLOT\n","\n","fig = plt.figure()\n","ax = fig.add_subplot(111)\n","\n","plt.plot(hist['epoch'], hist['mse'], color = 'blue')\n","\n","plt.ylabel('Cost Function (MSE)', fontproperties = fm.FontProperties(fname = f))\n","plt.xlabel('Number of Iterations', fontproperties = fm.FontProperties(fname = f))\n","\n","plt.ylim(0, 50)\n","plt.xlim(0, 1000)\n","\n","plt.axis(True)\n","plt.show()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAmQAAAF0CAYAAACNLyW6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAMTQAADE0B0s6tTgAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5wkVbn/8c/Tk8PORthdFmQBAQWUIEq6ElZFFEREBFQQFPGHVyQpZnRVDFeRIHjFKwoCKiAIGBAEyawJFQFBkLDLZtg0u5NDP78/TvVMM9s903Gqe+b7fr36VdNVp6qfqZruefqcU+eYuyMiIiIi8UnEHYCIiIjIZKeETERERCRmSshEREREYqaETERERCRmSshEREREYqaETERERCRmSshEREREYqaETERERCRmFZOQmdliM/Msj1VZ9tnfzG4zs3Vm1m1mj5rZWWZWM97xi4iIiBSqNu4ARmgHLs6wvmPkCjN7J3AT0ANcD6wD3gFcBBwAvKd8YYqIiIiUjlXK1ElmthjA3efnULYNeAaYChzg7g9H6xuBu4H9gPe6+3XlildERESkVCqmyTJPxwBbANelkjEAd+8BvhA9/WgcgYmIiIjkq9KaLBvM7ATgFUAn8Chwv7sPjii3IFrenuEY9wNdwP5m1uDuvWWLVkRERKQEKi0hmwNcM2Ld82b2QXe/L23dztHy6ZEHcPcBM3se2BXYHniyLJGKiIiIlEglJWRXAg8A/wI2EZKp04GPAL8zs/3c/Z9R2anRsj3LsVLrp2XaaGbnAOekntfU1MybPXt23gGvXWf09kBrq9PWlvfuIiIiUgFWrFjR5+4NccZQMQmZu395xKrHgdPMrAP4BLAQeFeJXutC4MLU83nz5vmTzz6R93Hef2wzv/l1HR86tZevfr2nFKGJiIjIOJvaNO2luGOohk79l0fLA9PWpWrAppJZav2GskQUqa0Ld6gO9JfzVURERGSiq4aELJW1tqSteypa7jSysJnVAtsBA8Bz5QysNqpfHBgo56uIiIjIRFcNCdm+0TI9ubo7Wh6WofyBQDOwqNx3WNbVhWW/EjIREREpQkUkZGb2ajNrybB+PnBZ9PTatE03AmuA481s77TyjcD50dPvlyXYNDVDNWRW7pcSERGRCaxSOvUfB3zCzO4HlhDustwBOBxoBG4DLkgVdveNZnYqITG718yuI0yddCRhSIwbCdMplVWd+pCJiIhICVRKQnYPIZHakzAPZQuhQ/6DhHHJrvERczy5+y1mdhDweeDdhMTtGcJwFt8dWb4caqMpzNWHTERERIpREQlZNOjrfWMW3Hy/h4C3lz6i3Az1IVMNmYiIiBShIvqQVSv1IRMREZFSUEJWhKE+ZGqyFBERkSIoISuCxiETERGRUlBCVoThPmRqshQREZHCKSErQqoP2aBqyERERKQISsiKUFcb+pDpLksREREphhKyIqgPmYiIiJSCErIiDM9lqT5kIiIiUjglZEVQHzIREREpBSVkRUiNQ6Y+ZCIiIlIMJWRFUB8yERERKQUlZEWo09RJIiIiUgJKyIpQEw17MaAmSxERESmCErIi1NeHZb+aLEVERKQISsiKkErI+nrVZCkiIiKFU0JWhLr60GTZ1xdzICIiIlLVlJAVoT4aGFYJmYiIiBRDCVkRGhrCcnDQGByMNxYRERGpXkrIipBqsgTVkomIiEjhlJAVIdWpH5SQiYiISOGUkBWhLi0h6+/TnZYiIiJSGCVkRaivU5OliIiIFE8JWRFSnfoBepWQiYiISIGUkBUhvVO/mixFRESkUErIiqBO/SIiIlIKSsiK8PKETDVkIiIiUhglZEWorR3+ua83vjhERESkuikhK4IZNDRE81n2xxyMiIiIVC0lZEVKNVuqyVJEREQKpYSsSKk7LfvVqV9EREQKpISsSPV1YdmrPmQiIiJSICVkRUo1WWocMhERESmUErIi1atTv4iIiBRJCVmRhjv1xxuHiIiIVC8lZEVKDXvR060mSxERESmMErIiNTeHZZcSMhERESmQErIiNTWHGrLurpgDERERkaqlhKxITU1h2d2lGjIREREpjBKyIjVHNWRd3TEHIiIiIlVLCVmRhpssVUMmIiIihanNp7CZvQ7YC9gD2AqYCrQCHUA7sAJ4BPi7u/+ttKFWppZUp34lZCIiIlKgMRMyM9sZ+BjwbmBOavUou3i03wrgRuD77v50kXFWrKEaMjVZioiISIFGTcjM7EfAiVG5HuB24B/Ao4TasI1AJ9ACtBFqzV5LqEE7BDgT+JiZXe3uHy7T7xCrplQNWadqyERERKQwY9WQfRC4H/gucLu75zK4ww0AZtYMvA34eHScCZmQNTdFnfo17IWIiIgUaKyE7L/cfVEhB46St5uAm8xsv0KOUQ2GmyxVQyYiIiKFGfUuy0KTsQzH+WMh+5nZCWbm0SNjDZuZHWFm95pZu5l1mNmfzeyk4iLOXbM69YuIiEiRxhz2wsw+YGZvzvWAZrbQzG4qLiwws22Aywh3cGYrczrwa2A34Frgh4R+bFeZ2QXFxpCLZnXqFxERkSLlMg7ZVcC5I1ea2RfN7IQM5Q8AjiomKDMz4EpgLXB5ljLzgQuAdcDe7v4xdz+bcFPBs8AnxqOpNNWpX+OQiYiISKFyHRg2U7axEPhA6UJ5mTOABYSbATqzlPkQ0ABc5u6LUyvdfT3w9ejpaWWKb0h6p373cr+aiIiITEQVN1K/mb0a+CZwibvfP0rRBdHy9gzbfjeiTNmkOvUPDhr9/eV+NREREZmIKiohM7Na4BrgBeBzYxTfOVpuNuisu68k1KxtHQ2/UTbNaUfvzFaXJyIiIjKKikrIgC8CewInu/tY3eSnRsv2LNvbR5QbYmbnmNmy1KOjI+t9A2Nqahpup1Q/MhERESlExSRkZrYPoVbsO4UOk5Erd7/Q3bdOPVpbWws+VnoNmcYiExERkUJUREIWNVVeTWh+PC/H3bLWgI1Yn60GrSSaW4ZryDRav4iIiBRizMnFI9uY2Rk5rt+mgDhagZ2in3vCqBeb+aGZ/ZDQ2f8s4ClgVrTfy2rUzGwuYX7NZTlO91Swxsbhn9VkKSIiIoXINSHbGbgox/UG5DsARC/woyzb9iL0K3uQkISlkq+7CWOeHcaIhIwwh2aqTFklEqEfWXe30aXBYUVERKQAuSRkXy53EFEH/mxTIy0kJGQ/cfcr0jZdCXwKON3MrkyNRWZm0xm+QzPjoLKl1tQcEjLVkImIiEghxkzI3L3sCVkh3P15MzsX+C7wsJldD/QBxwBbMw43B6Q0N4XpAjSfpYiIiBQi1ybLiuTul5rZYuCThFkDEsATwBfc/SfjFUdqcFh16hcREZFCFJ2Qmdl2hFqpZuBOd19UdFRp3H0hYZqmbNt/TZhgPDap+Sy7OlVDJiIiIvkbc9gLM5thZn8zsw9l2HYw8C/CVEdfAh4ws2+XPMoK1xoNfdGhhExEREQKkMs4ZKcAewB16SvNLAFcQZjg+1LgM8CLwDlm9qYSx1nR2qaGhGzTxpgDERERkaqUS5Pl4cAg8PMR6xcA2wM/j8YFw8weAB4CTgX+UMI4K1pbW0jINm5UDZmIiIjkL5cast2Af7v7yPqfwwnjjQ2NQxbd1bgE2K9kEVaBVA3ZxnYlZCIiIpK/XBKyNmBthvWHABvc/eER65cBWxQbWDVRDZmIiIgUI5eEbAMjEqxo8NXdgPszlK8D+osPrXoMJWSqIRMREZEC5JKQPQnsaGZbpa17P2GKpDszlN8BWFqC2KrGlLawVA2ZiIiIFCKXhOwmQq3XT8xsJzPbnzDERV+0bYiZ7Q3MJMw7OWlMVQ2ZiIiIFCGXuyy/D5wMvIlQWwahduxr7r56RNlzCB39ry1VgNVgqFP/JiVkIiIikr9c5rLsN7NDCLViC4Ae4Hrg4vRyZjYLeD1wu7tPqhqyVB+yTRvBHUx5mYiIiOQhp6mT3L2dUPs1Wpk1ZrYzYQqlSSVVQ5ZMGh0dMGVKzAGJiIhIVcmlD1nO3D3p7h2lPGY1SNWQgfqRiYiISP5KmpBNVqkaMtCdliIiIpK/MZsszey5Ao7r7r5DAftVpSlTwMxxN9WQiYiISN5y6UM2G2gi3D2Za7bhYxeZOBKJkJRt3KgaMhEREclfLk2WOwM3EJKxAeB7wFbA9FEeM8oRbCWbMiXkoO2qIRMREZE8jZmQufsydz8eOAj4F/Ax4I/Am9y9PdujzHFXnFQ/sk2bYg5EREREqk7Onfrd/QFgL+CjQAvwCzO7IxrqYtJL3WnZvkE1ZCIiIpKfvO6y9OAHwI7AZcAhwKNm9i0zaylHgNVi+oyQkK1fr4RMRERE8lPQsBdRs+SZwJ7AA8AngafN7H2lDK6azIgSsnXrNJKIiIiI5Keo7MHd/+Xubwb+F5gLXGNmby1JZFVmKCFbqxoyERERyU9OUydlY2avIMxp+U6gG/gmcE8J4qo602dGTZbrlJCJiIhIfgpKyMysAfh09GgCbgXOcvclJYytqgw3WSohExERkfzknZCZ2TuBC4HtgGeAM9z99lIHVm1mzEgCSshEREQkfzn3ITOzHc3sd8AvgTnAecBuSsaCGTOH+5D5pJqnQERERIo1ZkJmZi1m9j/AY8ChwM3Aq9z9a+7eV+4Aq0WqybK/3+joiDkYERERqSq5NFk+RbiDEsIclQuAf5qN2jTn7j6zyNiqysyZw9Via9fa0FRKIiIiImPJJSHbKu1nA6aVKZaqNiMtIVu3NsH8+YMxRiMiIiLVZMyEzN010mkOGhqgpcXp7DR17BcREZG8KNkqoVQ/svUaHFZERETyMGpCZmYlaZ4s1XEqXWpwWNWQiYiISD7GqiFbYmb/E43Inzcz2ya6Q3NSDBirschERESkEGP1IVsMnAt80sz+BtwG/AN4HFjh7t2pgmbWRLgBYDdgD+AIYC/CjQD/LHnkFUjzWYqIiEghRk3I3H13M3svcAbwBmBvwtAXAJjZIGEOyyagJm1Xi8otAi519xtKHHdFmqEmSxERESlALndZ/hz4uZltAxwO7AnsThibbCrQAnQC7cAK4FHg78Bv3H15meKuSJrPUkRERAqR81yW7r4UuLyMsVS9VEK2dq1uXhUREZHcKXMooVSTpYa9EBERkXwoISshNVmKiIhIIZSQlVCqhqyjw+jpiTkYERERqRpKyEpozpzk0M+rV6uWTERERHKjhKyEtpztmIVastUrdWpFREQkN8oaSqiuDmbOCgnZypWqIRMREZHcKCErsblzoxqyVTq1IiIikpuisgYzazKztkyPUgVYbWZH/chWrVINmYiIiOQm54FhU8xsR+CrwFuBbImXF3LsiWDOnFBDtkp9yERERCRHeSVNUTL2Z8KUSQZsIEyZVDQz+x/CXJk7AbMIc2QuAW4BLnP3tRn22R/4ArAvYT7N/wA/JsyfOViKuPI1Z65qyERERCQ/+dZifRWYBtwOfNTdl5QwlrMJc2DeCbxImCNzX2Ah8BEz2zeavgkAM3sncBPQA1wPrAPeAVwEHAC8p4Sx5SxVQ6Y+ZCIiIpKrfBOyBcBG4Dh331TiWNrcfbPhVM3sa8DngM8C/x2tawN+CAwCB7v7w9H684C7gWPM7Hh3v67EMY4p1YdMd1mKiIhIrvKtxmkDnihDMkamZCxyQ7TcMW3dMcAWwHWpZCztGF+Inn601DHmInWX5do1Cfr64ohAREREqk2+CdlSYMtyBDKKd0TLR9PWLYiWt2cofz/QBexvZg3lDCyT2Wmj9b+o0fpFREQkB/kmZLcA881sv3IEA2BmnzSzhWZ2kZk9QOi39ijwzbRiO0fLp0fu7+4DwPOE5tjts7zGOWa2LPXo6OgoWfyzoz5koDstRUREJDf59iH7GqHG6hdmdg7wuzI0X34SmJ32/HbgZHd/KW3d1GiZ7Q7P1PppmTa6+4XAhann8+bN80zlCtHQADNmJlm3NqE7LUVERCQn+Vbh/IXQj2wu8HNgg5ltMLN1Ix6bDVGRK3ef4+4GzAGOJtRy/cPM9ir0mONtaCwy3WkpIiIiOcg3Y3glIVGytEcboSYq/TG92MDcfbW73wwcCswErk7bnKoBm7rZji9fv6HYOAoxNBaZ7rQUERGRHOSVkLl7ItdHqQKMxjp7AtjVzGZFq5+KljuNLG9mtcB2wADwXKniyMdWUQvosqWqIRMREZGxVUvGsFW0TI2+f3e0PCxD2QOBZmCRu/eWO7BM5s8PNWRLFlfL6RUREZE4FTu5+LZm9gYze0WRx9nJzDZrfjSzRDQw7JaEBGt9tOlGYA1wvJntnVa+ETg/evr9YmIqRiohW/y8EjIREREZW0ETgJvZhwgj52+ftu4/wNfd/eqsO2b3duAbZvYgYciKtYQ7LQ+KXmMVcGqqsLtvNLNTCYnZvWZ2HWHqpCMJQ2LcSJhOKRbbzh8erb+nBxob44pEREREqkHeCZmZXQx8nNCh/zFgObANsCtwpZnt7u6fyPOwdxFuGPgvYE/CjQGdhHHGrgG+6+7r0ndw91vM7CDg88C7gUbgGeCcqHzJhrLI1/ztklGMxtIXEuy4U3KMPURERGQyyyshM7O3A2cQarGOS5+2yMz2IdRKnWVmd7p7plH0M3L3x4HT84kl2u8hQu1aRdliS6e52enqMpYsVkImIiIio8u3k9PpgAPvSU/GANz9z4Q5Jo1QgzZpmQ03Wz6vfmQiIiIyhnyzhb0Ik4v/PdPGKEl7AnhdsYFVu211p6WIiIjkKN9sYQrZpytK2QC0FhbOxKE7LUVERCRX+WYLS4BdzKwh08Zo2Ildo3KT2lAN2RIlZCIiIjK6fLOFmwnTEn0ly/avEaZSurmYoCaC1J2WS1RDJiIiImPId9iLi4CTgU+a2auAK4FlhGEvPkwYOX95VG5SS9WQtbcb69YaM2bGNgqHiIiIVLi8EjJ3XxMNffFL4B3AEWmbjTAO2NHuvrZ0IVanHV6ZJJFwkknjyScTHPBfg2PvJCIiIpNS3u1p7v5PYDfgFOCnwO8Jg7d+EHhtNKbYpNfYyND4Y48/WhNzNCIiIlLJCpo6yd27Cc2VV5Y2nIllt9cM8tS/a3j8cSVkIiIikp16nJfRrq9J1ZDpNIuIiEh2o9aQmdkXgGMJk4ZfZ2ZfzPG47u5fLTq6KrfbbqHf2JNP1DA4CDWqKBMREZEMbLQ5uM2snTAY7F/dfR8zSxKmTrIxjuvuXjXpx7x58/zJZ58o+XFXLDde/co2ABb9dRO77qY5LUVERCrN1KZpy9196zhjGKsP2ceB9wDfjZ6fXd5wJpat5jnbzk+yZHGCB+6rZdfd+uIOSURERCrQqAmZu18NXJ32/JKyRzTBHHjwANdcVc/999Zy2seUkImIiMjmSt7b3MwSZtZS6uNWq4MOGgDgwQdqGdRQZCIiIpJBXgmZmX3RzE4YZXsD8BTDTZyT3hujhKy93XjowarpViciIiLjKN8asoXAB7JtdPde4Hng8CJimlDmzHXeeGBIyn52TX3M0YiIiEglKscAWVsCyjzSvO/E0Hfs1pvr2Lgx5mBERESk4ow5Ur+ZHQgcnLZq+yzjkdUA+wGvAe4qSXQTxJFH9fOpc5xNm4yrflTPGWerc7+IiIgMG3UcMgAz+xLwJXIbfwxgJfDWaprTslzjkKX7wmcbufTiBlpbnQf/sontthv9vIuIiMj4qIRxyHJJyPYB9iUkYxcSOu1fnqHoIPACcI+7bypxnGU1HglZezsc8PopLF2aYJ99B7jtzk5qC5pJVEREREqpKhKylxUOI/Xf5e6Hli+k8TceCRnAgw/UcMRbW3A3PvuFHj7z+d6yv6aIiIiMrhISsnw79R8CfLIcgUwG//XGwaH+Y984v5Gbb1IVmYiIiOSZkLn7fUCtmX0003YzO93MXluSyCao8xb2cPCCfgBO+3AzD/9FY5OJiIhMdvkODLsdcDfwHTObOWLbLOAC4A9mtk3pQpxY6urgJz/t4lWvHqSnxzj+Pc0sWZLLvRIiIiIyUeXbZPlZYArwSXdfm77B3dcAZwEzo3KSxbRpcP0vO5m1RZKXXkxw3NEttLfHHZWIiIjEJd+E7K3AKnf/3yzbf0AY9uJtRUU1Ccyf71z3iy4aGpwnn6jh5BOa6e+POyoRERGJQ74J2ZbAM9k2erhl81lgdjFBTRav32eQy6/oBuDuu+o495xG8rjpVURERCaIfBOyl4Cx+odtDawpLJzJ5+hj+jnvyz0AXHlFAxde0BBzRCIiIjLe8k3IHgS2NbPDMm00s0OB+cADRcY1qXzi3F5OOCkMh/GVLzbykyvrYo5IRERExlO+CdklhCmUrjWzo83MACx4N/BTYAC4uLRhTmxmcMll3bzt8NCJ7KzTm/jdbzVGmYiIyGSR7zhkfwbOBaYDvwDWmNnjhCbKG4AZwCfc/a+lDnSiq62FH1/dxRv2GSCZND54YjP336cxykRERCaDfGvIcPeLgMOA+4GpwC5AG3Av8BZ3v7SUAU4mzc1w/U1d7LTzIN3dxvHvbuGpf+d9iURERKTKFPTf3t3vdPdDgEZgLtDo7m9y97tLGt0kNGOm86vbOnnFK5J0dhqHLmhh0YOqKRMREZnIiqp+cfcBd1/t7oOlCkhg7lbOtdd3Mndukg3rExxzVAt/WqSkTEREZKIyL2DgKzNLAK8BtgIy3hLo7r8qLrTxM2/ePH/y2SfiDmMzK5YbRxzWwrPP1NDa6vz8xk4OPEi5r4iISClNbZq23N23jjOGvBMyMzsO+A6hqTIrd6+aKp1KTcgAli8z3n5oC4ufr6G21rn6Z10c/o6BuMMSERGZMCohIctrbAUzezPwM8CA54BHAc3CWEbztnZu+30n7z22hX/+o4aT3t/MFVd1cdTRSspEREQmirxqyMzsbuAg4HxgoRfS3lmBKrmGLGX9ejjy7a08+kioeLz4si4+eIomvxQRESlWJdSQ5dupf2/C5OJfmijJWLWYPh1u/W0nbzwo1IyddXozV/1YI/qLiIhMBPkmZAlgcRnikBzMmOHceEsnb3rL8Ij+C89roK8v5sBERESkKPkmZE8B25cjEMlNYyP89PouDjx4AHfjogsaed+xzXR2xh2ZiIiIFCrfhOxKYLaZfagcwUhumprg+ps6+fhZvQDceUcdhx/awourLebIREREpBD5dup/BXANsD/wQ+B3wFIgObKsuz9aohjLrho69Wfz02vqOOO/mxgYMF6xbZJf3NzJq1692eUQERGRLCqhU3++CVkScMKwF6Pt6O6e85AaZjYTeBdwOGHA2XlAH/AYoVbuSnffLMsws/2BLwD7Ak3Af4AfA5fmM3tANSdkAHf/oZaT3tfMxo3G1KlhlH8NICsiIpKbakzI7mX0RGxINNdlrsc9Dfg+sBK4B3gBmA0cTZjA/CbgPel3dprZO6P1PcD1wDrgHcDOwI3u/p5cX7/aEzKAfz2e4Nh3tbBsWYK6Oucb3+7hwx/pw9SKKSIiMqqqS8jKFoTZAqAF+G16TZiZzQH+AmwDHOPuN0Xr24BnCMnaAe7+cLS+Ebgb2A94r7tfl8vrT4SEDGDlCuO4Y8IAsgDHHt/HxZd109ISc2AiIiIVrBISsqImFy8Vd7/b3X89slnS3VcBl0dPD07bdAywBXBdKhmLyvcQmjABPlq+iCvT3K2c2+/q4L3vD+Ng3HBdPW9d0MqSJaomExERqWQVkZCNITUcffpcQQui5e0Zyt8PdAH7m1lDOQOrRM3N8P0fdnPxZV3U1TmPPVrDwQe0cuWP6qiAylARERHJIN+5LL+YY1F3968WEM/I16sFPhA9TU++do6WT2d44QEzex7YlTBm2pPFxlFtzOCDp/Szyy5JTnxfM6tXJTjr9GZ+++t+/vcH3Ww5W5mZiIhIJSnmLst06QcxQkJWU3RwZhcAnwBuc/fD09Y/DewI7Ojuz2TY7yHC0Bz7u/sfM2w/Bzgn9bytrW3e0tUvFBtuRVq10jj/K41cc1U9AFOnOd/8djfvfX+/OvyLiIhQGX3I8k3IzsywOgG8AngLsAtwH3C5u19fVGBmZwCXAP8mdNxfl7atqIRspInSqX80v/11Lad/tIl1a0Mr9ZFH9fOdi1VbJiIiUgkJWV5Nlu5+SbZtZpYALgVOAy4oJigzO52QjD0BvCk9GYu0R8upWQ6RWr+hmDgmksPfMcD+/7WJc85o4pc31vOrW+r4w521fOXrPZxyqobHEBERiVPJOvVHd0ieDWwCzi30OGZ2FiGxexw4JLrTcqSnouVOGfavBbYj3ATwXKFxTETTp8OV13RzxZVdzNoiSWen8YkzmzjqiGaefqoa7u8QERGZmEr6X9jd+wid6PcqZH8z+zRwEfAIIRl7MUvRu6PlYRm2HQg0A4vcvbeQOCa69xzfzyP/2sTJp4TTc+/ddeyzVyvnf7mBdWtVVSYiIjLeylEtsiWbd/ofk5mdB3wT+BuhmXLNKMVvBNYAx5vZ3mnHaATOj55+P98YJpMpU+CSy3q46VedbL/DIMmk8e1vNrL7LlO46RcaIkNERGQ8lXSkfjM7FrgO+JO775/HficBVwGDhObK9gzFFrv7VWn7HEVIzHqi11wHHEk0dRJwrOf4y02GTv2j6emB717UwPcurWfD+pCj77f/AF//Vg97vU5zYoqIyMRWCZ36873L8owsm9oIzZRHEmrHTnD3n+dx3IXAl8Yodp+7HzxivwOAzxOmSmokTKf0Y+C7k2ly8VJZsdw46+NN3PG7uqF1Rx7Vz+e/2MOrXr3Z3O4iIiITQjUmZKlxyDbbFC0HgK+7+8LiQxs/Sshe7k+Lavjcpxv528PhJlwz561vG+DL5ysxExGRiacaE7KFZE7I+oEVwF3uvqw0oY0fJWSbcw9jl53/5UaefCKM8WvmHPfefj5yWh+ve72aMkVEZGKouoRsolJClt3gIPz+9lo++6lGnn9uePKF/Q8Y4PQze3nb4QMkNGKGiIhUsUpIyPSvVEZVUwNvO3yAvz3aweVXdPG6vcMc74sequV9x7aw+y5T+M63G1i9SsNliIiIFGrMGjIzGyQ0Rb51xPotgCZ3r/pJIFVDljt3uP++Gr5/WQO331aLe0jEEgnnTW8Z4AKtWHAAACAASURBVP0n9vH2IwZoaIg5UBERkRxVQg1ZLglZkpCQHTpi/Z2EwVvzmn6pEikhK8zixcYVP2jgih/U0909XEM2fUaSNx44yDuP6mfBWwaYMUPN4iIiUrkqISErtslS7VST2Pz5zvnf6OGFVRu54ZedHHlUP3V1zvp1CX51Sx2nnNzMjttO4cT3NnPnHbW0ZxpdTkRERIquIVvg7jWZ96weqiErnXVrjd/8upZrf1LPn/+0eeXpXq8bYMGbBzjq6H522TVJTdX/9YiISLWrhBoyJWQoISuXtWuMG66r447ba7nnD3WbbZ85K8mCNw2w+x6DHLxggN1ek8RU5yoiIuNMCVmFUEJWfqtWGo8/XsNdd9Tyu9tqWfz85n82LS3OvvuHxGzf/QZ47e6DzNrCaWyMIWAREZk0lJBVCCVk42/FcuP239Vx9121/P3hGpYvz9ydsaHB2WPPQXbcOclerxtkx50G2WGHJHO3co1/JiIiJVFNCdm9wDt5eSf+W4EDgWlk6Nzv7htLFmWZKSGL37PPJPjH32t48IEannm6hr/+pYaenuztl01NzvztkszdKskrtk2y/fZJZs9xdt9jkC23dGbMdDV/iohITqopIct33AKvpuEwlJBVnr4+WLo0wZ8W1bBieYLHH0vw+GM1LH4+wcDA2JlWfb3T1ATztg5JW3MzbL9DmIdzzpwkO7wyyZQpTnOL0zaVofXNzeX8rUREpBJVQkKWS9L0AvknZCJFqa+HHXZIssMOL5/MfGAAli41nn2mhkf+UcPyZcaypQleetFYvDjB+nWhHbOvz+jrg/b2Gp74V+6t6lOnOclB6O+HV+6YZM7cJA31MJiE6dOd1lZn+gzHHWbPdhoandraMKNBY6MzdZrT2BCer19vuMNOrxqkpRlqa6G2zuntMdqmOvX1bFaLl/p+pNo9EZHJRXNZohqyiSKZhPZ2eHF1gpUrjc4O47lnE6xckWDdOmPDBqNjk7F+vbFkcYLOTkgm4818ttgySWM0q4E7bNgQkrjZc5LU1obfqbEx1PglEuF5QwMkEtDVBW1ToWMTzJgZtk+bFpYDA6HswIAxdZpTWxsSQYC6eqer06ithYZGp6YGujqN3t6QCDY2MTSYbzK9ftyGf57S5iSTMGVKWK5ckWDuVkl6e42BgfC7NDZBc5PT0wPTpjvJpFFb6/T1hmS5vh4SUeK65ZbJod/PPSyTyXAu3MPrdHSE5/190NwSbvbo6AjHbG0J68xCMr1mTYK6Omf2bCdRAwP94ZyE82LU1zs93UZTc/j9UzHX1kJTs9PcBF3dsGJ5gtZWp21qOH+1ddA2xdmwwahvcNavC+c3HMOoq3PqaqGuDja0G8lBsATU1jqDg0ZNTShbkwhJe01N2F5TE64JhGuwYYOx5Wynri51nkM5T0J7u9HfBytXGvX1sPOrkrhDT0+4NokE9PQYM2clGRgwBgfCMRM1YUaNRAISFsolEtDVbby4ymidEr5opPpmTp3qJB26u8LfRkMDdHeH7bNmOf0DUFsD/dE5ra11GuphxYoEM2YmmdIaru2MGU5vb7guSR++ri3NTle30VDvtLeH36Wu3qmvY+jnujpo32Akk9DXH74Upc5TY1P4G1jzktHSEv5m2qaGv5OWlnCeN20K57ivN/U+CDGZOc3N8NKLxhZbOmtesqG/qZUrE9TXOVPaQvn+/nCM1tbw+3d1GjNnhb+1wcHwt9PZEf4eamujv+sEbNgALS3hbyG8F8P6mprRv3Cl3gOp6+AeWgwaGob36+mJvuDVhu3d3eHnRAJ6e8NnRk3N8Be81N9VusHBUD61PvVeS/0NjYxpZAz5SCaH90udt9RrjHU891AmFdual4xZWwx3SRkYCL/7WFLHyaZaashEqkIiAdOnw/TpSXZ+1djl3cOH18Z2Y3AQVq0yVq1MUBeN0PHUv0Mit2mj0dcfyrmHf/b9fdDZZXR3hQ/ont7wodC+wUbt+zbSSy9mvjPhuWer/l4ZkYrT3Ox0dRX2JSyR8KEvcImERwlM5mO1tDidnSHxg83L1dT40BeuVGIyMACDg8PlGhtDIpvat67O6e8PSf/gYPj8amkJX8xScZk57uF129pCEpX6nJs+w9m0MXwR6+sLr93SEpKj1Jex1JeSRCKsHxyEpibojJLghgandYrT3WX09IQWhb7ecEwsfFkaGGAouW+oh64uY+NGG/pSmfqd6uvDuWlqCuuaW0LrQndPSCb7+4yOjnBOEolUAmxD56ahEfDwBWX6jCT19eG89feHcoMDwwkzhJ+nTR9OpJODw18eZ82qjIop1ZChGjIprWQyfCh0bDKmtDmLn08MfTvr6AgfKO6wfp3RHiV5EGo/evtCLdfatUZXZ6i16O01ujqHv8329YUm2c5Oo7MTWltDbU9jk7N2bWLow76n25gyJdQ+DA6GGquuzpBAJmrCh+lAf/gwrKuD1asTtLeHD6f164zWKdDWFoLr7h7+sO7vH/7Gu2xpgrY2Z8vZzvPPJYZqoWbOdBoaoKPTqK9zVq9KYAnAiZprnf6BUIOUSITjW1qtTernVI3XSy8ZjQ3hg7R/IMSVTA7XOPV0G13dw9+iZ892+vpg9apw7lO1LbVR7UR3D6xfl6ClxYdqNiDEHmoLw3WaOzfJYDIk4xDt2x1eN/UPsa8vxFVfH2Lt79/8n3RdXWiiTv2TGxwcvXa2psZf9s9ZRMrNVEMmMtEkEqFqv6EhZAc77ZwcYw+Jw2hNGKmkuqHh5eUBOjsZuvkj29ArqRqMwcFQu9DW5ps1q6SaZgcHQxNUXR1s2hiS+IaG4SZIs9Akljpm29RQQ5JqCtuw3mhoCE14qdqAmpqwT01NaKpkqBl484cDLc1RjUmrDzVdrVtnNNSHJtz6+hBPc3NIWtesCds6u4zmppDs9vfDM88k2G67MMDzpk3G9OmhebexKTTlphJu91AjPW26MzBgzJqVZHDAoi8bIalN/VxTE5Ldmppw7lunhJqg3h6juzt8gVizxmhqGj5/3d2hZqa2JjR1TpsWyr/0UmgKbmx0OjaFZsbuLmP6jHDuNmwwtprn9PaE69KxKXwpMgtJeXNL1NT5klGTCNegpjas64tqh3q6jdWrja23cQajrgO1teE6uMNg9KWlP2rCTEYfD7W1w03ZvX2hqdgS0NoSmq0HBqG+LuxXkwjlN24MzbONTcNN+g1RTdGGDUYiEa5LYxMsfcGYt7XTH9WOTWkL56SuPjQ19/XD2jXhC1ZtXfh7TVh4jSlt4fp1dIYvh80t4UvGpo1GfUP4nUItFySjWimzsK6pKZz/7p5Qc7VsqbHVVj60va4O6hvC9ejrC03fZgw92qYOd9dYsjjB+nXGtvOTQy0ZrVPCF8hUM2jqb6Wmlpd9ObUEdHeFY6b6/aauyZqXjI98KNdPjvJRDRmqIRMREZnMKqEPmYbWFBEREYmZEjIRERGRmJUlITPTKEoiIiIiucorITOzD5jZm0fZ3mBmDwAXFB2ZiIiIyCSRbw3ZVcCnsm10915gADi2iJhEREREJpVyNFk2AG1lOK6IiIjIhDTmOGRm9lpgj7RVc83sAxmK1gD7A/sAfyxNeCIiIiITXy4Dw74L+BLDM9rtAlyZpawBXcBniw9NREREZHLIJSH7E3BJ9POZwFLg5gzlBoEXgFvc/YXShCciIiIy8Y2ZkLn7HcAdAGZ2JvC0u59d7sBEREREJot857L8ILCiHIGIiIiITFb53mV5PaFZ8mXM7INmdouZ/cTM9siwn4iIiIhkkW9CdirwmJkdl1phZicDVwBHAicC95nZ9iWLUERERGSCyzchOypaPpC27uNAD/AO4IvAFOCc4kMTERERmRzyTch2Bxa7+woAM2sjjFH2M3f/rbufT2jSzDq9koiIiIi8XL4JWSuwMu356whjjy1KW7cY2Lq4sEREREQmj3wTsuXAVmnP9yUMGPvPtHUtQLLIuEREREQmjXwTsr8A25vZKWa2I/BRoJMoITOzOsJI/s+VNEoRERGRCSzfhOwbhBH5/w/4N6Fp8kp3H4y2vxtoAu4uWYQiIiIiE1xeCZm7P0q4m3IR8ARwMfCptCLbE+ayvLxUAYqIiIhMdObuY5fK54Bmbe6+saQHLbN58+b5k88+EXcYIiIiEoOpTdOWu3usNyTmO3XSEDNrBl4PzAZWAX919+5qS8ZERERE4pZ3QmZmCWAhcAZhENiUjWZ2EfBVL3W1m4iIiMgEVkgN2Q3Au4A+4DeEoTC2IQwG+yVgV+C4rHuLiIiIyMvklZCZ2fuBo4G/A+909+Vp214B3AocY2bvdfeflzRSERERkQmqkMnFB4Fj05MxAHd/AXgPYaDYU0sTnoiIiMjEl29CthvwmLtnHPjV3Z8BHgVeW2xgIiIiIpNFvglZPdAzRpneqFxezOwYM7vUzB4ws41m5mZ27Rj77G9mt5nZOjPrNrNHzewsM6vJ9/VFRERE4pJvp/5ngddmG2vMzKYSaseeLiCWLwC7Ax3AMuBVoxU2s3cCNxESxOuBdYRBay8CDiA0n4qIiIhUvHxryG4AmoHvRcNfDImefx9oBK4rIJazgZ2ANsIcmVmZWRvwQ0J/toPd/RR3PxfYA/gj4caC4wuIQURERGTc5ZuQXQI8BbwP+GvUPHiMmZ1NuPPyeOBJ4Lv5BuLu97j7f3Icw+wYYAvgOnd/OO0YPYSaNhgjqRMRERGpFHk1Wbp7l5m9hVBTti+hRirFgIeA49y9u3QhZrQgWt6eYdv9hPk09zezBnfvLXMsIiIiIkXJe2BYd19GSHYOJiRGWwCrgXvc/b7ShpfVztFys75q7j5gZs8TBqjdnlBjJyIiIlKxCp7L0t3vBe4tWST5mRot27NsT62flmmjmZ0DnJN63tbWVrrIRERERPKUUx8yM9vGzA7LodzbzGxe8WGVl7tf6O5bpx6tra1xhyQiIiKT2JgJWTSm12+AX5rZ9qOU2wm4GbjFzKx0IWaUqgGbmmV7av2GMschIiIiUrRcashOAF4D3JpthH4Ad38a+DWwF+Fuy3J6KlruNHKDmdUC2wEDQNZ4RURERCpFLglZan7Kz+ZQ9jOEuy2PKyaoHNwdLTM1ox5IGCttke6wFBERkWqQS0K2F/C0uy8eq6C7P0u483HvIuMay43AGuB4Mxt6LTNrBM6Pnn6/zDGIiIiIlEQud1nOIL+pkFYD2+YbiJkdBRwVPZ0TLfczs6uin9e4+ycB3H2jmZ1KSMzuNbPrCFMnHUkYEuNGwnRKIiIiIhUvl4RsDbBlHsfcgpAc5WsP4KQR67aPHgBLgE+mNrj7LWZ2EPB54N2EKZueIQxn8d0cR/wXERERiZ2NlbeY2f3APsA27v7iGGVnA0uBP7v7G0sWZZnNmzfPn3z2ibjDEBERkRhMbZq23N23jjOGXPqQ3QTUAWfkUPYMoIbQZCgiIiIiOcglIbuG0AT5KTM7IlshMzsc+BSwFri6NOGJiIiITHxjJmTuvg44hdDf7GYzu8TMdrFhrzazi4FbCENefMjd15c3bBEREZGJI6epk9z9VsJ4ZD3Ax4HHop97gMcJTZVdwDHu/pvyhCoiIiIyMeU8ubi732Rmi4D/Bg4njIYP8DzwK+Byd19V+hBFREREJracEzIAd18JnBc9RERERKQEcmqyFBEREZHyUUImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxU0ImIiIiEjMlZCIiIiIxq/qEzMy2NrMfm9kKM+s1s8VmdrGZTY87NhEREZFc1MYdQDHMbAdgEbAlcCvwb+ANwJnAYWZ2gLuvjTFEERERkTFVew3Z/xKSsTPc/Sh3/4y7LwAuAnYGvhZrdCIiIiI5qNqELKodOxRYDHxvxOYvAZ3AiWbWMs6hiYiIiOSlahMy4JBo+Xt3T6ZvcPdNwENAM7DveAcmIiIiko9qTsh2jpZPZ9n+n2i50zjEIiIiIlKwau7UPzVatmfZnlo/beQGMzsHOCdtVXJq07SVJYxNSqcV6Ig7CMlI16Yy6bpULl2byjUn7gCqOSErmLtfCFyYem5my9x96xhDkix0bSqXrk1l0nWpXLo2lcvMlsUdQzU3WaZqwKZm2Z5av2EcYhEREREpWDUnZE9Fy2x9xHaMltn6mImIiIhUhGpOyO6Jloea2ct+DzObAhwAdAF/yuFYF45dRGKia1O5dG0qk65L5dK1qVyxXxtz97hjKJiZ3UEYi+wMd780bf2FwNnAD9z9tLjiExEREclFtSdkI6dOehLYhzBG2dPA/po6SURERCpdVSdkAGa2DfAV4DBgJrASuBn4sruvjzM2ERERkVxUfUImIiIiUu2quVN/UcxsazP7sZmtMLNeM1tsZheb2fS4Y5sIzGymmX3YzG42s2fMrNvM2s3sQTM7ZeSNGGn77W9mt5nZumifR83sLDOrGeW1jjCze6Pjd5jZn83spPL9dhOPmZ1gZh49PpylTN7n2cxOMrO/ROXbo/2PKM9vMbGY2Zui98+q6DNqhZndYWZvz1BW75txYGaHm9nvzWxZdJ6fM7NfmNl+WcrrupSQmR1jZpea2QNmtjH6vLp2jH3G5RqU5LPO3SfdA9gBWA04cAvwTeDu6Pm/gZlxx1jtD+C06HyuAH4KfAP4MWFcOAduJKqhTdvnncAAYSTrHwHfjq6HA7/I8jqnR9vXECaZvwhYGq27IO7zUA0PYJvoumyKztuHS3GegQui7Uuj8t8D1kbrTo/7967kB/CttHP3f8DXgR8Cfwe+NaKs3jfjc03+J+2cXRH937gR6AOSwAm6LmW/Bo9E52IToc+4A9eOUn5crkGpPutiP8ExXdQ7ohP18RHrL4zWXx53jNX+ABYA7wASI9bPAV6IzvO709a3AS8CvcDeaesbCTduOHD8iGPNB3qiP/z5aeunA89E++wX97mo5AdgwF3As9GH1WYJWSHnGdg/Wv8MMH3EsdZGx5tfrt+rmh/AqdG5uwqoz7C9Lu1nvW/G55rMAQaBVcCWI7YdEp2z53Rdyn4dDiGMMWrAwYySkI3XNSjlZ13sJziGC7pDdPKeZ/NkYQohk+4EWuKOdaI+gM9F1+DStHUfitb9JEP5BdG2+0as/0q0/ssZ9sl6PD1edp7OJHy7PxBYSOaELO/zDFwdrf9ghn2yHm+yP4CG6J/IEjIkY7me/2ib3jeluy77ROfl1izbNwKbdF3G9ZoczOgJ2bhcg1J+1k3GPmSHRMvfu3syfYO7bwIeApqBfcc7sEmkP1oOpK1bEC1vz1D+fsIgv/ubWUOO+/xuRBkZwcxeTWh2ucTd7x+laCHnWdemMG8BtgB+CSSjPkufNrMzs/RT0vtmfPyH0DT5BjOblb7BzA4kfJm/K221rkv8xusalOy6TcaEbOdomW1Kpf9Ey2xTMkkRzKwW+ED0NP0POOt1cfcBQo1mLbB9jvusJNR0bm1mzUWGPeFE1+EaQvPx58Yontd5NrMWYB7QEW0fSe+x7F4fLXuAfwC/ISTNFwOLzOw+M9sirbzeN+PA3dcBnwZmA0+Y2f+Z2TfM7Abg98CdwP9L20XXJX5lvwal/qybjAlZatLx9izbU+unjUMsk9E3gd2A29z9jrT1hVyXXPfJNgH9ZPZFYE/gZHfvHqNsvudZ77HCbRktzyU0dbyRUPvyWsI//gOBX6SV1/tmnLj7xcDRhH/ipwKfAd5D6Mh9lbu/mFZc1yV+43ENSvpZNxkTMomJmZ0BfIJwl8uJMYczaZnZPoRase+4+x/jjkdeJvWZPAAc6e4PunuHuz8GvAtYBhyUbZgFKR8z+xThrsqrCH2RW4DXAc8BPzWzb8UXnUwEkzEhG+tbRmr9hnGIZdIws9OBS4AngEOiJoB0hVyXXPfJ9u1l0omaKq8mVMmfl+Nu+Z5nvccKlzon/3D3xekb3L2LcIc4wBuipd4348DMDiYMe/Erdz/H3Z9z9y53/zshUV4OfMLMUs1fui7xG49rUNLPusmYkD0VLbO16e4YLbP1MZM8mdlZwKXA44RkbFWGYlmvS5REbEeoNXgux33mEr7BLov+kUnQSjhfrwZ60gaDdeBLUZkfRusujp7ndZ7dvZPwD6o12j6S3mPZpc51tg/w1HRwTSPK631TXqkBPu8ZuSE6T38h/D/dM1qt6xK/sl+DUn/WTcaELPWGOtRGjBZvZlOAAwh3X/xpvAObiMzs04SB8h4hJGMvZil6d7Q8LMO2Awl3vi5y994c93nbiDIS9BIGSMz0+EdU5sHoeao5s5DzrGtTmD8Q+o7tMvLzKbJbtHw+Wup9Mz5Sd+JtkWV7an1ftNR1id94XYPSXbe4xxKJ44EGhh2v83xedD4fBmaMUbYNeIn8BvHbDg2kWMrrtZDM45DlfZ7RwLDFXIdbo3N39oj1hxLGjFsPTI3W6X0zPtfk2Oi8rALmjdj2tui6dBPN8qLrMi7X5GDGHhi27NeglJ91sZ/UmC7kyKmTvsHw1ElPoamTSnGOT4rO5wChhmxhhsfJI/Y5iuFpLq4gTB8zNM0FI6Zaivb5OJpqpFTXbCHZp07K+zwD32Hz6UTWoKmTxroOWzM8m8VdhBkUbozeG/2kzXARldf7pvzXJEEY2sIJg8D+hKhPGSEZc+BMXZeyX4ejCDdVXEUYNskJs4yk1l2QoXzZr0GpPutiP8ExXthtgCuBlYRq5iWEsX6mxx3bRHgw/M99tMe9GfY7ALiNUAvQDTwGnA3UjPJa7wDuI8xv1gn8FTgp7nNQbQ9GScgKPc/AyVG5zmi/+4Aj4v5dK/1BaAK7NPpc6os+3G8G3pClvN435b8mdcBZhO4sG6N/9C8Sxoo7VNdlXK7BWP9XFsd1DUrxWWfRgUREREQkJpOxU7+IiIhIRVFCJiIiIhIzJWQiIiIiMVNCJiIiIhIzJWQiIiIiMVNCJiIiIhIzJWQiIiIiMVNCJiIiIhIzJWQikjMzczPbEHccpWRme5rZ7Wa2xszWmtkNZjYl7rjGm5m9xsxWRufC4o5HZLJRQiZSZma22Mw2mdmuY5R7JEp45o9PZGJmcwnzRb6VMBHwf4A9CHPfZdsnY1JqZp8xs4VlCrVoZjbHzBaa2clZirwVmBMtZ45bYCICKCETGS+twK1mNiPuQORl3g7MAK5z953dfV9gNy9sTrnPAF8qaXSlNYcQ38lZtt9AmH/vQndfM15BiUhQG3cAIpPIDsD1ZnaYuw/GHYwAsHW0/Etqhbv3xRRLrNz9BeDguOMQmaxUQyYyftYCbwa+nc9OY/XbMrMNZuYj1h0c7XeFme1hZr+M+kh1RU2j/y8qZ2b2QTNbZGbtZtYZ/Xz0GDE1mdkXzezfZtZjZsvN7Hoze8Mo+xxvZveZ2brotf5kZh/O1F8piv0ZM9vCzL4XNfveMPbZAjM7wczuiX7fDjN71Mw+a2YtI88PsDBadWH0mp5vs6OZ3Rsda2pa7D7ymkTbXm1mV5rZC2bWbWbPmtlFUdPpyLJXRcd5o5l9xMz+GjV9p16n0cxOM7M/RL9rv5mtMrPrzGyXEcdy4B/R04PSYrw3rcweI9elbas1s49F12xDdP3+Eq3b7It96pyY2XwzO93M/hb9bW2Krs1bspzLI83sLjNbYWZ9ZrYs+ts9OtPficiE4u566KFHGR/AYsCBQ4Hu6OcTM5R7JNo2f8R6BzaMcvwN4a38snUHR/s9AvREr/s34OlovQMXAT+Nfm4H/gwsSdv+sQyv5YT+VYuin5cBD0ZLB/qB00bsY8BV0faeKKa/AV3RuluBmgyvsyKKNwk8B3x1jPNcC9wc7ZsE/gn8EeiM1j0BbB2V3Z3Qd+y5aNtT0fO7gA+M8Tovux7Ad6L9+qNtqePcNWK/I9JieRZ4CFgZPV9JaCpNL586Z/dFy/XRPq3R9l9H6zuBv0bbXkq7Rq9JO9ZdwJ/SjpOK8TtpZfaItt87Io626NgO9AEPR6/XF617CJgyYp97o233R8vV0bVI/b5J4L0j9vlKtG0wunYPpf1dPQ/Ux/1e1kOPcj5iD0APPSb6g+GEbBrwgejnbuD1I8qVIyFz4A/AFmnbjoz+6aW2/whoStt+JsNJWkuGWJyQTL0PsGi9ASdGv1d/+u8GfCza5xZgdtr6mcDPo22fyfI6q4GDcjzP32Q4Cd0pbf1U4Npo2wMj9lkYrT8rj+uZ8Xpkug5p2+ZF21cDB6atN+CkKLn5F2mJKcMJmQPnA7UjjnlQtG9D2ro64IJon5tGlM+YcI21HbiO4URzq7T1c4DfR9uuHbHPvQwnV6emrU8AX2U44a6N1jcz/MXhtSOOtTdwSJzvYT30GI9H7AHoocdEf5CWkEXPL2S4dmlOWrlyJGQ9RLVCI7bfxCg1D8Dfo+2HZIjFgbOzxHJutP1n0XOLfv+VWV6nhlA7tWJEMpJ6nZNzPMdTCLVC3cA2GbbXEmpdnLQEj/FLyFK1P0dm2X5etP3taeuuGi2BGiW+JmAAeGnE+rwTMmB7Qm3WSkbUgqWd95VR4rVd2vp7o2P9IMM+CUINoQN7Rutmpb0navL5ffXQY6I81KlfZPydC7yG0J/sl2Z2sJevI/m/3H1ZhvX/BI4G7sny2o8AewJbZTnuVVnW/wj4FvDG6Pn2wLb8//buQEgv0gAABVtJREFUL8SqKorj+HelZiGVmmEFRUFppNZYNNIfo2kSDAMfsqLEQiwwSl+CXirE6qWHrESyoocUi8JKUBgfTEoHjRjBFKMoSNO0FLHU/miMrh7WPs71zrl37vXe8aj8PjAcuHufs/fZc+CuOXvvNbAf6KiwDOgi4IpU96eyspUV2inXCgwBVrn7rvJCd+82syXE9OK9xDTg6dSWjs+Z2dyc8mHpeCfQUVZWcQzM7B5gGnANMJgIgCGCm2akrmhL11zu7ofLC939sJktB+YQfwRsL6tSfi+4+3Ez20r8vq8ENrv7fjP7lggKN5rZB8TU6FZ3P9qE+xA54ykgEznN3P2YmT1CrMO5HXgbeLKBS1bbnLO7wueH0jEvWCstH5RT9q+7/5F3krsfMLMjwGXpo+w4Amiv0s+sbmlA9re7H+jjnNJzofL9QM9YjKzxms2U9e/uGuuV2plX0cxeJKb/+lOj49orOE4OpmPp8zUVWEzkQcs2h/xnZuuA+e6+oe/uipy9FJCJFCAFLlOJhc6zzGxLlerdwBAzG+Bl6TLMbCjxhqmS4310pa/yPBea2fC8YMnMRgAX0PNFvDcdN7v7LXW2011H3aydq6rUycp+r7MfzbAXGA3c7O5b6zy31ziY2aVETrEjwExgtbsfLCn/k7Trs0GNjmvNz5dH2o0pZnYx8Xa2BZgETCF2hra4+/e1Xk/kbKO0FyIFcfdt9CzyX0BMO+XZQ/zxNC6nbHK/dK5vMyt8nr3p60zHHelnvJlNyDuhSekMuog1ZJPM7OqcNgYSYw2xyaE/HEttDcgp+yodZ+edeApjMJp4Jja6+8dlwdhQIijO7R+xbq9W2Q7Ph7J0G6VS8DSNCLy+rOO6Fbn7IXdf5+5vufsDxDTz+cD9zbi+yJlKAZlIgdx9BbHgeyCV32isTseF6Q0UACnAeb1/e1jRq2Y2PQskLMwgFsl3A29CWuEOr6VzPjOz+0ovYmbtwCYzu7yRzrj7X8BCYh3VKjMbVdLGJcASIqBd7+6d+VdpWDa1eEdq9zwzy6bk3gUOAE9b/PuiEwFTuvdVafxqlU0Tjjez60uuNZxIZTK4Sv/GlOYyq9aIu/9M7IQdCawwsxNrClO/P09ly9z9lzr6f5I0Vu+Z2cSyzwcB2b8cO9T7TJFzhwIykeLNJ/JnVfIKsZNtIrAjJeT8kZ4cW7/1fxdP8iuRZHQZsMvMOokpyqVEYDnX3buyyu7+DrCISP2wxsx2mVmnmW0nUimMAcY2oV/ziADhJuAHiwS4XxNvGB8j8pA92oR2KvkoHTvMrIPYPToLwN33AA8TOzHnAfvS73ETMXZTiDQWNUnBz3JiM8B3KWnsN0SgNpa45/JzDhKL7IcB21Ifd5rZDX00N5vICdZGPH9dZtZF5KxrJ/LQPVNr3ytoB54C1qcktxvMbAMxDTqZyEf3SYNtiJzRFJCJFCy9RXoc2FahfDdwG/A+sRi6hUhtsIjYFHC63xwcJb5AXybykbUS02CfAhPdfXH5Ce4+h/hiXZnqthILupcCLe7+RaOdcvdud3+QyIe2nljbNI5IsfAC0JoCo/7yBpG+Yh+x4/AfShbDu/taIlhcQEoEC1xHBDvT3b3ejR0zgOeJjRBjgGuJdCZ3Ab12RCZPEDtkB6Z61dYuZv0+nO7nWSIQH5V+thC7K9vSG8pT5u5riF3HHxLP863AjcT4vQRMyNvlKXIuyZI6ioiIiEhB9IZMREREpGAKyEREREQKpoBMREREpGAKyEREREQKpoBMREREpGAKyEREREQKpoBMREREpGAKyEREREQKpoBMREREpGAKyEREREQK9j+l68ULuiaCewAAAABJRU5ErkJggg==\n","text/plain":["