{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Solving classification problems with CatBoost" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/catboost/tutorials/blob/master/events/pydata_nyc_oct_19_2018.ipynb)\n", "\n", "In this tutorial we will use dataset Amazon Employee Access Challenge from [Kaggle](https://www.kaggle.com) competition for our experiments. Data can be downloaded [here](https://www.kaggle.com/c/amazon-employee-access-challenge/data)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Libraries installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "#!pip install --user --upgrade catboost\n", "#!pip install --user --upgrade ipywidgets\n", "#!pip install shap\n", "#!pip install sklearn\n", "#!pip install --upgrade numpy\n", "#!pip install --upgrade pandas\n", "#!jupyter nbextension enable --py widgetsnbextension" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.10.2\n", "Python 3.5.5 :: Anaconda custom (64-bit)\r\n" ] } ], "source": [ "import catboost\n", "print(catboost.__version__)\n", "!python --version" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The most simple way — read everything in pandas data frame" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import os\n", "import numpy as np\n", "np.set_printoptions(precision=4)\n", "import catboost\n", "from catboost import *\n", "from catboost import datasets" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# import ssl\n", "# ssl._create_default_https_context = ssl._create_unverified_context" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "(train_df, test_df) = catboost.datasets.amazon()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ACTIONRESOURCEMGR_IDROLE_ROLLUP_1ROLE_ROLLUP_2ROLE_DEPTNAMEROLE_TITLEROLE_FAMILY_DESCROLE_FAMILYROLE_CODE
013935385475117961118300123472117905117906290919117908
11171831540117961118343123125118536118536308574118539
21367241445711821911822011788411787926795219721117880
31361355396117961118343119993118321240983290919118322
4142680590511792911793011956911932312393219793119325
\n", "
" ], "text/plain": [ " ACTION RESOURCE MGR_ID ROLE_ROLLUP_1 ROLE_ROLLUP_2 ROLE_DEPTNAME \\\n", "0 1 39353 85475 117961 118300 123472 \n", "1 1 17183 1540 117961 118343 123125 \n", "2 1 36724 14457 118219 118220 117884 \n", "3 1 36135 5396 117961 118343 119993 \n", "4 1 42680 5905 117929 117930 119569 \n", "\n", " ROLE_TITLE ROLE_FAMILY_DESC ROLE_FAMILY ROLE_CODE \n", "0 117905 117906 290919 117908 \n", "1 118536 118536 308574 118539 \n", "2 117879 267952 19721 117880 \n", "3 118321 240983 290919 118322 \n", "4 119323 123932 19793 119325 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preparing your data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Label values extraction" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "y = train_df.ACTION\n", "X = train_df.drop('ACTION', axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Categorical features declaration" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0, 1, 2, 3, 4, 5, 6, 7, 8]\n" ] } ], "source": [ "cat_features = list(range(0, X.shape[1]))\n", "print(cat_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking on label balance in dataset" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Labels: {0, 1}\n", "Zero count = 1897, One count = 30872\n" ] } ], "source": [ "print('Labels: ', set(y))\n", "print('Zero count = ' + str(len(y) - sum(y)) + ', One count = ' + str(sum(y)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To train model in CatBoost we need to create wrapper class for data: Pool.\n", "This class stores the data in CatBoost internal format.\n", "\n", "There exists several ways to create pool. \n", "The most simple one: create it from pandas dataframe or numpy array" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "pool1 = Pool(data=X, \n", " label=y,\n", " cat_features=cat_features) #Indicies of categorical columns in X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This approach is not the most efficient, especially for big dataset: we'll need to copy everything from pandas to our internal format.\n", "\n", "So CatBoost could create Pool direclty from file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets look how we could load Pool from file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Firstly, lets save data frame to disk" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "dataset_dir = './amazon'\n", "if not os.path.exists(dataset_dir):\n", " os.makedirs(dataset_dir)\n", "\n", "# We will be able to work with files with/without header and with different separators.\n", "train_df.to_csv(os.path.join(dataset_dir, 'train.tsv'), index=False, sep='\\t', header=False)\n", "test_df.to_csv(os.path.join(dataset_dir, 'test.tsv'), index=False, sep='\\t', header=False)\n", "\n", "train_df.to_csv(os.path.join(dataset_dir, 'train.csv'), index=False, sep=',', header=True)\n", "test_df.to_csv(os.path.join(dataset_dir, 'test.csv'), index=False, sep=',', header=True)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\t39353\t85475\t117961\t118300\t123472\t117905\t117906\t290919\t117908\r\n", "1\t17183\t1540\t117961\t118343\t123125\t118536\t118536\t308574\t118539\r\n" ] } ], "source": [ "!head -n2 amazon/train.tsv" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ACTION,RESOURCE,MGR_ID,ROLE_ROLLUP_1,ROLE_ROLLUP_2,ROLE_DEPTNAME,ROLE_TITLE,ROLE_FAMILY_DESC,ROLE_FAMILY,ROLE_CODE\r\n", "1,39353,85475,117961,118300,123472,117905,117906,290919,117908\r\n", "1,17183,1540,117961,118343,123125,118536,118536,308574,118539\r\n" ] } ], "source": [ "!head -n3 amazon/train.csv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have dataset in 2 different formats:\n", "\n", "1) tab-separated without header\n", "\n", "2) comma-separated with header\n", "\n", "\n", "CatBoost, like pandas, could load data from different formats, we just need to pass proper options\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we load data, we need to set types of each column\n", "\n", "Also, we need to specify columns type. For this CatBoost uses special file, column description\n", "And we have helper-function to easily do this" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from catboost.utils import create_cd\n", "\n", "feature_names = dict()\n", "for column, name in enumerate(train_df):\n", " if column == 0:\n", " continue\n", " feature_names[column - 1] = name\n", " \n", "create_cd(\n", " label=0, \n", " cat_features=list(range(1, train_df.columns.shape[0])),\n", " feature_names=feature_names,\n", " output_path=os.path.join(dataset_dir, 'train.cd')\n", ")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\tLabel\t\r\n", "1\tCateg\tRESOURCE\r\n", "2\tCateg\tMGR_ID\r\n", "3\tCateg\tROLE_ROLLUP_1\r\n", "4\tCateg\tROLE_ROLLUP_2\r\n", "5\tCateg\tROLE_DEPTNAME\r\n", "6\tCateg\tROLE_TITLE\r\n", "7\tCateg\tROLE_FAMILY_DESC\r\n", "8\tCateg\tROLE_FAMILY\r\n", "9\tCateg\tROLE_CODE\r\n" ] } ], "source": [ "!cat amazon/train.cd" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "create_cd(\n", " label=0, \n", " cat_features=list(range(1, train_df.columns.shape[0])),\n", " # feature_names=feature_names,\n", " output_path=os.path.join(dataset_dir, 'train_without_names.cd')\n", ")" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\tLabel\t\r\n", "1\tCateg\t\r\n", "2\tCateg\t\r\n", "3\tCateg\t\r\n", "4\tCateg\t\r\n", "5\tCateg\t\r\n", "6\tCateg\t\r\n", "7\tCateg\t\r\n", "8\tCateg\t\r\n", "9\tCateg\t\r\n" ] } ], "source": [ "!cat amazon/train_without_names.cd" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "create_cd(\n", " label=0, \n", " cat_features=list(range(2, train_df.columns.shape[0])),\n", " feature_names=feature_names,\n", " output_path=os.path.join(dataset_dir, 'train_with_num.cd')\n", ")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\tLabel\t\r\n", "1\tNum\tRESOURCE\r\n", "2\tCateg\tMGR_ID\r\n", "3\tCateg\tROLE_ROLLUP_1\r\n", "4\tCateg\tROLE_ROLLUP_2\r\n", "5\tCateg\tROLE_DEPTNAME\r\n", "6\tCateg\tROLE_TITLE\r\n", "7\tCateg\tROLE_FAMILY_DESC\r\n", "8\tCateg\tROLE_FAMILY\r\n", "9\tCateg\tROLE_CODE\r\n" ] } ], "source": [ "!cat amazon/train_with_num.cd" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "? create_cd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's load pool from file now:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "pool2 = Pool(\n", " data=os.path.join(dataset_dir, 'train.tsv'), \n", " #delimiter=',', \n", " column_description=os.path.join(dataset_dir, 'train.cd'),\n", " # has_header=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Loading pool from file is the fastest way to build Pool if you don't have Pool in RAM yet" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Exercices: load the same pools from csv file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, if you want maximum performance and you data is already in RAM, than in some cases we could do better, than simply passing dataframe to Pool constructor\n", "\n", "We have class FeaturesData that is a fast way to pass data from numpy matrices to catboost" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "? FeaturesData" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "# Fastest way to create a Pool is to create it from numpy matrix. This way should be used if you want fast predictions\n", "# or fastest way to load the data in python.\n", "\n", "X_prepared = X.values.astype(str).astype(object)\n", "# For FeaturesData class categorial features must have type str\n", "\n", "pool3 = Pool(\n", " data=FeaturesData(cat_feature_data=X_prepared, cat_feature_names=list(X)),\n", " label=y.values\n", ")\n" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset shape\n", "dataset 1:(32769, 9)\n", "dataset 2:(32769, 9)\n", "dataset 3:(32769, 9)\n", "\n", "\n", "Column names\n", "dataset 1:\n", "['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n", "\n", "dataset 2:\n", "['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n", "\n", "dataset 3:\n", "['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n" ] } ], "source": [ "print('Dataset shape')\n", "print('dataset 1:' + str(pool1.shape) + '\\ndataset 2:' + str(pool2.shape) + \n", " '\\ndataset 3:' + str(pool3.shape))\n", "\n", "print('\\n')\n", "print('Column names')\n", "print('dataset 1:')\n", "print(pool1.get_feature_names()) \n", "print('\\ndataset 2:')\n", "print(pool2.get_feature_names())\n", "print('\\ndataset 3:')\n", "print(pool3.get_feature_names())\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Split your data into train and validation" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/noxoomo/anaconda3/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2069: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.\n", " FutureWarning)\n" ] } ], "source": [ "from sklearn.model_selection import train_test_split\n", "X_train, X_validation, y_train, y_validation = train_test_split(X, y, train_size=0.8, random_state=1234)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
RESOURCEMGR_IDROLE_ROLLUP_1ROLE_ROLLUP_2ROLE_DEPTNAMEROLE_TITLEROLE_FAMILY_DESCROLE_FAMILYROLE_CODE
1492674463105908117961118225129617118702132654118704118705
794017278120340120342120343119076118834311236118424118836
247687932517733117961118300119984118890125128118398118892
163351733075117961117962120677120357120678118424120359
2743156723745117961118300118360124435118362118363124436
\n", "
" ], "text/plain": [ " RESOURCE MGR_ID ROLE_ROLLUP_1 ROLE_ROLLUP_2 ROLE_DEPTNAME \\\n", "14926 74463 105908 117961 118225 129617 \n", "7940 17278 120340 120342 120343 119076 \n", "24768 79325 17733 117961 118300 119984 \n", "1633 5173 3075 117961 117962 120677 \n", "2743 15672 3745 117961 118300 118360 \n", "\n", " ROLE_TITLE ROLE_FAMILY_DESC ROLE_FAMILY ROLE_CODE \n", "14926 118702 132654 118704 118705 \n", "7940 118834 311236 118424 118836 \n", "24768 118890 125128 118398 118892 \n", "1633 120357 120678 118424 120359 \n", "2743 124435 118362 118363 124436 " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.head()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(26215, 9)" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.shape" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(6554, 9)" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_validation.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Selecting the objective function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Possible options for binary classification:\n", "\n", "`Logloss`\n", "\n", "`CrossEntropy` for probabilities in target" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "\n", "model = CatBoostClassifier(\n", " iterations=5,\n", " learning_rate=0.1,\n", " #loss_function='Logloss',\n", " #loss_function='CrossEntropy'\n", ")\n", "\n", "train_pool = Pool(data=X_train, \n", " label=y_train, \n", " cat_features=cat_features)\n", "\n", "validation_pool = Pool(data=X_validation, \n", " label=y_validation, \n", " cat_features=cat_features)\n", "model.fit(\n", " train_pool,\n", " eval_set=validation_pool,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model is fitted: True\n", "Model params:\n", "{'iterations': 5, 'learning_rate': 0.1, 'loss_function': 'Logloss'}\n" ] } ], "source": [ "print('Model is fitted: ' + str(model.is_fitted()))\n", "print('Model params:')\n", "print(model.get_params())" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " iterations=5,\n", " learning_rate=0.1,\n", " #loss_function='Logloss',\n", " #loss_function='CrossEntropy'\n", ")\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model is fitted: True\n", "Model params:\n", "{'iterations': 5, 'learning_rate': 0.1, 'loss_function': 'Logloss'}\n" ] } ], "source": [ "print('Model is fitted: ' + str(model.is_fitted()))\n", "print('Model params:')\n", "print(model.get_params())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stdout of the training" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.2922585\ttest: 0.2894772\tbest: 0.2894772 (0)\ttotal: 50.6ms\tremaining: 708ms\n", "1:\tlearn: 0.2200332\ttest: 0.2193253\tbest: 0.2193253 (1)\ttotal: 103ms\tremaining: 667ms\n", "2:\tlearn: 0.1882770\ttest: 0.1794720\tbest: 0.1794720 (2)\ttotal: 169ms\tremaining: 676ms\n", "3:\tlearn: 0.1787690\ttest: 0.1655785\tbest: 0.1655785 (3)\ttotal: 237ms\tremaining: 652ms\n", "4:\tlearn: 0.1748384\ttest: 0.1586419\tbest: 0.1586419 (4)\ttotal: 309ms\tremaining: 619ms\n", "5:\tlearn: 0.1737776\ttest: 0.1564081\tbest: 0.1564081 (5)\ttotal: 420ms\tremaining: 630ms\n", "6:\tlearn: 0.1723479\ttest: 0.1542367\tbest: 0.1542367 (6)\ttotal: 508ms\tremaining: 581ms\n", "7:\tlearn: 0.1719611\ttest: 0.1540092\tbest: 0.1540092 (7)\ttotal: 593ms\tremaining: 518ms\n", "8:\tlearn: 0.1718945\ttest: 0.1536590\tbest: 0.1536590 (8)\ttotal: 676ms\tremaining: 450ms\n", "9:\tlearn: 0.1697911\ttest: 0.1524950\tbest: 0.1524950 (9)\ttotal: 738ms\tremaining: 369ms\n", "10:\tlearn: 0.1684481\ttest: 0.1509084\tbest: 0.1509084 (10)\ttotal: 832ms\tremaining: 303ms\n", "11:\tlearn: 0.1673863\ttest: 0.1495139\tbest: 0.1495139 (11)\ttotal: 920ms\tremaining: 230ms\n", "12:\tlearn: 0.1642212\ttest: 0.1473859\tbest: 0.1473859 (12)\ttotal: 1s\tremaining: 154ms\n", "13:\tlearn: 0.1631211\ttest: 0.1468169\tbest: 0.1468169 (13)\ttotal: 1.08s\tremaining: 77.5ms\n", "14:\tlearn: 0.1628934\ttest: 0.1465499\tbest: 0.1465499 (14)\ttotal: 1.15s\tremaining: 0us\n", "\n", "bestTest = 0.1465498889\n", "bestIteration = 14\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=15,\n", " #verbose=5,\n", " logging_level='Verbose'\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", ")" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "{ROLE_FAMILY} pr1 tb0 type0, border=1 score 71.20196708\n", " tensor 0 is redundant, remove it and stop\n", "0:\tlearn: 0.3020322\ttest: 0.3024549\tbest: 0.3024549 (0)\ttotal: 46.3ms\tremaining: 648ms\n", "\n", "{ROLE_FAMILY} pr0 tb0 type0, border=6 score 22.73090753\n", "{ROLE_FAMILY, ROLE_CODE} pr0 tb0 type1, border=1 score 22.85737971\n", "{ROLE_TITLE} pr0 tb0 type0, border=13 score 23.18484659\n", "{ROLE_DEPTNAME, ROLE_FAMILY, ROLE_CODE} pr2 tb0 type0, border=10 score 23.4271605\n", "{ROLE_ROLLUP_2} pr1 tb0 type0, border=13 score 25.42952513\n", "{ROLE_TITLE, ROLE_FAMILY} pr2 tb0 type0, border=7 score 25.91828995\n", "1:\tlearn: 0.2126748\ttest: 0.2068150\tbest: 0.2068150 (1)\ttotal: 137ms\tremaining: 890ms\n", "\n", "{MGR_ID} pr2 tb0 type0, border=11 score 9.55130831\n", "{MGR_ID, ROLE_CODE} pr2 tb0 type0, border=14 score 12.59291657\n", "{ROLE_FAMILY} pr0 tb0 type1, border=9 score 14.15678078\n", "{ROLE_TITLE, ROLE_FAMILY} pr1 tb0 type0, border=6 score 14.34218082\n", "{ROLE_ROLLUP_1, ROLE_FAMILY} pr2 tb0 type0, border=3 score 14.2932558\n", "{ROLE_FAMILY, ROLE_CODE} pr0 tb0 type1, border=3 score 14.4073355\n", "2:\tlearn: 0.1856771\ttest: 0.1759821\tbest: 0.1759821 (2)\ttotal: 282ms\tremaining: 1.13s\n", "\n", "{MGR_ID} pr2 tb0 type0, border=10 score 6.594784046\n", "{MGR_ID, ROLE_CODE} pr2 tb0 type0, border=12 score 8.339746264\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=11 score 9.115213253\n", "{ROLE_ROLLUP_2} pr0 tb0 type0, border=12 score 10.5940908\n", "{MGR_ID, ROLE_FAMILY_DESC} pr0 tb0 type0, border=5 score 10.45322949\n", "{ROLE_TITLE} pr2 tb0 type0, border=6 score 10.66637876\n", "3:\tlearn: 0.1756533\ttest: 0.1621181\tbest: 0.1621181 (3)\ttotal: 397ms\tremaining: 1.09s\n", "\n", "{MGR_ID} pr2 tb0 type0, border=10 score 4.816932629\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 5.826899508\n", "{ROLE_ROLLUP_2} pr2 tb0 type0, border=13 score 6.455962742\n", "{ROLE_FAMILY} pr2 tb0 type0, border=10 score 6.668192917\n", "{ROLE_TITLE} pr2 tb0 type0, border=8 score 6.787022309\n", "{ROLE_FAMILY, ROLE_CODE} pr2 tb0 type0, border=1 score 6.784150409\n", " tensor 5 is redundant, remove it and stop\n", "4:\tlearn: 0.1732514\ttest: 0.1587816\tbest: 0.1587816 (4)\ttotal: 526ms\tremaining: 1.05s\n", "\n", "{MGR_ID} pr2 tb0 type0, border=7 score 2.297190459\n", "{RESOURCE} pr2 tb0 type0, border=11 score 3.692999825\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=9 score 4.566200375\n", "{RESOURCE} pr0 tb0 type0, border=13 score 5.195022352\n", "{MGR_ID, ROLE_FAMILY_DESC} pr0 tb0 type0, border=10 score 5.882398347\n", "{MGR_ID, ROLE_FAMILY_DESC} pr1 tb0 type0, border=6 score 5.715399768\n", "5:\tlearn: 0.1693831\ttest: 0.1534864\tbest: 0.1534864 (5)\ttotal: 633ms\tremaining: 950ms\n", "\n", "{MGR_ID} pr2 tb0 type0, border=9 score 2.56631672\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 3.053372671\n", "{ROLE_FAMILY} pr0 tb0 type1, border=14 score 3.662099526\n", " tensor 2 is redundant, remove it and stop\n", "6:\tlearn: 0.1692429\ttest: 0.1530965\tbest: 0.1530965 (6)\ttotal: 688ms\tremaining: 786ms\n", "\n", "{MGR_ID} pr2 tb0 type0, border=9 score 2.662106324\n", "{MGR_ID, ROLE_CODE} pr0 tb0 type0, border=14 score 3.271764978\n", " tensor 1 is redundant, remove it and stop\n", "7:\tlearn: 0.1692302\ttest: 0.1530455\tbest: 0.1530455 (7)\ttotal: 740ms\tremaining: 647ms\n", "\n", "{RESOURCE} pr2 tb0 type0, border=9 score 2.704934036\n", "{ROLE_ROLLUP_2} pr1 tb0 type0, border=9 score 3.380733801\n", "{ROLE_FAMILY_DESC} pr0 tb0 type1, border=8 score 3.485919025\n", "{ROLE_ROLLUP_2, ROLE_DEPTNAME} pr2 tb0 type0, border=8 score 4.040211274\n", "{ROLE_DEPTNAME} pr2 tb0 type0, border=4 score 4.662311897\n", "{ROLE_ROLLUP_1} pr2 tb0 type0, border=10 score 4.587959116\n", "8:\tlearn: 0.1678218\ttest: 0.1518261\tbest: 0.1518261 (8)\ttotal: 843ms\tremaining: 562ms\n", "\n", "{RESOURCE} pr2 tb0 type0, border=6 score 3.22558481\n", "{MGR_ID} pr0 tb0 type0, border=6 score 3.598670764\n", "{MGR_ID} pr2 tb0 type0, border=6 score 4.442316954\n", "{ROLE_ROLLUP_1} pr1 tb0 type0, border=14 score 4.534285954\n", " tensor 3 is redundant, remove it and stop\n", "9:\tlearn: 0.1675718\ttest: 0.1514379\tbest: 0.1514379 (9)\ttotal: 900ms\tremaining: 450ms\n", "\n", "{RESOURCE} pr1 tb0 type0, border=8 score 1.706869104\n", "{RESOURCE} pr2 tb0 type0, border=12 score 2.541399767\n", "{MGR_ID} pr0 tb0 type0, border=12 score 3.042148696\n", "{MGR_ID, ROLE_DEPTNAME} pr1 tb0 type0, border=6 score 3.056465669\n", "{ROLE_DEPTNAME} pr0 tb0 type1, border=14 score 3.417063506\n", " tensor 4 is redundant, remove it and stop\n", "10:\tlearn: 0.1662480\ttest: 0.1499766\tbest: 0.1499766 (10)\ttotal: 974ms\tremaining: 354ms\n", "\n", "{ROLE_FAMILY_DESC} pr1 tb0 type0, border=13 score 1.188282501\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=6 score 2.265758008\n", "{RESOURCE} pr2 tb0 type0, border=12 score 2.620514967\n", "{ROLE_CODE} pr0 tb0 type0, border=14 score 3.027549244\n", " tensor 3 is redundant, remove it and stop\n", "11:\tlearn: 0.1660153\ttest: 0.1497512\tbest: 0.1497512 (11)\ttotal: 1.04s\tremaining: 260ms\n", "\n", "{RESOURCE} pr2 tb0 type0, border=3 score 1.942379095\n", "{ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 2.328950537\n", "{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr2 tb0 type0, border=13 score 2.976624744\n", "{RESOURCE, ROLE_ROLLUP_1} pr2 tb0 type0, border=11 score 3.247835595\n", "{RESOURCE, ROLE_ROLLUP_1} pr2 tb0 type0, border=4 score 3.98226103\n", "{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr0 tb0 type0, border=14 score 3.884491014\n", " tensor 5 is redundant, remove it and stop\n", "12:\tlearn: 0.1655821\ttest: 0.1492094\tbest: 0.1492094 (12)\ttotal: 1.14s\tremaining: 175ms\n", "\n", "{RESOURCE} pr2 tb0 type0, border=6 score 1.401892445\n", "{ROLE_DEPTNAME} pr2 tb0 type0, border=5 score 2.17295077\n", "{RESOURCE, ROLE_ROLLUP_1} pr1 tb0 type0, border=6 score 2.344410053\n", "{RESOURCE, ROLE_ROLLUP_1, ROLE_ROLLUP_2} pr0 tb0 type0, border=6 score 2.784440779\n", "{ROLE_CODE} pr2 tb0 type0, border=8 score 3.478854515\n", "{ROLE_DEPTNAME, ROLE_CODE} pr2 tb0 type0, border=7 score 3.183561803\n", "13:\tlearn: 0.1637591\ttest: 0.1478251\tbest: 0.1478251 (13)\ttotal: 1.27s\tremaining: 91.1ms\n", "\n", "{MGR_ID} pr2 tb0 type0, border=5 score 1.678525175\n", "{RESOURCE} pr2 tb0 type0, border=2 score 1.913548971\n", "{ROLE_ROLLUP_2} pr1 tb0 type0, border=4 score 2.09899603\n", "{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr0 tb0 type0, border=14 score 2.529101972\n", " tensor 3 is redundant, remove it and stop\n", "14:\tlearn: 0.1637261\ttest: 0.1477449\tbest: 0.1477449 (14)\ttotal: 1.34s\tremaining: 0us\n", "\n", "bestTest = 0.1477448839\n", "bestIteration = 14\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=15,\n", " #verbose=5,\n", " logging_level='Info'\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metrics calculation and graph plotting" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b78812f3160e458dbd4fdd81998054e0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=500,\n", " random_seed=63,\n", " learning_rate=0.5\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Eval metric, custom metrics and best trees count" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a99d9adb234d42a9a82ff7812ebd0718", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " eval_metric=\"Accuracy\",\n", " use_best_model=False\n", ")\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "50" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.tree_count_" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3237fc1e455c42079880250710ad16a5", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " custom_loss=['AUC', 'Accuracy']\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "28" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model._tree_count" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metric hints" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "987ef484e520494a820090a2bb88ac50", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " eval_metric='AUC:hints=skip_train~false' #default\n", ")\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "328bd48940a24bdc807e5bc149b56072", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " eval_metric='AUC:hints=skip_train~false', #default\n", " metric_period=10\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model comparison" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model1 = CatBoostClassifier(\n", " learning_rate=0.5,\n", " iterations=100,\n", " train_dir='learing_rate_0.5'\n", ")\n", "\n", "model2 = CatBoostClassifier(\n", " learning_rate=0.01,\n", " iterations=100,\n", " train_dir='learing_rate_0.01'\n", ")\n", "\n", "model1.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=False\n", ")\n", "model2.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d4ec633ced854e4f99dd4731264681ec", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from catboost import MetricVisualizer\n", "MetricVisualizer(['learing_rate_0.01', 'learing_rate_0.5']).start()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overfitting detector" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "80e70b13a7564741b84cd3c7328d04e0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_with_early_stop = CatBoostClassifier(\n", " iterations=200,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " early_stopping_rounds=20\n", ")\n", "model_with_early_stop.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "28\n" ] } ], "source": [ "print(model_with_early_stop.tree_count_)" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "32fc59c9d0204b41b75e2bb13327d8be", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_with_early_stop = CatBoostClassifier(\n", " eval_metric='AUC',\n", " iterations=200,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " early_stopping_rounds=20\n", ")\n", "model_with_early_stop.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cross-validation" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "447c85da2d564f4890220e78ff7eb230", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from catboost import cv\n", "\n", "params = {}\n", "params['loss_function'] = 'Logloss'\n", "params['iterations'] = 80\n", "params['custom_loss'] = 'AUC'\n", "params['random_seed'] = 63\n", "params['learning_rate'] = 0.5\n", "\n", "cv_data = cv(\n", " params = params,\n", " pool = Pool(X, label=y, cat_features=cat_features),\n", " fold_count=5,\n", " type = 'Classical',\n", " shuffle=True,\n", " partition_random_seed=0,\n", " plot=True,\n", " stratified=False,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
test-AUC-meantest-AUC-stdtest-Logloss-meantest-Logloss-stdtrain-AUC-meantrain-AUC-stdtrain-Logloss-meantrain-Logloss-std
00.5000000.0000000.3022170.0038710.4999960.0000090.3021890.002418
10.6252870.0984860.2255020.0143720.6189810.0883960.2277760.009594
20.7973810.0318020.1820740.0101770.7585750.0369030.1908740.004790
30.8185420.0190810.1686300.0068340.7784740.0125470.1806570.004351
40.8238750.0132420.1635910.0071060.7846640.0076620.1767910.003464
50.8420550.0068450.1594240.0062780.8014860.0082650.1735450.003944
60.8525050.0115920.1558350.0072500.8120290.0048140.1708150.003318
70.8592900.0130550.1526860.0073740.8215730.0041500.1681680.002686
80.8624000.0129300.1514120.0077500.8255370.0021660.1668540.002284
90.8643130.0121270.1503880.0073120.8276990.0043550.1659800.002763
100.8646210.0122510.1499810.0072680.8282380.0049670.1657100.002916
110.8699110.0121210.1481500.0080580.8323910.0040980.1642500.002763
120.8711780.0114280.1474790.0081750.8342700.0054160.1633280.003049
130.8716630.0112810.1473030.0082300.8348860.0055300.1630570.003074
140.8722520.0103880.1472000.0082470.8355800.0060420.1627110.003282
150.8725060.0105180.1470030.0084780.8361770.0065050.1623500.003406
160.8737820.0095030.1463930.0082120.8375870.0066700.1617290.003425
170.8757300.0084380.1457390.0079610.8404220.0083380.1606300.003884
180.8766550.0092060.1454210.0080270.8425640.0054160.1599100.003345
190.8769610.0095290.1451510.0083680.8437620.0042230.1594560.002908
200.8771310.0096570.1450950.0083610.8438620.0041370.1593370.002843
210.8777970.0096270.1446910.0083320.8457420.0049770.1585030.002997
220.8784560.0098850.1444740.0083870.8470260.0043510.1579640.002731
230.8780980.0104340.1444970.0084840.8480290.0050310.1575160.002853
240.8785180.0104050.1443710.0086340.8485540.0051700.1572130.002953
250.8791140.0107060.1441430.0088710.8497410.0050650.1566560.002807
260.8790010.0109420.1440830.0089580.8501910.0049550.1562960.002907
270.8793220.0110280.1439800.0087610.8509110.0054070.1558610.002955
280.8796940.0113260.1436920.0091070.8514710.0049560.1555990.002789
290.8797810.0110450.1436150.0091350.8516300.0050690.1554550.002823
...........................
500.8825120.0125000.1426040.0104090.8602300.0053780.1501630.003652
510.8824530.0123850.1426210.0104000.8603100.0053310.1500980.003616
520.8823850.0124150.1426370.0104030.8604540.0054180.1499820.003705
530.8824430.0120710.1425240.0103840.8610730.0056050.1496850.003798
540.8822970.0121770.1425170.0104840.8612480.0054200.1495610.003734
550.8821990.0122240.1425560.0105130.8613190.0054840.1494840.003754
560.8821940.0122050.1425740.0104950.8613690.0054920.1494240.003744
570.8822770.0121280.1424930.0105130.8616460.0054240.1492690.003714
580.8823210.0122030.1424890.0105290.8617080.0054150.1492290.003720
590.8821940.0121650.1425550.0105000.8617390.0053970.1491710.003693
600.8821520.0121890.1425790.0104940.8617420.0053930.1491480.003703
610.8824420.0122750.1425140.0104800.8620270.0051360.1489430.003607
620.8825310.0121480.1424920.0104390.8621210.0051560.1488570.003626
630.8823840.0123050.1424500.0104020.8622490.0052530.1487370.003715
640.8823270.0123620.1426040.0103500.8624040.0053270.1486280.003732
650.8823940.0124760.1425540.0104030.8626640.0053930.1484640.003785
660.8827180.0120370.1424750.0102940.8629640.0057430.1482850.003972
670.8824410.0125220.1425240.0102630.8633910.0061730.1480760.004102
680.8824100.0125630.1425350.0102840.8634870.0062850.1479850.004199
690.8826880.0128170.1424290.0103690.8639470.0061930.1477170.004175
700.8828250.0127290.1423430.0103210.8640800.0062830.1475740.004268
710.8829480.0128390.1422890.0104180.8642430.0063870.1474330.004301
720.8830910.0128950.1421790.0104930.8647030.0060830.1471740.004146
730.8830800.0129290.1422130.0105550.8647840.0060460.1471110.004117
740.8829210.0129680.1422230.0105040.8650530.0060980.1469300.004142
750.8829270.0129930.1421920.0105950.8651550.0060750.1468420.004101
760.8829420.0129870.1422210.0105740.8653040.0059870.1466950.004043
770.8833050.0129680.1421000.0107140.8655120.0060000.1465370.003998
780.8835230.0131460.1421310.0106810.8656170.0059380.1464480.003941
790.8836610.0130470.1420580.0106380.8657270.0059610.1463410.003937
\n", "

80 rows × 8 columns

\n", "
" ], "text/plain": [ " test-AUC-mean test-AUC-std test-Logloss-mean test-Logloss-std \\\n", "0 0.500000 0.000000 0.302217 0.003871 \n", "1 0.625287 0.098486 0.225502 0.014372 \n", "2 0.797381 0.031802 0.182074 0.010177 \n", "3 0.818542 0.019081 0.168630 0.006834 \n", "4 0.823875 0.013242 0.163591 0.007106 \n", "5 0.842055 0.006845 0.159424 0.006278 \n", "6 0.852505 0.011592 0.155835 0.007250 \n", "7 0.859290 0.013055 0.152686 0.007374 \n", "8 0.862400 0.012930 0.151412 0.007750 \n", "9 0.864313 0.012127 0.150388 0.007312 \n", "10 0.864621 0.012251 0.149981 0.007268 \n", "11 0.869911 0.012121 0.148150 0.008058 \n", "12 0.871178 0.011428 0.147479 0.008175 \n", "13 0.871663 0.011281 0.147303 0.008230 \n", "14 0.872252 0.010388 0.147200 0.008247 \n", "15 0.872506 0.010518 0.147003 0.008478 \n", "16 0.873782 0.009503 0.146393 0.008212 \n", "17 0.875730 0.008438 0.145739 0.007961 \n", "18 0.876655 0.009206 0.145421 0.008027 \n", "19 0.876961 0.009529 0.145151 0.008368 \n", "20 0.877131 0.009657 0.145095 0.008361 \n", "21 0.877797 0.009627 0.144691 0.008332 \n", "22 0.878456 0.009885 0.144474 0.008387 \n", "23 0.878098 0.010434 0.144497 0.008484 \n", "24 0.878518 0.010405 0.144371 0.008634 \n", "25 0.879114 0.010706 0.144143 0.008871 \n", "26 0.879001 0.010942 0.144083 0.008958 \n", "27 0.879322 0.011028 0.143980 0.008761 \n", "28 0.879694 0.011326 0.143692 0.009107 \n", "29 0.879781 0.011045 0.143615 0.009135 \n", ".. ... ... ... ... \n", "50 0.882512 0.012500 0.142604 0.010409 \n", "51 0.882453 0.012385 0.142621 0.010400 \n", "52 0.882385 0.012415 0.142637 0.010403 \n", "53 0.882443 0.012071 0.142524 0.010384 \n", "54 0.882297 0.012177 0.142517 0.010484 \n", "55 0.882199 0.012224 0.142556 0.010513 \n", "56 0.882194 0.012205 0.142574 0.010495 \n", "57 0.882277 0.012128 0.142493 0.010513 \n", "58 0.882321 0.012203 0.142489 0.010529 \n", "59 0.882194 0.012165 0.142555 0.010500 \n", "60 0.882152 0.012189 0.142579 0.010494 \n", "61 0.882442 0.012275 0.142514 0.010480 \n", "62 0.882531 0.012148 0.142492 0.010439 \n", "63 0.882384 0.012305 0.142450 0.010402 \n", "64 0.882327 0.012362 0.142604 0.010350 \n", "65 0.882394 0.012476 0.142554 0.010403 \n", "66 0.882718 0.012037 0.142475 0.010294 \n", "67 0.882441 0.012522 0.142524 0.010263 \n", "68 0.882410 0.012563 0.142535 0.010284 \n", "69 0.882688 0.012817 0.142429 0.010369 \n", "70 0.882825 0.012729 0.142343 0.010321 \n", "71 0.882948 0.012839 0.142289 0.010418 \n", "72 0.883091 0.012895 0.142179 0.010493 \n", "73 0.883080 0.012929 0.142213 0.010555 \n", "74 0.882921 0.012968 0.142223 0.010504 \n", "75 0.882927 0.012993 0.142192 0.010595 \n", "76 0.882942 0.012987 0.142221 0.010574 \n", "77 0.883305 0.012968 0.142100 0.010714 \n", "78 0.883523 0.013146 0.142131 0.010681 \n", "79 0.883661 0.013047 0.142058 0.010638 \n", "\n", " train-AUC-mean train-AUC-std train-Logloss-mean train-Logloss-std \n", "0 0.499996 0.000009 0.302189 0.002418 \n", "1 0.618981 0.088396 0.227776 0.009594 \n", "2 0.758575 0.036903 0.190874 0.004790 \n", "3 0.778474 0.012547 0.180657 0.004351 \n", "4 0.784664 0.007662 0.176791 0.003464 \n", "5 0.801486 0.008265 0.173545 0.003944 \n", "6 0.812029 0.004814 0.170815 0.003318 \n", "7 0.821573 0.004150 0.168168 0.002686 \n", "8 0.825537 0.002166 0.166854 0.002284 \n", "9 0.827699 0.004355 0.165980 0.002763 \n", "10 0.828238 0.004967 0.165710 0.002916 \n", "11 0.832391 0.004098 0.164250 0.002763 \n", "12 0.834270 0.005416 0.163328 0.003049 \n", "13 0.834886 0.005530 0.163057 0.003074 \n", "14 0.835580 0.006042 0.162711 0.003282 \n", "15 0.836177 0.006505 0.162350 0.003406 \n", "16 0.837587 0.006670 0.161729 0.003425 \n", "17 0.840422 0.008338 0.160630 0.003884 \n", "18 0.842564 0.005416 0.159910 0.003345 \n", "19 0.843762 0.004223 0.159456 0.002908 \n", "20 0.843862 0.004137 0.159337 0.002843 \n", "21 0.845742 0.004977 0.158503 0.002997 \n", "22 0.847026 0.004351 0.157964 0.002731 \n", "23 0.848029 0.005031 0.157516 0.002853 \n", "24 0.848554 0.005170 0.157213 0.002953 \n", "25 0.849741 0.005065 0.156656 0.002807 \n", "26 0.850191 0.004955 0.156296 0.002907 \n", "27 0.850911 0.005407 0.155861 0.002955 \n", "28 0.851471 0.004956 0.155599 0.002789 \n", "29 0.851630 0.005069 0.155455 0.002823 \n", ".. ... ... ... ... \n", "50 0.860230 0.005378 0.150163 0.003652 \n", "51 0.860310 0.005331 0.150098 0.003616 \n", "52 0.860454 0.005418 0.149982 0.003705 \n", "53 0.861073 0.005605 0.149685 0.003798 \n", "54 0.861248 0.005420 0.149561 0.003734 \n", "55 0.861319 0.005484 0.149484 0.003754 \n", "56 0.861369 0.005492 0.149424 0.003744 \n", "57 0.861646 0.005424 0.149269 0.003714 \n", "58 0.861708 0.005415 0.149229 0.003720 \n", "59 0.861739 0.005397 0.149171 0.003693 \n", "60 0.861742 0.005393 0.149148 0.003703 \n", "61 0.862027 0.005136 0.148943 0.003607 \n", "62 0.862121 0.005156 0.148857 0.003626 \n", "63 0.862249 0.005253 0.148737 0.003715 \n", "64 0.862404 0.005327 0.148628 0.003732 \n", "65 0.862664 0.005393 0.148464 0.003785 \n", "66 0.862964 0.005743 0.148285 0.003972 \n", "67 0.863391 0.006173 0.148076 0.004102 \n", "68 0.863487 0.006285 0.147985 0.004199 \n", "69 0.863947 0.006193 0.147717 0.004175 \n", "70 0.864080 0.006283 0.147574 0.004268 \n", "71 0.864243 0.006387 0.147433 0.004301 \n", "72 0.864703 0.006083 0.147174 0.004146 \n", "73 0.864784 0.006046 0.147111 0.004117 \n", "74 0.865053 0.006098 0.146930 0.004142 \n", "75 0.865155 0.006075 0.146842 0.004101 \n", "76 0.865304 0.005987 0.146695 0.004043 \n", "77 0.865512 0.006000 0.146537 0.003998 \n", "78 0.865617 0.005938 0.146448 0.003941 \n", "79 0.865727 0.005961 0.146341 0.003937 \n", "\n", "[80 rows x 8 columns]" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cv_data" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best validation Logloss score, not stratified: 0.1421±0.0106 on step 79\n" ] } ], "source": [ "best_value = np.min(cv_data['test-Logloss-mean'])\n", "best_iter = cv_data['test-Logloss-mean'].idxmin()\n", "\n", "print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n", " best_value,\n", " cv_data['test-Logloss-std'][best_iter],\n", " best_iter)\n", ")" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f52baa02b8ad418ea07b4b50be5def76", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Best validation Logloss score, not stratified: 0.1412±0.0055 on step 67\n" ] } ], "source": [ "cv_data = cv(\n", " params = params,\n", " pool = Pool(X, label=y, cat_features=cat_features),\n", " fold_count=5,\n", " type = 'Classical',\n", " shuffle=True,\n", " partition_random_seed=0,\n", " plot=True,\n", " stratified=True,\n", " verbose=False\n", ")\n", "\n", "best_value = np.min(cv_data['test-Logloss-mean'])\n", "best_iter = cv_data['test-Logloss-mean'].idxmin()\n", "\n", "print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n", " best_value,\n", " cv_data['test-Logloss-std'][best_iter],\n", " best_iter)\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Select decision boundary" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ba81252dc8bb4d3e858b3627977b24ba", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=200,\n", " learning_rate=0.03,\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import sklearn\n", "from sklearn import metrics\n", "from catboost.utils import get_roc_curve" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [], "source": [ "? get_roc_curve" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4FNX6wPHvmw1pJBASikhvUqUIUgRpShEQCyqgFxW5KiIiwkVUsKFcERAEqaLIT7nKVbwolyogyFXp0qR3CCCdhJaQZM/vj9kkSwhhA9lMdvN+nifPTp93J8m8M+fMnCPGGJRSSqlrCbA7AKWUUrmbJgqllFKZ0kShlFIqU5oolFJKZUoThVJKqUxpolBKKZUpTRQqy0TkCRH5ye447CYipUXkvIg4cnCfZUXEiEhgTu3Tm0Rki4g0v4H19G8wB4m+R+HbRGQ/UAxIBs4DC4Dexpjzdsblj1zH+u/GmMU2xlAW2AfkM8Yk2RWHKxYDVDLG7PbyfsqSS75zXqV3FP7hfmNMOFAbqAO8bnM8N8TOq2R/uULPCj3eylOaKPyIMeYvYCFWwgBARIJFZKSIHBSRYyIySURC3eY/ICIbRCRORPaISFvX9IIi8rmIHBWRwyLyfkoRi4g8LSK/uoYnichI9zhE5EcR6ecavlVEvheREyKyT0T6uC33jojMFJHpIhIHPJ3+O7ni+NK1/gERGSwiAW5x/CYin4hIrIhsF5F70q2b2Xf4TURGi8hp4B0RqSAiP4vIKRE5KSL/EpFI1/JfAaWB/7qKm15NXwwkIstE5D3Xds+JyE8iUtgtnidd3+GUiLwpIvtF5N6MfpciEioiH7mWjxWRX91/b8ATrt/pSREZ5LZefRFZISJnXd97nIgEuc03IvKiiOwCdrmmjRGRQ66/gXUicrfb8g4RecP1t3HONb+UiCx3LbLRdTw6u5bv4Pp7Oisiv4tITbdt7ReRgSKyCbggIoHux8AV+1pXHMdEZJRr1ZR9nXXtq5H736Br3eoiskhETrvWfSOj46pukDFGf3z4B9gP3OsaLglsBsa4zf8YmA1EARHAf4EPXPPqA7FAK6yLhhJAFde8H4DJQH6gKLAaeN4172ngV9dwU+AQacWYhYBLwK2uba4D3gKCgPLAXqCNa9l3gETgQdeyoRl8vy+BH12xlwV2Aj3c4kgCXgHyAZ1d3yfKw++QBLwEBAKhQEXXsQgGimCdoD7O6Fi7xssCBgh0jS8D9gC3uba3DBjmmlcNq2iwietYjHR993uv8Xsd71q/BOAA7nLFlbLPKa591AISgKqu9eoCDV3fqSywDejrtl0DLML6ewh1TfsbEO1apz/wFxDimjcA62+qMiCu/UW7baui27bvAI4DDVwxP+U6ZsFux28DUMpt36nHFFgBdHMNhwMNMzrOGfwNRgBHXbGHuMYb2P2/6U8/tgegPzf5C7T+0c4D51z/TEuASNc8AS4AFdyWbwTscw1PBkZnsM1irpNPqNu0rsBS17D7P6kAB4GmrvFngZ9dww2Ag+m2/TrwhWv4HWB5Jt/N4Yqjmtu054FlbnEcwZWkXNNWA908/A4Hr7Vv1zIPAuvTHevrJYrBbvN7AQtcw28B37jNCwMuk0GiwEqal4BaGcxL2WfJdN+5yzW+Q19gltu4AVpe53ufSdk3sAN44BrLpU8UE4H30i2zA2jmdvyeyeDvNyVRLAfeBQpf4ztfK1F0df896U/2/2g5oX940BizWESaAV8DhYGzWFfFYcA6EUlZVrBOwGBd2c3LYHtlsK7Qj7qtF4B153AFY4wRkRlY/6zLgceB6W7buVVEzrqt4gD+5zZ+1TbdFMa6+j7gNu0A1lV2isPGdbZwm3+rh9/hin2LSFFgLHA31lVpANZJMyv+chu+iHVljCum1P0ZYy6KyKlrbKMw1pXxnqzuR0RuA0YB9bB+94FYd3Xu0n/v/sDfXTEaoIArBrD+RjKLw10Z4CkRecltWpBruxnuO50ewBBgu4jsA941xszxYL9ZiVHdAK2j8CPGmF+AaVjFGgAnsa5MqxtjIl0/BY1V8Q3WP22FDDZ1COtqvLDbegWMMdWvsetvgEdEpAzWXcT3btvZ57aNSGNMhDGmnXvYmXylk1jFM2XcppUGDruNlxC3TOCaf8TD75B+3x+4ptU0xhTAKpKRTJbPiqNYRYOAVQeBVdyTkZNAPBn/bq5nIrAd62mkAsAbXPkdwO17uOojBgKPAYWMMZFYxXcp61zrbyQjh4Ch6X7fYcaYbzLad3rGmF3GmK5YxYQfAjNFJH9m69xAjOoGaKLwPx8DrUSktjHGiVWWPdp1tYyIlBCRNq5lPwe6i8g9IhLgmlfFGHMU+An4SEQKuOZVcN2xXMUYsx44AXwGLDTGpNxBrAbiXBWYoa6K0RoicqcnX8QYkwx8CwwVkQhXIupH2h0LWCeVPiKST0QeBaoC87L6HVwisIrxzopICazyeXfHsOpZbsRM4H4RuctVufwuV5/AAXD93qYCo8R6GMDhqsAN9mA/EUAccF5EqgAveLB8EtbvL1BE3sK6o0jxGfCeiFQSS00RSUlw6Y/HFKCniDRwLZtfRNqLSIQHcSMifxORIq7vn/I3lOyKzcm1j/0c4BYR6SvWwxsRItLAk30qz2ii8DPGmBNYFcBvuiYNBHYDK8V6smgxVsUkxpjVQHdgNNZV5C+kXb0/iVVssBWr+GUmUDyTXX8D3ItV9JUSSzJwP9ZTWPuwrpQ/Awpm4Su9hFXPshf41bX9qW7zVwGVXNseCjxijEkp0snqd3gXq0I2FpgL/Cfd/A+Awa4nev6Rhe+AMWaL67vMwLq7OIdV8ZtwjVX+gVWJvAY4jXWF7cn/6z+wiv/OYZ24/32d5RcC87EeEjiAdSfjXjw0CitZ/4SVgD7HqkQHq47p/1zH4zFjzFqsOqpxWMd7Nxk8yZaJtsAWETkPjMGqd4k3xlzE+t3+5tpXQ/eVjDHnsB5CuB+rSG4X0CIL+1XXoS/cKZ8lIk9jvQDXxO5YskpEwrGumisZY/bZHY9SmdE7CqVyiIjcLyJhrnL3kVh3DPvtjUqp69NEoVTOeQCrov0IVnFZF6O39MoHaNGTUkqpTOkdhVJKqUz53At3hQsXNmXLlrU7DKWU8inr1q07aYwpciPr+lyiKFu2LGvXrrU7DKWU8ikicuD6S2VMi56UUkplShOFUkqpTGmiUEoplSlNFEoppTKliUIppVSmNFEopZTKlNcShYhMFZHjIvLnNeaLiIwVkd0isklE7vBWLEoppW6cN9+jmIbV3PCX15h/H1Z7N5WwOruZ6PpUSimVEWMgOQGS4sEkg3Fan063YZMMyYlw6bi1vDORy/GXb2q3XksUxpjlIlI2k0UeAL50NYq2UkQiRaS4q8MZpZTKXsZpnUCdieBMsn5MElw4BkmX3E68TreTbrrhi8fBEWQNp2zDmQTnDkJAPggITDtxO5PSTtzH10NoERC5ep4zGY6ugEKVwXkZzh2ChFgIjnTNT3L7dGb5aw/4byvWH8msG5brs/PN7BJc2UFKjGvaVYlCRJ4DngMoXbp0jgSnlMqipASIPw2Xz115EnS/0nUmw6UT1gk1ORHO7ITkeEDSljFO13pu6x77AyIrXHmCdybBntnW9NR1nFeul/IZn9Wuz21w/I8rxxPOZrxcipBCIA6QANenazjANXz+CESUpEb1woz9rexNhWZnosioG8gMm7I1xnwKfApQr149be5W5W3OJFfxQ4J1kk2KTyuOyGg8tajCdXWMsT7/Wg1hxVxX2InWifvQUihQOu2KOf3V8cnNUKCMa1smbVsXcqAg4PD/Mp5+ZlfWthMYBo58IIFWwsJYdwrFG7lOsgFuP46rh09vh+IN0u4gUrZz4QhEVoTAUNd01wk7wGEtE38aoqtmPC/AYR3PsCLgCLbuWoIKWsu6LyMO664kE1u3nuCPP47ytx41AXiyu6HZgFjKlRuStePkfshueM2bFwOUchsvidVOv1L+5/J5uPCXVbSQfNm6wr1wxDoBGKd1Qt84CUKirJP24V8hopRr3mU4H2NtRwJuqPghS2L3Zj4/LpMmgxzBVmIqVNntxOa4ctgkW9+nWD3rRBh3EMrcCwFBbssFuK3nOlHHn4XI8mkn5pQTdVA4hJdMu5omwG09t09HEOQLy9ZDlZtcvJjI++8vZ8SI33E4hIYNS1KxYhQiQtmykTe1bTsTxWygt4jMwKrEjtX6CZVrGWOdQI/9cWUxSvpy6L1zIKKMdbI/thbCilpFMUmXsr7Ps7sziMNpnfgcQdaVqyMYHCEQGOL6dB9PNxzgAMRaP+WG/nIsFKvrOum6fpyJUOi2K0/wKVfBAQ5rmcCQK7clYiW5wJAbP8bqhs2fv4sXX5zHvn1WcVWPHnWJjg69zlqe81qiEJFvgOZAYRGJAd4G8gEYYyYB84B2WB2wXwS6eysWpa7p9E7Y9R/AejqEXd9DaFFr+OhK6+o45eTpqfNuN8YXj1ufKduIqmoVewQEWVfV0dUhtLB1wnUmQ1QVKNEk7Yo5/y1WUnAEWSf7oAKuE75ScPhwHH37LmTmzK0A1KxZjEmT2tOoUanrrJk13nzqqet15hvgRW/tX+Vxl07D+k8gZhmcPwyIdaJNLaM3cHqbZ9tKSRIpxT63Pea6sk5f1uywimUqPgBFaruuvEMh/FYIirhu2bJSWfXii/P48ccdhIXlY8iQ5rz8ckMCA7P/9Tif649CKSDtefK4A9bjhGd2wqmtcGAxnNmR9e2Vvx+K1korfine0PoMCk+7CxBtyEDZLynJmZoMPvzwXvLlc/DRR60pXbqg1/apiULlXkkJ1ueFoxCz3EoIR36zil92/NuzbURVhZrPWpWr4SVcFZtCavl6vjDrKR6lcrnY2HgGD/6ZnTtPs2DBE4gIlSsX5rvvHvX6vjVRKHsZp1XRe3wjbPnCqhA+ts7z9QMCrQrl8h2sl5RKNoXQaKj0CESU1OIe5fOMMXz33Vb69l3A0aPncTiEDRv+ok6dm3uJLis0Uaic4UyCE5utZ9DP7oaDi627hKwoUsu6+g8pBGVawy31oVBF78SrVC6wZ89peveez4IF1hNwjRqVZNKkDtSsWSxH49BEobzryArYNQvWjrj2MgGBVkVz4gWo+bz1fH3RWlC4pvW4p1J50MiRv/Pmm0uJj08iMjKEDz+8l7///Q4CAnL+LlkThfKeRT1h0+Srp99S33qDNaQQNBgE4Tl3C62Ur7h4MZH4+CS6davJyJGtKVo0v22xaKJQ2efSKesls63TYdv0K+c1eAMqd4YiNe2JTalc7sSJC+zYcYomTaz27AYObEzz5mVp2tT+hy00Uaibc+wP+PFBSLwI8aeunh8YBr3PWC+MKaWu4nQapk5dz6uvLiIwMIDt23sTFRVKcHBgrkgSoIlC3QhjYN88WNzLal45vejqUL49VOgIt96lTx4pdQ1//nmcnj3n8NtvVkParVqV5+LFRKKisq/5jeygiUJ5LikBfukPG8ZfPa/pcKtoKaKUJgalruPChcsMGfILo0atJCnJSbFi+fn447Z07lwdyYX/P5oolGecSTA+0mqu2l29AdDgdatiWinlkUce+Y4FC3YjAr161WPo0HuIjMy9DSpqolCZi91nPd76S/+0aUXvgDZTrYrpXHj1o1RuN3BgY44dO8/Eie1p0KCk3eFclyYKlcaZDJdOwo4ZsPtHqxOb9Ko8Du3/lfOxKeWjkpKcfPLJKvbvP8uYMfcB0Lx5Wdaufc6WdyJuhCYKZZnX7epHWt0VKAsdZ1p9FyilPLJ69WGef34OGzb8BcBzz9WlevWiAD6TJEATRd5lDBxdBYd+hl8HXT0/KMJ6Ga5kM6vbRy1iUspjZ8/G88YbS5g0aS3GQJkyBRk3rl1qkvA1mijymrhDMK2q1VxGevlvgeditGMcpW7CjBl/0rfvAo4du0BgYAD9+zfizTebkj+/775LpIkirzBOOLISZjS+cro4rErpu9613n3QPheUuik//bSHY8cu0LhxKSZObM/tt+dsA37eoInC38Xuh597w965V05v/RnUeEaLlJS6SQkJSRw+fI7y5a1HxIcPb8Xdd5fmqadq+1Q9RGY0UfijczGw5EXYMzvj+a0mw+09cjYmpfzQzz/v44UX5hIQIGzc2JOgIAeFC4fRvXsdu0PLVpoo/EXsPjj8G+z5L+z89ur5tz8L9V+DyPI5H5tSfubYsfP84x+LmD59EwBVqhQmJiYu9a7C32ii8GWHlsGCp61+ozNSqgU0HwVFa+dkVEr5LafTMGXKOl57bQlnz8YTEhLI4MF3M2BAY4KC/PchEE0UvurkFvi2xdXTKzxgfTb9EKIq52xMSvm5hx76N7Nn7wCgTZsKjB/fjgoVomyOyvs0UfiqC3+lDdd+ERoOth5vVUp5zcMPV2H16sOMGdOWRx+tlisb8PMGTRS+rnRLuGec3VEo5Zdmz95BTEwcvXrdCcCTT9bi4YerEhGRt7ro1UThi5ITYc6jdkehlN86eDCWPn3m8+OPOwgOdtC2bUXKly+EiOS5JAGaKHyPccLHbm94RpS2Lxal/ExiYjJjx67i7beXceFCIhERQbz/fkvKlClod2i20kThS45vhK/cnmAq1w7unWRfPEr5kZUrY3j++Tls2nQMgEcfrcbo0W0oUaKAzZHZTxOFL3HvE6LSw9Dxe/tiUcrPvPnmUjZtOka5cpGMG9eOdu0q2R1SrqGJwlcc/h0OLrGGG70Nd71jazhK+TpjDOfOXaZAAavOYdy4+/jyy40MGtSUsLB8NkeXu2gLcL7CvZ/qGt3ti0MpP7Bjx0nuvfcrHn743xhjAKhcuTBDh96jSSIDekfhCy6ehO1fW8ON3oECZWwNRylfFR+fxAcf/I9hw37j8uVkoqND2b//LOXK+WfTG9lFE0Vul74C+5Y77YtFKR+2aNEeevWax+7dpwF45pnaDB/eiujoMJsjy/28WvQkIm1FZIeI7BaR1zKYX1pElorIehHZJCLtvBmPz/n55SuTRMlmUKaVffEo5YOMMTzzzI+0bj2d3btPU61aEZYvf5rPP39Ak4SHvHZHISIOYDzQCogB1ojIbGPMVrfFBgPfGmMmikg1YB5Q1lsx+ZSlfWH92LTx5qPhjpe1/wilskhEKFs2ktDQQN56qxn9+jXy6wb8vMGbRU/1gd3GmL0AIjIDeABwTxQGSHlIuSBwxIvx5H7HN8CGCbBrJsSfSZv+/BEIL25fXEr5mA0b/uLo0XPcd5/1iOvAgY3p1q2m1kXcIG8WPZUADrmNx7imuXsH+JuIxGDdTbyU0YZE5DkRWSsia0+cOOGNWHOHVR/A5ilXJokXjmmSUMpD584l0K/fQurW/ZSnnvqB06cvARAcHKhJ4iZ4844iozISk268KzDNGPORiDQCvhKRGsYY5xUrGfMp8ClAvXr10m/DfyQnWJ+1e0ORWnDbIxASaW9MSvkAYww//LCdPn0WEBMTR0CA8Pjjt5Mvn74BkB28mShigFJu4yW5umipB9AWwBizQkRCgMLAcS/GlfuVvgcqPWh3FEr5hAMHztK793zmzNkJQL16tzJ5cgfuuEPvxLOLN9PtGqCSiJQTkSCgC5C+E+eDwD0AIlIVCAH8uGxJKZWdjDF06vQtc+bspECBYMaNu4+VK3tokshmXrujMMYkiUhvYCHgAKYaY7aIyBBgrTFmNtAfmCIir2AVSz1tUl6TVEqpa3A6DQEBgogwcmRrJk1ay+jRbShePMLu0PySV1+4M8bMw6qkdp/2ltvwVqCxN2NQSvmPU6cu8tpriwGYMqUjAM2bl6V587I2RuX/tKZHKZXrGWP4v//bQJUq4/nss/V8+eUmYmLi7A4rz9AmPHILZzIc+d3uKJTKdbZtO8ELL8zll18OANYdxMSJ7SlZUvuJyCmaKOxmDGyZBgufSZsWoG+NKmWM4a23lvLhh7+RmOikcOEwPvqoNd261US0hYIcpYnCLs4kWPVP2PY1nNmRNj2sGJRqbltYSuUWIsLhw+dITHTy7LN3MGzYvURFhdodVp6kicIuX90BJzdfOe3Rn6F0C3viUSoXOHLkHCdPXqRmzWIADB/eih496tC4sfYNbydNFDkt/ix8VQfi9lvj4SWh+UdQ7j4I0kf7VN6UnOxk4sS1DBr0MyVKRLBhQ0+CghwULhxG4cKaJOymiSKnLX05LUkULA89dmuLsCpP++OPozz//BzWrrUabmjatAxxcQkULqxNgOcWHiUK15vVpY0xu70cj/+76GqdpHgD6LpCk4TKs+LiEnjzzZ8ZN24NTqehZMkCjB3blgcfrKKV1bnMdROFiLQHRgFBQDkRqQ28bYx5yNvB+bVGb2uSUHmWMYamTb9g48ZjOBxCv34Neeed5kREBNsdmsqAJy/cDQEaAGcBjDEbgIreDEop5d9EhFdeaUj9+iVYu/Y5PvqojSaJXMyToqdEY8zZdLeC2h7TjTi7B/YvsDsKpXLc5cvJjBq1AodDGDDAarXnySdr8be/1cTh0AYicjtPEsU2EXkMCBCRcsDLwErvhuWn/nDr2jS0sH1xKJWD/ve/A/TsOZetW08QHOzgySdrUaxYOCKCw6HFr77Ak1TeG6gLOIH/APFYyUJ5KikBFvdK6wO7wgNQrJ69MSnlZSdPXuSZZ36kadNpbN16gkqVopgz53GKFQu3OzSVRZ7cUbQxxgwEBqZMEJGHsZKGuh5nEnxRGeIOpE2r9bxWZCu/ZYxh2rQNDBiwiFOnLhEU5OD115vw2mtNCAnRJ/J9kSe/tcFcnRQGZTBNZeTYuiuTRNff4dZG9sWjVA6YPn0zp05domXLckyY0I7KlbWo1ZddM1GISBusbkpLiMgot1kFsIqhlCecSWnDfS5CPm2rRvmfixcTiY2Np3jxCESECRPasWbNEZ544nZ9J8IPZHZHcRz4E6tOYovb9HPAa94Myi/depcmCeWX5s/fxYsvzqN8+UIsWtQNEaFy5cJ6F+FHrpkojDHrgfUi8i9jTHwOxuQ/Lp2CGU3sjkIprzh8OI6+fRcyc+ZWACIigjl16pI2veGHPKmjKCEiQ4FqQEjKRGPMbV6LytcZA3M6w87v0qaV0ISh/ENyspPx49cwePDPnDt3mfz58zFkSAv69GlAYKC+E+GPPEkU04D3gZHAfUB3tI4ic8teuTJJ3P0h1H/VvniUyiZOp6FZs2n89tshAB58sApjxrSldOmCNkemvMmTRBFmjFkoIiONMXuAwSLyP28H5tNObLQ+HcHQ54L2WKf8RkCA0Lp1BQ4ejGXcuHZ07FjZ7pBUDvAkUSSI9djCHhHpCRwGino3LD/x8HxNEsqnGWP49tstBAYG0KlTNQAGDmxMv36NCA8Psjk6lVM8SRSvAOFAH2AoUBB4JtM18jJj4OSfdkeh1E3bs+c0vXrN46ef9lCkSBgtW5ajUKFQgoMDCdb2+/KU6yYKY8wq1+A5oBuAiJT0ZlA+bWF3uHTSGhat2FO+JyEhiREjfmfo0P8RH59EoUIhDB3akoIFQ66/svJLmSYKEbkTKAH8aow5KSLVsZryaAlosnB36Bf4tvmV026pb0soSt2oZcv288ILc9m+3brY6datJiNHtqZo0fw2R6bsdM1LXhH5APgX8ASwQEQGAUuBjYA+GuvuzO50SULg+cP6gp3yKcnJTnr1spJE5crR/Pzzk3z55UOaJFSmdxQPALWMMZdEJAo44hrfkTOh+RD3tpwenA0V7rcvFqWywOk0xMcnERaWD4cjgIkT27N8+QFefbUxwcHagJ+yZPaXEG+MuQRgjDktIts1SVzDoZ+tz9ItNUkon7F58zF69pxLlSrRfP75AwA0a1aWZs3K2huYynUySxTlRSSlhVgByrqNY4x52KuR+YqLx2HVP63hfBH2xqKUBy5cuMyQIb8watRKkpKc7Nt3hjNnLlGokBaVqoxllig6pRsf581AfNaK99KGW4y2Lw6lPPDf/+6gd+/5HDwYiwj06lWPoUPvITJSn2hS15ZZo4BLcjIQn+RMgg2u/FmiCRQsZ288Sl1DUpKTzp1n8p//bAOgdu1bmDy5A/Xrl7A5MuULtLbqZhiTNtxxln1xKHUdgYEBFCwYTHh4EO+914LevetrA37KY179SxGRtiKyQ0R2i0iGfViIyGMislVEtojI196Mx2sCAiFM295XucuqVTGsWhWTOj5iRCu2bXuRvn0bapJQWeLxHYWIBBtjErKwvAMYD7QCYoA1IjLbGLPVbZlKwOtAY2PMGRHxjTakjm+EVUPh9Da7I1HqKmfPxvP664uZPHkdVaoUZsOGngQFOYiO1n4i1I257mWFiNQXkc3ALtd4LRH5xINt1wd2G2P2GmMuAzOw3s1w9yww3hhzBsAYczxL0dtl3SirGfGUNp0qdLQ3HqWwGvD7+uvNVKkyjkmT1uFwBNCxY2WSk7VXAHVzPLmjGAt0AH4AMMZsFJEWHqxXAjjkNh4DNEi3zG0AIvIb4ADeMcYs8GDb9kq+bH3ePQxqvQDBBeyNR+V5u3adoleveSxevBeAxo1LMWlSB2rU8I2bdJW7eZIoAowxB9J1kJ7swXoZ9ahu0o0HApWA5lhtR/1PRGoYY85esSGR54DnAEqXLu3Brr0oOREOLLKGI0prklC2S0xMpmXLL4mJiSMqKpThw++le/c6BARk9C+oVNZ5kigOiUh9wLjqHV4CdnqwXgxQym28JFYzIOmXWWmMSQT2icgOrMSxxn0hY8ynwKcA9erVS59scs7FkzCxSNq4I59toShljEFEyJfPwdChLVm6dD/Dh99LkSLaNpPKXp48+vAC0A8oDRwDGrqmXc8aoJKIlBORIKALMDvdMj8ALQBEpDBWUdRez0K3we4f0oYDQ6FMa/tiUXnWsWPn6dZtFu+/vzx12pNP1uKLLx7QJKG8wpM7iiRjTJesbtgYkyQivYGFWPUPU40xW0RkCLDWGDPbNa+1iGzFKs4aYIw5ldV95RhnovUZWQG6b7cei1UqhzidhilT1vHaa0s4ezaeyMgQ+vZtSESE9iKkvMuTM90aV5HQv4H/GGPOebpxY8w8YF66aW9DRCFlAAAgAElEQVS5DRusu5V+nm4zVyjTSpOEylEbN/5Fz55zWbnSei+ibduKjB/fTpOEyhGe9HBXQUTuwio6eldENgAzjDEzvB5dbnL5HCzpZXcUKo9JTEzm9deX8PHHK0lONhQvHs6YMW155JFqpHvARCmv8ej1TGPM78aYPsAdQBxWh0Z5y+Hf0oYLaJtOKmcEBgawfv1fOJ2Gl16qz7ZtL/Loo9U1Sagcdd07ChEJx3pRrgtQFfgRuMvLceU+xvXSUvitcOcAe2NRfu3gwViSk52UK1cIEWHSpPbExiZQr96tdoem8ihPCtr/BP4LDDfG/M/L8eR+RWqBXs0pL0hMTGbMmFW8/fYyGjUqyaJF3RARKlWKtjs0lcd5kijKG2O0DYCr3hVUKvusWHGInj3nsmnTMQCiokK5eDGR/PmDbI5MqUwShYh8ZIzpD3wvIledJfNUD3frx8HPL9kdhfJDZ85c4rXXFvPpp38AUK5cJOPHt+O++yrZHJlSaTK7o/i36zNv92yXeOnKJFGunX2xKL+SkJBE7dqTOXgwlnz5Ahgw4C4GDWpKWJi+8a9yl8x6uFvtGqxqjLkiWbhepMsbPeAZt2atnj0IBUpde1mlsiA4OJAePeqwZMk+Jk5sT7VqRa6/klI28OTx2GcymNYjuwPJ9fLl1yShbkp8fBJvv72Ur7/enDrtjTfuZtmypzRJqFwtszqKzliPxJYTkf+4zYoAzma8lh/SenyVDRYt2kOvXvPYvfs0RYvm56GHqhAamk97mlM+IbM6itXAKaxWX8e7TT8HrPdmULnGjm9hTmdr2OhTTyrr/vrrPP36LeSbb6xOrqpXL8KkSR0IDdV6COU7Mquj2AfsAxbnXDi5zJoRacNVH7cvDuVzkpOdTJ68jjfeWEJsbAKhoYG8/XYzXnmlEUFBDrvDUypLMit6+sUY00xEznDlSwSC1Z5flNejs1tKRfZDc6G8Pu2kPJecbPjkk9XExibQrl0lxo27j3LlCtkdllI3JLOip5TuTgvnRCC5Wv5b7I5A+YBz5xJITjZERoYQFORgypT7OXbsPA8/XFXbZlI+7Zo1aW5vY5cCHMaYZKAR8DygvaMo5WKM4T//2UbVquPp339h6vQmTUrTqZO28qp8nyePXPyA1Q1qBeBLrIYBv/ZqVLmBMXBqq91RqFxu//6zdOw4g06dvuXw4XP8+ecJ4uOT7A5LqWzlSaJwuvq0fhj42BjzElDCu2HlAv97DZITXCN6RaiulJiYzIcf/kq1auOZM2cnBQoEM27cffz++zOEhGinVsq/eNQVqog8CnQDHnRN8+9n+07vhDXDreGAQChcw954VK5y8WIiDRt+xubNxwHo0qUGo0a1pnjxCJsjU8o7PEkUzwC9sJoZ3ysi5YBvvBuWzTZPSRvuvBwc/p0XVdaEheWjXr1buXgxkQkT2tO6dQW7Q1LKqzzpCvVPEekDVBSRKsBuY8xQ74dmI2ei9VmpExRvaG8synbGGL78ciMVKkTRpElpAEaPbkNQkENfnFN5gic93N0NfAUcxiqsv0VEuhljfst8TT9Qool2UpTHbdt2ghdemMsvvxygatXCbNjQk6AgBwULhtgdmlI5xpOip9FAO2PMVgARqYqVOOp5MzCl7HTpUiJDh/6P4cN/IzHRSZEiYbz+ehPy5dO2mVTe40miCEpJEgDGmG0i4t/dbiUn2h2BstGCBbt58cV57N17BoBnn72DYcPuJSoq1ObIlLKHJ4niDxGZjHUXAfAE/too4Pmj8Fk5t8diVV5z/vxlunWbxcmTF6lRoyiTJrWncePSdoellK08SRQ9gT7Aq1h1FMuBT7wZlC3WjoJf+l85rcy99sSiclRyshOn05Avn4Pw8CDGjGlLTEwcr7zSkHz5tAE/pTJNFCJyO1ABmGWMGZ4zIdkgIfbKJFHvH9BsxLWXV35j3bojPP/8HB54oDJvvtkMgMcfv93mqJTKXa5ZMycib2A13/EEsEhEMurpzj+410k8uUmTRB4QF5fAyy/Pp379z1i37ihffbWJxMTk66+oVB6U2R3FE0BNY8wFESkCzAOm5kxYOWzNh9ZnSDQU0atJf2aMYebMrbz88gKOHj2PwyH069eQd99tocVMSl1DZokiwRhzAcAYc0JE/Pe5wA3jrM/w4vbGobzq3LkEOneeyfz5uwFo0KAEkyZ1oHZtbUZeqcxklijKu/WVLUAF976zjTEPezWynBJ3AJLireFHl9gbi/Kq8PAgEhKSKVgwmGHD7uW55+oSEKAvVCp1PZklik7pxsd5MxDbzPtb2nBQAfviUF6xfPkBihcPp1KlaESEqVM7EhISSLFi4XaHppTPyKzPbP+/vD6yEg7/ag3f+SoEarMM/uLkyYu8+uoivvhiA/fcU45Fi7ohIpQpE2l3aEr5nLzdcP7qD9KG6/azLw6VbZxOw7RpGxgwYBGnT18iKMjB3XeXJjnZEBioxUxK3QivVlCLSFsR2SEiu0XktUyWe0REjIjkbPtRyZetz7uGQP5iObprlf22bDlO8+bT6NFjNqdPX+Kee8qxefMLvP12cwID/fdZDKW8zeM7ChEJNsZ43LaFiDiA8UArIAZYIyKz3duNci0XgfXm9ypPt53tbtH2DX1dbGw8DRt+zvnzlylaND+jRrXm8cdv1/6qlcoG173MEpH6IrIZ2OUaryUinjThUR+r74q9xpjLwAzggQyWew8YDsR7HnY22PEd7F+Qo7tU2c8YA0DBgiEMHNiYnj3rsn37izzxRE1NEkplE0/ux8cCHYBTAMaYjUALD9YrARxyG48hXV/bIlIHKGWMmZPZhkTkORFZKyJrT5w44cGur+PEZpjzWNp4kVo3v02Vow4fjuORR75l+vRNqdMGDbqbiRM7UKiQtvKqVHbyJFEEGGMOpJvmSVsHGV3OmdSZ1gt8o4H+GSx35UrGfGqMqWeMqVekSBEPdn0de+emDT+2FMJvvfltqhyRlORkzJiVVKkynu+/38bbby8jOdkJoHcQSnmJJ3UUh0SkPmBc9Q4vATs9WC8GKOU2XhI44jYeAdQAlrn+wW8BZotIR2PMWk+Cv3GufFW9O5Rq7t1dqWyzZs1hevacyx9/HAXgwQerMHZsWxwOrahWyps8SRQvYBU/lQaOAYtd065nDVBJRMphdaPaBXg8ZaYxJhYonDIuIsuAf3g/SbjRJ518woULlxk4cDETJqzBGChduiCffHIfHTtWtjs0pfKE6yYKY8xxrJN8lhhjkkSkN7AQcABTjTFbRGQIsNYYMzvL0WYHY+CvNbbsWt2YwMAAFi/eS0CA0K9fI95+uxn58/t3J4tK5SbXTRQiMgW3uoUUxpjnrreuMWYeVquz7tPeusayza+3vZtmDPz8EuyeZY0H5O33DXOzPXtOExkZQnR0GMHBgXz11UOEhARy++16F6hUTvOkcHcxsMT18xtQFPDNvkJPbYEN49PGq3azLxaVoYSEJN5/fzk1akxk4MDFqdPvvLOEJgmlbOJJ0dO/3cdF5Ctgkdci8qYNE9OG/74XCpazLxZ1lWXL9vPCC3PZvv0kYD3hlJzs1MpqpWx2I2Uv5YAy2R2I1106BRsnWMPl7tMkkYscP36BAQMW8eWXGwGoXDmaiRPb06KF/o6Uyg08qaM4Q1odRQBwGrhmu025VtKltOGW/tliui86efIiVauO5/TpSwQHOxg06G5efbUxwcFaf6RUbpHpf6NYLzjUwnq8FcBpUtpM8FXhJSCyvN1RKJfChcN44IHKxMTEMWFCeypWjLI7JKVUOpkmCmOMEZFZxpi6ORWQ8m8XLlxmyJBfaN/+Npo2tUowJ0xoT3CwQ9+sViqX8qSWcLWI3OH1SLxp/yL4tNT1l1Ne9d//7qBatQkMH/47vXrNxem0bk5DQgI1SSiVi13zjkJEAo0xSUAT4FkR2QNcwGrDyRhjfCN5HFgC37dOG7/lTvtiyaMOHYrl5ZcXMGvWdgDq1LmFyZM7aH/VSvmIzIqeVgN3AA/mUCzesebDtOH7v4NK6bsCV96SlORk7NhVvPXWUi5cSCQ8PIj332/Biy/W146ElPIhmSUKATDG7MmhWLzDmWR93jsRbnvE3ljymLi4BD744FcuXEikU6eqfPxxW0qWLGB3WEqpLMosURQRkWt2JG2MGeWFeLyn0G12R5AnnD0bT2hoIMHBgURFhTJ5cgeCgx20b6/HXylfldn9vwMIx2oOPKMfpVIZY/j6681UrjyO4cN/S53+8MNVNUko5eMyu6M4aowZkmOReMPO7+HQUruj8Hs7d56iV6+5LFmyD4Dlyw9ijNEnmZTyE9eto/Bpe916WI2qal8cfio+PokPP/yVf/7zVy5fTiYqKpQRI1rx9NO1NUko5UcySxT35FgU3nbvJAgvbncUfuWvv87TtOkX7Np1GoCnn67NiBGtKFw4zObIlFLZ7ZqJwhhzOicD8SqHdnKT3YoVy0+pUgUJDAxg4sT2NGtW1u6QlFJe4r8trx1aBlum2R2F33A6DVOmrKNFi3Lcdls0IsLXXz9MoUKhBAU57A5PKeVF/vvW06FlacPF6tkWhj/YuPEvGjeeSs+ec+nVay4p7UIWKxauSUKpPMB/7yhSNBwMRW63OwqfdP78Zd55Zxkff7yS5GTDrbdG0LOnJl2l8hr/TxSiV7w34ocftvPSS/OJiYkjIEB46aX6vP9+SwoUCLY7NKVUDvP/RKGy7PDhOLp0mUlCQjJ16xZn0qQO1Kt3q91hKaVs4r+J4qxvN1GV0xITkwkMDEBEKFGiAEOHtiQoyEGvXndqn9VK5XH+eQbY9jVsm24Ni39+xez0+++HqFv3U6ZP35Q6rX//u3jppQaaJJRSfpgozu6BeU+kjWuLsdd0+vQlnn/+vzRuPJXNm48zYcJafL2nW6VU9vO/oqe4A2nDnX+B6Gr2xZJLGWOYPn0T/fv/xIkTF8mXL4BXX23MoEF3a9MbSqmr+F+iSFGqBZRsancUuc6xY+fp2vV7li7dD0CzZmWYOLE9VasWsTcwpVSu5b+JQmUoMjKEo0fPU7hwGCNHtuLJJ2vpXYRSKlOaKPKARYv2cMcdxYmODiM4OJDvvnuU4sXDiY7WBvyUUtfnf5XZKtXRo+fo2vV7WreezsCBi1On16hRVJOEUspj/nVHYQws7mV3FLZLTnYyefI6Xn99CXFxCYSGBlK5crR2JqSUuiH+lSji9sOZHdZwgTK2hmKXP/44Ss+ec1iz5ggA7dtXYty4dpQtG2lzZEopX+VficKZnDbcarJ9cdhk//6z1K8/heRkQ4kSEYwdex8PPVRF7yKUUjfFq4lCRNoCYwAH8JkxZli6+f2AvwNJwAngGWPMgas25Alj4Ps21nDB8nmys6KyZSPp3r02ERHBvPtucyIitAE/pdTN81pltog4gPHAfUA1oKuIpH/7bT1QzxhTE5gJDL/hHZpkiN1rDZfvcMOb8SX795/l/vu/4Zdf9qdO+/TT+xk1qo0mCaVUtvHmHUV9YLcxZi+AiMwAHgC2pixgjFnqtvxK4G83vVdxQMsxN72Z3CwxMZlRo1bw7ru/cOlSEidPXmTFih4AWsyklMp23kwUJYBDbuMxQINMlu8BzM9ohog8BzwHULp06eyKzyf9+utBevacw5YtJwDo0qUGo0a1tjkqpZQ/82aiyOjSNsMW50Tkb0A9oFlG840xnwKfAtSrVy9Ptlp35swlBgxYxOefrwegQoVCTJjQntatK9gcmVLK33kzUcQApdzGSwJH0i8kIvcCg4BmxpgEL8bj05xOw48/7iBfvgBee60Jr7/ehNDQfHaHpZTKA7yZKNYAlUSkHHAY6AI87r6AiNQBJgNtjTHHvRiLT9q+/STlykUSHBxIdHQY//rXw5QuXZAqVQrbHZpSKg/x2lNPxpgkoDewENgGfGuM2SIiQ0Sko2uxEUA48J2IbBCR2d6Kx5dcvJjIoEFLqFlzIsOH/5Y6vXXrCpoklFI5zqvvURhj5gHz0k17y234Xm/u3xctWLCbXr3msm/fWQBOnrxoc0RKqbzOv97M9mFHjpyjb98FfPed9fTw7bcXZdKkDtx1V6nrrKmUUt6liSIX2LnzFPXqfcq5c5cJC8vHO+80o2/fhuTL57A7NKWU0kSRG1SqFMWdd5Ygf/58fPLJfZQpow34KaVyD00UNoiLS+Ctt5bSq9ed3HZbNCLC7NldyJ8/77VPpZTK/fwnUcx/0jWQe9/HM8Ywc+ZWXn55AUePnmf79pMsWGC1WqJJQimVW/lHoji2HrZ/Yw0Xb2RvLNewd+8Zeveex/z5uwFo2LAkH36oD30ppXI//0gUywekDT80x744MnD5cjIjR/7Oe+8tJz4+icjIEIYNu4dnn61LQIA24KeUyv38I1Ekut41uOtdCMldFcGHDsUyZMgvJCQk88QTt/PRR60pVizc7rCUUspjvp8o4g7B0RXWcOncUZRz5swlIiNDEBEqVIhizJi2VKwYxT33lLc7NKWUyjKvNeGRY1a+lzYcFGFfHFgN902dup6KFT9h+vRNqdOff76eJgmllM/y/URx+Zz1WaY1FK5hWxhbthynefNp9Ogxm9OnL6VWWiullK/z7aKnM7thxwxruPrTYEPvbhcvJvLee78wcuQKkpKcFC2an9Gj29C1q31JSymlspPvJoozu2DqbWnj+W/J8RB27jxFmzbT2b//LCLQs2dd/vnPeyhUKDTHY1FKKW/x3URxYHHacIM3oFTzHA+hTJmChIQEUqtWMSZN6kDDhiVzPAaVeyUmJhITE0N8fLzdoag8JCQkhJIlS5IvX/Z1bOa7iSLFbY9Ak6E5squkJCeTJq2la9caREeHERwcyIIFT1CiRAECA32/ukdlr5iYGCIiIihbtixiQ7GoynuMMZw6dYqYmBjKlSuXbdv1/bNbaM505LN69WHq15/CSy/NZ+DAtLuZMmUiNUmoDMXHxxMdHa1JQuUYESE6Ojrb72J9/47Cy2Jj4xk06GcmTFiDMVC6dEEeeKCy3WEpH6FJQuU0b/zNaaK4BmMM//73Fl55ZSF//XWewMAA+vVryFtvNdMG/JRSeYpvlpnEn4Ulvby6i40bj9G16/f89dd57rqrFH/88RwffthKk4TyKQ6Hg9q1a1OjRg3uv/9+zp49mzpvy5YttGzZkttuu41KlSrx3nvvYUxa68vz58+nXr16VK1alSpVqvCPf/zDjq+QqfXr1/P3v//d7jAy9cEHH1CxYkUqV67MwoULM1xmyZIl3HHHHdSuXZsmTZqwe7f1HtbBgwdp0aIFderUoWbNmsybZ/UsvXnzZp5++umc+grWlbMv/dStW9eYvfOMGYn1s26MyS5JSclXjL/yygIzZco6k5zszLZ9qLxj69atdodg8ufPnzr85JNPmvfff98YY8zFixdN+fLlzcKFC40xxly4cMG0bdvWjBs3zhhjzObNm0358uXNtm3bjDHGJCYmmvHjx2drbImJiTe9jUceecRs2LAhR/eZFVu2bDE1a9Y08fHxZu/evaZ8+fImKSnpquUqVaqU+vcyfvx489RTTxljjHn22WfNhAkTUrdVpkyZ1HXuuecec+DAgQz3m9HfHrDW3OB51zeLnlKueqKqwB19smWTS5fuo1eveUye3IGmTcsAMGpUm2zZtlJ85KW6iv6e97/SqFEjNm2ympb5+uuvady4Ma1btwYgLCyMcePG0bx5c1588UWGDx/OoEGDqFKlCgCBgYH06nX1Xfz58+d56aWXWLt2LSLC22+/TadOnQgPD+f8+fMAzJw5kzlz5jBt2jSefvppoqKiWL9+PbVr12bWrFls2LCByEirMc+KFSvy22+/ERAQQM+ePTl48CAAH3/8MY0bN75i3+fOnWPTpk3UqlULgNWrV9O3b18uXbpEaGgoX3zxBZUrV2batGnMnTuX+Ph4Lly4wM8//8yIESP49ttvSUhI4KGHHuLdd98F4MEHH+TQoUPEx8fz8ssv89xzz3l8fDPy448/0qVLF4KDgylXrhwVK1Zk9erVNGp0ZXcIIkJcXBwAsbGx3HrrrZlOB7j//vuZMWMGr7766k3F6AnfTBQpCt7841/Hj19gwIBFfPnlRgBGjVqRmiiU8hfJycksWbKEHj16AFaxU926da9YpkKFCpw/f564uDj+/PNP+vfvf93tvvfeexQsWJDNmzcDcObMmeuus3PnThYvXozD4cDpdDJr1iy6d+/OqlWrKFu2LMWKFePxxx/nlVdeoUmTJhw8eJA2bdqwbdu2K7azdu1aatRIawGhSpUqLF++nMDAQBYvXswbb7zB999/D8CKFSvYtGkTUVFR/PTTT+zatYvVq1djjKFjx44sX76cpk2bMnXqVKKiorh06RJ33nknnTp1Ijo6+or9vvLKKyxduvSq79WlSxdee+21K6YdPnyYhg0bpo6XLFmSw4cPX7XuZ599Rrt27QgNDaVAgQKsXLkSgHfeeYfWrVvzySefcOHCBRYvTnvisl69egwbNkwThTc5nYbPP/+DgQMXc+ZMPMHBDgYPbsqAAXfZHZryR1m48s9Oly5donbt2uzfv5+6devSqlUrwCpyvtbTMVl5ambx4sXMmDEjdbxQoULXXefRRx/F4XAA0LlzZ4YMGUL37t2ZMWMGnTt3Tt3u1q1bU9eJi4vj3LlzRESkNfx59OhRihQpkjoeGxvLU089xa5duxAREhMTU+e1atWKqKgoAH766Sd++ukn6tSpA1h3Rbt27aJp06aMHTuWWbNmAXDo0CF27dp1VaIYPXq0ZwcHrqjzSZHR8R09ejTz5s2jQYMGjBgxgn79+vHZZ5/xzTff8PTTT9O/f39WrFhBt27d+PPPPwkICKBo0aIcOXLE41huhm8min3zb271fWf4299m8fvvhwBo3boC48e3o2LFqOyITqlcIzQ0lA0bNhAbG0uHDh0YP348ffr0oXr16ixfvvyKZffu3Ut4eDgRERFUr16ddevWpRbrXMu1Eo77tPTP9OfPnz91uFGjRuzevZsTJ07www8/MHjwYACcTicrVqwgNPTazeGEhoZese0333yTFi1aMGvWLPbv30/z5s0z3Kcxhtdff53nn3/+iu0tW7aMxYsXs2LFCsLCwmjevHmG7yNk5Y6iZMmSHDp0KHU8JibmiuIjgBMnTrBx40YaNGgAWMmzbdu2AHz++ecsWLAg9VjFx8dz8uRJihYtSnx8fKbHJzv53lNPzsuwYZw1HBh2Q5soUCCYnTtPccst4cyY0YkFC57QJKH8WsGCBRk7diwjR44kMTGRJ554gl9//TW1KOPSpUv06dMntRhjwIAB/POf/2Tnzp2AdeIeNWrUVdtt3bo148aNSx1PKXoqVqwY27ZtSy1auhYR4aGHHqJfv35UrVo19eo9/XY3bNhw1bpVq1ZNfToIrDuKEiVKADBt2rRr7rNNmzZMnTo1tQ7l8OHDHD9+nNjYWAoVKkRYWBjbt29PLf5Jb/To0WzYsOGqn/RJAqBjx47MmDGDhIQE9u3bx65du6hfv/4VyxQqVIjY2NjUY71o0SKqVq0KQOnSpVmyZAkA27ZtIz4+PvUuaufOnVcUvXmTDyaK5LThxkM8Xm3hwt0kJCQBEB0dxuzZXdi+/UU6d66hL0WpPKFOnTrUqlWLGTNmEBoayo8//sj7779P5cqVuf3227nzzjvp3bs3ADVr1uTjjz+ma9euVK1alRo1anD06NGrtjl48GDOnDlDjRo1qFWrVuqV9rBhw+jQoQMtW7akePHimcbVuXNnpk+fnlrsBDB27FjWrl1LzZo1qVatGpMmTbpqvSpVqhAbG8u5c1ZXA6+++iqvv/46jRs3Jjk5+arlU7Ru3ZrHH3+cRo0acfvtt/PII49w7tw52rZtS1JSEjVr1uTNN9+8om7hRlWvXp3HHnuMatWq0bZtW8aPH59a7NauXTuOHDlCYGAgU6ZMoVOnTtSqVYuvvvqKESNGAPDRRx8xZcoUatWqRdeuXZk2bVrq+Wrp0qW0b9/+pmP0hGRUhpab1atd3aztthWiq8HTW667/KFDsfTps4AfftjOe++1YPDgpjkQpVLWFWDKlaHyjtGjRxMREZHr36XIbgkJCTRr1oxff/2VwMCraxAy+tsTkXXGmHo3sj/fu6OI3evRYklJTkaNWkHVquP54YfthIcHERWlzX8r5U9eeOEFgoOD7Q4jxx08eJBhw4ZlmCS8wfcqs5NclUuZ9D+xcmUMPXvOYePGYwB06lSVMWPaUqJEgZyIUCmVQ0JCQujWrZvdYeS4SpUqUalSpRzbn+8lCoC206BcxmVzq1bFcNddn2MMlC0bybhx99G+/W0ZLquUt2X2GKpS3uCN6gTfTBTlO0BodIaz6tcvQZs2FalT5xYGD25KWFj2dd6hVFaEhIRw6tQpbWpc5Rjj6o8iJCQkW7frm4nCza5dp3jllYWMGtWG226z/iHnzn2cgAD9x1T2KlmyJDExMZw4ccLuUFQektLDXXby2USRkJDEsGG/8sEHv5KQkExISCAzZz4GoElC5Qr58uXL1l7GlLKLV596EpG2IrJDRHaLyFVvo4hIsIj82zV/lYiU9WS7S5YeombNSbzzzi8kJCTTvXttJk3qkN3hK6WUwot3FCLiAMYDrYAYYI2IzDbGbHVbrAdwxhhTUUS6AB8Cna/eWpp9pyO5t/2PAFStWphJkzpoI35KKeVF3ryjqA/sNsbsNcZcBmYAD6Rb5gHg/1zDM4F75Dq1fmcuhhIS4uCf/2zJhg09NUkopZSXee3NbBF5BGhrjPm7a7wb0MAY09ttmT9dy8S4xve4ljmZblvPASkNw9cA/vRK0L6nMHDyukvlDXos0uixSKPHIk1lY0zE9Re7mjcrszO6M0iflTxZBmPMp8CnACKy9kZfQ/c3eizS6LFIo8cijR6LNCKy9kbX9WbRU+noAVAAAAd+SURBVAxQym28JJC+8fTUZUQkECgInPZiTEoppbLIm4liDVBJRMqJSBDQBZidbpnZwFOu4UeAn42vtVKolFJ+zmtFT8aYJBHpDSwEHMBUY8wWERmC1cn3bOBz4CsR2Y11J9HFg01/6q2YfZAeizR6LNLosUijxyLNDR8Ln2tmXCmlVM7yvWbGlVJK5ShNFEoppTKVaxOFt5r/8EUeHIt+IrJVRDaJyBIR8du3EK93LNyWe0REjIj47aORnhwLEXnM9bexRUS+zukYc4oH/yOlRWSpiKx3/Z+0syNObxORqSJy3PWOWkbzRUTGuo7TJhG5w6MNG2Ny3Q9W5fceoDwQBGwEqqVbphcwyTXcBfi33XHbeCxaAGGu4Rfy8rFwLRcBLAdWAvXsjtvGv4tKwHqgkGu8qN1x23gsPgVecA1XA/bbHbeXjkVT4A7gz2vMbwfMx3qHrSGwypPt5tY7Cq80/+GjrnssjDFLjTEXXaMr+f/27j1EyiqM4/j311XLMkKKbrRF98ysLKz+6KJFF7IScQutNqxIutDF/giDjPojsohKTStCgwpTrMQMi7AL4lYSpSV2QcWEqAiTKIvSX3+cs+20zc68u7mzs7PPBwZ2zsz7nmcOs+8z73lnnpN+s9KIirwvAB4EHgF+r2VwNVZkLG4EZtreAmD7hxrHWCtFxsJA2xKXg/nvb7oagu33qfxbtMuBF5y0AvtJOqjafus1URwCfFtyf3NuK/sc238BW4Hyqxn1bUXGotQk0ieGRlR1LCSdAhxme0ktA+sFRd4XxwDHSFohqVXSRTWLrraKjMU0YKKkzcBS4LbahFZ3uno8Aep3PYqdVv6jARR+nZImAiOAc3o0ot5TcSwk7QI8DrTUKqBeVOR9sRtp+ulc0lnmB5KG2v65h2OrtSJjcTUw1/Zjks4k/X5rqO0dPR9eXenWcbNezyii/Ee7ImOBpNHAVGCM7T9qFFutVRuLfUhFI9+VtJE0B7u4QS9oF/0fed32n7Y3AF+SEkejKTIWk4BXAGyvBAaQCgb2N4WOJx3Va6KI8h/tqo5Fnm6ZQ0oSjToPDVXGwvZW20NsN9luIl2vGWO728XQ6liR/5HXSF90QNIQ0lTU+ppGWRtFxmITMApA0vGkRNEf16hdDFybv/00Ethq+7tqG9Xl1JN7rvxHn1NwLKYDg4AF+Xr+Jttjei3oHlJwLPqFgmOxDLhQ0lpgO3CP7Z96L+qeUXAs7gaelXQnaaqlpRE/WEp6mTTVOCRfj7kf2B3A9mzS9ZlLgG+A34DrC+23AccqhBDCTlSvU08hhBDqRCSKEEIIFUWiCCGEUFEkihBCCBVFogghhFBRJIpQdyRtl/Rpya2pwnObOquU2cU+383VRz/LJS+O7cY+bpZ0bf67RdLBJY89J+mEnRznx5KGF9jmDkl7/d++Q/8ViSLUo222h5fcNtao3wm2TyYVm5ze1Y1tz7b9Qr7bAhxc8tgNttfulCjb45xFsTjvACJRhG6LRBH6hHzm8IGkT/LtrDLPOVHSR/ksZLWko3P7xJL2OZJ2rdLd+8BRedtReQ2DNbnW/565/WG1rwHyaG6bJmmKpHGkmlsv5j4H5jOBEZImS3qkJOYWSU91M86VlBR0k/S0pFVKa088kNtuJyWs5ZKW57YLJa3M47hA0qAq/YR+LhJFqEcDS6adXs1tPwAX2D4VaAaeLLPdzcATtoeTDtSbc7mGZuDs3L4dmFCl/8uANZIGAHOBZtsnkSoZTJa0P3AlcKLtYcBDpRvbXgisIn3yH257W8nDC4GxJfebgfndjPMiUpmONlNtjwCGAedIGmb7SVItn/Nsn5dLedwHjM5juQq4q0o/oZ+ryxIeod/blg+WpXYHZuQ5+e2kukUdrQSmSjoUWGT7a0mjgNOAj3N5k4GkpFPOi5K2ARtJZaiPBTbY/io/Pg+4BZhBWuviOUlvAIVLmtv+UdL6XGfn69zHirzfrsS5N6lcRekKZeMl3UT6vz6ItEDP6g7bjsztK3I/e5DGLYRORaIIfcWdwPfAyaQz4f8sSmT7JUkfApcCyyTdQCqrPM/2vQX6mFBaQFBS2fVNcm2hM0hF5q4CbgXO78JrmQ+MB9YBr9q20lG7cJykVdweBmYCYyUdAUwBTre9RdJcUuG7jgS8bfvqLsQb+rmYegp9xWDgu7x+wDWkT9P/IulIYH2ebllMmoJ5Bxgn6YD8nP1VfE3xdUCTpKPy/WuA9/Kc/mDbS0kXist98+gXUtnzchYBV5DWSJif27oUp+0/SVNII/O01b7Ar8BWSQcCF3cSSytwdttrkrSXpHJnZyH8IxJF6CtmAddJaiVNO/1a5jnNwOeSPgWOIy35uJZ0QH1L0mrgbdK0TFW2fydV11wgaQ2wA5hNOuguyft7j3S209FcYHbbxewO+90CrAUOt/1RbutynPnax2PAFNufkdbH/gJ4njSd1eYZ4E1Jy23/SPpG1su5n1bSWIXQqageG0IIoaI4owghhFBRJIoQQggVRaIIIYRQUSSKEEIIFUWiCCGEUFEkihBCCBVFogghhFDR30CkDmpVOc0WAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "eval_pool = Pool(X_validation, y_validation, cat_features=cat_features)\n", "curve = get_roc_curve(model, eval_pool)\n", "(fpr, tpr, thresholds) = curve\n", "\n", "plt.figure()\n", "lw = 2\n", "roc_auc = sklearn.metrics.auc(fpr, tpr)\n", "plt.plot(fpr, tpr, color='darkorange',\n", " lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)\n", "\n", "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n", "plt.xlim([0.0, 1.0])\n", "plt.ylim([0.0, 1.05])\n", "plt.xlabel('False Positive Rate')\n", "plt.ylabel('True Positive Rate')\n", "plt.title('Receiver operating characteristic')\n", "plt.legend(loc=\"lower right\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": [ "from catboost.utils import get_fpr_curve\n", "from catboost.utils import get_fnr_curve" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": [ "? get_fpr_curve" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8VfX5wPHPk5tJCCuAIkNQ2bKDA6viqHWiuHFU66xKrbM/cBVntWrr1rr3QIuK1lkUtQrWoIgCRVEZERAIOyTkJnl+f3zvTW5CxiXcc8+9N8+7r/O6Z+WcJ6d4n3zH+X5FVTHGGGMakuZ3AMYYYxKbJQpjjDGNskRhjDGmUZYojDHGNMoShTHGmEZZojDGGNMoSxTGGGMaZYnCJBwRWSQipSKyKWLZSUR6iohG7FskIhMifk5FpCR07GcR+ZuIBBq5z5kiUlnnPveFjj0Zut4eEefvJiIasT1dRMpCP7daRKaISBevnosxfrFEYRLVUaraOmJZFnGsnaq2BsYB14nIoRHHhoSO7Q+cBJzVxH1m1LnP+Ihja4Cbmvj58aH77Qa0Bu6I5pfbHiKS7vU9jIlkicIkLVWdAcwFdq/n2ELgU2DodtziKWCwiOwfRSzrgNcau5+I5IjInSKyWETWi8h/QvtGi0hRnXMXicjBofVJIvKKiDwrIhuAq0Ilrg4R5w8LlWoyQttnich8EVkrIu+KyM6h/SIifxeRlaEY5ojIVs/PmEiWKExSCn3h7QMMBL6q53g/YF9g4XbcZjNwC3BzFPHkA8c2cb87gBHAKKAD8CegKspYjgZeAdoBtwMzgOMijp8CvKKqQRE5BrgqFE8n4BPghdB5hwD7AX1C1zoJKI4yBtNCWaIwieo1EVkXWl6rc2w1rlroUWCCqk6LOPaliJQA84HpwANN3GeviPusE5G96hz/B9BDRA5r4OfvEZH1oZg6An+o7yQRScNVg/1RVX9W1UpV/UxVtzQRX9gMVX1NVatUtRR4Hlf1hogIcHJoH8D5wF9Udb6qVuCS3dBQqSII5AH9AAmdszzKGEwLZYnCJKpjVLVdaDmmzrGOqtpeVfur6j11jg3HtRWcBOwJ5AKIyL4RDdZzI86fGXGfdqo6M/JioS/yG0OL1BPnxaraFhgMtAe6NfD7dASygR+a/M3rt7TO9ivA3iKyE66EoLiSA8DOwN3h5IdLqgJ0VdUPgPuA+4FfRORhEWnTzJhMC2GJwqQcdSbjqmeuC+37JKLBeuA2XvIJoC0wtpF7foNr+L4/9Bd+XauBMmDXeo6VAK3CG6GeWp3q3qLO/dYB7wEn4qqdXtCaoaCXAufXSYA5qvpZ6GfvUdURuGq7PsCVDf1exoAlCpPabgXOE5Edt+cioeqbScD/NXHqU0BnYEw916gCHgf+FurqGxCRvUUkC/gOyBaRI0KN0dcAWVGE9jzwW1xbxfMR+x8CJorIQAARaSsiJ4TWR4rInqH7lOCSV2UU9zItmCUKk7JCf+V/RGz+Yn4BaLQuX1XLgXuAaxs45QrgG+ALXHXQbUCaqq4HLsS1ufyM+wIvauAakaYCvYFfVPXriDheDV37xVAvqW+BcBtLG+ARYC2wGNeQ7XmXXpPcxCYuMsYY0xgrURhjjGmUJQpjjDGNskRhjDGmUZYojDHGNCrpBhfr2LGj9uzZ0+8wjDEmqcyaNWu1qtZ9PycqSZcoevbsSWFhod9hGGNMUhGRxc39Wat6MsYY0yhLFMYYYxplicIYY0yjLFEYY4xplCUKY4wxjbJEYYwxplGeJQoReTw0L++3DRwXEblHRBaG5u0d7lUsxhhjms/L9yiexM2k9XQDxw/DDZHcGzcT2YOhz0Zt2ADvvw/p6RAIuCW8Xt++aI/XO9WMMcYkqdJgKd+v+Z5N5ZvYVL5pu67lWaJQ1Y9FpGcjpxwNPB2alWumiLQTkS5Nzd/7/fdwyCExDDREZPsSTeRnbi706AHdu7vPHj2gdeumY8jKgg4dID/frRtjTHOoKiMeHsH81fNjcj0/38zuSu15gItC+7ZKFCJyHnAeQHb2IPbZByoroaLCfUau17cvmuOqEAy6JRG0bu0SRnjp1Kkm8ey8s/ts29YluJ12cgnKGGMA5q6ay/zV88lJz2HIjkPIzchlGtOafT0/E0V9lT31zqKkqg8DDwMUFBTov/8d20BUoaoqdslnwwZYuhSWLHHL0qVQWtp0HKWlUFzslk2b3LI4ipfuR42CTz/d/udgjEkNHy36CIBj+x/Ls8c+C4Cc0fz6dT8TRRHQPWK7G7DMj0BEaqqRMjP9iKA2VZdswklj9WpYudIlnMWLXfJZvBg2boRly+Czz6CoCLp18ztyY0wiWLJ+CQADOw2MyfX8TBRTgfEi8iKuEXt9U+0TLYWIq1Zq2xZ22aXxc8eOhddec9VSV14JJ50EI0bEJ05jTGJaU7oGgPxW+TG5npfdY18AZgB9RaRIRM4Wkd+LyO9Dp7wF/AgsxE32fqFXsaSyBx6Avn3d+u23w8iRLlk8+yxMnQrLLfUa0+IUlxYD0CGnQ0yu52Wvp3FNHFfgIq/u31J06QJz58KHH8JDD8GUKTB5slsA0tLggANg331dW8b++ydG9ZoxxjvhEkXCJwoTP4EAHHywW374wSWLGTNcG8ZHH8G0aW4B6NoVrr4azj7bEoYxqWpt2VoA2me3j8n1LFGkmF13dW0VYcXFrrTx6afw9tuwYAFceCH8+c9w9NGwww5wySXQsaN/MRtjYmtzcDMArTOjeIErCjbWU4rLz4fjj4e//x3mzYObbnIvBK5aBY8+CjffDJ07w5YtfkdqjImVkvISAFpltIrJ9SxRtCBpaa7aafVqeP55uO46t18Vdt8dnngicV44NMY0X7hEYYnCNFt2NowbB9dfDy+/DL17w8KFcNZZsNtucO+9rsrKGJOcwokiNzM3JtcT1/koeRQUFGhhYaHfYaSUYBBefBH+8heYHxoaJhBwpYx+/eC3v609VlVeHgwdagMpGpOIgpVBMm/KJCABgtcGkdB/qCIyS1ULmnNNa8w2ZGTA6afDqafC66+7dzOmT4evv3bLSy9t/TP9+8Pdd8Ovfx33cI0xjYisdpIY/TVnicJUS0tzb3qPHQvr1sHs2fDcc66nVKSFC13J49BD4aijXNXV5ZfDjjv6E7cxpkZJ0DVkx6raCSxRmAa0awejR7ulrmDQ9Z666SZXAgF45RUoLHS9rIwx/ol1QzZYY7ZphowM1xA+Zw488ogb5nzRIjdPyIMPunVjjD+qG7IzYleisERhmm3gQDjnHDeuFMCXX7qX+Xr1gjFj4Isv/I3PmJYo1u9QgCUKEwMHHODe/L7kEjjhBNf99o03YI89XE+qJOtYZ0xSs6onk7BGjXJvf0+e7Bq/Tz7Z7b/qKjcg4Y8/+hufMS2FF43ZlihMzPXoAS+84LrVpqW50sbuu8PTT/sdmTGpz0oUJqmceKIbAv03v3HTvJ59Nrz3nt9RGZParDHbJJ1+/eDNN13jdkWFSxp77QXr1/sdmTGpyRqzTVJKT4dXX4XLLnPbn3/uEkZVlb9xGZOKrOrJJK20NLjzTje/d06OSxYdO8L991vCMCaWqhuzrerJJKujj3ZzYACsXQvjx8OkSb6GZExKsRKFSQmXXALff+/e7ga48UY47LCakWuNMc1nbRQmJYi4eS8uucQN+wHwzjswaJDrJWUv6BnTfJsrYjsXBViiMD5q0wbefde9Z5GeDpWV7n2LXr3cTHzTpvkdoTHJx6qeTEoaNcrNqHfGGS5hLF4Mt9ziqqN++cXv6IxJLuGqJ2vMNimnTRt48knYssUNAwJuOPPwgIPGmOhYicKkvLQ0N7Dgq6+67T//2U2gZIyJjiUK02IccwyccgqUlLg3uSdOdKUNY0zjSitKAcjJyInZNS1RmIT12GNuFNotW+DWW91wILfeCsuX+x2ZMYmrrKIMgOz07Jhd0xKFSVjZ2W7O7kcfha5d3cx5EydCz56uO60xZmvhRJGTbiUK00KkpblRZxcvhn/9Cw4/HMrLXY+oQYPgq6/8jtCYxFIadFVPWelZMbumJQqTFAIBlySmToUDD3T7vv0Whg+HQw+FVav8jc+YRGHDjJsWLxCAf//bvXdxxBFu37vvQu/ebjhzY1oyVbXGbGPADQHSoYObl/vaa10V1Pr1rlvtG2+4qVh//tnvKI2Jv8iG7DSJ3de7p4lCRA4VkQUislBEJtRzvIeIfCgiX4nIHBE53Mt4TGoRgRtugK+/hmOPhbIyN0FSv37QrRtMmeJ3hMbEV7jaKZYN2eBhohCRAHA/cBgwABgnIgPqnHYNMFlVhwEnAw94FY9JXSLw/PNw0UXQp49rAAc49VSbetW0LF68bAfelij2ABaq6o+qWg68CBxd5xwF2oTW2wLLPIzHpLCsLLjvPlftVF4ORx3lShgnnghr1vgdnTHxkYyJoiuwNGK7KLQv0iTgNBEpAt4C/lDfhUTkPBEpFJHCVda9xTQhEIAXX3TvW6xf797wtjm6TUuQjIlC6tlXd6aBccCTqtoNOBx4RmTrFhhVfVhVC1S1oFOnTh6EalJNq1bwz3+6aql3362Zr9uYVJaMiaII6B6x3Y2tq5bOBiYDqOoMIBvo6GFMpgUZPhz+8he3/sEH/sZiTDyEu8YmU6L4AugtIr1EJBPXWD21zjlLgIMARKQ/LlFY3ZKJmSuugLw8N/zHEUe4OS8uuww2bfI7MmNir7rXUwzfoQBIj+nVIqhqhYiMB94FAsDjqjpXRG4AClV1KnA58IiIXIqrljpT1SbCNLETCMDYsfD00/DWWzX7y8vh3ntd1ZQxqcKrqifPEgWAqr6Fa6SO3HddxPo8YB8vYzDmvvvcMB/l5bB2LVx6Kdx/vxuddto0N8OeMakgKROFMYkgLw/GjavZ7tzZvWNRVuZKG9984/YZk+yqE0V68rRRGJOQTjkFNm50b2+vXAk77FC7WsqYZJWMvZ6MSVitW8Mrr7g2DICbbvI3HmNiITzEuCUKY2Jkzz3dKLQAM2bUzNNtTLLyqteTJQrTorVtC/fc49avu67xc41JdFb1ZIxHzj3Xfc6bB5WV/sZizPawRGGMR7KzoVMnqKqCW25xDdzGJKPNFZYojPHM8OHu87rroGtX927FkiX+xmTMtrIShTEeuu8+uOACGDwYKipc4/bYsVYVZZKL9XoyxkO77QYPPACzZ8PU0IhkX37phio//ni46y5fwzMmKlaiMCYORNykR/ffDz16QFGRG6780kvhvPPcMCDGJKqkmwrVmGR24YXw00/wySdQUOD2PfII7LGH6x1lTCKyEoUxcZaWBr/6FXzxBfz3v6531Ndfu5KFMYnIEoUxPho5Ej791K1/+qlryzAm0ViiMMZnw4fDH//o1g89FJ59FoJBf2MyJlIyznBnTMq57jr3jsUvv8Dpp8OgQfDhh7B8ud+RmZZOVW2sJ2MSQYcO8PHHcOONkJkJCxbAgQfCTjvVdKs1xg9lFWUAZAWySJPYfrVbojBmGwUCcM01NfNwh40dC3Pm+BaWaeG8ap8ASxTGNFuXLvDmm1BaCjvv7MaK2ndfWL3a78hMS2SJwpgElp3tZsjLy4MNG+Cyy9w0q8bEkyUKYxLcgAE181k88wz07evGizImXrzq8QSWKIyJmfPPdwML9u3rRp496CCYNs3vqExLYSUKY5JAXp4bWPDbb+Gss1zbxcEHw4oVfkdmWgKvusaCJQpjYi493Y0LtfPObvvII13yMMZLVqIwJsmkpcGtt7r1WbPci3mPPmqN3MY7liiMSUInn+ze2g4791x44QX/4jGpzRKFMUlq9Gh4//2a7bPOgrlzfQvHpLDq2e3SfUoUIpIjIn1jfndjWoCDD4YpU2q2zz3Xplg1sedriUJEjgJmA++EtoeKiI1qY8w2GDsW3n7brc+YASNGwPr1/sZkUovfvZ4mAXsA6wBUdTbQM+aRGJPiDj3UTakKbgKkwYNdF1pjYsHvNooKVbW/fYyJgb/9zU2lmpHhXsqzxm0TK34nim9F5BQgICK9ReRe4LOYR2JMC9G/Pzz4oFt/7jl/YzGpw+9E8QdgILAFeB5YD/wxmouLyKEiskBEForIhAbOOVFE5onIXBF5PtrAjUlmxx0HWVnwwQeulLF5s98RmWTn91hPR6jq1ao6MrRcA4xp6odEJADcDxwGDADGiciAOuf0BiYC+6jqQOCSbf4NjElC7drB//2fW7/8cigosPYKs338LlFMjHJfXXsAC1X1R1UtB14Ejq5zzrnA/aq6FkBVV0ZxXWNSwqRJcNddIALz58O118KWLX5HZZKVl4kivaEDInIYcDjQVUTuiTjUBqiI4tpdgaUR20XAnnXO6RO616dAAJikqu9EcW1jkp4I/PGPsMsuMGYM3Hkn/OtfUFgIubl+R2eSTXX32PT4do9dBhQCZcCsiGUq8Jsori317NM62+lAb2A0MA54VETabXUhkfNEpFBECletWhXFrY1JHkceCWec4db/9z84+mj46Sd/YzLJx5eqJ1X9WlWfAnZT1acilinhqqImFAHdI7a74ZJP3XNeV9Wgqv4ELMAljrqxPKyqBapa0KlTpyhubUzyEIEnn3SDB+bmujkshg+Hl17yOzKTTPxuo+gpIq+Eeib9GF6i+LkvgN4i0ktEMoGTcaWRSK8BBwCISEdcVVQ01zYm5QwfDp9/7rrPrlvnBhX8z3/8jsokC797PT0BPIhrlzgAeBp4pqkfUtUKYDzwLjAfmKyqc0XkBhEJ95p6FygWkXnAh8CVqlq87b+GMalh4EA3xEefPm7797+Hr77yNyaTHLwsUYhq3WaDOieIzFLVESLyjaoOCu37RFX3jXk0USgoKNDCwkI/bm1M3BQXw157wcKFbvvYY+Gxx1y3WmPqk3tLLpuDm9k0cRO5mVv3hgh9lxc059rRlCjKRCQN+F5ExovIWKBzc25mjIlOfr5rqzjwQLc9ZYorZWza5G9cJjGpqu+DAl4CtAIuBkYApwNnxDwSY0wtPXq4ZDE11LK3ahVcfDFURNM53bQoZRVu6sSsQBZpEvtphpq8oqp+oaqbVLVIVX+nqscCy2MeiTGmXkcd5XpAicATT0BODnTq5KqnjAFv2yegiUQhInuLyPEi0jm0PTg0HpP1xTAmjk480ZUuevVyJYrVq12yuP9+vyMzicDLHk/QSKIQkduBx4HjgH+JyJ+B94HPqeddB2OMtw44AH74AT76CNq2BVUYP95NglRQUHvZf383hHkTfVVMivC6RNHgEB7AEcAwVS0Tkfa4l+UGq+r3nkRijGmSCOy3H6xZA3vvDf/9L3z5Zf3nfvwxvPYaPP20G6nWpC4/E0WpqpYBqOpaEVlgScKYxJCWBjNnwjffQHl57WNVVfDGG/CXv8DkyS653HuvO9a+PaQ39l+9SUpe9niCxhPFrnXmxu4Zua2qTQ41bozxjoibTrU+e+wBxxwDo0a5hvDwcCBdusAzz8BBB8UvTuM9P0sUdYcEv9OTCIwxnhgxws2g96c/wcaNrgF8+XI4+GDX3vH665CX53eUJhZ8SxSq+pEndzTGxM3xx7sF3MRIhx8O06fDhx+6brdnngmBABxyCOywg5+Rmu1RGvS215PVVhrTQuTkuASxYIGrkvroI7eEjR8Pp58OI0e6ai2TPHx9j8IYk3r69nU9pcaPh9NOg512cvvvuw/23BN2373hnlQmMVUninR/XrgLhN6nMMakkJ13dj2hnnkGiopc76hDD3XH5s1zM+6ttImJk4afvZ5Q1UoRGSEiok0NM2uMSUoicMIJblm0yL2nsXSpa7M47zxoFfojtXVruPRS18XWqqYSi5+9nsK+Al4XkZeBkvBOVZ3iSUTGGN/07AnvvOO61n7/PTz8cO3jN93kutjecQeccoovIZp6JEKi6AAUAwdG7FPAEoUxKWjAAPjuOze73hdfuH3LlsGjj8KGDa6L7amnurfCR4xwVVY2Q7G/vB7rqclEoaq/8+TOxpiE9qtfuSXs9tvd2FF33glXXgl33+32t2kDZ50F55zjZugz8ed7rycR6SYir4rIShH5RUT+KSLdPInGGJPQROCKK+Ddd11iGD3alTLuuguGDIFrrvE7wpYpEaqengCeB04IbZ8W2vdrTyIyxiS8Qw5xC0BhoRvu/Mkn4eaboV8/2LfORMki0K2bG6PKxJ7vJQqgk6o+oaoVoeVJwGokjTGAG9b88cdr3uw+/XTXKB657LwzHHecDXvulerusenedI+NJlGsFpHTQu9UBETkNFzjtjHGAK7EMH26G6SwR4+tl/R0N+R5z56u2srEViKUKM4CTgRW4KZAPT60zxhjqvXrB19/DYsXb73ceKMbU2rJEtdLqlUr6NwZJk6E9ev9jjz5ef3CXZNvZgPHqeoYVe2kqp1V9RhVXexJNMaYlDRhApSVwQ03QGamG6Bw1Sq49Vb3XsZ++7k3wk3zFJe6Sp4OOR08uX6jiUJVK9l6uHFjjNlm6elw7bWul9SaNXDddW5sqdJS+OQTN2PfqFG1lwkT/I46Oawvc8Wy9tntPbm+NDUyh4jcDLQFXqL2m9m+DBtWUFCghYWFftzaGBNjqrBwIVx2Gbz5Zv3nHHCAGzrksMNsdr76qCqBGwIoSvDaIOlp9T8kEZmlqgXNuUc0j31U6POGyNio/aa2McZsMxHo3RumTnXtG5s31xx7/XU3VMiHH7qlTRs46SR3rFMnl1zS093+ljz21IYtG1CU3IzcBpPE9mr0qiKSBjyoqpM9ubsxxuC+6IcOrb1v1Cg3sdJ998EDD7gqq0ceqTl+yy3uMz8fVqxouaWNdWXrAGif4021EzTdRlEFjPfs7sYY04j+/d3LfN98A//4h1tOOQU6dKiZxrW4GO65x984/VSdKDxqn4Douse+LyJXiEh3EekQXjyLyBhj6th9dzfk+XnnuXnAi4tdCeO229zx997zNz4/rS1bC0C77Hae3SPa9yguAj4GZoUWa002xvhu7Fj32ZJf4otH1VM0o8f28uzuxhizHXbZBbKyYMsW18Zx+umud9Quu0B2tt/RxUc4UbTNauvZPRosUYjInyLWT6hz7BbPIjLGmCgFAu4lPnC9pq64wg11npMDF1wAn34Kq1fXLGvX+huvFzZs2QBAm6w2nt2jsaqnkyPWJ9Y5dqgHsRhjzDb705/ghx9cI3f//jX7H3rIzafRqVPN0qGDSyIffOBfvLEWLlH41UYhDazXt13/BUQOFZEFIrJQRBp8x1JEjhcRFZFmvQxijGnZdtnFNXLPm+dm53v0UfjNb1zX2fASVlYGBx3khhJ5/nn/Yo4VX6uecC/V1bde3/ZWQuNE3Q8cBgwAxonIgHrOywMuBj5vMlpjjGlC795w9tlu7u/IaidVePllV10FEAy6KV3bt4f993dDiSQjv9+jGCIiG0RkIzA4tB7eHhTFtfcAFqrqj6paDrxI/eNG3Qj8FSjb1uCNMWZbHH88VFTAL7+4HlPp6bBuHXz8sauW+vlnvyPcduu3uHGefClRqGpAVduoap6qpofWw9sZUVy7K7A0YrsotK+aiAwDuqtqA6O8VJ93nogUikjhqlWrori1McY0rHNnmDLFlTQuu8ztKyuD7t3dm+DffgtVVf7GGK1wY3bbbH+qnrZXfe0Y1VVWoeFB/g5c3tSFVPVhVS1Q1YJOnWxyPWNMbLRtC3fe6Zbu3V311B/+AIMGuVn5IseeSlThkWP96vW0vYqA7hHb3YBlEdt5wO7AdBFZBOwFTLUGbWNMvF12mStFHHaYSx4ARUWQm+vmBE9kXs9FAd4mii+A3iLSS0Qycd1tp4YPqup6Ve2oqj1VtScwExijqgn+f4sxJhW1aQNvveXaLK6/vmb/yJGJ3di9pnQNAB1bdfTsHp4lClWtwA0o+C4wH5isqnNF5AYRGePVfY0xZntddx1Mm1az/fHHbvrW//7Xv5jqU6VVSV/1hKq+pap9VHVXVb05tO86VZ1az7mjrTRhjEkUBx7o2izOPrtm36RJvoVTr41bNqIoeZl5ns1FAR4nCmOMSXaPPuoauwHefhu6doXbb3dJxG/xGDkWLFEYY0yTzjrLdakFWLbMDRty5JGuwdvPhLG21CUKL1+2A0sUxhjTpHbtXIKYMaNm31tvuS61Q4f6N9hguETh5aRFYInCGGOiEgjAXnu5CZMGD3ZVUABz5sA55/gTk5UojDEmAeXluSHNi4rg2mvdvilTXOnikUdg6dLGfz6Wwl1jrURhjDEJatIkGBAa6rSoyE3VOnx4/O4fThRevmwHliiMMabZ0tJg7lxXojj2WLdv9WqYODE+jdzht7K9fNkOLFEYY8x2GzsW/vlPN2Q5wK23uiTy9NPe3nfVZjdIan5OfhNnbh9LFMYYEyNz59bePuMMuOQSWLLEmxLG6s2rAeiU6+1gqZYojDEmRrp0cfNdvP9+zb6773Yj0WZkwAMPxHb48lUlrkTRqZUlCmOMSRqBABx8MHz1FZxwQs3+ykq46CK44ILY3ctKFMYYk8SGDoXJk12V0/vvQ7dubv/DD8PlTc7CE51wG4U1ZhtjTJI7+GA330XY3/7mZtTbHuWV5WzYsoGABGysJ2OMSQVt29aeMS8nB554ovnXC1c75bfKJ028/Sq3RGGMMXGSk1O72umss1yyaE6PqOr2CY8bssEShTHGxNUdd7ghQMLOOgvOP3/brxPu8eR1+wRYojDGmLgbPNiNRivith95BDp2rN2O0ZQVm1YAsEPrHTyIsDZLFMYY44MuXSAYhGHD3HZxMQwaFH01VNGGIgC65XXzKMIaliiMMcYngQDMmgXPPFOzb/Lk6H72540/A9CtjSUKY4xJaSJw2mkwZIjbHj8+up+rLlFYojDGmJbh+uvd5+rVMHt20+eHE0XXNl09jMqxRGGMMQngqKNq1v/1r6bPt6onY4xpYdLSakoV11wD33zT8LnByiDLNy5HELq07uJ9bJ7fwRhjTFSOP75m/eKLGz5vxaYVKMoOrXcgI5DheVyWKIwxJkEMGAAPPujWp09veEjyeFY7gSUKY4xJKOecU7Me2W02Ujx7PIElCmOMSSjp6e6ujTJaAAAUXUlEQVTFO4Azz3TzWNS1ZP0SALrmed/jCSxRGGNMwpkypWZ91qytj3+z0rV09+vYLy7xWKIwxpgEs9tu0LmzW7/99q2Pz1rmsseILiPiEo8lCmOMSUCnnOI+33qr9v5gZZB5q+YhCEN2HBKXWCxRGGNMArr0Uve5eXPtUWV/XPsjlVpJ97bdaZXRKi6xWKIwxpgE1L17zfo999Ssh9snBnUeFLdYPE0UInKoiCwQkYUiMqGe45eJyDwRmSMi00RkZy/jMcaYZCECkya59X//u2b/nF/mADB4h8Fxi8WzRCEiAeB+4DBgADBORAbUOe0roEBVBwOvAH/1Kh5jjEk2v/qV+/zpJzd3BcDXv7jp8VIiUQB7AAtV9UdVLQdeBI6OPEFVP1TV8HTjM4H4vD1ijDFJYP/9a9YzM2HZMmVm0Uwgfj2ewNtE0RVYGrFdFNrXkLOBt+s7ICLniUihiBSuWrUqhiEaY0ziSk+HK6+s2R530Y+sLFlJp1ad2K3DbnGLw8tEIfXsq3eSPxE5DSgA6ukxDKr6sKoWqGpBp06dYhiiMcYktr/+FS680K1/vPgjAPbpsQ8i9X3FesPLRFEERLTb0w1YVvckETkYuBoYo6pbPIzHGGOS0i23hFZ6fQDA/j0OjOv9vUwUXwC9RaSXiGQCJwNTI08QkWHAP3BJYqWHsRhjTNJq2xbmzNHqRJFRFN9Eke7VhVW1QkTGA+8CAeBxVZ0rIjcAhao6FVfV1Bp4OVSMWqKqY7b1XsFgkKKiIsrKymL4GySG7OxsunXrRkaG92POG2MSV0aXBZC3HDZ15rYrBnDRCfG7t2eJAkBV3wLeqrPvuoj1g2Nxn6KiIvLy8ujZs2dc6+28pqoUFxdTVFREr169/A7HGOOjD35ypQl+OhA0vt9zKfFmdllZGfn5+SmVJABEhPz8/JQsKRljts37P77vVn46kKVLXXfZ996Lz71TIlEAKZckwlL19zLGRG9zcDPv/+ASxR75vwHcC3hPPRWf+6dMojDGmFT16ZJPKQmWMLzLcGa+24PbbnP7FyyIz/0tUcRIIBBg6NCh1cuiRYuYPn06bdu2ZdiwYfTv35/rr78eoNb+fv36ccUVV/gcvTEmkb353ZsAjN55NCIwIvRSdn2TGnnB08bsliQnJ4fZs2fX2rdo0SL23Xdf3nzzTUpKShg6dChHHnkkQPX+0tJShg0bxtixY9lnn338CN0Yk8Aqqip4/tvnARjbfywA/fvXHN+wAdq08TaGlCtRiHizbK/c3FxGjBjBDz/8UGt/Tk4OQ4cO5eeff97+mxhjUs5/lvyH1ZtX07tDb/bp7v6Y3GmnmuPjx3sfQ8olCr+UlpZWVzuNHTt2q+PFxcXMnDmTgQMH1tq/du1avv/+e/bbb794hWqMSSJT5rsJtMf2G1urc8u4ce5z2jTvY0i5qietdzQp79VX9QTwySefMGzYMNLS0pgwYQIDBw5k+vTpfPLJJwwePJgFCxYwYcIEdtxxRx+iNsYksoqqCl6a+xIAJwys/YbdJZfACy/AsmVuFrxWHk52l3KJItGE2yIa2v/dd9/xq1/9irFjxzJ06FAfIjTGJKo3v3uTlSUr6ZPfZ6thxUeOrFn//nsY4uH02Vb15LM+ffowceJEbgv3dzPGGFxp4qppVwFwQcEFW71TJQLhGusXXvA2FksUCeD3v/89H3/8MT/99JPfoRhjEsSTs59k/ur57NJ+Fy4ouKDecyoq3Gd49juviPpVqd9MBQUFWlhYWGvf/Pnz6R/ZXyzFpPrvZ4yprbyynD739mHx+sU8f+zzjBs0rt7zHn0Uzj3XrTf1VS4is1S1oDnxWInCGGMSzGNfPsbi9Yvp37E/Jw48scHz+vWrWfdySDhLFMYYk0Bmr5jN5e9dDsD1o68nkBZo8NxRo2rWv//eu5gsURhjTIIoDZZy+qunU1pRynH9j+P4Acc3en5aGhSEKpM+/NC7uCxRGGNMAlhftp4jnj+Cb1d+S5/8Pjx1zFNRjR7doYP7nDDBu9gsURhjTAL4w9t/4MNFH9KxVUemnDiF3MzcqH7unHPcZ2lpTS+oWLNEYYwxPlJVLn3nUp6Z8wyZgUw++d0nDOw8sOkfDDkh4oXtF1/0IEAsUcRMQ8OMiwhvvPFG9XlHHnkk06dPB2D06NH07duXIUOGMHLkyHqHADHGpK6FaxZyzEvHcNfnd5GRlsHjYx6nX8d+Tf9gHeHe8w89FOMAQyxRxEh4rKfw0rNnTwC6devGzTff3ODPPffcc3z99ddceOGFXHnllXGK1hjjp+LNxZz9+tn0ubcPUxdMJS8zjzfGvcGpg09t1vXOP999LlkSwyAjpNxYT3K9N1OH6p+b92LikCFDCAaDvP/++/z6179u8Ly9996b22+/vbnhGWOSQGmwlLtm3sUdM+5gTeka0tPSOXXQqdx4wI10b9u92dc95hg3SKBXsxWkXKLwS3iYcYBevXrx6quvVh+75ppruOaaaxpNFO+88w7HHHOM53EaY+JLVfli2Rc8O+dZnp3zLGvL1gIwuudoHjriIfp27Lvd98jPd59VVVBZCYGGX71olpRLFM39y397NTTMOLiRYsENOV7XqaeeSklJCZWVlXz55ZeexmiMiZ9N5Zt4Z+E73P353fxnyX+q94/oMoJr97uWMX3HRNX9NRqtW9esL1gAAwbE5LLVUi5RJKqrr76am2++mfT02o/8ueeeY8iQIUyYMIGLLrqIKVOm+BShMWZ7banYwmdLP+PleS8zee5kikuLAcjPyee3Q37LaYNPY3iX4Z7cOz8fiovhnXcsUSStQw45hGuvvZZly5ZtdSwjI4ObbrqJXXfd1QYANCZJLN+4nP+t/h/fFX/Hl8u/pHB5IfNXzae0orT6nBFdRnDiwBM5f8T5tM1u62k8gwe7t7O9GMrDEkUcXX311Rx99NH1HsvJyeHyyy/njjvu4LHHHotzZMaY+myp2MLSDUuZv2o+3xV/x7KNy/hh7Q/MWj6Log1F9f7MwE4DOarPUZy8+8kM3mFwzKqXmnLccS5RvPNO7K9tiSJGNm3atNW+0aNHM3r06OrtMWPGEDmse/h9irDLL7/cq/CMMQ1YWbKSxesWM3/1fH7e8DM/b/yZBcUL+K74O5auX4pSf7tnm6w2DOo8iN75vemX349R3Uexe+fdaZ/TPs6/gdM91Glq8+bYX9sShTEmpVRWVbJs4zJWlqxkTeka1patZW3pWtaWrWVVySqWbFjCik0rWL5xOSs2raAkWNLgtdIkjR5tetCrfS8GdR5Etzbd2ClvJ4Z3GU6/jv1Ik8R5FW3QIPe5cmXsez5ZojDGJDRVpayijNWbV/NLyS+sLFnJqpJVrNq8yq2HPleWrGTZxmWs2LSCKq2K+vp5mXns1mE3erbrSe8Ovemc25m+HfvSJ78PPdv1JDOQ6eFvFzu9etWsv/02HHlk7K6dMolCVeNWFxhPyTYDoUldVVpFWUUZFVUVVFRVUFlVWb1eUVXBlsotbA5upqS8xH0GS2pth5eSYAmbyjdRXFpMSXkJZRVl1cuWyi21tssqyiivLN/mWHfI3YEueV3okNOB9tntqz/b57Rn57Y70yWvC11ad2HH1jvSJqtNynx39OsH//sf/P3vlii2kp2dTXFxMfn5+Snzfzi4JFFcXEx2drbfoZgkUllVSWlFaa0v583BzZQGSymtKHVfyBXuC3lT+SY2lm9k45aN1Z/hc8oqytiwZQPFpcUUby5mTekaKrXSl98pM5BJx1Yd6ZzbuXrp1KpTrc/OuZ3ZKW8ndmi9Q9KUAmLtt7+Fq66CDz6ADRugTZvYXDcl5swOBoMUFRVR5uVcgD7Jzs6mW7duZGRk+B2KiZEqraI0WFr913VJeUnUn5srNjd4PJwMtlRu8Sz27PRs0tPSay0BCZCelk5mIJPczFxyM3JpldGK3Ez32Sq9Va3t3IxccjNz6ZDTgTZZbcgKZJGdnr3VkpXu9mcGMhOqLSCRbdgAbUO9cKdOhaOOqjm2PXNmp0SJIiMjg16RFXTGNFOwMljvl3BpsOav7MgqlHC1ypbKLVRUVRCsDLrPqiDBqiAbt2xkw5YN1cvasrWsK1vn+e/RKqNVvUt2enb1F3NWehatM1qTl5VHXmZe9WduZm71l3XrzNbk5+ST3yqfDjkdWuxf6smiTRv3PsWcOfDtt7UTxfbwNFGIyKHA3UAAeFRVb61zPAt4GhgBFAMnqeoiL2MyiaFKq6q/WMN10+HqkGBVsFbdd2RdeLAqWKvqJFynXV5ZXn29YFWQYGWw+lhpRWl1tUv1l3x5CeWV5dXnhuvNm1Mf3hzZ6dm1/rqu97OxY3X+cg/va5XRiqxAVkpVwZptM3KkSxRXXQUTJ8bmmp4lChEJAPcDvwaKgC9EZKqqzos47WxgraruJiInA7cBJzV23c3BzRQuc1VPdavN6vZ3TuXjqkqwKkhpsJRgVZAqraKyqpIqrXLrGrEe2l+pEV+4ob98K7Wy1vHKqsrqnw2vV1ZVVn95V39G/uVc2fCx8P3CX+7h/dvSKyWe0iStwS/k8F/ZORk51V/k4S/qrEAWGYEM0tPSyUgLfQYyaJ3ZmrZZbWmT1aZ66ZDTgUBajEdtMybk5JMh/M7uZ59BVtb2X9OzNgoR2RuYpKq/CW1PBFDVv0Sc827onBkikg6sADppI0HJTqKc70nIJs7CX6iR9dHhL9yMtAwCaYGt6sIzAhk1ddiBmp/JDGRWfzlnpGWQEcggM5BJTnoOORk51Z/hKpjWma1rnZeRlkG77HZkp2fbX+Mm6dX/Tzgx2yi6AksjtouAPRs6R1UrRGQ9kA+sjjxJRM4DzgttbmES33oScfLpSJ1nlUyCof+VUtr0yU1L6mcRY/YsatizqNHs8cy9TBT15bS6JYVozkFVHwYeBhCRwuZmxVRjz6KGPYsa9ixq2LOoISKFTZ9VPy/7nBUBkVM2dQPqDp1afU6o6qktsMbDmIwxxmwjLxPFF0BvEeklIpnAycDUOudMBc4IrR8PfNBY+4Qxxpj486zqKdTmMB54F9c99nFVnSsiNwCFqjoVeAx4RkQW4koSJ0dx6Ye9ijkJ2bOoYc+ihj2LGvYsajT7WSTdm9nGGGPiy96LN8YY0yhLFMYYYxqVsIlCRA4VkQUislBEJtRzPEtEXgod/1xEesY/yviI4llcJiLzRGSOiEwTkZ39iDMemnoWEecdLyIqIinbNTKaZyEiJ4b+bcwVkefjHWO8RPHfSA8R+VBEvgr9d3K4H3F6TUQeF5GVIlLvu2bi3BN6TnNEZHhUF1bVhFtwjd8/ALsAmcDXwIA651wIPBRaPxl4ye+4fXwWBwCtQusXtORnETovD/gYmAkU+B23j/8uegNfAe1D2539jtvHZ/EwcEFofQCwyO+4PXoW+wHDgW8bOH448DbuHba9gM+juW6ilij2ABaq6o+qWg68CBxd55yjgadC668AB0lqjr3Q5LNQ1Q9VNTxT7kzcOyupKJp/FwA3An8FUm/c+RrRPItzgftVdS2Aqq6Mc4zxEs2zUCA8O0Nbtn6nKyWo6sc0/i7a0cDT6swE2olIl6aum6iJor7hP7o2dI6qVgDh4T9STTTPItLZuL8YUlGTz0JEhgHdVfXNeAbmg2j+XfQB+ojIpyIyMzSacyqK5llMAk4TkSLgLeAP8Qkt4Wzr9wmQuPNRxGz4jxQQ9e8pIqcBBcD+nkbkn0afhYikAX8HzoxXQD6K5t9FOq76aTSulPmJiOyuqt5PiBFf0TyLccCTqnpnaMDSZ0LPIjGHMfZOs743E7VEYcN/1IjmWSAiBwNXA2NU1bspzvzV1LPIA3YHpovIIlwd7NQUbdCO9r+R11U1qKo/AQtwiSPVRPMszgYmA6jqDCAbN2BgSxPV90ldiZoobPiPGk0+i1B1yz9wSSJV66GhiWehqutVtaOq9lTVnrj2mjGq2uzB0BJYNP+NvIbr6ICIdMRVRf0Y1yjjI5pnsQQ4CEBE+uMSxaq4RpkYpgK/DfV+2gtYr6rLm/qhhKx6Uu+G/0g6UT6L24HWwMuh9vwlqjrGt6A9EuWzaBGifBbvAoeIyDygErhSVYv9i9obUT6Ly4FHRORSXFXLman4h6WIvICrauwYao/5M5ABoKoP4dpnDgcWApuB30V13RR8VsYYY2IoUauejDHGJAhLFMYYYxplicIYY0yjLFEYY4xplCUKY4wxjbJEYVoMEckXkdmhZYWI/BxaXxfqQhrr+40WkW0aSkREptf3gqCInCki98UuOmOiZ4nCtBiqWqyqQ1V1KPAQ8PfQ+lCgyaEcQiMAGNPiWKIwxgmIyCOheRveE5EcqP4L/xYR+Qj4o4h0EpF/isgXoWWf0Hn7R5RWvhKRvNB1W4vIKyLyPxF5LjzCsYgcFDrvm9AcAll1AxKR34nId6F77xOn52DMVixRGOP0xg3JPRBYBxwXcaydqu6vqncCd+NKIiND5zwaOucK4KJQCWVfoDS0fxhwCW4OhF2AfUQkG3gSOElVB+FGSLggMpjQ0M/X4xLEr0M/b4wvLFEY4/ykqrND67OAnhHHXopYPxi4T0Rm48bNaRMqPXwK/E1ELsYllorQ+f9V1aLQKKWzQ9ftG7rfd6FznsJNOBNpT2C6qq4KzbHwEsb4xOpcjXEiR9ytBHIitksi1tOAvVW1lNpuFZF/4cbRmRkazbe+66ZT/1DP9bHxdUxCsBKFMdvmPWB8eENEhoY+d1XVb1T1NqAQ6NfINf4H9BSR3ULbpwMf1Tnnc2B0qKdWBnBCrH4BY7aVJQpjts3FQEFoYvp5wO9D+y8RkW9F5Gtc+0SDswyqahlu1M6XReQbXI+rh+qcsxw3K9sM4N/Al7H+RYyJlo0ea4wxplFWojDGGNMoSxTGGGMaZYnCGGNMoyxRGGOMaZQlCmOMMY2yRGGMMaZRliiMMcY06v8B4mO7HDDPik8AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.figure()\n", "lw = 2\n", "(thresholds, fpr) = get_fpr_curve(curve=curve)\n", "(thresholds, fnr) = get_fnr_curve(curve=curve)\n", "plt.plot(thresholds, fpr, color='blue', lw=lw, label='FPR')\n", "plt.plot(thresholds, fnr, color='green', lw=lw, label='FNR')\n", "\n", "plt.xlim([0.0, 1.0])\n", "plt.ylim([0.0, 1.05])\n", "plt.xlabel('Threshold')\n", "plt.ylabel('Error Rate')\n", "plt.title('FPR-FNR curves')\n", "plt.legend(loc=\"lower left\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.4719014981770065\n", "0.9876472944285266\n" ] } ], "source": [ "from catboost.utils import select_threshold\n", "\n", "print(select_threshold(model=model, data=eval_pool, FNR=0.01))\n", "print(select_threshold(model=model, data=eval_pool, FPR=0.01))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Snapshotting" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [], "source": [ "#!rm 'catboost_info/snapshot.bkp'\n", "\n" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.3019732\ttest: 0.3022828\tbest: 0.3022828 (0)\ttotal: 67ms\tremaining: 1.27s\n", "1:\tlearn: 0.2238474\ttest: 0.2233617\tbest: 0.2233617 (1)\ttotal: 148ms\tremaining: 1.33s\n", "2:\tlearn: 0.1904507\ttest: 0.1818139\tbest: 0.1818139 (2)\ttotal: 252ms\tremaining: 1.43s\n", "3:\tlearn: 0.1802458\ttest: 0.1659696\tbest: 0.1659696 (3)\ttotal: 363ms\tremaining: 1.45s\n", "4:\tlearn: 0.1754740\ttest: 0.1587777\tbest: 0.1587777 (4)\ttotal: 485ms\tremaining: 1.46s\n", "5:\tlearn: 0.1703625\ttest: 0.1514527\tbest: 0.1514527 (5)\ttotal: 566ms\tremaining: 1.32s\n", "6:\tlearn: 0.1684087\ttest: 0.1485265\tbest: 0.1485265 (6)\ttotal: 654ms\tremaining: 1.22s\n", "7:\tlearn: 0.1672054\ttest: 0.1477918\tbest: 0.1477918 (7)\ttotal: 741ms\tremaining: 1.11s\n", "8:\tlearn: 0.1662518\ttest: 0.1470028\tbest: 0.1470028 (8)\ttotal: 842ms\tremaining: 1.03s\n", "9:\tlearn: 0.1656720\ttest: 0.1469340\tbest: 0.1469340 (9)\ttotal: 929ms\tremaining: 929ms\n", "10:\tlearn: 0.1646958\ttest: 0.1458786\tbest: 0.1458786 (10)\ttotal: 1.02s\tremaining: 832ms\n", "11:\tlearn: 0.1639292\ttest: 0.1456439\tbest: 0.1456439 (11)\ttotal: 1.1s\tremaining: 737ms\n", "12:\tlearn: 0.1631213\ttest: 0.1453225\tbest: 0.1453225 (12)\ttotal: 1.18s\tremaining: 638ms\n", "13:\tlearn: 0.1628037\ttest: 0.1449395\tbest: 0.1449395 (13)\ttotal: 1.25s\tremaining: 538ms\n", "14:\tlearn: 0.1626140\ttest: 0.1450753\tbest: 0.1449395 (13)\ttotal: 1.32s\tremaining: 440ms\n", "15:\tlearn: 0.1614957\ttest: 0.1448378\tbest: 0.1448378 (15)\ttotal: 1.42s\tremaining: 356ms\n", "16:\tlearn: 0.1614173\ttest: 0.1448587\tbest: 0.1448378 (15)\ttotal: 1.51s\tremaining: 267ms\n", "17:\tlearn: 0.1614154\ttest: 0.1448968\tbest: 0.1448378 (15)\ttotal: 1.61s\tremaining: 179ms\n", "18:\tlearn: 0.1613764\ttest: 0.1448062\tbest: 0.1448062 (18)\ttotal: 1.65s\tremaining: 87ms\n", "19:\tlearn: 0.1612640\ttest: 0.1445666\tbest: 0.1445666 (19)\ttotal: 1.72s\tremaining: 0us\n", "\n", "bestTest = 0.1445666012\n", "bestIteration = 19\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#!rm 'catboost_info/snapshot.bkp'\n", "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=20,\n", " save_snapshot=True,\n", " snapshot_file='snapshot.bkp',\n", " snapshot_interval=1,\n", " random_seed=43\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=True\n", ")" ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "902ccb0a2fef4f25b41b4697f325192a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "20:\tlearn: 0.1607445\ttest: 0.1440626\tbest: 0.1440626 (20)\ttotal: 1.82s\tremaining: 17.1s\n", "21:\tlearn: 0.1606770\ttest: 0.1439965\tbest: 0.1439965 (21)\ttotal: 1.9s\tremaining: 15.8s\n", "22:\tlearn: 0.1606616\ttest: 0.1440083\tbest: 0.1439965 (21)\ttotal: 1.97s\tremaining: 14.8s\n", "23:\tlearn: 0.1606540\ttest: 0.1440366\tbest: 0.1439965 (21)\ttotal: 2.04s\tremaining: 14s\n", "24:\tlearn: 0.1606450\ttest: 0.1440838\tbest: 0.1439965 (21)\ttotal: 2.11s\tremaining: 13.6s\n", "25:\tlearn: 0.1604540\ttest: 0.1439186\tbest: 0.1439186 (25)\ttotal: 2.22s\tremaining: 14.3s\n", "26:\tlearn: 0.1600134\ttest: 0.1436727\tbest: 0.1436727 (26)\ttotal: 2.33s\tremaining: 15s\n", "27:\tlearn: 0.1600132\ttest: 0.1436712\tbest: 0.1436712 (27)\ttotal: 2.4s\tremaining: 14.5s\n", "28:\tlearn: 0.1597968\ttest: 0.1437510\tbest: 0.1436712 (27)\ttotal: 2.49s\tremaining: 14.6s\n", "29:\tlearn: 0.1597306\ttest: 0.1437718\tbest: 0.1436712 (27)\ttotal: 2.62s\tremaining: 15.3s\n", "30:\tlearn: 0.1595170\ttest: 0.1437298\tbest: 0.1436712 (27)\ttotal: 2.77s\tremaining: 16.1s\n", "31:\tlearn: 0.1593775\ttest: 0.1439295\tbest: 0.1436712 (27)\ttotal: 2.88s\tremaining: 16.2s\n", "32:\tlearn: 0.1589895\ttest: 0.1438260\tbest: 0.1436712 (27)\ttotal: 2.99s\tremaining: 16.2s\n", "33:\tlearn: 0.1589845\ttest: 0.1438430\tbest: 0.1436712 (27)\ttotal: 3.04s\tremaining: 15.6s\n", "34:\tlearn: 0.1589319\ttest: 0.1438151\tbest: 0.1436712 (27)\ttotal: 3.12s\tremaining: 15.4s\n", "35:\tlearn: 0.1588918\ttest: 0.1438180\tbest: 0.1436712 (27)\ttotal: 3.19s\tremaining: 15s\n", "36:\tlearn: 0.1588731\ttest: 0.1438125\tbest: 0.1436712 (27)\ttotal: 3.24s\tremaining: 14.6s\n", "37:\tlearn: 0.1588159\ttest: 0.1438478\tbest: 0.1436712 (27)\ttotal: 3.32s\tremaining: 14.4s\n", "38:\tlearn: 0.1588010\ttest: 0.1438314\tbest: 0.1436712 (27)\ttotal: 3.4s\tremaining: 14.2s\n", "39:\tlearn: 0.1586945\ttest: 0.1436908\tbest: 0.1436712 (27)\ttotal: 3.49s\tremaining: 14.1s\n", "40:\tlearn: 0.1586894\ttest: 0.1436654\tbest: 0.1436654 (40)\ttotal: 3.56s\tremaining: 13.9s\n", "41:\tlearn: 0.1582296\ttest: 0.1432243\tbest: 0.1432243 (41)\ttotal: 3.66s\tremaining: 13.9s\n", "42:\tlearn: 0.1582208\ttest: 0.1432282\tbest: 0.1432243 (41)\ttotal: 3.71s\tremaining: 13.6s\n", "43:\tlearn: 0.1582196\ttest: 0.1432159\tbest: 0.1432159 (43)\ttotal: 3.78s\tremaining: 13.4s\n", "44:\tlearn: 0.1579211\ttest: 0.1430249\tbest: 0.1430249 (44)\ttotal: 3.86s\tremaining: 13.3s\n", "45:\tlearn: 0.1578847\ttest: 0.1431223\tbest: 0.1430249 (44)\ttotal: 3.96s\tremaining: 13.2s\n", "46:\tlearn: 0.1578482\ttest: 0.1430689\tbest: 0.1430249 (44)\ttotal: 4.05s\tremaining: 13.2s\n", "47:\tlearn: 0.1578470\ttest: 0.1430681\tbest: 0.1430249 (44)\ttotal: 4.11s\tremaining: 13s\n", "48:\tlearn: 0.1578462\ttest: 0.1430580\tbest: 0.1430249 (44)\ttotal: 4.17s\tremaining: 12.7s\n", "49:\tlearn: 0.1577480\ttest: 0.1432538\tbest: 0.1430249 (44)\ttotal: 4.25s\tremaining: 12.6s\n", "50:\tlearn: 0.1575513\ttest: 0.1433252\tbest: 0.1430249 (44)\ttotal: 4.34s\tremaining: 12.6s\n", "51:\tlearn: 0.1571386\ttest: 0.1428501\tbest: 0.1428501 (51)\ttotal: 4.43s\tremaining: 12.5s\n", "52:\tlearn: 0.1571009\ttest: 0.1428871\tbest: 0.1428501 (51)\ttotal: 4.52s\tremaining: 12.5s\n", "53:\tlearn: 0.1570543\ttest: 0.1428245\tbest: 0.1428245 (53)\ttotal: 4.6s\tremaining: 12.4s\n", "54:\tlearn: 0.1569937\ttest: 0.1428190\tbest: 0.1428190 (54)\ttotal: 4.67s\tremaining: 12.2s\n", "55:\tlearn: 0.1568604\ttest: 0.1428687\tbest: 0.1428190 (54)\ttotal: 4.79s\tremaining: 12.3s\n", "56:\tlearn: 0.1568583\ttest: 0.1428933\tbest: 0.1428190 (54)\ttotal: 4.84s\tremaining: 12s\n", "57:\tlearn: 0.1566636\ttest: 0.1425799\tbest: 0.1425799 (57)\ttotal: 4.92s\tremaining: 11.9s\n", "58:\tlearn: 0.1566105\ttest: 0.1425226\tbest: 0.1425226 (58)\ttotal: 5s\tremaining: 11.9s\n", "59:\tlearn: 0.1565786\ttest: 0.1425208\tbest: 0.1425208 (59)\ttotal: 5.1s\tremaining: 11.8s\n", "60:\tlearn: 0.1565691\ttest: 0.1425103\tbest: 0.1425103 (60)\ttotal: 5.2s\tremaining: 11.8s\n", "61:\tlearn: 0.1564572\ttest: 0.1425675\tbest: 0.1425103 (60)\ttotal: 5.3s\tremaining: 11.8s\n", "62:\tlearn: 0.1564145\ttest: 0.1426340\tbest: 0.1425103 (60)\ttotal: 5.42s\tremaining: 11.8s\n", "63:\tlearn: 0.1555005\ttest: 0.1421151\tbest: 0.1421151 (63)\ttotal: 5.52s\tremaining: 11.7s\n", "64:\tlearn: 0.1548624\ttest: 0.1414761\tbest: 0.1414761 (64)\ttotal: 5.63s\tremaining: 11.7s\n", "65:\tlearn: 0.1548316\ttest: 0.1415012\tbest: 0.1414761 (64)\ttotal: 5.73s\tremaining: 11.7s\n", "66:\tlearn: 0.1546954\ttest: 0.1414039\tbest: 0.1414039 (66)\ttotal: 5.83s\tremaining: 11.6s\n", "67:\tlearn: 0.1545918\ttest: 0.1412461\tbest: 0.1412461 (67)\ttotal: 5.94s\tremaining: 11.6s\n", "68:\tlearn: 0.1545238\ttest: 0.1412214\tbest: 0.1412214 (68)\ttotal: 6.05s\tremaining: 11.6s\n", "69:\tlearn: 0.1543865\ttest: 0.1412259\tbest: 0.1412214 (68)\ttotal: 6.17s\tremaining: 11.5s\n", "70:\tlearn: 0.1541683\ttest: 0.1414024\tbest: 0.1412214 (68)\ttotal: 6.27s\tremaining: 11.5s\n", "71:\tlearn: 0.1540775\ttest: 0.1414143\tbest: 0.1412214 (68)\ttotal: 6.37s\tremaining: 11.4s\n", "72:\tlearn: 0.1540220\ttest: 0.1413343\tbest: 0.1412214 (68)\ttotal: 6.45s\tremaining: 11.3s\n", "73:\tlearn: 0.1539815\ttest: 0.1413039\tbest: 0.1412214 (68)\ttotal: 6.52s\tremaining: 11.2s\n", "74:\tlearn: 0.1539774\ttest: 0.1413123\tbest: 0.1412214 (68)\ttotal: 6.62s\tremaining: 11.1s\n", "75:\tlearn: 0.1539583\ttest: 0.1413918\tbest: 0.1412214 (68)\ttotal: 6.71s\tremaining: 11s\n", "76:\tlearn: 0.1538207\ttest: 0.1412925\tbest: 0.1412214 (68)\ttotal: 6.8s\tremaining: 10.9s\n", "77:\tlearn: 0.1537301\ttest: 0.1412290\tbest: 0.1412214 (68)\ttotal: 6.89s\tremaining: 10.9s\n", "78:\tlearn: 0.1536745\ttest: 0.1411666\tbest: 0.1411666 (78)\ttotal: 6.99s\tremaining: 10.8s\n", "79:\tlearn: 0.1536233\ttest: 0.1412135\tbest: 0.1411666 (78)\ttotal: 7.07s\tremaining: 10.7s\n", "80:\tlearn: 0.1533953\ttest: 0.1413119\tbest: 0.1411666 (78)\ttotal: 7.18s\tremaining: 10.6s\n", "81:\tlearn: 0.1533845\ttest: 0.1412750\tbest: 0.1411666 (78)\ttotal: 7.27s\tremaining: 10.6s\n", "82:\tlearn: 0.1532740\ttest: 0.1413232\tbest: 0.1411666 (78)\ttotal: 7.35s\tremaining: 10.4s\n", "83:\tlearn: 0.1531701\ttest: 0.1416607\tbest: 0.1411666 (78)\ttotal: 7.45s\tremaining: 10.4s\n", "84:\tlearn: 0.1530255\ttest: 0.1416930\tbest: 0.1411666 (78)\ttotal: 7.53s\tremaining: 10.3s\n", "85:\tlearn: 0.1530003\ttest: 0.1416376\tbest: 0.1411666 (78)\ttotal: 7.6s\tremaining: 10.2s\n", "86:\tlearn: 0.1529827\ttest: 0.1415299\tbest: 0.1411666 (78)\ttotal: 7.7s\tremaining: 10.1s\n", "87:\tlearn: 0.1529783\ttest: 0.1415534\tbest: 0.1411666 (78)\ttotal: 7.78s\tremaining: 9.98s\n", "88:\tlearn: 0.1529650\ttest: 0.1415712\tbest: 0.1411666 (78)\ttotal: 7.86s\tremaining: 9.87s\n", "89:\tlearn: 0.1529603\ttest: 0.1415611\tbest: 0.1411666 (78)\ttotal: 7.95s\tremaining: 9.79s\n", "90:\tlearn: 0.1526304\ttest: 0.1415594\tbest: 0.1411666 (78)\ttotal: 8.03s\tremaining: 9.69s\n", "91:\tlearn: 0.1525361\ttest: 0.1417001\tbest: 0.1411666 (78)\ttotal: 8.14s\tremaining: 9.62s\n", "92:\tlearn: 0.1525222\ttest: 0.1416753\tbest: 0.1411666 (78)\ttotal: 8.25s\tremaining: 9.56s\n", "93:\tlearn: 0.1523982\ttest: 0.1417682\tbest: 0.1411666 (78)\ttotal: 8.34s\tremaining: 9.48s\n", "94:\tlearn: 0.1520970\ttest: 0.1417407\tbest: 0.1411666 (78)\ttotal: 8.44s\tremaining: 9.4s\n", "95:\tlearn: 0.1520886\ttest: 0.1416787\tbest: 0.1411666 (78)\ttotal: 8.53s\tremaining: 9.31s\n", "96:\tlearn: 0.1519551\ttest: 0.1416528\tbest: 0.1411666 (78)\ttotal: 8.62s\tremaining: 9.22s\n", "97:\tlearn: 0.1518608\ttest: 0.1417604\tbest: 0.1411666 (78)\ttotal: 8.72s\tremaining: 9.15s\n", "98:\tlearn: 0.1515918\ttest: 0.1418670\tbest: 0.1411666 (78)\ttotal: 8.81s\tremaining: 9.06s\n", "99:\tlearn: 0.1514836\ttest: 0.1421103\tbest: 0.1411666 (78)\ttotal: 8.88s\tremaining: 8.95s\n", "100:\tlearn: 0.1513803\ttest: 0.1424265\tbest: 0.1411666 (78)\ttotal: 8.94s\tremaining: 8.82s\n", "101:\tlearn: 0.1513654\ttest: 0.1424137\tbest: 0.1411666 (78)\ttotal: 9s\tremaining: 8.7s\n", "102:\tlearn: 0.1513427\ttest: 0.1423719\tbest: 0.1411666 (78)\ttotal: 9.06s\tremaining: 8.57s\n", "103:\tlearn: 0.1511778\ttest: 0.1422654\tbest: 0.1411666 (78)\ttotal: 9.14s\tremaining: 8.48s\n", "104:\tlearn: 0.1511117\ttest: 0.1423467\tbest: 0.1411666 (78)\ttotal: 9.24s\tremaining: 8.41s\n", "105:\tlearn: 0.1510436\ttest: 0.1423221\tbest: 0.1411666 (78)\ttotal: 9.35s\tremaining: 8.33s\n", "106:\tlearn: 0.1508977\ttest: 0.1423137\tbest: 0.1411666 (78)\ttotal: 9.43s\tremaining: 8.24s\n", "107:\tlearn: 0.1508959\ttest: 0.1423254\tbest: 0.1411666 (78)\ttotal: 9.5s\tremaining: 8.13s\n", "108:\tlearn: 0.1508424\ttest: 0.1424438\tbest: 0.1411666 (78)\ttotal: 9.59s\tremaining: 8.04s\n", "109:\tlearn: 0.1508101\ttest: 0.1424697\tbest: 0.1411666 (78)\ttotal: 9.67s\tremaining: 7.95s\n", "110:\tlearn: 0.1507799\ttest: 0.1424306\tbest: 0.1411666 (78)\ttotal: 9.75s\tremaining: 7.85s\n", "111:\tlearn: 0.1507635\ttest: 0.1424197\tbest: 0.1411666 (78)\ttotal: 9.84s\tremaining: 7.76s\n", "112:\tlearn: 0.1507610\ttest: 0.1423976\tbest: 0.1411666 (78)\ttotal: 9.92s\tremaining: 7.67s\n", "113:\tlearn: 0.1507207\ttest: 0.1423904\tbest: 0.1411666 (78)\ttotal: 10s\tremaining: 7.58s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "114:\tlearn: 0.1506871\ttest: 0.1423714\tbest: 0.1411666 (78)\ttotal: 10.1s\tremaining: 7.52s\n", "115:\tlearn: 0.1506365\ttest: 0.1423830\tbest: 0.1411666 (78)\ttotal: 10.2s\tremaining: 7.43s\n", "116:\tlearn: 0.1505736\ttest: 0.1425609\tbest: 0.1411666 (78)\ttotal: 10.3s\tremaining: 7.35s\n", "117:\tlearn: 0.1505321\ttest: 0.1425929\tbest: 0.1411666 (78)\ttotal: 10.4s\tremaining: 7.29s\n", "118:\tlearn: 0.1505028\ttest: 0.1425452\tbest: 0.1411666 (78)\ttotal: 10.5s\tremaining: 7.21s\n", "119:\tlearn: 0.1497712\ttest: 0.1424179\tbest: 0.1411666 (78)\ttotal: 10.7s\tremaining: 7.17s\n", "120:\tlearn: 0.1497006\ttest: 0.1423512\tbest: 0.1411666 (78)\ttotal: 10.8s\tremaining: 7.11s\n", "121:\tlearn: 0.1496638\ttest: 0.1423660\tbest: 0.1411666 (78)\ttotal: 11s\tremaining: 7.1s\n", "122:\tlearn: 0.1496092\ttest: 0.1423799\tbest: 0.1411666 (78)\ttotal: 11.1s\tremaining: 7.04s\n", "123:\tlearn: 0.1495761\ttest: 0.1422939\tbest: 0.1411666 (78)\ttotal: 11.2s\tremaining: 6.95s\n", "124:\tlearn: 0.1495648\ttest: 0.1423047\tbest: 0.1411666 (78)\ttotal: 11.3s\tremaining: 6.86s\n", "125:\tlearn: 0.1493551\ttest: 0.1421014\tbest: 0.1411666 (78)\ttotal: 11.5s\tremaining: 6.8s\n", "126:\tlearn: 0.1492764\ttest: 0.1420590\tbest: 0.1411666 (78)\ttotal: 11.6s\tremaining: 6.72s\n", "127:\tlearn: 0.1490840\ttest: 0.1417298\tbest: 0.1411666 (78)\ttotal: 11.7s\tremaining: 6.65s\n", "128:\tlearn: 0.1490683\ttest: 0.1417569\tbest: 0.1411666 (78)\ttotal: 11.8s\tremaining: 6.58s\n", "129:\tlearn: 0.1490611\ttest: 0.1417673\tbest: 0.1411666 (78)\ttotal: 12s\tremaining: 6.53s\n", "130:\tlearn: 0.1490453\ttest: 0.1417709\tbest: 0.1411666 (78)\ttotal: 12.1s\tremaining: 6.45s\n", "131:\tlearn: 0.1490231\ttest: 0.1417575\tbest: 0.1411666 (78)\ttotal: 12.3s\tremaining: 6.39s\n", "132:\tlearn: 0.1489009\ttest: 0.1419888\tbest: 0.1411666 (78)\ttotal: 12.3s\tremaining: 6.29s\n", "133:\tlearn: 0.1486744\ttest: 0.1420246\tbest: 0.1411666 (78)\ttotal: 12.4s\tremaining: 6.2s\n", "134:\tlearn: 0.1485575\ttest: 0.1421380\tbest: 0.1411666 (78)\ttotal: 12.5s\tremaining: 6.11s\n", "135:\tlearn: 0.1485306\ttest: 0.1421969\tbest: 0.1411666 (78)\ttotal: 12.6s\tremaining: 6.01s\n", "136:\tlearn: 0.1483855\ttest: 0.1420489\tbest: 0.1411666 (78)\ttotal: 12.7s\tremaining: 5.91s\n", "137:\tlearn: 0.1483428\ttest: 0.1421026\tbest: 0.1411666 (78)\ttotal: 12.8s\tremaining: 5.83s\n", "138:\tlearn: 0.1483134\ttest: 0.1420898\tbest: 0.1411666 (78)\ttotal: 12.9s\tremaining: 5.72s\n", "139:\tlearn: 0.1483052\ttest: 0.1420878\tbest: 0.1411666 (78)\ttotal: 13s\tremaining: 5.63s\n", "140:\tlearn: 0.1482157\ttest: 0.1420241\tbest: 0.1411666 (78)\ttotal: 13.1s\tremaining: 5.53s\n", "141:\tlearn: 0.1481112\ttest: 0.1421268\tbest: 0.1411666 (78)\ttotal: 13.2s\tremaining: 5.45s\n", "142:\tlearn: 0.1479973\ttest: 0.1422019\tbest: 0.1411666 (78)\ttotal: 13.3s\tremaining: 5.36s\n", "143:\tlearn: 0.1479948\ttest: 0.1422188\tbest: 0.1411666 (78)\ttotal: 13.4s\tremaining: 5.28s\n", "144:\tlearn: 0.1479705\ttest: 0.1421976\tbest: 0.1411666 (78)\ttotal: 13.6s\tremaining: 5.22s\n", "145:\tlearn: 0.1479037\ttest: 0.1423085\tbest: 0.1411666 (78)\ttotal: 13.8s\tremaining: 5.16s\n", "146:\tlearn: 0.1478758\ttest: 0.1424303\tbest: 0.1411666 (78)\ttotal: 13.9s\tremaining: 5.08s\n", "147:\tlearn: 0.1478342\ttest: 0.1424301\tbest: 0.1411666 (78)\ttotal: 14s\tremaining: 4.98s\n", "148:\tlearn: 0.1478086\ttest: 0.1423590\tbest: 0.1411666 (78)\ttotal: 14.1s\tremaining: 4.89s\n", "149:\tlearn: 0.1477729\ttest: 0.1422865\tbest: 0.1411666 (78)\ttotal: 14.2s\tremaining: 4.8s\n", "150:\tlearn: 0.1474847\ttest: 0.1422271\tbest: 0.1411666 (78)\ttotal: 14.3s\tremaining: 4.71s\n", "151:\tlearn: 0.1474169\ttest: 0.1423728\tbest: 0.1411666 (78)\ttotal: 14.4s\tremaining: 4.63s\n", "152:\tlearn: 0.1472865\ttest: 0.1422631\tbest: 0.1411666 (78)\ttotal: 14.6s\tremaining: 4.54s\n", "153:\tlearn: 0.1472423\ttest: 0.1422256\tbest: 0.1411666 (78)\ttotal: 14.7s\tremaining: 4.46s\n", "154:\tlearn: 0.1471948\ttest: 0.1421481\tbest: 0.1411666 (78)\ttotal: 14.8s\tremaining: 4.37s\n", "155:\tlearn: 0.1468854\ttest: 0.1419873\tbest: 0.1411666 (78)\ttotal: 14.9s\tremaining: 4.28s\n", "156:\tlearn: 0.1468790\ttest: 0.1420101\tbest: 0.1411666 (78)\ttotal: 15s\tremaining: 4.18s\n", "157:\tlearn: 0.1468687\ttest: 0.1419619\tbest: 0.1411666 (78)\ttotal: 15.1s\tremaining: 4.09s\n", "158:\tlearn: 0.1468658\ttest: 0.1419551\tbest: 0.1411666 (78)\ttotal: 15.2s\tremaining: 3.99s\n", "159:\tlearn: 0.1468288\ttest: 0.1420261\tbest: 0.1411666 (78)\ttotal: 15.3s\tremaining: 3.89s\n", "160:\tlearn: 0.1467160\ttest: 0.1419321\tbest: 0.1411666 (78)\ttotal: 15.4s\tremaining: 3.8s\n", "161:\tlearn: 0.1467084\ttest: 0.1419409\tbest: 0.1411666 (78)\ttotal: 15.5s\tremaining: 3.7s\n", "162:\tlearn: 0.1464073\ttest: 0.1416654\tbest: 0.1411666 (78)\ttotal: 15.7s\tremaining: 3.61s\n", "163:\tlearn: 0.1462285\ttest: 0.1416156\tbest: 0.1411666 (78)\ttotal: 15.8s\tremaining: 3.52s\n", "164:\tlearn: 0.1460339\ttest: 0.1416086\tbest: 0.1411666 (78)\ttotal: 15.9s\tremaining: 3.42s\n", "165:\tlearn: 0.1460218\ttest: 0.1416287\tbest: 0.1411666 (78)\ttotal: 16s\tremaining: 3.33s\n", "166:\tlearn: 0.1460092\ttest: 0.1416437\tbest: 0.1411666 (78)\ttotal: 16.1s\tremaining: 3.23s\n", "167:\tlearn: 0.1456841\ttest: 0.1418764\tbest: 0.1411666 (78)\ttotal: 16.2s\tremaining: 3.13s\n", "168:\tlearn: 0.1456011\ttest: 0.1419543\tbest: 0.1411666 (78)\ttotal: 16.3s\tremaining: 3.04s\n", "169:\tlearn: 0.1455905\ttest: 0.1420247\tbest: 0.1411666 (78)\ttotal: 16.4s\tremaining: 2.94s\n", "170:\tlearn: 0.1455581\ttest: 0.1420612\tbest: 0.1411666 (78)\ttotal: 16.5s\tremaining: 2.85s\n", "171:\tlearn: 0.1455441\ttest: 0.1420924\tbest: 0.1411666 (78)\ttotal: 16.6s\tremaining: 2.75s\n", "172:\tlearn: 0.1454990\ttest: 0.1421376\tbest: 0.1411666 (78)\ttotal: 16.7s\tremaining: 2.65s\n", "173:\tlearn: 0.1453813\ttest: 0.1420980\tbest: 0.1411666 (78)\ttotal: 16.9s\tremaining: 2.56s\n", "174:\tlearn: 0.1451034\ttest: 0.1420903\tbest: 0.1411666 (78)\ttotal: 17s\tremaining: 2.46s\n", "175:\tlearn: 0.1450984\ttest: 0.1421000\tbest: 0.1411666 (78)\ttotal: 17s\tremaining: 2.36s\n", "176:\tlearn: 0.1449863\ttest: 0.1421289\tbest: 0.1411666 (78)\ttotal: 17.1s\tremaining: 2.26s\n", "177:\tlearn: 0.1447876\ttest: 0.1420145\tbest: 0.1411666 (78)\ttotal: 17.2s\tremaining: 2.16s\n", "178:\tlearn: 0.1447701\ttest: 0.1420666\tbest: 0.1411666 (78)\ttotal: 17.3s\tremaining: 2.06s\n", "179:\tlearn: 0.1446955\ttest: 0.1420109\tbest: 0.1411666 (78)\ttotal: 17.4s\tremaining: 1.96s\n", "180:\tlearn: 0.1446544\ttest: 0.1420329\tbest: 0.1411666 (78)\ttotal: 17.5s\tremaining: 1.86s\n", "181:\tlearn: 0.1445431\ttest: 0.1421508\tbest: 0.1411666 (78)\ttotal: 17.6s\tremaining: 1.76s\n", "182:\tlearn: 0.1442690\ttest: 0.1421298\tbest: 0.1411666 (78)\ttotal: 17.7s\tremaining: 1.67s\n", "183:\tlearn: 0.1441832\ttest: 0.1421534\tbest: 0.1411666 (78)\ttotal: 17.8s\tremaining: 1.57s\n", "184:\tlearn: 0.1440706\ttest: 0.1423811\tbest: 0.1411666 (78)\ttotal: 17.9s\tremaining: 1.47s\n", "185:\tlearn: 0.1439856\ttest: 0.1425963\tbest: 0.1411666 (78)\ttotal: 18s\tremaining: 1.37s\n", "186:\tlearn: 0.1439761\ttest: 0.1426506\tbest: 0.1411666 (78)\ttotal: 18.1s\tremaining: 1.28s\n", "187:\tlearn: 0.1439714\ttest: 0.1426176\tbest: 0.1411666 (78)\ttotal: 18.2s\tremaining: 1.18s\n", "188:\tlearn: 0.1439492\ttest: 0.1426109\tbest: 0.1411666 (78)\ttotal: 18.4s\tremaining: 1.08s\n", "189:\tlearn: 0.1439454\ttest: 0.1426164\tbest: 0.1411666 (78)\ttotal: 18.4s\tremaining: 984ms\n", "190:\tlearn: 0.1439413\ttest: 0.1426552\tbest: 0.1411666 (78)\ttotal: 18.5s\tremaining: 885ms\n", "191:\tlearn: 0.1438359\ttest: 0.1427087\tbest: 0.1411666 (78)\ttotal: 18.6s\tremaining: 787ms\n", "192:\tlearn: 0.1438287\ttest: 0.1427033\tbest: 0.1411666 (78)\ttotal: 18.7s\tremaining: 689ms\n", "193:\tlearn: 0.1437879\ttest: 0.1426357\tbest: 0.1411666 (78)\ttotal: 18.8s\tremaining: 590ms\n", "194:\tlearn: 0.1437860\ttest: 0.1426468\tbest: 0.1411666 (78)\ttotal: 18.9s\tremaining: 492ms\n", "195:\tlearn: 0.1437606\ttest: 0.1426516\tbest: 0.1411666 (78)\ttotal: 19s\tremaining: 393ms\n", "196:\tlearn: 0.1437591\ttest: 0.1426400\tbest: 0.1411666 (78)\ttotal: 19.1s\tremaining: 294ms\n", "197:\tlearn: 0.1437577\ttest: 0.1426297\tbest: 0.1411666 (78)\ttotal: 19.2s\tremaining: 196ms\n", "198:\tlearn: 0.1437447\ttest: 0.1426317\tbest: 0.1411666 (78)\ttotal: 19.3s\tremaining: 98ms\n", "199:\tlearn: 0.1437230\ttest: 0.1426660\tbest: 0.1411666 (78)\ttotal: 19.4s\tremaining: 0us\n", "\n", "bestTest = 0.1411666436\n", "bestIteration = 78\n", "\n", "Shrink model to first 79 iterations.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " iterations=200,\n", " save_snapshot=True,\n", " snapshot_file='snapshot.bkp',\n", " snapshot_interval=1,\n", " random_seed=43\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=True,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model predictions" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [], "source": [ "? model.predict_proba" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.0688 0.9312]\n", " [0.008 0.992 ]\n", " [0.0068 0.9932]\n", " ...\n", " [0.0129 0.9871]\n", " [0.0184 0.9816]\n", " [0.0273 0.9727]]\n" ] } ], "source": [ "print(model.predict_proba(data=X_validation))" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1. 1. 1. ... 1. 1. 1.]\n" ] } ], "source": [ "print(model.predict(data=X_validation))" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2.6056 4.8177 4.9809 ... 4.3367 3.9791 3.5714]\n" ] } ], "source": [ "raw_pred = model.predict(data=X_validation, prediction_type='RawFormulaVal')\n", "print(raw_pred)" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.9312 0.992 0.9932 ... 0.9871 0.9816 0.9727]\n" ] } ], "source": [ "import math\n", "def sigmoid(x):\n", " return 1 / (1 + math.exp(-x))\n", "probabilities = [sigmoid(x) for x in raw_pred]\n", "print(np.array(probabilities))" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.0688 0.9312]\n", " [0.008 0.992 ]\n", " [0.0068 0.9932]\n", " ...\n", " [0.0129 0.9871]\n", " [0.0184 0.9816]\n", " [0.0273 0.9727]]\n" ] } ], "source": [ "X_prepared = X_validation.values.astype(str).astype(object)\n", "# For FeaturesData class categorial features must have type str\n", "\n", "fast_predictions = model.predict_proba(data=FeaturesData(cat_feature_data=X_prepared, \n", " cat_feature_names=list(X_validation)))\n", "\n", "print(fast_predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Staged prediction" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Iteration 0, predictions:\n", "[[0.3065 0.6935]\n", " [0.2317 0.7683]\n", " [0.2317 0.7683]\n", " ...\n", " [0.2317 0.7683]\n", " [0.2317 0.7683]\n", " [0.2746 0.7254]]\n", "Iteration 1, predictions:\n", "[[0.3002 0.6998]\n", " [0.1762 0.8238]\n", " [0.1422 0.8578]\n", " ...\n", " [0.1422 0.8578]\n", " [0.1916 0.8084]\n", " [0.2116 0.7884]]\n", "Iteration 2, predictions:\n", "[[0.3307 0.6693]\n", " [0.13 0.87 ]\n", " [0.1038 0.8962]\n", " ...\n", " [0.1454 0.8546]\n", " [0.1956 0.8044]\n", " [0.1938 0.8062]]\n" ] } ], "source": [ "predictions_gen = model.staged_predict_proba(data=X_validation, ntree_start=2, ntree_end=8, eval_period=2)\n", "for iteration, predictions in enumerate(predictions_gen):\n", " print('Iteration ' + str(iteration) + ', predictions:')\n", " print(predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Solving MultiClassification problem" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1536e93770b14eb48d913a0209269c87", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 69, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=150,\n", " random_seed=43,\n", " loss_function='MultiClass'\n", " #loss_function='MultiClassOneVsAll'\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metric evaluation on a new dataset" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.6569860\ttotal: 53.2ms\tremaining: 10.6s\n", "50:\tlearn: 0.1950260\ttotal: 3.33s\tremaining: 9.74s\n", "100:\tlearn: 0.1700584\ttotal: 6.72s\tremaining: 6.58s\n", "150:\tlearn: 0.1641016\ttotal: 10.6s\tremaining: 3.42s\n", "199:\tlearn: 0.1604074\ttotal: 14.1s\tremaining: 0us\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=200,\n", " learning_rate=0.03,\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=50\n", ")" ] }, { "cell_type": "code", "execution_count": 71, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "metrics = model.eval_metrics(data=pool1, \n", " metrics=['Logloss','AUC'],\n", " ntree_start=0,\n", " ntree_end=0, \n", " eval_period=1,\n", " plot=True)" ] }, { "cell_type": "code", "execution_count": 72, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AUC values:\n", "[0.4998 0.538 0.5504 0.5888 0.6503 0.6487 0.6487 0.6601 0.6601 0.6612\n", " 0.6614 0.67 0.6699 0.6697 0.6697 0.6698 0.6698 0.6698 0.6698 0.7265\n", " 0.7338 0.7376 0.7478 0.7478 0.7506 0.7591 0.7642 0.7877 0.8148 0.8265\n", " 0.8353 0.8459 0.8533 0.8564 0.8573 0.8748 0.8874 0.893 0.8948 0.898\n", " 0.9003 0.9052 0.9122 0.9159 0.92 0.9204 0.9209 0.9246 0.9262 0.9266\n", " 0.9279 0.9303 0.9311 0.9312 0.9327 0.9329 0.9337 0.9341 0.9341 0.9349\n", " 0.9354 0.9365 0.9391 0.9411 0.9424 0.9435 0.9446 0.9458 0.9463 0.9471\n", " 0.9478 0.9481 0.9482 0.9483 0.9486 0.9497 0.951 0.9514 0.9515 0.9519\n", " 0.9522 0.9526 0.953 0.9537 0.9543 0.9544 0.9545 0.9548 0.9548 0.9548\n", " 0.9548 0.9548 0.955 0.9551 0.9553 0.9554 0.9557 0.9557 0.9557 0.9557\n", " 0.956 0.9566 0.9566 0.9569 0.9568 0.957 0.9569 0.9572 0.9573 0.9575\n", " 0.9576 0.9576 0.9577 0.958 0.9587 0.9594 0.9595 0.9594 0.9601 0.9609\n", " 0.9609 0.9608 0.9608 0.9612 0.9616 0.9618 0.9622 0.9623 0.9623 0.9625\n", " 0.9625 0.9628 0.9629 0.9629 0.9629 0.963 0.9631 0.9632 0.9635 0.9636\n", " 0.9637 0.9638 0.9638 0.964 0.9641 0.9643 0.9645 0.9645 0.9645 0.9647\n", " 0.9647 0.9647 0.9647 0.9648 0.9648 0.9648 0.965 0.9652 0.9652 0.9653\n", " 0.9653 0.9654 0.9655 0.9655 0.9655 0.9656 0.9656 0.9657 0.9657 0.9657\n", " 0.9658 0.9658 0.9659 0.9659 0.966 0.966 0.966 0.966 0.966 0.966\n", " 0.966 0.966 0.9661 0.9662 0.9663 0.9663 0.9663 0.9663 0.9663 0.9663\n", " 0.9665 0.9664 0.9666 0.9666 0.9666 0.9666 0.9666 0.9667 0.9667 0.9666]\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "79673ab90b80409f9098024a23d609ec", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print('AUC values:')\n", "print(np.array(metrics['AUC']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Saving the model" ] }, { "cell_type": "code", "execution_count": 73, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 73, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_best_model = CatBoostClassifier(iterations=10)\n", "my_best_model.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [], "source": [ "my_best_model.save_model('catboost_model.bin')" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'iterations': 10, 'loss_function': 'Logloss', 'logging_level': 'Silent'}\n", "17379826207872\n" ] } ], "source": [ "my_best_model.load_model('catboost_model.bin')\n", "print(my_best_model.get_params())\n", "print(my_best_model.random_seed_)" ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [], "source": [ "my_best_model.save_model('catboost_model.json', format='json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Feature importances" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.6569860\ttotal: 56.9ms\tremaining: 11.3s\n", "50:\tlearn: 0.1950260\ttotal: 4.06s\tremaining: 11.9s\n", "100:\tlearn: 0.1700584\ttotal: 7.18s\tremaining: 7.04s\n", "150:\tlearn: 0.1641016\ttotal: 10.8s\tremaining: 3.5s\n", "199:\tlearn: 0.1604074\ttotal: 14.3s\tremaining: 0us\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=200,\n", " learning_rate=0.03)\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=50\n", ")" ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [], "source": [ "fstrs = model.get_feature_importance(prettified=True)" ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{b'MGR_ID': 18.398056223683028,\n", " b'RESOURCE': 21.27625707750949,\n", " b'ROLE_CODE': 11.943230282191124,\n", " b'ROLE_DEPTNAME': 15.252806962601875,\n", " b'ROLE_FAMILY': 2.4789173515872176,\n", " b'ROLE_FAMILY_DESC': 9.984073192415533,\n", " b'ROLE_ROLLUP_1': 2.6278918788673633,\n", " b'ROLE_ROLLUP_2': 13.582536028250486,\n", " b'ROLE_TITLE': 4.456231002893895}" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "{feature_name : value for feature_name, value in fstrs}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Shap values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "https://github.com/slundberg/shap" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [], "source": [ "def object_predictions(model, obj):\n", " print('Probability of class 1 = {:.4f}'.format(model.predict_proba([obj])[0][1]))\n", " print('Formula raw prediction = {:.4f}'.format(model.predict([obj], prediction_type='RawFormulaVal')[0]))\n", " " ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "The model has complex ctrs, so the SHAP values will be calculated approximately.\n" ] } ], "source": [ "import shap\n", "explainer = shap.TreeExplainer(model)\n", "shap_values = explainer.shap_values(Pool(X, y, cat_features=cat_features))\n" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3.345740996864643" ] }, "execution_count": 82, "metadata": {}, "output_type": "execute_result" } ], "source": [ "explainer.expected_value" ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Probability of class 1 = 0.9798\n", "Formula raw prediction = 3.8820\n" ] } ], "source": [ "object_predictions(model, X.iloc[3,:])" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "
\n", " Visualization omitted, Javascript library not loaded!
\n", " Have you run `initjs()` in this notebook? If this notebook was from another\n", " user you must also trust this notebook (File -> Trust notebook). If you are viewing\n", " this notebook on github the Javascript has been stripped for security. If you are using\n", " JupyterLab this error is because a JupyterLab extension has not yet been written.\n", "
\n", " " ], "text/plain": [ "" ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "shap.initjs()\n", "shap.force_plot(explainer.expected_value, shap_values[3,:], X.iloc[3,:])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above explanation shows features each contributing to push the model output from the base value (the average model output over the training dataset we passed) to the model output. Features pushing the prediction higher are shown in red, those pushing the prediction lower are in blue" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Probability of class 1 = 0.6404\n", "Formula raw prediction = 0.5772\n" ] } ], "source": [ "object_predictions(model, X.iloc[91,:])" ] }, { "cell_type": "code", "execution_count": 86, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "
\n", " Visualization omitted, Javascript library not loaded!
\n", " Have you run `initjs()` in this notebook? If this notebook was from another\n", " user you must also trust this notebook (File -> Trust notebook). If you are viewing\n", " this notebook on github the Javascript has been stripped for security. If you are using\n", " JupyterLab this error is because a JupyterLab extension has not yet been written.\n", "
\n", " " ], "text/plain": [ "" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import shap\n", "shap.initjs()\n", "shap.force_plot(explainer.expected_value, shap_values[91,:], X.iloc[91,:])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get an overview of which features are most important for a model we can plot the SHAP values of every feature for every sample. The plot below sorts features by the sum of SHAP value magnitudes over all samples, and uses SHAP values to show the distribution of the impacts each feature has on the model output. The color represents the feature value (red high, blue low)." ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAFZCAYAAAC4x3ouAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XecXFd58PHfuWX6zO7O9tVqtatVlyxL1rXcccfGxg1MdQwGm+QlIQTCG8qLwUBICIlDCRhCMG6AKQ5Vxr3LtlyuZFm9S6vV9jq93Pb+MaPVqtjYktaSlvP9fEaaueWcc++dnfvMc889IzzPQ5IkSZIkSXpzlGPdAEmSJEmSpBORDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSTouCCF2CSEWHDDNFEKcJ4T4mhDifW+gjK8IIW6duFbuo70VlUiSJEmSJB0Jz/O+fKzbcCCZiZIkSZIk6bgnhLhLCPGJ8vMKIcRvhBCbhBCPCyHuOSD7NEUI8UB5/p+EEKGJaJPMREmSJEmSdDz5XyFEftzrWYdY5svAiOd5c4QQcWAl8Jtx8w3gVCABPAxcB/z4aDdUBlHSseYd6wZIkjR5LVu2DIArrrjiGLfkhCEmpNTY+w/+rE/+8rXqutbzvHVjDRLCPMQy5wN/D+B53rAQ4vcHzH/Y87zR8vovAu2H0+w/RwZRkiRJkiRNKO8QsdkRRmuC1/8SPj6T5QDBI6vu0GSfKEmSJEmSJpQnDn4coSeBDwMIIaqAq464xMMggyhJkiRJkiaUhzjocYS+BtQJIdYDPwOeo9T/6S0lL+dJkiRJkjSh3mjmyfO81kNMM8pPnxo3OQN8wPO8vBAiBjwL3F1e/isHrL/f66NJBlGSJEmSJE2wo95fvQp4UAihAgHgXs/zHjvalfw5MoiSJEmSJOmE4nleP7DkWLdDBlGSJEmSJE2oo9CR/LgkgyhJkiRJkibUUehIflySQZQkSZIkSRNKZqIkSZIkSZIOg8xESZIkSZIkHQZPyCBKkiRJkiTpTZusP5IqgyhJkiRJkibUZM1EyZ99kSRJkiRJOgwyEyVJkiRJZX3Deb70yUfpq4xjCcG0/l4+9rULOWV2xbFu2gltsmaiZBAlSZIkSWVf+eSjDFRVE3RsAsCrrTP47I+24HNc6hND3HnXZce6iSekydonSl7OkyRJkqSydY1T8Ts2UPq1t6pCjt7KagajMbKBMK++2nNsG3iC8oQ46DEZyCBKkiRJksqmJEdxx41plPX5x/4fDYX5+D3dDIwWjlXzTljeIR6TgQyiJEmSJKns3B2b2F5VzXAwRFe0goFwdGxe2uen1irynq+vx/MmSxggHQkZREmSJElSWdrvx1UUdsTr6IvEcIXAAxwh6I9E2VpTT9i2+OUTnce6qSeUyXo5T3YslyRJkv5iff5HG7Cf3MQZu7eR1AP4AgFA4AIIgee65DUdhEABLCFIBEKse7EXLmw5to0/gciffZEkSZKkE5RrO3z5/9xPd0Eh4FoEbQfhuKTDEb743GP4HJtHZi3g5WkzyPl8+FwH4XkUdR1PUQFwPA/VEwSLBUYrY/uVn88W+MGyDn621iavqBRVjfMri/z4H+Yei8097sgfIJYkSZKkE5BruXzgb54gHa5DC3ic3LObKYkRXEXBSSV4vnUGg+Eot592HgBByyKvaSieh6vs3+ulMpdm5mAvDUvmMtIxxK2ffpgNtU20JofZXttIvaLi4jEQjfFYNsKpX1jNIzfPpyqsH4MtP35M1kyUkJ3jpGNMvgElSXrTPM/j1U1D7BnK8sMHh8il88zNjtChhmkfHSLguayvn0LO9cgGQ+QCAU7p3s2F2zagUOrjlNH9eELgCEHaH+CXi88YK98CLti+kWXzF4MoBVK6bRMt5GgZHsRnW7SNDtMfiuApCnsqqpg70MPZOzcTtiw84P45C1neNpuCpvHfV1Vw0UlVx2RfvUkTEu0MNd500Gd9dc/tJ3xkJTNRE8gwjKeAc4H3mab563HTTwNeADpM02wtT5sBfAm4CKgEhoF1wI9N0/xteZmvADcDeUrBxwBwD/BV0zT/bDBiGIYHnGOa5rOGYZwHPAlkAJfSZ8Ym4DfAbaZpynt4pTFrugrc+WKCsE/hia1Z0okCV68x6Y9VsLGuiayuU5XNMHOgl/WNU6nI59AdmwW9XYwGQ3RU1VBQVWb197Czph5L03A9CDgWBVXFEQoKkPH5Sfn8TB8eJO33010RxxUCzbbxuw4+u8iHzeeYPjxAR0U1K9pmsquqhgu2bWDWQC/3LDmLVCCI8DyW7NnJeds2UZ3LsKm2kfvnncxwMExlIYfftpkyOsxIIIjR3UFB1fntQoOColKfTuKoKsJzyep+HEWhfaif97z6MmGrwMbaBhKBEJFiAd11yOg+nm6fQ1HVyWsatqJQ0HTO6NjGNetW8sK0dnqjlbSMDCFwUTxBQVXZVl3H5vomovkcOd1Hxh8g5ffTPDrMtOEBZg710xutoDdaia2qjPqDdMRrcBWFqSODTB0doS9aQV7TeNe6lbSMDvOnmfNQFJWVza0UdJ2mxChG5w5S/iA74zUIz6WoafRFK7FUjWg+i89xyGs6toBPrHiSqlyGZ1tnMnOgl6dmzKOzooqMPwCA8DwQEM+kcYVCTvfxjk1rqMsk2V1Zjc+yWN4+BxtoSo0yv68bV1FYMW0GfZEY4UKeD6x+ke6KKlY3tZDXNFTXoy6T5LxtG1g2dxGrm9twhcDn2ISLBWxFxRKCeD7LBds3EsvleL5lOkWfH92x0R2basdBdR3au3ppUwTba+qZ17WbilyWnxtn4SHwPI/ZA71jd1Opnofu2ATtUsBjj8s4uUC0WODxWQuIZzMonktlPsdNK57ka2+/hu019WiuQyoQZGdNHUWtlGVqH+ojZFmlfQVcsmU9j85aQKhY5OO/G2HDvEp09YSPGw7LZM1EySBq4m0EPgb8ety0j5WnhwAMwzgJeBb4LXAesAPQy8+vK0/f6ynTNC8yDEMAZwMPA7uAuw6jbY5pmpFyGwLAOcC3gfcYhnGuaZrFwyhTmmTSBZebftFHMu+OTTO6dtNTUcXy9jmlCZ7H9atW8PPFZ9A+PIDulpbtj1bQXVH69u3zPHbWNIAi0B0HxXNRgJDjUFTAVlUiVpGAbZMIhtlTWQXlO3iKmkZKUbnlkd9xg/ksAFndx7KTloCisHz6HNY0NJPxB9DK2fVVzW38zYtPEynk+da5l2JrOjGrSNCyCNo2PRVVzB7oIRkI8svFZ5AIhgAYisaIFAo0JUcIOjY4cP3K56nOZQBY1NPJ2sap+BwHAN3JEbAdhFDwFR1G/UFUz+Pmx5exua6Rl1ra+Ui5zR6Q1f1srq3nwfmLx/ZnU2KEinyOeDaN5nks7u5kTeNU1jdOHbucFM1nmZoYYVttA4lgBLV8UtI8j3uMs/if++4gatv87qSTCZTbtjteg+J5xAo5iroPPK90PAT4bYtI+YQ/GAjyb3/6NZdtXgPA6R3b+dXJSxF4ZAPBsSAKz8NnFUtlAe/Y9Crn7toKwNyBXn6zYAk5n4/RYIiLdmxiSioBQO36V/if08/nfWteZlN9E+sap45tu6PCzngtubmLWNPciiJAwaOo6eR1fSwouXLDappSo6R9fgr+AJ4QFBRfaS84DtvidSzs6uSOpeeS8Qd4aPZCQlZxbP/5LYuQte+7oQdECnlE+T0Wz2awFAEIXCFIlt8Ped3H5RtX86XH/kikkOe751zCLY/9gdaRQR6euYDvNlwyVubGhilctnktavk9WLoDTWDpOrpt84WHRrj18jh/iSZrnyg5xMHE+y2w2DCM6QCGYUSBdwN3jlvmO8DLpml+xDTNraZpOqZp5k3TfMg0zesPVahpmp5pmsuB9YBxpI0s1/cocA2wGPjwkZb5RqRSKfn8OH/e2Z/cL4ACqM8k6Yvt+y0xzXWJFgvYqjoWQAGk9p589xL7/hv/4aN4pXU8KPVDEWIsgAIQ5TzrnP7usWkhq0hNptROS9NI+YMHfaA1J4bZXVU9likAsMudhIUHAdsi5QuMBVBQOvFprrPf9+ZIcd/J1+c4KO6+xK8CBO3i2HbZmsrUxDDRYoEtNQ00JUf323zVcxkM7Rt7CKColr7P7j35xnNZEsHQfv1xLFUbG0nbV/5/b5kg2FrbQEHXOfBqjKWpqHvbKwRuebY27jgVNI324f6x17XZNL3RGKrr7ZehQQjUcTnv2kx6v7r8tk1FPkvGH8Bv7fsOFnBsArZFbSbFrqqa/dYRgKsodFXE9z/meCCUsev9lfksAHlN3+/2eFcIcppOX7SSjnjNvoCvPG/8Nib8IUYCIXqiFfRGYuiuMzbf59gI1z2oDxRAXSpJpJCno7KaG19+htN3b6chleDDq55jYVfH2HKK66E6DjXpJHWpBOdu30Tr0ECpfNdlfV8pIDwe/q5f6/lE8RAHPSYDGURNvDzwc+DG8usPAE8DPQCGYQQpXfL7xZsp1DAMxTCM84EFwOaj1VjTNLcCK4ELj1aZrycajcrnx/nz2c0VnDsjyHirmts4c+dWRDn4sVWVV5paaB/sYzRQWtYD2gf70ced8IOFwthoxU75BOcBVjmwKaoqo8FQ6VJOIT8231EU8Dx+v2DJWFnd0Qq6YqUsV0NylCs3rMIqnwA9QHcdnm2dxez+XuJ7T/aeNxaAhIoFuqOV1GVSTB/cF0AorktO1/dr32Mz5uGWn2+qaaCoqmMndw9I+UonbkcIfJbNzqoadsRrSwM3VtdSVEvbZwsFWygs7Okkms8BpX42sXKAkCmPjr2tuo7W4QF8tjXW7qBVpDdaClyT/kDp0hpgKQr1qQQXbN1AZS5DTSY5NuK25jjEMxmy5cwRnke4fAzy2r5tDBfy3Lvo9LF98MjM+cwZ7CXpDxDJ58fqEp5LUVXZG36tmDadXDlAHQ6GwfPYWtNAqFhgZ7xurB2baxpI+QP8Yf4pLNmzi/GKqoriupzesRVt3HvFQyDcfWOHv9jSDkA8m6ainBXE83BK4RYCWNU0Dd0uleEqCsq4QHH6UD+KgNFQmLUNU9lc20hzYqS0bZ6Hrag0pNMEiwVq0ilah/oJFov4rSJTR4dY1zCFB+aeTLC4f4J+QW8Xum0TyeeIFgsUdR+nde7k1D27qM5lWLpnB3geNekEnzq79Hd1PPxdv9bzieKJgx+Tgbyc99b4MfCIYRi3AH8N3ALs7WEYB1Sga+/ChmEsAp4qvwwAs03T3Pt151zDMEaBIOADflh+HE17gOqjXKZ0glKE4AfvreOVPQXiIYWtAxbL1ocQrSrXb9xEB0Fcn0a4OoA/XSQnFAYCAQLFIgOBIM39fWxuaMJv27ieS8PIEH2RCiwhKCoCVyh4nkc4l0UXCrFUippsill9XeyoqGY0FCLrD1CZTbOqoZm/veo6alIJ9lTU0DLQy7a6RhSrSFpRuXDDK6xrmIrmuUQLBVY0t/F883SmDvcTy6boCcfIazrRbIpooUBeUdhZEeecTWsItM7EURRChTy2olLUdWwUIvkcm2rqqci0MhQMMhqOkVN1LFXD5znkdB/9/iCKItCLBXKahuu4fOGSa3jHpjWc0rGdZbNPwnNcYvkcw5EYddk0l6xbSU9dPeGQykjGZSASJR8No6U8NjVPYcaePZy1dSPpQICEP0BfNEoslSJQDr5CVp7aeICOrMvMvi6++PZrGAhHMXZuZTgSoytWQctAH2d27aInUkFPRZxIPsueqmpGAkEEgqpsmow/QEDX+NVCA3NKC1NGR/AVCzSlErQO9ePaFq7rlvoF5TJYmk7rQD+D0QhpVeeX8xdTncsiHJdEIEDzyBCq49AfCvPSlBbyms7m2gYahwbojFQwomnUJIZJ+4KkAwFimRRLujqY39cNjsvz09pLwwoIhWAhj6so5DSd7RWV2K2ziRTzpFDw5/LM6e8mahVY3jqLaDbDaDBEbTqJ8Dw8AVWpJEVVx+9YtCWGgdKo45trG/CEoCKX4+Tcbgqqxm1nX4yCR2MyQdAuErIsqvI51tdP4cuXvpuZg31cvX4Vg+EIfZEY9ekkK6dMozscI1AskgwE6ais5uUp07hm3UoAiorK9ngtU0cGyQeDXDIr9Fp/ZpPeZMk8HUgGUW8B0zTXGYbRQanjeD3wEKWMFMAI4ADN45ZfDVQahtEMdLJ/fv7pcp8oH/AZ4HpKfauSR7HJe+uVJABURWC0lLIt02t8XDI3DNQBJx3Tdr213n4Y67ztqLfi0I74in7ZjKNUzpt1NgDvOQolJVIFbvvn5VTlMyy8eCYjxc2s/3k/52/eye1L38aq5jYsXSeSz/KJq69j2sgQqWCYkVApk9aUGBnrD9Udq2I0FAZgfUMzRVXl4s1rufim/8u1K58nEYrQPtjHvz9wH9d/8K/ZUVPPLxefQV+0gsVdHaxsbqUrWoEQgmf/ZcFR2LoT12QZofxAMoh66/wP8BPga6ZpOoZR+tAzTTNrGMYzwPuB299oYeVO398wDONS4KvAp49GI8t3CS4B7j4a5UmSJL2VKqJ+/t+/XzT2etmy7cy7sYnHf6DydyueYCAU4WsXXUU6GCSv6SiOQyYYwlFUcrpOd7SC2myG5sTwQeOvdFbWcP+8xTQkR8lOb+L2/zgHgB9dC+9Z/RL3z17AhsZm1jY0s7GuCUsoPPiFGdTE/G/hHjg+TdaxbGQQ9db5BaXszspDzPtHYLlhGHcA/wrspHSJ76w3UO7NwGOGYXxn3CW/N80wDD+lr4PfBl5FBlGSJE0iX3jwprHnF42bPv+WrVSnU2QiodJPu7guA+EoiUCQhV27SPkDDIUjaK6L5jrkdT/BYoHWln39iP7mf28A4COAW+5jJYQYu/NPmrxkx/K3SPnut8dM0xw5xLzVlPLxOvAMkAK2U/qbvAZ4zeCofIfeckrZqDdLNQwjbRhGklJH968DPwPOleNESZL0l+CZz7aR8euEClk0q8CMgV6ihSyDkSijoQjvffVF2of6CdoWiucRtAponstpJ9ccsjxFCBRFkQHUQcQhHic+OWK5dKzJN6AkSRNm2bJlAFxxxRVvaPkbP/owJ/V18fv5p9ATidGQTlHUNIqqSsoXYFpimLbBXr7+T4upm984kU0/ViYkuuma+vGDPuundP7whI+kZCZKkiRJkspqM0ku37Cau371Y65d+zL9sQpGQ2Gy/gC663DN6hdLQ2NMzgBqwkzWcaJkn6hJxDCM9cC0Q8zqME1z/lvdHkmSpBPNB1e9wMyh0rhhzYnR/ebVppNsPc/g/ptPPRZNO6FNlnGhDiSDqElEBkqSJElHpi6zb/Tua9at5L/OeTsIURpQM5Pmm994I/f7SAeaLJmnA8nLeZIkSZJU9sD8k8dGZH+6fQ4FVSWczTC9q4Nbv7QInyZPm4dDjlguSZIkSZPcRx/9O5770wXced8O+kJhPt6S5Z/+aj4gE/1HYrJmomQQJUmSJEnjnHX5XM66fO6xbsakIkcslyRJkiRJOgyTdSwbGURJkiRJkjSxZCZKkiRJko4/ruPymwu/xeZoHVduXoPPsSm6HuqtHwLfsW6dBDITJUmSJEnHpcd+/DIPT5vHVx//I7pburfOA4Y/9RO8712KUCc+C7Lj3hU8e8cqOmrihC5ZyNL5VZyztGnC6z1RyD5RkiRJknQcev75LuJZi/X1U8jpPub1dVORzxJwXbI9RcLN/gmt/97zv8ODMxbQdfJp1GfS1D2+nf6HivwkUsFdP7l4QuuWji054IUkSZJ0QutTA5za08mL02awpqmFP85fjO44ZP1+iv6Jv543qgbwOQ5TkwnClkXGH2Q4FCXv8/GC2T3h9Z8IvEM8JgMZREmSJEknNN2ySAaCY6+zPj+DoTD9oQghtzjh9b/aNLXUcdrzyKsqeU3DVRTyisLmhzdPeP0nAk+Igx6TgbycJ0mSJJ3QapJJpg/2010RB6AulcBsbuPknt30785C/b7LeXbvMD33reDuHj+Fjf3MG+hlbn833bFK/jD3ZJbu3Iqt6+yqquHqf1jKhuW7eXFjiqnJESpyWYZCUZKKQAn5COeKBAoFds1eQPtgPzHXQXNdXCHojsaY1d/LlDkNx2q3HFfkYJuSJEmSdBwq+Hy0D/bgd2wcVWXa8CBPt82iM1bFA521ZAbDrF79PHOXPU13dS2NqQTXd2znXy94J7lAkB019QjPpTabIR+OsKeiikQwxK9+tI7egJ8GPEajFfRUVbOnspqr15pc8tJa0v4APZEo79y6HkcIlrfNpiNegwLEczlUAfVu4VjvnuPCZPmZlwPJIEqSJEk67jiPrWH1tx5md08WLaTzjic/i+Y79CmrK1bFK02t9FRVk/b5eaVhKrtq6vDZNv/vqYfQXYdb33YJmxafjqWVyrAVhYu2b2RXvBYBCNclZNtYmk5tJk1e92ErCrXFIqpX6sETtG3es/oFqnI5bj/tPJoTw8zp6yaj+3i+bSaeUKjMZRkNhvA5Nif17GFgZXKsna/sSnHXFx6jPp3CUgQd8VoKmsqd/30RvrfgDsJjSWaiJEmSJGmCWUWH+8+9lfO3buCUYoElQFLX+eHlt7O7spr87Cl8/MYFzGuLja0TzmXYWVNPVteZmhgm5QuS9PnZ2TiVtN/P93//M87dsZnfLDptbJ3OyjjNo8Njp/bxp/h9nYU9OqqqqcxlOX33dnK6H3+xyO9PWlIqo6qawWCErN9PtFjKOAVti6TncdbOLVy+aQ0/aL6UC4CC5fDTzz/CYEUVO2sb6I9EGQ1FALj2k8/RruT5j2+dj6arE7Fbj73JGUMhPG+y9JE/MoZhPAWcAViAA+wE/sU0zfsOMX+8M0zTXGsYRi3wb8ClQAWQAlYDHzVNs6dcRgD4IvABYAqQBB4EvmSaZmd5mdZy3VNN09wzrn37TTcM4yvAzUCe0o0OA8A9wFdN0/TGrdcIfBm4DKgBhoAXgG+aprnSMIwbgDuA7AHbdZtpmp97E7vwcMk3oHTU2LaLpimMjGTZtX0Yn1+jMqRR1RCBZI6+rYNUzazF1nXiQRXH70NVxNhgyp7rIhQFcYSdXj3P26+MTLZIQAHHsvBFQmOjNxfyFj6/hmvbFEez+EJ+cukctqqjeR7+iiDZZJaeVV0Mbuyhcck0ambU4mkqPbuHyDy7hcYaH+qUegpVEZRUjmhDjHBLnKLlYRUstj26kaf/sJWEDbZQqC+kaQtBTtWwdwyS9vvZGq/FUVQ0x6EhnSDj8xMvpumtqmGOkiXeWo3vzFn0v7qbbNKm6eRG5nlpAhURqi+cy0jGIR7xIbTSvitaDh2vdoMCen0FuucRqwzi8wmGejKsfrUfJZnGF4/S3F7F9OYIu7rTdO5O8fv7tvDuFc9yXsc2AJa3zWJFSzs+x6E7WkFftAJLCNKBANXpFKlgmFguy4L+Ls7euYWle3aV9q2qsmLaDJbNXURDOsn6uiaEgIw/gOK6fPCVFSybs4ioVaAplcDzPDTXRQA5TWdTXSOuELgC/u2B+whbRVzg7iVnsbl+ytix3VFVw8zBPmYO9e89+Lzc2MIFOzczc6gPs7EZCxVXVfEJD79jk9V8/Pi083CFIGhbVOayLNqzk201DXz08nrOmx8nHA8R0uGZ7Uk++pvMfu+v81rgv6+tI+8oRAMKytHtpD0h4c6GmZ856LN+3tb/POFDK5mJ2t8/m6b5dcMwNOATwL2GYbximua28fNfY92fAQlgsWma/YZh1FEKqDwAwzBU4E9AA3AdsBKYCvwn8KJhGKeaptn1Jtv7lGmaFxmGIYCzgYeBXcBd5TqbgJcoBXOXAZuAIHAN8K5yGwB2mKY5403Wfcx1/PIlvv1gP9FslpBnc9oVc+iIVrHp92uYOjRIVWOMy79yMZXNlXjpPMW/+jb285vorKpm87svYu3ONF7BJnbyVP7hlnPYuD3Jd+7eTL7gcuO1bZwdyJH/q+9g947yy4VLeXjOQurifr75Twv5+bLdPGsO0NYc4XN/PYdoWD/Wu+M1pYayfPcfH2JjoIqG1CifWv4Imuvw8rsv4Z23vJ3EjT+kf2MfD8xcwEAowkgoTLhY5Kbkdmbf9/e423tJfPA7bPRCrJ4yjbyusz1eSy4U4rqPLuSiU6vZ9swOnrntOVRN4Up9FP8TqxFNcdyRNEJV4bs38a2OAKs2jNDW10vr8AAA3bFKGt4+n6suaOJ/fvgqo0NZ6kdHUXSFCz9/IWecHOfz317Hqb95mFO6OuiuruH7p57Hou4O3rVuFY4Q/ODMC0j7A+Q0nZzuY0pimJbECLai0Jgc5ZIt69geDHH70nPpiNcyZXQLl29YzeMz5+IJhfbhAZoSI5y9YzP/euGVjIZCNI0Oc+G2jbzcMp2G5ChtI4PcveRs2kYGqEsniebzdMYqEYqCC5zZsY0ZQ/30RWLcceo55HUfUxNDRAoFovkcswb6OKW7A4BH22bx0yVn4XNsCpqO6ro0JUeozOeIFAv0hWP0RStQPZfm0WHqU6Psiteiuw7DK7N0V/QiPKjKZWhKjuAhEF4PZ+/cwtz+bm459x0MhSKkAwGEB/Fsml1Ns6nOJJnb10Pe0diacNkTq6A9XOBjLz7NyuZp/OKUM7CFYFskQjyT5ppVz1OVy3D3krMZGO4n+HI3W2vrqUmnuPRnf6A+ncQWCs9Nm84dp76Ns3Zt47pVK8hpGqOhMO3Dg7jAw7MW8L2zL8YVgsXdu0gGwmPbFikWWCcUeqIV7IrXoHgeRT3CC22zOK9jG0P+IC9NaWUkEGT1lFYEHjP6e1EUQcBx2FVb6rRd0HVGkiMs6ukce9/7HIdFPZ1sr6lnJBShIZNkZ2Ud7NPfAAAgAElEQVQ1c3u78NsW6+qmUJ8aZUtdI2fs3MpAJMpIKEJPtIJUIAhCoAA+u9RZHEoZqqZUgpda2qko5MlrGrviNXRWVVPUdIJWgemDfXz18T8yNTGMoyi0D/Xx0ffcyEVbN5Dx+ekOVNFRWc2vf3obPsfmXy+6khemzaA+nWRqYoTvPebn2XvWsLF+CjvitYf8uZSndsOcb/XvN01X4NGP1TOt6vg8rU/WPlFyiINDME3TBn5MKchc9AZXOxO4yzTN/nIZ/aZp3mOaZm95/geAc4CrTNN80TRN2zTNncD7gDTw1SNor2ea5nJgPWCMm/U1IANcY5rmetM0HdM006Zp/tQ0zS8ebn3HA8/zuOc3O1Bsh6Bt4TkeL/x+I8/fv4Vp/f0ojktizygr7ngJAOcnj8Iz69Fsh7aBfoKPrMKfyaF6HunVu3nppR5+9KvtDI4USWdtbrt3G9bn70HtHMBvWVy/6nmi+Rz9wwX+655tPPZ8H/miy8YdSX7/2JuNfd9af/re82z3xRgNhfnkc49Rl0kRz2U551f3Y37rCayXtvFs83Syuo/+aAVFTWckFOanoRa483GKN99LcTDNS9PaKWoaiufRkEqgF4rc+fNNeK7HE//5NPlEntiOLvzLXoRMAW9rDwym8PoTPPKfz/DiuhEsF1pGBlE9D9XzaE6MsO7ZDu66az2DgzlsT9AVq6QykeTXt73MMysH8a/YyJkd2wnYNtP7erlg+0au2rCaSLHAQCRG1h9AgfJJrMjMoQGCtkVDKsFVG14haFs0pRJcueEVALoq4/RHY7QPDzJjeAAB9FRUsbOmjve/+gIjoQjpQJBnp88ir/s4o2M7P1l6Ln7HpjqbwVFURkNhfJ6HpyhMSY4yr7+nNE5QYoTzt2+kITVKyLJwFYVEKMyswd5SvxvgbTu3MHVkiILuAyEoahphyyJaLGCpKn2xShACR1EZiMQYjMSIWEV8jkNPrApPKCieR+vIIMIr9zQRgmenz+YP808hHQiSDgTwhIKjKOyM16I7Dgt79hBwbFxFwVYUpiZHuHLDK2iey5Mz542drDXP491rTaYkS9vwsRefpjE1yqa6RhxF5ar1q2hKJVA9D5/rYOzeha2ofPSlZwjaFvF8junDgwhABd6xZR3z+7sJWUXSgRAK4AmFzqoaBsJRcrrO7qrq0slICAQeT86YR04o9MQq6YzXMBKJMSU1Sl+0klXlwLa3onLsPZ7xB1CEoC+y7xKfIxTyqkZPtAKAmQN9tI4MoXserqphaxojoRCLu3ezo7aeRDCEpaoUdH2/2+9tVaUrtq+uzbUNrGlo5unps1neNhtL0wkXCzzZPodVU1q5eOt6WkeHSvvHcajI5zln51ZcIXhh2gzWNTbz9Yd+w6KeTub19/D9394DnkdO15k92Et/tJJosYDRueNN/Z1bLnz+weE3tc5byUMc9JgMZBB1CIZh+ICPl19ueYOrPQP8h2EYf20YxuJy5mm8y4AXx2W1ADBN0wJ+BbzjCNqrGIZxPrAAGD8oyWXAfeU6jkupVOqwn3vj/uWAqXtZRbs8+bWXE+XZ+13a9sCx7XGvS3/2AJa1/+50y+sdybZM6PNxm6WUv1EDKN6+bTnUmC2eEOB6OLZ90F4eW7q83/buu9f6WHy9XgMC76DD6FEuFxAHrKyUDhYA7iHavfc47Q1a9luvLGDbB63rCYHiMbbu+LVdIQ5qx4H1ja/nwFY5+9UlSt/Kx5W3t21i3LYd2GaPfcdpb/C0X6meh+44h2gf6I491jl6b1m2UMiXO1lX5vZdzXeEIDDuPS7w0F1nrC7fuDrEIbb/UBTPxRMHdy4+VH8kEDQnhumM19AZryGvlwbL1FyXeDZNxudHAEFr3/hPPttC8TwemzGfh2fOZ1t1Lc+0zeKzl7+PRDCEcF3Sfj/NyZF9bQKihQIK4LcsWkeGiOayDIciDAdDY8tV5LI0pEYBcIFIocDcgR7U8qW/BV0dzOnv4aS+LmoyKTbXNmIp+06teU1jNBgk5Q/gKgoCiBXyY/PDxQKK53L+1g08NnM+Iasw1r43a+8hPpLPjYniiYMfk4EMovb3RcMwRoEc8HXgJtM01xw4f/xj3Lz3Ubqk9xHgeWDIMIzvlPtBAdQCr5Wy6AbqDqO9545r7xPAncAPx81/vTrHaztwuwzD+OBhtOdNi0ajh/VcCMH172zB0jRymo5QBEuvnMPSS2eyq6YWVxFEGqKc/ddnAKDeeDHemXNwFIVd1bWkL1hMPhjEEYLgwqksXdrAx97bTmVMJxhQ+Zv3txP45g24jXEsVeXni88gGQhRXenjUzfM5byldWiaYGZrhKsvnHJE2zLRzy/7+9OZXkhSkc3ygzMvJOkPkNV9PHPNJZz+TxejndLGWbu3EbAsatNJdMcmlsvyV8md8JELCH7jevzVEZZ07kR1HVxK3/iLPp0PXzcLRVU4/9NvwxfSGWlpJP8OA/w6oq0OYiGIR3j7p85iydwKVAG7q6pxhMBF0BWrYs6Z0/jQh+dRWeVHwaMpOUoiGuXd/8fgbafUkDltDi9NbcVWFHbX1PL4jLn8cd5icppOU2KEUKGAB/itIjndx67KGjxgJBji4ZnzcYRgKBjmgdkLqcqmWdqxnVguQ2dFFbsqq/GA+mSC9oE+fnnyUuKZNNFchjN3bUFzHJ5rncmHzGcZCQYZCobB84jmc7ieh3Bd9sSq2FJTjysEvZEYT7TPpTdSQVFVwfOoSSXoj8QQnocLPN02i46qWvyWheY4Y2VZioLmurSMDKLbNn6rSHUmRWNiGI/Sh3V9chQ8D0tV2ROrQnVdwMNnW1y66VWu2PAKdakEsVwWzXEIWkVO6u7EFgr94dL7wQPyms7W6lpuO+si1tc1cdqubSiOg+eVLrHefvp5jARC2IrCsrknMxoMM2OwD82xeWzmPPKaVg7EBM+1zURzHe5ceg45TWc4EGR9XWP5GMMjM+extmEqOd1PoJgHz0NzbNoH+6jMZfA5Did176aoKDhCMBoM4XkerqLQnNg/s5L1+TilqwPheSzo2UNjcoTaVIKF3Z0IoKMyztz+burSKYK2hSNg6ugwmlf68rCwe/dYMBzPpvFbFqFigQ++soKr16/iRvNZlnTuIBEMs7mmnuaRAT71zIPUZtKEC3k21TRw0db1nL9tE9e++jKffPoh2kYGyfgD9EZivDplGp+56jpufO+NAKR9Pv40bxGb6prYWls/th13nHo2BVXFEYJ/O/+d2IrKfQuXsidWyWkd2wHYUlPPm6Er8M3L4kf8uTFRJmsmSnYsLyt3HH+s3CeqCvgJkDNN87oD57+BsnyU+kP9FPiuaZpfNgzjXkqdws85xPL/DNxommaTYRhTgD3A9PLlvr3LzKSUFWswTbOv3LH87HKfKB/wGeB64HTTNJPldbqBO0zTvPl12noDcPMx7BMl34DSEfM8D9eFfMFGAHs6RrGyefKjBRxNo63Gz66iIBpW8RxBpZsnPiVOUVfxA6qm4gkPVdewbBfXdgkEtLGMmKIoY53FxzqNe95+/VUcp5ytEcrYKcL1ACHI54vYOZuiZROtCuEr36pftFxcBFoyiZcpMqroOBkL4VfRBGiWBbrKOnMPieVbaXz7PBrnNqLkciS2D9HRl6Uh6GGjkxnNMe20FnK6j4jPI+sLUlHI05m02HP7cnZ3Z5gy2M9oKMyeuTOozyTYnNHRVIWiECzs6WQoFGV7XQO2UGjr28O0TJLB9mm0t8fIZIr0FxQKDXFq7SxN9REWNel4S2dTTFk0tFRANo/l9yNch5HOYbJDGV7udqkVOc68cAYpB7o7k/S9sJNQMUOfCLDwjBbCMT+b1g9ANMSqO1+iZdsu/Aiu3vAKLze3saG+CUsIhkIRKgt5tlbGidg2tqqS0XRUoD45wgU7tjBnsHfsmLwwtY11TS1jr0/t2EZDOkUiEKQhmeDfzr+MU/fs5N3rVo0ts6Klnaenz2ZzbQNdsSoevf0/xrJve2JVfPXiq/HKmSbNsSkqCiORGBtrGyno+/pGzuzrpn24nwXdnfz2JIPeWBVzBrrpqoyT8gfxANV1SQeCaI7N1OEhpiSGiUY0PvnpJVRX+REIntg6xBceGZcVB+oD8L1rqmiu1KkOK/i0o3pH34REN6/O+exBn/Unb/r3Ez6SOj57oB1jpmmOGIZxE7DdMIyrTNP8w5tcvwj80TCMx9jXp+oh4HbDMKabpjl2sbvcif29lO7SA+gBCsAMSnfj7TWDUv+m/XsT7qvvG4ZhXEqpb9Wny7MeAK41DOOrx/MlPUk6UkIIVBXCodJJbNbc2oOWOXgKHOpnaX26CuXbzEW5bA7x/4EdflV1/ImsNG9vqj8U8kPo4Nr8vvI6NZVQ89rp6LNba+HaxftNq5/fzKzXWH68WuCUM6a9gSWPwN6dGwuVTyoqjbNKnb7bxy0WBOriITj54FG8p8wsZV4uuKCNr/39g9zw+2X8dPEZ+G2LuX1dNI0O88c5C+mdN52v/ftFhEL7fhPva9f8HEdVCdr7/8SLo+w7Jh6wurGFyzavJWhZ/GnuyfTFKljf0My71q0aixwGwxFcIfjBb+9ma3U9lqKOBVG1mSSesu+4W6rGSCiMBwTs4lgQVZtO0jg6xEgwTOyi+Tz52SVYtsv1f7uTYKFAVvcRKeSpSybI+/w4rksNRT7zrimc8o45+23D+0+p5/2nvIFjIB0TMoh6DaZpDhuG8S3gXw3DWPbnli8v+wtgLVAE3gacD3yjvMi9wI3AHwzDuJF9d+f9B6UhEb5Srtc1DOOnwNcMw9gB7ACmUwqO7h4/fMEh3Aw8ZhjGd0zT7ABuAV4E/tcwjM9TymQFgCuB+a+XoZIkSToWhBDc8v3LSN1yDjXvuo3I8DDDVVUsfOXr/K3/0Kesc7eu4/4Fp/LA3JO58cWn0V2XlD/AnaecRcwuECoW6YnE+O/f/RS/Y1NUVS7esp5IMceTM+bz+Iy5nNq5E7O5lbUNzVy4bQOxYoHN9Y14isLSPaXvsytaZpQzkACly5WRYp5kIERzYoSAZaG5Lp974n6Wdu7ge2dcQF9LKwC6pvDNzy7iu198iq6qOG0DfRR9Pmyh8PXvv51YPHTIbZssJuslBxlEvb7vUsrqfKj8+kvlYGS895umeT+lL513Ai2U3i9dwK2UhjDANE27nCm6Gfgl0ERpnKiHgKV7x4kq+xSlsZ0eofTltJ9S5/N/fr3Gmqa53DCM5ZQCrhtM0+wyDONUSsHUI0Cc0jhRKyiNabXXdMMw0gcUt8w0zQ+8Xn2SJEkTJVob5R3LD/y4PbSYZbO0czsvtM7kGxdeQTSfo6DpVOezbKqppzqX4X3T4d6Fp3Lm7m2MhiJsqa3n9F3buGjrRjTPZSQQQnNsPvvknwg6Np0VcVY2TWPF1HaSuo+dNfUMRGOlsnUdn20j8GgZHWZtw1SSgSBzBnrRXBcF0F2Hk/p7iKv7cqDTZlTzrV+9e4L22PFtsvzg8IFknyjpWJNvQEmSjsjvl36Ds3dtZTAcJR0IorgufzhpCT7LouXqCJU1KldccQWe55Fau4e1y7fyp1eT5JIeVfksVbk0Od1PLJ+lYXSEkXCYzoo4V/7jGax5eBubtyRR7CIzhwfwPJdVTa2EC3lsn4+C7kN4LkVfgLzuI55J0ZBKYKkqum1TvaCej3zzsmO9i96MCYl2Xpn7uYM+6xdv/OYJH1nJTJQkSZJ0Qsv6/YwGQ9TkMtTmMnjAvN499IejBDI+qAkCpUuFsYVTOWvhVM56g2UvOm/mQdM+fMDrmz7yEMPBMEv2dKB7zljfqIKuE+wcPPwNm0Qmy914B5JDHEiSJEkntI21jWyvqd9v3Kmm5CiLujooRg91+8DRZQlIhML83DiTnljF2HRXCGa/c/6E138ikONESZIkSdJxqKuhgQdnLsAdNy2r+VjQ143VceDPgh59tU6BBd27uXqtSevQIOFCHhdY0ziVxR9cMuH1nwgm6zhRMoiSJEmSTmhve3sbLaPD/Oj089lSU8/zLe2c1LOb7mgF1a0T32vl1ruv4oYPzSEdDtJRUYE7u4n3/MvF3PeDt0143SeKyZqJkn2iJEmSpBPaDVe38s17fUzv7mN101TO27aJjKIS/6d3sqMq95a0YfGlc1h86Zw/v+BfqMmSeTqQDKIkSZKkE97nfn39oWcs+7PD/Elvgck6xIEMoiRJkiRJmlCTdSwbGURJkiRJkjSxZCZKkiRJkv5y/O3fPkx8Tz++xihf/tHVx7o5J7TJmomSd+dJkiRJ0gE+/qE/Yec8+mvqWK9U8dEPyb5V0sFkECVJkiRJ43zkpocIOBa2UKjOpGhOjDBYUX2sm3VC84Q46DEZyMt5kiRJkjSOV3TIqRqnd24nZFkA6LZ9jFt1Ypusl/NkECVJkiRJ41ieS4XjEi4WUV0XTwiaEiPHulkntMmSeTqQvJwnSZIkSeN4QmE0EERzXRRA9Tz8js1nb37uWDfthOUd4jEZyEyUJEmSJI0jPJfWxPB+Y2wrwNrk5MymvBVkJkqSJEmSJrk1q3tYMDRAR6ySvFrKM3jAtnjt5EmfHAOT9QeIZSZKkiRJmlSyBYuvfukpoiGdBYs8VPWNn7C/f+vLzFAUIo6NOXUa0UKBs3dsZkFfFxsamiaw1ZPbZPnB4QPJIEqSJEk6oa3dNsInvr0BoekErDzTUwk0IEeeh3aoLDrHfd31Pc9jsHeUO3+wGheFoqZz1q5tOIrCprpGnp4xl4s2r8NS9bdmgyalyRlFCc+b+PykYRhPAWcAFuAAO4F/MU3zvnHLnAHcUl5OBTYD/2Wa5t3jlvkKcLZpmhcdoo4bgDuA7AGzbjNN83NvsH1FwAWGgOeA75imufI1tmO8M0zTXGsYxl3AdUChXM6e8jb8t2EY6XHL+8v/F/ZOME0zUi7/XOBc0zSfGVfvNuDrpmneNW6aoLSPGoAm0zTT4+adBzwJbDBNc/4B2/ogcCnwEdM07zIMo5XS8ciyf7J6jWmaZx5yhx1dMkEu/eV4ZQe8soO0ovKx5RZdldUorkvb8AC12TRtgwN4nsva5lbSmo6jaWi2zSmJXk7dvIVYPo0rFCxV4/mW6aybMYu4sBmsr6HCLtJ+cj1FVaPvqc3M2LSVUc3Pqimt+FyXtuE+6nJpNlfXE83l6IlUsrmugVgui8+2yasatfkcolig0raY0d/LQDRKTvOxuX4KarFIfT7FYCCCIjxaRkYYDoXZXN+E57n4HIe8puFoGrrjEM7nCDgOBVUDzyNUtOgLh9GEQHcdKrMZRv0B5gz0MBKO0B2pxOe5LNm9nc6qGnZX1+AhyKsqyVAU3bFRHQef52F7Ho6u43dsLCHwPA+/4+ApCp4QxFNJ4vkcYbtAMhBkV7QKv2NjazqOoqA5NrplYwtBxHHwORaVuSwZf4CM7ucjK58dO+WnfX5ebJmOz7ZZMW0Gt/a+yNCGPpoyCdRPXcnS6w0UdVL1jJmQaOfJU7560Gf9+atuOeEjq7cyE/XPpml+3TAMDfgEcK9hGK+YprnNMIy3A38EvkEpCMkB7wR+ZBjGdNM0b3mDdewwTXPGkbQPwDCMacDHgBcMw3ivaZq/O9Ryr+Fu0zRvMgxDAd5HaTu3mKYZ2buAYRi3A5ppmjccYv0h4FbDME4zTfP1AozzgelAGvgA8OMD5juAbhjGWaZpPleutwU4Deg+RHmzTdPc8zr1SZJ0JG57AL5wDwBfvOID9Da1oApByLLor6pmV30Tu2vq+cxTD/DQwlMpqipZTSNk2zxSVY1XcPjUs48igKQ/wI6aen4Xq0F1HK55YRWRYoGO9XGemT6bT7y8moW9e3CEIB2MsK6hmbkDfZzUs4drV5vUZDNYisL/fecHeHLGXPK6TlHTCOfznLVrK56AbU3NtA328cv5pyA8j08vf5jLNq3BBW4780KenzaTXfEaXFUFIFAsELYt8DxsVaM7Xks8lwUh8ID+Kj+OouCWB1rsr6zCUVT6qmtRHIdEKAxCsL2+ib9f/gjJcITZg31kdR9Ptc8hEwiWfn/N81Bcl8p8jmghj+46FDUdPI8+f5SGdIJ3bFtPyLLoqKxmRUs7CgLHdaCcNChqOvghYFukyocn4w/icx00xyGj64QtCwHYioKtqiQDAVwhGFzfz7XrVqIAPf/vdm57YCt/94sPTLZA6qibLH2gDvSWH3XTNG1KJ3wNWFSefBvwC9M0v2qa5pBpmlnTNH8NfBr4Yjlb8la2scM0zZuBe4DvlbM+b7YM1zTNX1AKiha/iVV/DDRTCoxez98ADwE/LT8/lP/P3nmHx1FdC/x3Z2abVqsuWe6WewUMl2J6CxCIIZBGAiE8YhIgPSGk8QJppLwkkEryEgjJS4BUSExC6L3EvgZsDO69qHdt35n7/piVLMlykbEt2b6/75tPd28598zuaOfsuWfO/TW+MdjNh4H78I1Ug8FwMPnpP3uKy0aN69mQNREOkwj6zun6WDEby6uY2NKABQS07vmSvnDl6z23oaJ0itn12wnlsmghKMz4Tu1NZRVUxTs5qs7/PWRrnTeKBGrMREJujopEHICA5/H+115CAKF8IsnyRBeRnO9o10KwfNQ4BFCQSXPhymWAf9O49I1X8Cyrx4ACSAVDdC+a2doj5OZ6zlEAQdf1n9DK17mWP9azLFLBYE99zrZ5YPZxTGppRABLxtaQCIV3bGArBJ5t01IQJZjL+gZRvr44k2Lu9s09CTLHtzUzubmRoOf29On+K+h7A3QtCw0kg0F+L0/lT0efQF1hjDdGjMZD0BAtIpTJMG/Lup5xI7s6GLt5C3Vrm3f1qRvyaLHzcThw0I0oKWUQuD7/crWUciowGfj9AN3vxf//e9tBUq8/9wOjgWmDHSiltKWUHwDKADWIoXHgq8BtUsrQQB2klJXAO/GXL+8CjpNSHjdA13uAd0opi6WUNnANO3ushpTOzk5TNuUjozy2oue147o9Zby+8Tql8S6aCmIAuHkvDkBDrKinjysE8UCQjO1g6R3jY+kUHaFwz1NlAM0FUcA3kHKWhdfLI7CtqNRXIW9cJAN9Y34KU0k0kHICtIUjPfX1hUV95gWwvB2SPSFwLavH89OtM7sIH7F612vNuLZmMvlzSDu7WDARgrTTV18tINWvf/9z6p6jf66icDbjv99C+J6uZII1ldXY2iPj2KQCQUZ1tlIbK9kh2wnQEI1hR3fIGTbX2z6WDxTm6by3zleklDcCMfyYogVKqWVSylPy7dv6D1BKZaSUTUDVXs5RI6Vs61d3g1Lq3n3UuXt5q/emSd3n0VvPkl4vPyilfDf+ctom4MNKqWcGOe9vgE/lj+8N0P5fQDuwUCmVlVK+CnyEfh4ppVSDlPJx4Mq8LnVKqdeklAPN+YaUsvd3yr1KqRsGqfegicVipmzKR0b5j5+H934PvWYbX3riH3z73IvJWjYViS4Crks4l+P4LetJBIII7aG1xsnlQHtUdHWysaScEyybnG2xrqySB2bNJZzNUJJMsHj0eEZ0dZCxLHKWxa9POJ0z16+kKxji4WlHEUslqGmuZ0NpJYtHT2BqUz11hcXcftr5CM8jlMtieS4ugtrCIsa1NVOeiDOxsY72cIS1ldXcfP5lXLXkRbqCIZ6YPIPiZIKaxnq2lZShhWBkewvN0RgCyFoWkxrrsLWmPRIlks1gh8K4kSgFqQRB10VoTWtBFMvTZCyB0Jq0E2B0eyuTmht4fvxkxna0MbqtlbZwge898ly0sPwlQq3pCIaZ3lRHWyRK2rFpLyzi1VHjKE0mKEynWDFiFFuKywi6OUKuS2E6RXVHG8lAkI1llUSyGaLpNNFMmmtfepL7jp3Huspqqro6iGVSgO8pO3PdKtaVtbChuIQlI8ci0BSnktRGCznm42cwYkzl8LjG9kPZMDgOphH1rXxMVCm+9+Ts/N/GfPtoYGXvAXmvVUWvPntiw1uIiRqIMfm/vX2139pDTNT/KaUWvJVJlVKulPIm4D4p5V292/JLi9cCv1dKdQe43wV8R0r5ud4B5nl+BXwX34janRdqlomJMhgOIKWF8NjXEfhffmfvpuvFA9b6z9OEgKPyx97wgZ7Szg79dww4YkafV+/r8+o8AObv1cwz9txlQMbSfXa5nMvFt7xGcWM9IWBcewttkQLWV1UTTqWZ0FTHlkiUs9avZuXI0XQFQxRkM5DL8dSk6eRshwtWvc7yWJSz55TwzGvttAcCJFxNJJ0ikMkwurmOLifIX+ZIGsMRgll/q5feeEIwpq2F1cVlXPf49Yi8527qPp7hkcjhsnzXn4Oe4kAp1SqlXACsk1Jegh9Qvh7/f/2Jft0vx/e4PnZwtezhffgeslUHe2Kl1MNSykX4S3u9OQd/+fOa/HIh+J9jIf57+L/9+j8K/AI/EP3KA6exwWAw7F8cx+Zf3xooUqGbmfm/J/ap/eSla/igeoHtsWKWllbwqYuSzJ9/Ll/eiznPueEFRre34FkWGduhIt5FWSKOGj2O0Z2tPQaUYXAcLst3/RmSPFFKqRYp5Q+B24CF+E/rPSil3AD8HD/w+SLgDuC7SqkNvYZbUspwP5GZ/amflHIssAC4GnjfHp6SO5B8HniZvuf3EeBZ+v9A9N/Lj9LPiFJKaSnlRUBEKXXgF74NBoNhiPnxAzuey1m4cOGgxhaHPIrjXUxsbWZ9uf+E4SujxrF81Diq2prRWhtDah84XLd9Gcpkmz/Cf/ruqny+onPwvS434eeJWg3cqJS6u9+4s9j56bLu/5iJ/fIxgR83tKcn3QD+W0r5BXzPVzPwInCyUmrRAP2+2K/ucqXUQ3sxx6BQSi2VUt6Pb8whpazCDyh/l1KqrndfKeV3gRVygIAnpdSbezHdqn4xUW1KqTG77G0wGAyHIX+7/TQu/q848zat5XcnnLajQWtaIiZ2yMwCB8QAACAASURBVNCXg5Js02DYDeYCNBgMB4xuT9T8+XsXyQXwnmse5arXF/P3OZJtJWUARNIpXGHx95+esofRhzwHxGX07xNu2+m7/oJFXz7k3VNm2xeDwWAwGHqRcT3eGDmWlRXVHFO3hYztUJBKcvu97xxq1Q5ZDtdfy0eEESWl/AW7DqqeqZTafDD1MRgMBsPwJSwEKcfhhK0bcLSmIJsl4QRMLNRbwMREHcIopa4DrhtqPQwGg8Ew/HFtm6Tt4PQKd+nO5G7YNw5XT5TZ7MdgMBgMhl586EPTCeeydOeV10Bnry1uDIPncM1Ybowog8FgMBh6Mf/0MWwcO4pOO4CTzbAxGuPMk0qHWq1DmsN177wjYjnPYDAYDIbB8NsfnzXUKhxWHC6ep/4YI8pgMBgMBsMB5XANLDfLeQaDwWAwGAz7gPFEGQwGg8GwC17+z1Z+9r9v4toOLoKI7fG/PzqTYNAEmh/qCCFmAO8GqrXWHxNCTAeCWutleyvDGFEGg8FgMADv+cRzOOkMTiZHLKj55GeP4c5frcRzAgAEtCblWiy44Unu/sU5OI5ZzNlbhttynhDiPcDPgL8BHwA+BhQC3wHO3Vs55gowGAwGwxHPFQseY1J9HWO6OhmRSdIkQlx9TyOeZZGyHSytydk2NpC1HT5z49NDrfIhhR7gGGK+Dpyntb4OerJZLAWOHowQ44kyGAwGwxHPqM423LzHSQDTmxuIujlSTgDHc+kKhUEIcDSRTJr2Lnf3Ag19GG6eKKAK32iCHTbdoO0744kyGAwGwxHN5255iYwd2Onu2RkK0xUKkw4EfQMKQAhcyxoOnpRDimHoiVoCfLBf3eXAosEIMZ4og8FgMBzRzH3qZVojUR6ZdhQj4p0kAwHeGDGKrG0TTacZ3dm+o7PWeK7LYZr26IAxDD1RnwQeFUJ8GIgKIR4BpgLnDUaIMaIMBoPBcEQTcl1emDgNB01drJh15VVMamlgUnMDnhA0R6IkgyEAwtkMYc/DLOYNjmHgeeqD1npl/mm8dwAPAVuAh7TWXYORY4wog8FgMBxWfOxjj7G0cAQVXR0UpSJ0hMP898srmNTcgBCQtSyOn13CzdfPoaU1xWOTZ5IKBAEIeC6VXe1Mam4AwNKagmyGrlCYSCZDNJdFA5nh51kZ1gxDTxRa6wTwp7ciwxhRBoPBYDjkWb14EzfdtYHyZJL6oiI8y6K+uIQR8S2Eu7Ksq6zmtfGTAKhpqmfp0hY+9NnnqNhaTwWahuIde+MlnQA5y8LxPMDf5y0eCFKcSqABFzEsjYLhzfB6v4QQz7ELB5nW+vS9lWOMKIPBYDAMW7TWpDMuX7ljKWvXd1DgeRRmUqwvq6IlWkjWCWBrj8JUEqcgRmkmS2OshLq8UVRbVMKV6gVGdrazLhQGoL6omIDnEmlvo62wiM8+92/WVIxg0dhJTG+s5XfHnowaPZ5Z9dvJWRYbSysY09EGwiLlWIRzWSKux92/eZ1r/mvOUL49hwzDcMPhX/d7XQ18GPj9YIQIrYdupVJK+TQwD8ji52nYAHxLKfXnXn3mAbfk+9nAKuDHSqnf9upzK3CqUmqnBFlSyquBu4FEv6afKaW+8Fb124863qyUmry3bf3rB9B1PfBNpdRf93COFwI3AkfldV8OfFkp9dzuxu1HhttSucGwd6Qy8Lm74fkVMG8afO9qeGEFLFkH0TD88t/QEYdYAcweD196N8id/sXhi7+FP78A1aW47zsN9/Fl2BvrsLpS5LRGl8Zw5k3FetfJZE6bzaPP1cLrG6l+6XVWFpSwaMJkarpaOfbJF3mjvJqVVaOIB4LMqN+G3LKB9kgB0xrr2Fxcxh/nnkR5vAvXtilMJUg7AQKuRyIQ4NUxEwhmcxTk0pQn45TFu2iKRNlaWk4yEGRMWwtTG+vYVF7JiI42zl+9nGdqpvFG9RgquzrYUlJK1glQnExy7LYNrKkYSdpxSASDNERjxINhAp5LLJHgmLoteEKwobQCAIHA1h6uJfCEhaehNJWgrKuDt695g/JEHDVmAn89+njiwRAIQTCXo6yrnZXVYwnksqSdAKPbW0kEQzQWFpFxdvgIzl21nPpYEasrRzKmvZVQLovwXIrSKQKex1G1W/jcMw9jATkhuPCaz+JoTdBzIb+c1/tRdtt1cbSH1pAT4AoLGyiO2hw9p4JIxCYSdpg1q5xp08oO5FV4IDgg5s79p9+x03f95c9+eliZVkKIycBvtNan7e2Y4eCJ+oZS6ptSSgf4OHCvlPJVpdRaKeV5wD+AbwNXAEn8ILBfSiknKqVu2cs51g9koLxV/QD2o477g966fhb4o5RyplJq9W7GlAI/AZ4CuoBrgYellDOUUlsOvMoGwyHKZd/2DSiADfW+AbWpced+XWmobYWnXoeXvwdTR+9o++Yf4ecP++XGDqzXNwH+rxmAAJBpiZNbX0/w/55m4fUf5Mk6wQ8W3kc4l0UCtaefz9WLnuXeY+fx9OSZPaL/UzOVmfW1zF+xFAuY1lRPRyTK01NmUB8r7qNiynbYVlJOYTrFxNZG2goKaYoWsam4jNZoIZ5lIbdt5IVJ08jZDhvLq5ja3MCmiiqwLRqLSwhqTTibwXNsnpoyi+aCQt/w0Jp4MNQTc+RaNtUJP3bXs206QhHCbg6AjGXz2qhxFKWSzGmsZVbdVqbmY5POX/MGaypH8FLNVL+v45AJBIlk0iSCIYQQbC8uJZzLEcmkGdPWQnmii0QwSCoYpK6oBNAEcxmqO9sJuC6pQBDPslg6ejz3yNM4fst6Hpk2m6xtM76lkaxt0xaK7JQLSOR/+2kBlrB2nGdXlpdequ3p9/C/N/HFL0gmTSrZ9XV0hDAMPVEDsQ3fobDXDJs8UUqpHPArfMPumHz1z4D7lFJfU0o1K6USSqk/AZ8BviKlnDDE+g0rHfvp+nP87+Ld+pqVUn9QSj2glGpTSuWUUnfiG4LyIKhKZ2enKZvyoVl+bQN9GMiA6k3OhSXr+shxH1rcp4tgZzeAoz1cy/+qDixZw4SWRsK5bE97dVcnpakkK6tG7TSltkTPl7wAJrQ10Zlf0upN2M0RcF3fy9NrXlt7ePm5K7s6ydk7fncvrx7jJ6DM0/tm4uu3I69Spte4tkgBbj6eqDwR7zFIAIKeSyiXoy1SgAZEv5WSwnS618lpuoIhgrncjhxOQCCX5fgtG5jRWEtVvJMJrc0E3BwF2QzasihOp4jkcjhaU5BJQ36Ou048g/dfeQOrqkYyu2E7kVyWonSK4nSKTK9PRXgeVn5M/8+q/2vP06xd56dHGDbX7R7KRwpCiGv6HR8H/gm8PBg5w8aIklIGgevzL1dLKacCkxl4ffJe/Ov1bQdJvZ30y9cNKx27yev6MfylvaV76N5/7FFAOf6y3gEnFouZsikfmuXTZ9GHmePA2s3P7XAA5k3vI8e+8sw+XTTg9RuWFfkAZ8fCPXMOaypH0JE3XjxgS3EJ9dFC5JZ+Rp3WBHPZHoNFA2sqqihN9o9sgJQTIGPbxNLJnjpXCHKWje36D/NvLyqhsqvD19t1OX/VMt8IydNbb1dYhHIZACzP6yN3WmMtdt4I6QyF+6znxwNBUoEABZk0FvDK6PG0RgoAWFNexerKamKpJJFMhpJkgrJEnIxtE8wblQKY3FRPyOuXgEDTIyec3WGAWvkxGcumvjBGSTLBihGjaSgs6unjeC4WmqwQ5PCf3hM7xPafpg+BgMXMGf5y3rC5bvdQPlBoxE7HEPPBfscFwIv4++jtNcNhOe8rUsobgRj+TX+BUmqZlPKUfPu2/gOUUhkpZRN+2va9oUZK2dav7gal1L37ql++rXI/6rg/6NY1A6wF3tW97Lg3SCmrgL8A31NKrTlAOhoMhwd/+Bz84EF45FU4ew58/jLfO/Xaeqgqhv/5GzR1QnkMTpgCH38HTOj3dfDxd4Blwd2PQ80I9AfOQD/yKu6GekRLJzlXI0YUEzh9Frz9OC45agIVSxpZct4oSl5+k1XhEtLVo/j7yZMY/6+XeMebr/LGiFE0RQqZ0lQHluBPcyTTG+tYV1rBUxOnU5ROMamxjuJEF13hCLbn4WKRsiysnAuex/jWJiY31NEYK2JdaRVbSst8mXVbGRWO4Gp4rXosKQTBdJryRBebSstJOwEmNdUzd9smXhkzgWgmzYbSSuqKS7Bcj4p4B6WdHQSzGZKBIA2RKNFMmrZwlK5QkPZQAZFUkrHtrSScABrNnSeeRcTNkrQdcgiKkwmytkMslaQ9GGJMRxtFyQSJUJhoJk00k+7jmfKA5kgU2/UIkaU+VkxNSyMC31AsSCU5qrGWteWVdOQNLRdBYTqFpT3iwRAegoKc74/KWpbvidKg0XhATlhEbI+RVQUcN7eKSMQmELCYPq2MMWMOvIFyKDDclvO01mftDznDwYj6Vj6OpxS4Czg7/7fbNz4aWNl7QN7TUtGrz57Y8BZionalH/tZx12RxQ+N6E8g37aTrvsyiZRyFPAY8CjwpX2RYTAcUdgW3HSZf3RzwhT/AHjnSXsn54YL/QPfM2JdumNcsF9XCzjt+Co4vgouP4q5vRu/cAq7Yy7w7r3TaFjyj0c28eAf3yTjCYoyKRKRKNPrt9NcWERTpICxHa24lo2VD/juDIbJOAFqY8VknACRbIaSzgThbJa2UIRz1yxnRt1WfnLa+awZMZpoNkO7EwCtmdJcD0LgCZtwNosW9HjPAlqTsmyE9rjtqycwflzxHjQ3AMPB84QQYuLe9NNar99bmcPBiAJAKdUqpVwArJNSXoIfrL0e37X2RL/ul+N7Th8bKv2UUn8H1hwEHTcCI6WUBUqp3n74yfm53zL5uK0ngAeUUjfuD5kGg8GwP7n4/PFcfP74XbbnXI87r7yX9TpCbXEZqWCIloJCEIKKeAcl8S4sIXA8j5wleGbCNB6Xx1GfdahpaWRkpICuUAQPCHg7FidtNL1DswT+F3sAbQyoQTBMPFFr8T++3Wmj2fFsxx4ZNkYUgFKqRUr5Q+A2YCH+03APSik34AdKJ4GLgDuA7yqlegcBWFLK/hGTmQOln5RyoVLKk1LuLx3FAG1Z/M0Q1wB3SCm/AHQApwALgOve6jlJKacDjwP3KKVufqvyDAaDYShwbItP3HclAD+4fiEjX32F5ydMJW47dIXDNBQUEkvGqeloIykEsy6dzXUfmEEymeOOdy/n0teXsLa8imdrppAWFiHdNzqt+86bth1yloXVP+7KsFuGgydKa73f48CHlRGV50f4T7ZdpZS6R0p5DvBV4CZ863A1cKNS6u5+487CN2B68/7834lSyv774SxUSr2fwdOjH77h8fD+0nGAti8ppb4jpbwI+C5+sHchvnfqxv75qvaRL+AvR35aSvnpXvUfVUr9YT/INxgMhoPK5+6cD/hLBAsXLgS6mD9/4BCYSMRBWDaLxtYQzmU5un47Qmvaw2Fc2wHt3/49IehyArjCwnUcfxNiw95zmGZ4H9JkmwYDJtmmwWA4gPhGFMyfP3+XfT525T/oCkWoSsZ76lK2TSoQJJzL9jzBlwwEiAdCaEvg5LL88Rf7JTZ5uHFArJ3fnv3Tnb7rP/Tkx4fMshJCOMANwBn48cs9ugxm25dhk+LAYDAYDIah4KizanaK2YnksjjujnQGfvZyv154Gtvrn4zCsDu0EDsdQ8ztwEeBZ4HjgL/iP03/5GCEDMflvIOGlPIXwJW7aJ6plNp8MPU5EEgpTwMe3kXzbUqp2w6mPgaDwTDc+OiH5/DPV9pwtKYs6W8y3B6J+oZS3n/SHRNla03QzfUJPjfsmWG45HAZME9rvVkI8TWt9Y+EEI8AvwRu3VshR7QRpZS6jv0QnD2cye+BVzjUehgMBsNwZoLXyTNjJlOYTiOAis52hOdRmUrgeK6fdT3vPbE8j4LwkHtSDimGgeepPwVA99ZmSSFEgdZ6pRBi7u4G9ccs5xkMBoPhiOfHd17IVbEWQukUZR2tVDk5fnHzMWS0pr6wiIzjp+vzAK09bv3qvKFV+BBDD3AMMSuA4/NlBdwqhLiZAZJn744j2hNlMBgMBkM3n7vxJD7Xr+6/b53HF761JJ/ZWFNRYvPN/z6F4qKd9yA07Jph6In6FND9iOVngTvxdyb5yGCEGCPKYDAYDIZdMGlCCX/51TlDrYZhP6O1XtyrvAY4d1/kmOU8g8FgMBgMB5ThtpwnhFgqhPi8EGLsW5FjjCiDwWAwGAwHGDHAMaTcih8TtUII8YwQ4qNCiLLBCjHLeQaDwWA44mmO53jPlxQdsSJAE+3q4pkfnTjUah02DLeYKK31A8ADQogYfrqD9wM/FEI8obW+eG/lGE+UwWAwGI54LrllGQXa48pXXuKcdSsJWRbnfvqloVbrsGG4Led1o7XuBO7FDyx/GbhwMOONJ8pgMBgMRzyj29u462+/JZLLkrYdbj33EpaNGEkqkyUcDAy1eoc8w80TJYQQwNn4WyxeCmzCN6auHowc44kyGAwGwxHD4jdbeN8tS/jDwjV96k/dtIZIzk9kEHJznLN+BR0FhRz7jfVDoeZhxzD0RG0HfpH/e4rWeq7W+n+01lv2MK4PxhNlMBgMhiOCsz/9MhurRqHDVbz6msuvn1Q8dbsEoCFW0qdvc0EhCIuCbHooVD3sGG6eKOCdWuv/vFUhxhNlMBgMhsMerTUbK0cSdF0q4nGKMylaYjE+e+sLANQWFfGTk89hyZjx/G3Ocbw4YTLJgINnG1/D/mC4eaL2hwEFxhNlMBgMhiOA2TevRESjFKeSvvdA+5sJL/JiPPjQepojBSwbMYp/zTwax3WpixaRs23SgSCvrWvjmEkle5rCsBuGoSdqv2A8UQaDwWA4rHlhWwGWbXPZssWE83FP4N8AXdvhZ0800xaN8erYiWRsh61FpTQXFdMRiZIOhrj211uHTvnDhOHmidpfGE+UwWAwGA4LLv7JWuq3JhjV2UZLQSEJ22beNk1rBI5zN/A///ozD804mu+e9Q5cIUgGAiSDQZKOQ3Gii85QmEgug6V73eKFIB0MDt1JHS4cpp4oY0QZDAbDYUQ8meHXj9dywawIlSOKuOeBjbR05Djm2HLOmVlCSSyM2MMNrT2RZfH6DuZNKSYa2v+3CdfTeK5LIODLXrW5k1AAtPB4alkXZ06PEA5YfHthHSXJODctmEUsGuYrP13Ko5s0WSCmM0wIeaxNOliBAIWpJGsqqymIRFhdEMXxPDTwxKTpYNlc+roC4B0rlnLequXcI0/ljtPPJ+kEcYVF1rIpSiWJZLJsKynvo2/Sdvj1E1u4+oxR2La1x/fPsDN66DOU74QQ4m3A5UCV1nq+EEICRVrrJ/dahtaHi1Ntz0gpnwbmAVn83Zs3AN9SSv25V595wC35fjawCvixUuq3vfrcCpyqlNppw0Ip5dXA3UCiX9PPlFJf2AsdJXAzcAoQAuqAfwHfVUrV5vvMBL4OnAVE8udxF3CHUsobQA8PSAHLgT8Av9lFv0Hrux84ci5Ag2E3fP3bi3m4I4IWgnPWLOfCN5eycPZxZG2btnCY5SPH0lxQSDibJZrNUJqI8+1//ZHiVIpPXfwBaotL0UJw8sY1TGhpYnushC2l5ZQm4wTcHIvGTqIzHEZoEGhszyOUTVOQzdEVDpMTFiE3hwDOWfU6L02YSmNhIVOaGhjT3kJxKklNSwM/n3c2HeECcraNFhYlXR3Mrd1KRzjM1uJy0o5DzrJwtSYbDKGBSNb37tjaw/I8skJgC4GtNcXJBNe+/BQ/OPNCkk4ALQRaCITWRNMpRnS2kwoGSQaCJAMBWiOFOJ7LmLYWxrS3kHKCvH3161R2dfLkpOk8PmUWtt5xy/YAW3tE02nuu/dORne00REKc9Xl17Js5Dhc2watKU4liabTdIbDdIYjfT8crSlNxolkMrSEI5SnkriWhYtgUksjxakE1R1tpAJBGgpjPD9hCtmA772yXJcx7a0ct3UDNQ31/OfDl/G795bjWMPPqMhzQBT78YV37/Rd/8l/XTNkb4IQ4hPAp4BfA1/SWhcLIWYBv9Jan7y3co5ET9Q3lFLflFI6wMeBe6WUryql1kopzwP+AXwbuAJIAu8AfimlnKiUumUv51ivlJo8WMWklG8DFgI/Aj6mlNompRwJLADOAO6XUh4FvAD8HpgNNOfbfgMcDXxoID2klIXAeXnZFwLveqv6GgyG/cOLz23lkfYQ2vbDVJ+cMpuQ61Ka8n/bLBs5hqbCItCaaDaDBbQXRLnnhDP40d//QEEui6M1SdvhH7OPA0BozfSG7RRnUqRtm5mNtSwZNQ7PtgGBa9sE0pqmwljPUouNJuS6PDr9aJLBIIXpJBPamgGIh8KUJRL86i/3cNmHPom2fF2Prt+OhWZzSQXasnC0RngeyXCYnO3guDm0ZeECLjYAjpvD8TwA2iMF3HbuJTsCj4Xw7+JCEA9HiGcz1MeKAf8Xl2tZuJbFhNZmitNJjtm2ginNDQC8e/kS1pZXUVtUgmvZPbLioTAd4QjvuuoTzGzYTllXJ6+OqenzGQityVkWwVwO23P98T2Ngo5QhEguR9jzyDh+8s1xrU2MiHcA0BYt5PjN63l08iyyzo7knJ5ts7msgs1lFXzy2UfYvmQr368O8sUzi/b5ejkU0cPPZvw0cI7WeqMQotthsBKYNhghR2xguVIqB/wK35A8Jl/9M+A+pdTXlFLNSqmEUupPwGeAr0gpJxxgtX4O3KuU+oJSaltez1ql1DeUUvfn+/zQr1bX59sySqnHgCuBq6SUpw4kWCnVpZT6G75xeFneYBtyOjs7TdmUj/hyc2MCLXZ8HfuemJ6XZPOP2Qut+3xpt0aiAHTkPScZZ8fvYi0EuV6GQMBzCXguvfGE6BOr4uV18Lq9JP18B54QTGus6+OqcDwXT1g9RtUORF5nBmCHBM3un9yyBlotyccz+fN7PdXbikpIBENoYSGAaCZFLJ3Eyhs+9bFiXhg3iZGdbYxvbqS6vZX/fuxBvv/Q/Zy4aS2W9rC1R0VnB3O3bNilTr3PvTdZ2ybo5XYZ/9MRjlCe7GJjc6qnbqivvf7lA4VG7HQMMTGgO7Fm90UWADKDEXLEGlFSyiBwff7lainlVGAyvoenP/fi/9cfMMOj1/z37qZPBDiTAXRUSj0NbAXevrt5lFLP4mdoPWfftd1/xGIxUzblI7587gU1VCU6IW8wzK7dQk1zA93mQXVHG7broi2LZN5Qsj2XaxY9A8C0hlo0EMple2RYnofd6yafsh0ytoPItwut/aW2TD6ZpNYE3ByW5xHOZvCA9nCEjSXlCK0Z3dbCuWve5K9zJEJ7PfNsKS7D8Vyqutp9MUDWssBzEVqTte0dtlh+TE4IcsLK97WZ2lDbM5ZeRpMLjGxvJZpOYeVjnBCCQC5HaSIOwJtVI2kNFwDw2JRZJIIhPMsiks0Q8DwcrRnR1YHtuWgBjtbcfeKZtBZEmbt9E1Ob6hnd0cYVr/2H47ZuJB4MkbMdVlWNxOo20LSmJBknnEmTDARwXBe0pjZWTNr2DdVuHePBEMHsjicAu89nXEsTY1ubWD5uAl88q/QtXzMHqnyg0GLnY4h5Fvhiv7pPAk8NRsiRuJz3FSnljfhWaBZYoJRaJqU8Jd++rf8ApVRGStkEVO3lHDVSyrZ+dTcopXZpIAGVu5q/F2X4cVq76rOdvdNxK9A7cnJf9DUYDPuJSEGAP3//BB56chsrWj0uuHwu4v44zSXVPJmOMW7TFsa11NMRjpC1A9QEUpR0dZLO5Xh0dA2rS8pI57I42SzFmSxuMEg4laCyvY1AKkVdrJhN5RVY6SQjujpIBUMk7QApJ4DtuhR0deJZFkJ7dDkOJ25YR31pGasqq1lVVMr2QJBjgU9c+B4Wj6mhsK2VeN77taUgSspzydoB2oUArbFyWQpTCQKuJhsMEEylOK52C8mgQ0O0hKZwiGwwgpPNMqWhluZYMaFEnLQTwAayAjwEtmXxRtVIJjfWUhpPosaNx85mcbTH4pHjKEt20RyN8cCsY4mmUwQ8r8dzJXq50QT4hg87PG+eZbG1pJzGaIyRne0EPBcPCOZyZB2Ho7dvoSTRxdOTZzC2uRHcHI1FJRQkEyTQRFNxElrzn+qxRJMJgtpfdvUKQ0SjNlfNDrKi0WVlbY65K1/nVKuLyb/+MK+MCRENHnn+i2HgeerPJ4CFQohrgZgQYhXQAcwfjJAj0Yj6Vj4mqhQ/GPvs/N/GfPto/HXRHvJeq4peffbEhn2IMeo9/4pd9GnB/3E2ehfto4An9mKuMfS1tvdFX4PBsB8JBGwuPX8cl3ZXzLmMufiRrzBn4EHfPwPwgx33L7P2u0Q4dhf1x+yivjcz99hj+bY47/r5FkrcLLbrkrIdbM+PH0vYDgHX9Q0orXsMqVgqmTeu4O8zjmHJ2Ilc8uYrjOpoY2txGfcdfSIC6CiI8p9vz96rsxyYXX1lHzkMA89Tf+qB4/PHePylvUVaa2+3o/pxJBpRACilWqWUC4B1UspL8APK1+Pv6NzfELkc39P82AHUZ7WUci3wfuDxXfRJSimfzet4V+82KeXp+MbRw7ubJx8zNQrY60c4DQaDYbgze3SUVd+a3qdu3boWvvLjV2jCoSswCixBKJcj4HmUJuJ0hCN856yL6AwEqYh3ML2hlrHtrQCMb2vmqLqtvDpqHIlQaChO6bBiOHmihBA20AWUaK0XAYv2VdYRa0QBKKVapJQ/BG7Dfyru48CDUsoN+EHeSeAi4A78FAO9Iw0tKWW4n8hBBaQNwA3AQillPfBTpdR2KWUV8GH8J+j+CHwOeE5K+VPgm/jeqdPwn867Vyn13ECCpZRR/JiuHwF/V0o9+hZ1NRgMhmHNpEllXHFuEoAvLgYQNOWf9KstLmVcSyOnrnmD5RNrSGsL3S+SPpzNMLqjlXBl9CBrfhgyjHJraa1dCq2j1wAAIABJREFUIcRq/LCW7W9F1hFtROX5Ef7Td1cppe6RUp4DfBW4CT/+aDVwo1Lq7n7jzsI3snrz/vzfiVLKrn5tC5VS72c3KKUey3uKbgZezy8j1gEP4RtJKKVelVKehJ8n6k0gDGwCfoL/5F5vuvXQQBo/T9Rt+HkxBuo3KH0NBoPhUOG1r05i6nf7bt9ia836kjIe/t4JzP3SG7RGotS0NTOyo42tJWUsHlPDxIbtPPKdo4ZI68OHYZgQ8A/AQ0KIH+HHCfd6/sEk2zQcOpgL0GAwHDAWLlwIwPz585lx61pSoTCeZRHI5ShKdLHkmzMQQvCRq/5JIlJASSpJ0gmwvGoU9cXFRJJJVnxtyhCfxUHlgLiMvnfJ/+30XX/T3z84lMk2d5XDQmutJ+6tHOOJMhgMBsMRgfpyDSd9YzWe5RBKp7jzihE9W7hkHYeSlL+4EMllmdjaSGs0Sm6n/FeGfWG4/VrWWtfsudeeMUbUQURK+Qv8pJgDMVMptflg6mMwGAxHEtGgzevfmDFgW0cwRDS5Y/ercC5LLBHnvKMLD5Z6hzW7S6h6KGOMqIOIUuo64Lqh1sNgMBgMfamPFuI6ASoScbQQZCybKbVb+PZtFwy1aocFw80TJYTYwi7U0lqP21s5xogyGAwGwxHPLRdVcONTKVZWjSScy6G1x9JvTN/zQMOhSv9VoZH4adnuH6DvLjFGlMFgMBiOeN529gReOcPj63cuozBi8/kP7yLBqWGfGG7LeVrrZ/rXCSGeBv6N/9T+XmGMKIPBYDAYANu2+NrH9yaDumGwDLflvF2QBgYVcG6MKIPBYDAYDAeU4eaJEkJ8vV9VAXAhe9j1oz/GiDIYDAbDEUN7PE3Q84jEIkOtyhHG8DKigLH9XsfxE1b/32CEGCPKYDAYDIc1//gjPHLvgywZP5GTNq8n5QR4buJUPlCV4ss3mNing8EwXM77kta6rn+lEKIaf6eQvcJkETMYDAbDYcuiJ3JcuvI1Hpt1DNqyqSsqZnxbMze8+CR/22rxnQUP9unvepqvXvsgt1z8W274wINsrO2/I5ZhX9BC7HQMMat3Uf/mYIQYT5TBYDAYDlu8DZ3ccsG7KUynuOSNV7h0+SvUF8Z4euI0Ll+2mP+MqeHOz/yT62+/CIBrP/wIbQUVbDpqOp5lcd13ljOnbhtJx+H4i2fwofdNHeIzOjQZhp6onaw4IUQR4A1GiDGiDAaDwXDYUtnRQWVXB5967lHetuYNLGB6Yy0Zx+Hu409nRfVo5i56FoDW9gxj2ltZWT0GR3to1+OMDWsoS/mZzOv+r5WfdKb5xAKzBDhYhoHnCeiTZDMihOi/S0g5cN9g5BkjymAwGAyHJbde8RdmpxNsLSnn6+ddyvlr3uhpK0kmebFmKpab46VxkwB4/tFVvDp6AsXxLpLhCK6GsmQc8gZARTLO8ifXgTGiBs0w8kRdie+F+hfwwV71GqjXWq8ajDBjRBkMBoPhsEJrzcUfe5501XjWRUpojxTQHingHnkqV6vniQeC3H76+QCUx+M0RWPcf9uTPLw2R5mA1VWjsLSmurONqq4OGmLFAFR3tpO2zW1zXxgunqjuJJtCiAqtdWJP/feEuRoMBoPBcNjgdSY56+bX2ThuIgjBqhGjetpuOf8yfnzq2xCeR8YJ8N7XXuady1/hE/PfT93Tq+mcPJMtZVW4lv/MVWukkMquDsa0t2JpjZPLsrGkdKhO7ZBmGHmiANBaJ4QQxwCnARX0ipHSWn91b+UYI8pgMBgMhwVf++IT/NUZjVdc3LMERz8PSFukgFv//QCz67ZSmk4CcMaG1Tw9aSaJULjHgAJoihZy5RXXccLm9WRtm/GtTWTc4WYOHCIME09UN0KIjwC3A48Cb8dPsnke8PfByDFGlMFgMBgOSRo6M3z/Z6/zTCO0lJRBdBKF2TTHbdvEyxOmkAiG+vQPZTPc/o97OXbbJi5YcCNfeOohpjXWMa25gUktTTxfMwXPcmiJFqKBeCiEFhaLxk3CQjOyo43t0cKhOVnD/uYm4AKt9XNCiFat9aVCiLcDlw9GiDGiDAaDYT/guR7ZdI7Ohi5KRheRbEkSrYjipnMECoK7HZtLZUknsoQscIUg40Fza4ryugbSZcUIx6IrFiWnLYLC4/7n68m1JCgTWZq2tlMeEayOVjBuxRqseILxy9fTEo7w56NPAAThbAYn59JYVMxNz/ybRCjMX+cchwdkLJsAmvqiEiKZNAXZDK6wWVVehSc0p6xfRVNRCS2BMO2RCIGATaWTJtOSojIZZ3NVFTXba0nbDptKKxE6x/i2FlrCBWwuryLa1cWx9Vup7mqntjDGiurRzKzbzoTWJpKhCK+PGE3YdUk7Dh2BAOlAiOqONkZ2tPNmZSWZUAG29kDD6RtWc8qWddRFi1hVWU11vAMvVkzdMSeBEAjt0eHYPDt5Bho4etsm3qgeTS4fx+QJi7nbNhPMuYzoaOO5mqlsLKlg/srXuPmCd+NaNuFcltPWvcnf5xzfE8fjeC7TGmpJCovmSITTPqOIZFIUZJNUpZKktMXZ61cRC7i0HjWNbeNGs6rLory2kY9eNJrq7duI17VTcsPbiYfC1HfmGFvioIUglcyxtC5DTUWQtaua6MwJTp1eRHkuTYMTpn57F8fMLGHjigbqU3CiF8c9fgqtSc3YUgchBF1pl0xWE3QE0ZCFGGZeHwA9/DKWV2mtn8uXPSGEpbV+WAjxh8EI2aMRJaV8GpgHZAEX2AB8Syn151595gG35PvZwCrgx0qp3/bqcytwqlLq3AHmuBq4G+gf5PUzpdQXBqFfN/crpRb06lMDrAMeV0qd12/8N4GvAD9RSn2yV30BsB0oBsYqpbZKKRcANyqlpuf7/B7oUkpd10/mR/Lvx1FKqeZe9T/EX389WSnVW9/eY6cDK/BT0GsgB6wF/gHcrpTqGqBfbxYppc7O9zkf+CowEz+x6vb8e/O1XvPNyPc5CygE6oEngO8opdYPpKPBYOhL7Zv1/OPmf5NJ7vi3Fp5H1M0htGbyBdM59fPnDDh22X1LeOXXL5MTFstGjmFm3TZ+Oe8scpbNp597gvlvvson33kFi8dNQgMVnR18/tl/M7KznVWV1fzhtPOxUyCSmkDZdE7pWM1s1yUFrB41nqzjgNbM2b6Z7z76AAHP5UtnXsDm0gpsz+OClUv509x5vjJa5yND/BteJJ1ibDxOVSbNfXPn9SzJlDTVcfHqN3C0Jr5tCydt3cCZ61dRFyvmysuv5bWaqdiuS0Br2ssreKqikqqONs5f9TrveP4JNpdW8IkXn+D7p5/PxxY/x0Url9JcEOWGSz/E2ctfoTCd4r65JzF/w2rOXLeCJybP5L65J/Hg3BOJejnOWr+KGS0N3HnimSweNxEAy/MQ+El+tBAIrXnf0kV8q2J+jxHlWhb1hUV89fxLmdzSwIJFz1KYTvG3WcfiWjahXJZwLktztIjzVr3OI9OPYnRbC2WdHWyoHMHaymqE1thuDkv7b9OWXI7iZJILnvgHZckEGzdt40PvW0A2EERXRXn5Py6nr49z/UvP0/KbZ7n99PNZOOtYbNelNBlHC0F7KMyU5gZimTQa+JPl0BUO8+UnFnLWupU0Rgv54ruvYW3lCJycg/v0ZjQQDNtceXQB/6sSaCGIZDJMK4KfXjOeESWB/XyVvzX0sLOh2CqEmKC13oifePMSIUQTkBmMkL3NWP4NpVQhfg6Fe4B7pZSTAaSU5wFPAS8BE4Eq4LvAHVLKrw0sbkDWK6UK+x27NaD669frWNCv/VqgFThXSjlpgPGrgCuklOFede/FNzoGjVLqf4FFwC+766SUZ+X1uGJXBlQ/JiilYvjv52fxN0ZcJKUsHqBf73PvNqCmAQ8APwEq8T+79+IbZN06HQssBjqAk4AYcCJ+xtYLBn3iBsMRysu/U30MKADH8xDaj59Z+++VNK2s32lcNpHhlbte9vtrjylNDfzqpDPJOAE8y+LHp76N5ydMYXH+EXwBnLlhFSM72wGY1lhH2M3l7R7/LvXO5a9goVk6apxvQOXbpjQ3UJDLEvA8GguLfIPDsnh86uwdCglB7xyEyVCYR6bPRm7diKV3xAIdt3UjTv71jKY6zlzvPxVe3dnOgsX+j/us7ZAMhnr0aigqYXtJGfFQmHesWEo8EGTJ2BouWrkUgPJEnKuWvEDOtinMpChOJfngKy8ytr2VReMm0R6JkgwEuevEM2kLR3i9egzPT5pO2glga43Vo7lACwvPsnlkyizioR1f62WJLj59yRUsGz2ef86cyy3nXUrA8whojXBdwjn/M9SWRWE6xfL/+RKnr1/FuupRuLYNWqOFwLVstGWhhUU6ECQeCvHy+MkAjGlvIesEej4v13a4esmLBD0XW2s+8fxjABRkM9hakwgGKUkniWXSPWPCXo6Z9ds5a91KACrjXVz7n6cByDm+BwshSKc97lLxHo9ZMhhkU0uW+17o+e0+bNCInY4h5nvAjHz568DvgSeBwdgtg9v2RSmVA36F78E6Jl/9M+A+pdTXlFLNSqmEUupPwGeAr0gpJwxmjv2NlDIA/BfwDXxj6doBum0ElgDv7lV3Lf657ivXAidLKT+UN3x+C3xOKbWrVPMDopTKKqWeAy4BqoFP7mFINxJoUkrdr5TK5Y/XlVK9XZU/Bp5TSn1UKbVRKaWVUk1KqTuUUj8fjJ77Smdnpymb8iFfDoQH+NXf7x7hRAI7jY0nE9gBu+d1zrIIuG7P63A2Q2H+5tpNwtm1h8ETFjnLl1cZ7+zTNqqjrafce45oJt1j7AG+N6oXQdelLlaM12uJqCO0Y/PeZD992sPdbf0CsLVHOJslmsmwsayCoJsjKyy8Xm9UezjC5KZ6HM+jNLnDyS70zsHcm0vKd7T3qu9dfmbKzD5jmgqL2FRW0fN6Q1klAKsrq9lQXoHwdiSr3lJazpXv/yj/mnF0j9zd3fYL0ykAbK376OsBycCO9yiTN2y7ewhNz2fWjQYSwb5LwJ2hMP3RgNVbqfzcDrkd4wZ5PR8otNj5GEq01vdorR/Olx8GSoFSrfWdg5EzKCNKShkErs+/XC2lnApMxrfg+nMv/jX3tsHMcQC4BCjD1/Fu4L/yhlV/fkXewJJSzsT3qi3c10mVUk3A1cCP8N+LV/Ieqn2V14Dv8Rt4TWBnFgFVUsq7pZQXSylH927MG3YnA4Na/93fxGIxUzblQ758+vXzqJ5e2ecu6zoOVjRE4cgiTrjhVErGl+00tqS8hDNuPo9gaQHxUIjG4iJuevZhJjY3MKa1mW/++6/IrRu57LXFWJ6HBp6eOI3nJ0yhNlbMQ9OPIh4I4uIvHwZyWe6Rp9JUEGV8SxM1TfWUJroY19JI2rLJWRauEBy3ZQNCawJujmsWPcuXH3uQaDoFaNAegVwOx80xsr2FUDbLMzVT/XPTGrTmqckzWFY9ho0l5Tw5aQa/OPFM1pRX8c/pR/HzeWejtSaSyfgytSaYy/K2VW9w4uZ1NBUU8MCsufx1juSUjWv45jnzWVUxgkVjarA8lxVVI3muZirVHW28NnIsAO9d+jKxZIKCTJobXnic4mSCcDaN5XkgBLrbaNF65z07+hlgvfMVzarbynfOvJBHp83h5E3rOX7LerKWRca2aQsX0BEpoCLe0SOjPN5JcSKOk8v21AVy/pJtXayI2lgxf5l9HOFMmpwQZC2LwnSSv884hs3FZbSHIvz3eZeC1sSDIdK2QziTJh4IUlcYI2tZuAg6AiE2F5fyk1POYVNJGU9NnMZPTjkXtKais51R7S0ATB4R4gfzy4nZmgAeI0WWc2bFuObcHWkdBns9HyiGoScKIUS5EOKDQoibtNYZoEgIMWYwMvY2sPwrUsob8Zd7ssACpdQyKeUp+fZt/QcopTJSyib85ai9oUZK2dav7gal1L2D0K+bC5RSL+fLHwEWKqWapJS/A24D3gn8uZ+MB4Gf5pfBPoK/bJnjLaCUelRKeR/wPmB/bLi0dQA566SUvb8l7lZKfVYptUZKeRK+5+oOYIKUcgVwk1Lqn+zIi7HTZ2cwGAZH0YgY77n9kn0aO+6UiXzglIl96i7rKR0PwP/kjx34XwPvAb7TT57WM9G502lri1Ndn6axJcObLzSyfUQ5t5eczuSGBq5Y/CwXvfkqz9ZM5XdHnUBHQZRAJk1lKs6M2u1sLS5lXGsTG0sriJaFKI2FOI8mxoyOcuV5YxhbHqQjNYpYyMKxLdY0JHhpQ5KQrfkBOcZWR9nUnGVDXZzyQmhd1kDVySUcf/1FZMaUc6ljs/jp8Wz41ypeKyznudHvobmygumBFMfoLlrjFi1VJbwk5lH9xhpEPMtZK5fRGCtmedUoFo8eT21JGZMaa8k6AeLBIPFg2DeogLGtzazuzg8lBOVdHTRHY9Q0N3D22jdZOHMuFV2d/Pb+X/PItDm8Y8VSTti0ji9d9F5aI1F/6a77/RQWkxtrmdrcwHGb13PHKW/jM8cI5kwrxvI8jj1qBC8ua6HsQx9k3NQyrhCCKwD3/9u78/i4yurx458zM0naJmm6t0ChC6VAWYWnQmVfrMgiIiIUEFDZFPSrLxH4irIXBBXBn/AFkZ0vqIiCRVAWAQFZPBYqfKF0p7SFrmmbNmmSmXl+fzw3ZTJJmqWZTDJz3q/XvHrnbnPuzTT35DzPfW4qTSwmUSfvvQiXAfhjp74dDgh/0bfoTJzhqD1ynwRtrXxXnrKJyCHAo4ACBxCa93YCLgKO6+h+OppETVfVa51zg4G7gMOjf1dGy7cDZmduEFWthmWs056Fqjqhg+u2Gl/2TOdc03fvWABVXe6cewI4j6wkSlUboyTr24Rv+/5djCXbLOCgqDK1tUYD2Y3dO7a1b1V9C/g6gHNuJKGz+x+jRHE1oRq8XWvbGmP6JhFBSuIMGT6Qz4TWKo4/uPX/5ke3OrdjjzQZVvFJc9MuoyrYZVTzW/933gHCJQA4qOXnH/eV3TnuK7u3mN9Sa91YP7GpLslVP32VR1Oj6Oc9n5vzDvstns9lR5/EppJShm6sYdqbr/GHvT7NwmEjeXjgIH72+EOMr17JxtLSzX2yFg4eSjoWI55OM3rdGj4YPAxEOObdt/jOy8/gY8IvDpzKDmtX842TDm0Ww4F7D2sRVzzeqYaegtcbKk9ZbgZO9t4/JyLV0bzXgU93ZiedGuJAVaujO9TmO+eOJ9wxtgA4lXBHV6ZTCBfpZzrzGd3sXEK15Z6Mak05UO6cm6Cq87LWv5Nwx9vzqjo/3/25MjnnhhPuoLu5K9tHCeQVhObYSar6pHPun8A0Wm+ONcaYXq9f/wTXX34Q1wN/eruGH/+2kcd235dLn59BXWkZjbE4/7P/4dRHfYxGbFhPQ0kJc0Zsy7xhaaYsmktZKsn/DR/FV/Vl7tzvEE6c9S/c0kXUlpQyZu0a5g8dzsvjJjJzmx0YtX5dfg+4j+ptlShgrPe+KW9pyg8a6GRe1OlxolR1TXSr/nWEPkMXAo855xYCtwF1wDGEi/0NqrowY/NY1h1wTUF3u6jf01nAdOBXWYtfIvR/anb3n6rOcc4dQueauOKtHFO9qnbLsLbRcXyaUGpcQegM3pHtDifcefBnQjNgJfADwpAIb0ar/RfwonPuNsIdlYsJnetOBdI91bncGGO6wwl7VHLCHrvx1XOf4fkxE5m0ajkPTD5wcwIFsPOKjzZ3Bk7HYqysGMgbo8fy2jbbc/7Mf/L592Zx6bEnM27NSn743AwWDxrKjYcdQyKd4lNLP2BxRfYN0qZjel0W9a6IfM57/7eMeUcCb3dmJ10dbPMWwt13Z6jqvc65IwhjDV1MGCdqDmE8pbuztjuMkGRlmhb9O945tyFr2QxVnUbXfBEYSBhbqVkTmHPuF8CVzrkfZ2+kqi938nPOjl6ZJhPaWbfGoqh6liIMS/AEcJOqrm9lvcz3y1V1R2ANoVP/D4FBhOTpLUJ/sY8AVPXfzrnJhGa+NwhVuhWE6mF2VwtjjOkTHvj1Z/nWN/7CkoFV1JZ8kkCJ91TV1RIN8YQHbjnwSD4or+Kxnzl++eVqHt17fxoTCeaM2IZvnPwN9lr6IeOrQ4+JslSKqrrsy5TpiN7yAOIM3weeEJG/AP1F5A5CX6hOdW4U38qto8b0IPsCGmNy4r+uVp5PDmTdgHIS6TSnvPUan/5wITWlZbwydidmj9iGgxfMpi6W4MZHTuKDFXWc8KvFn4wt5T1u8QKG1tVSWV9HLJ3mvaHD+cMdh+f3wHIrJ9nO9097vMXv+p//7/F5zaxEZFvgdGAM8CHwoPd+SWf2YT3fjDHGFKRbLneMXb6Ufg31xNMpasr68c8xE7jlwM/y3IRdOXTOO/RLJnl99BgARgwq4/Pvzdo8ztOo9Wup7j+ADaVlfFxRRW0sztdO2yWfh9RneZEWr3wQkVGbY/J+mff+Ru/9Bd77n3Q2gYI+8Ow859zthEyxNZNUdXFPxrO1nHNltLzDrskzqnpCT8ZjjDGFzMeEYRs3sGTwUB7ffd9m40bV9uvPH3bek9MmhSEN+pfGeH/YCK6b8Vs+HDyUJVVDaEwkeH/QcAbXbeC7Vx3GHrsObeujTN8wh9DVBwAR+aP3/ktbWH+Len0SFT2X7vx2V+wjVLWe8Iw6Y4wxOXbwbrW89m6MhkSCNf3L8TEh7j3JWIy3h4/i/H3KuOCLYzav/4f/OZT/fDiZf/xuHkNra7j40v0pK+31l8perxf128gugR26NTuzb4YxxpiCtcuYFDPfT7Fd9Wq2X7uGhkQiDCa6ZgX3/GASQ8e0rCztuX05N1+0Vx6iLVy9qGN5t+ZzlkQZY4wpaKdO3cBxx2UPQr1zXmIpVr2oEpUQkcP4pCKV/R7v/d87vLNuDs4YY4wxppleVIlaQXiObpPVWe894Uk7HWJJlDHGGGOKgvd+bHfuz5IoY4wxReeBVz/k6r81ECPN+1fulO9wCl4vqkR1KxsnyhhjTFF56J9Lue7ZJIlEHIknmHj1gnyHVPA80uJVCCyJMsYYU1R+9PQmiCojIkIsFmPcFXPzHFVh89LyVQisOc8YY0xREZEw6GaUSHkgXVqW36AKXKFUnrJZJcoYY0xR8T5NKkqkvPfUx+xSmGu+lVchsEqUMcaYoiISoyGeCBfyWAy8R1LJfIdV0KxjuTHGGFMA4slGBB8SKAARYr5QaiO9U6FWoiyJMsYYUxC+c8bjnH3u0+2u17+xgZJkRuXJe0qSqRxGZrxIi1chsOY8Y4wxfVZ9Q5Ky0gSPTLmRA+Jx7pl8MIddNJNFVUOYItVMa+UReMmyfpSmksQaIBmL0S/ZyKZYvOeDN32eJVHGGGP6nCn/rVSXVhED+tVv4vpNtZx78jmb77hLpJLMTA3iX6+VcdP+8wFYtrKG46+fQ1VZKcsrBpJMlACQlhipgmlgMj1JfJG2AzvnXgCmAI1AClgITFfVRzLWmQJcEa0XB94Hfqmq92WscyVwoKoe2cpnnEV4Jk9t1qJbVfWSLcS2IeNt03239U0zVLUiiv9Z4EXgqYz1BwANQFOt+iVV/bxzzgMHqerLrXzeImBUxjZNtlPVdW3F2U2K8wtojOmw8x74kFfmb0LSaYasX8/a8nIGNtSztryS+kSCtMTwsRhpIC0CIkg6Tf/GBvCe0mQjey9bzILho9gUT7Cu/wDqowQqnk5Tmmxkk8SYOqmCyw6vYszgkvwecH7lpJ3tnG883eJ3/Z13Te3zbXrFXom6RlWvdc4lgAuBh5xzb6rqPOfcVODPwPXAaUAdcCxwh3NuvKpe0cHPWKCqEzoTlKpWNE07534DJFT1rDbWfQnIXH8ecK2q3tuZzwTOVtUHO7mNMcbk1OduWsDSGo9IDOIxqgcOpL60jFIgKTEa4wligKTTJPCkJEYqFqN/YyP9ozvudlq1nFmjx1LZUM/Gsn4hgYoqVqlYjJj37LR2Ne+9vo7D5tXz8vkj2baq2C+P3cvGiSpgqpoE7iQklXtHs28FHlbVq1R1tarWqurvge8BlznnxuYnWmOMKR7LappfftOxOMl4go2l/dhUWgpECZRPE/eeklSKbdZVk8oY+2nJoCEk0mkAGuItk6NUPE4yHmNTPE7Kw+Oz63J6TMWoUEcstyQKcM6VAt+M3s5xzk0EJgCtVWYeIpQ7P9tD4RW0mpoam7Zpm7bpNqdLYtC824lH0mkaShIIkI7mNRGB6vIKUiI0Rp3FqzbVkYya+2LeE0+nw4jl3lOSTlGSSlGaTDIg2Qh4dh+SbBZDMU3nSqE+O6/Y+0TtR+hrVEnoG3WBqt7lnDsAeBnYVVVnt7LtckLfqOkd6BN1F5D9Df2Wqj7UwThbbc5r6hOlqtdmzW+1Oa8DfaKGE85Bk8WqumdHYtxKxfkFNMZ0SHVtksN/9gGN6XDZHbp+LasrKtlU2o90PI73nphPh0qTyOYxiFKxOCWpFJX1tUjKc8I7/+JvO+/JysoqUrEYHoinUpQ3NlBVV0dNWRkbBg/iR1MHM23v8vwedH7lJLv52jnPtvhdf8+dR/b5TKrYG32nR32iBhOSncOjf1dGy7cDmiVRUdVqWMY67VnY2T5ReXKe9YkyxvQ2gwckePPyHVvMH3/Zu5SVluElBqkUjSUJmgYpSMdiIEJjTNgYi/Pjnd/l9oZd2XblMso2buCjEaMQhIZ4goZ4gtrSEpaXV7Lo4tE9e3BFpFAqT9msOQ9Q1WrgbOBo59zxwFxgAXBqK6ufQvhD55mei9AYY0ymBdMnsU+/DSQa6zllN6FyUx396+vxaY+kUsQaGiivqeGGyXOoGJTgxZ/vx8P3Hcvf/udgypJJ1vUfQG1ZGWvLK1hRWRWa+EzOFGqfqGKvRG2mqmucczcB1wEzCHfrPeacWwjcRrg77xjgZuAGVV2YsXnMOdcva5cNPRB2Z5VmxZlW1d4YpzHGtOt/L/lDBnx7AAAgAElEQVRkJM1BMoeHZjZSmkoy/cQqpu4bqkozZixssd2mRKLZiNlpiZHwNmJ5bhVI1pTFkqjmbiHcfXeGqt7rnDsCuBy4mDBO1BzgIlW9O2u7wwhJVqZp0b/js8Z9ApihqtPoec9lvX8f2CWa/o1z7vas5VNU9e3ch2WMMVvnuydN5LsndWzdkatXsrFff1KxOHjPiA3rWN6/qPtB5VyhVJ6yFW3HctNr2BfQGJMzM2bMAOC4447bPO+A7/+bq/7+BD897GhGrV/LgYvm8tODj2LO5ePzFWZvkpN056vnPd/id/0DdxzW51Mrq0QZY4wpKisGDuLhfabww78/wdKBg5l+xHEkraCQU4XywOFslkTlSdR0dnobiyep6uKejMcYY4qFpNO8OH4Xnt9xV7wI4j2pVPZTr4xpnyVReaKq5wPn5zsOY4wpNjGgIRYjHg24mYrFkEarROVSoZ5dG+LAGGNMURkcT+KBZDxOKh6nrLGBRVdYf6hc8iItXoXAkihjjDFF5dXLd2VgzToq6moZsmE9j55ale+QCp5v5VUIrDnPGGNM0Xn7ut3yHUJRKZTKUzZLoowxxhiTU4VSecpmSZQxxhhjcssqUcYYY0zf05CEyZe+TXkqydpECW9dv3u+Qyo6hVqJso7lxhhjCtrPXtuBfT9ewvjqVey7Yhl7XfZevkMqOoV6d55VoowxxhS0wZvqeHncRGpLSumXbKRqY/bjTE2ueXsAsTHGGNP3LK+sIhWL0S+VJC1CQyKe75CKTqE+gNiSKGOMMQVtU0kpHiEtIN7TEC/Jd0hFxypRxhhjTB+UlBipeFR98h4Re05eTyvUSpR1LDfGGFPQ0rGMS50IUiCdmk3+WSXKGGNMwbr437tAwpP2HkRIpJKFe799L2bNecYYY0wfcfQPX2MpFfj+5ZQkkzT0HwCAAGNXr8xvcKZgWBJljDGmoOx18SySA4bhYzHEe+pLSzcvS8bj1JWUbmFrkwuFMi5UtrwmUc65F4ApQCOQAhYC01X1kYx1pgBXROvFgfeBX6rqfRnrXAkcqKpHtvIZZwF3A7VZi25V1Uu2Nr5ujPFHqjqho8uy57cS6wLgWlV9tJ1j3A64Ddgb2AH4qqo+uKVtjDGmq7z3PPfCEv705/ls2JhqvixrumLTJg6e9x7vbbM9c4aP4qS3XiMGvDtyW6o21bHvkkWUN9SzsqKSucNGsXDQUN4cPZbhEiOxrpryxnrGr1rBn/ZwmQFQ3X8Ai3Y8n7Fzb4NYDN5cAHc/C9sPg+9+AUqtvtDdCrUFtTd0LL9GVSuAocC9wEPOuabEYCrwPPAqMB4YAdwA3Oycu6oTn7FAVSuyXltMoDoSXzfH2B0yY30Y+J1zbmI726SBp4FTgSU5js8YU+Se/8dSHnh4TosECkJTW9Mr5j3n/OsfLByxDX/cazLvbLs9Nxz5BQ74YB6nvvU6p8x6gykfLmDP5UvZffkyxq1dzcdVgxmQSrJuwAAWDxnGeyO349mJu1OaSoL3xNIp+jU24JYsZPaA4XDID2F1DXzhWrjv73Dt7+HKh3v6lBSFQh2xvDckUQCoahK4k1Ad2zuafSvwsKpepaqrVbVWVX8PfA+4zDk3Ns/x9aoYs2K9jVAV26OddT9S1VtV9RVCBatH1dTU2LRN23QRTS9Z2rHRwmPpNIPralkwdPjmebWlZSwePJShG2sYVvvJfqrqQkPDuqjfUzL2yWCa9SUl1JaWgQhpiRHzno2l/egnkF60HJasgnUZDRX/t7jXnKt8TOeKR1q8CkGvSaKcc6XAN6O3c6IKygSgtaalhwh/rHy2h8JrEV80r1fF2CSK9QJC096snv78zqisrLRpm7bpIpr+tBtJR4oQ6Xic94eN5JD5s4ml0wCMW72CScuXMXvENnwwaOjmdZdXDgzLV60gKcLQ2g2btymvr8MDZY2NDGhsAGDZwMGUNtQTO+kA2GU0fGp82JEInHxQrzlX+ZjOFS8tX4WgNzT8XuacuwioJFz0z1bV/zjnDoiWL83eQFUbnHOrCE1nHTHOObc2a963VPWhrsYXLWv6E6k7YuwOTbE2APOAE1V1Xg9+vjHGbNEuEwfzk6un8NIry3j+paVs3Nh84Ms0of9MGrh7n88wYdVyvvPiUywZNBS3aC637ncI/95uLCM3rGOvZR8ycmMNqwZUUJ9IsOPKj3ljxChG1G1keXklgzbVsqp/OYlUioQPSRUi1PTvz2cumQpfi7qoPnk5vPgObDsU9h7Xk6ejaBRK5Slbb0iipqvqtc65wcBdwOHRv033oG4HzM7cIKq0DMtYpz0LW+u0vZXx0c0xtqURaO0ZBSXRshaxbuXnGWNMTo0aOYCTvjSBk77U+V/Lx7az/EJgr2/+g/GrlrO6YiCJZJJEVukrGYt9kkABlPeDox0mdwql8pSt1zTnqWo1cDZwtHPueGAu4Q6zU1tZ/RTCHyvP5DE+6JkYFwHbOOcGZM2fEH22McaYDLP+52Aev/VArjuuHJ9I4H3ze8N6vPOnKdg+Ub2hErWZqq5xzt0EXAfMIPxR8ZhzbiGho3QdcAxwM3CDqi7M2DzmnOuXtcuGXMXnnJuhqmnnXHfFKK0sawTeICRrNzvnLgHWAwcQErrzu+O4Mj5XgJLofTLqoG6MMX3SwQfuCDP+Q7qiknjGfCmQC3ifUqCnvNdUojLcAmwDnKGqTwFHAAcTKjKrgMuAi1T1sqztDiMkMJmvr0TLxjvnNmS9unof6+b4ALozxlaW/UBVGwlJWRXwDrAG+H/R/puNV7UVmj5vB8KYWnXAj7pp38YYkzc3HvhBi+t3oTYt9WaFWomS7DKnMT3MvoDGmJyZMWMGl8ycRMJ7krEY5Q31bEiU8N6VXe0mW/Bykt187vv/bvG7/m8/37fPZ1K9qjnPGGOM6W7br13DjtWr8IQMYWnFQEK3UtNTCrX6V9RJlHPuduD0NhZPUtXFPRlPLjjnDgKeamPxdap6XU/GY4wxPW34xjCYZNN1fIe1a/IXTJEqlOa7bEWdRKnq+XRT5+zeSlVfAiryHYcxxuRLI5BIJUnGEwyo30RNWVm+Qyo6hfKYl2y9sWO5McYY023eHjaSupJSJq5YSkM8zitDhre/kelWvpVXISjqSpQxxpjCd8OUBfz3C9vxyvbjoHYd7/50cr5DKjqFWomyJMoYY0zBe+fn++Q7hKJWKJWnbJZEGWOMMSa3rBJljDHGGNN5hVqJso7lxhhjClpDTSNzx1xI9fCvs2zb8/jnHS/mO6Si40VavAqBJVHGGGMK2rZXvcaSwcO4c8ohvDx+IpU3PpbvkIpOoT72xZrzjDHGFLR/bz+WeaNGgwjLqoZQlyhlj3wHVWRsxHJjjDGmD1o4fJtmHZtnjxiVx2iKU6FUnrJZc54xxpiCVV+XZuSG9c3mlTU05CkaU2isEmWMMaZgrb9xFi6Z5pkB5dSVljFu5Qri6VS+wyo6hdqcZ5UoY4wxBen+T11NRWk/ypKNHLjgfbatXo2PwfxhI/IdWtGxjuXGGGNMH7Fk4DSmAQk8a/v15+qpJ2weq6isoT6foRWlQq1EWRJljDGmzxv74/dJ9OuHFyGZSjGHpguc8NqYnRBACIM+NiYSpNNpYjFrjOk5hZlF2TfIGGNMn3XnF3/FjUf9hgOWLaY0maIklWLSxx+xoaQUgI0lpfxz7I6bq1ACpGNxnpt6Q95iLka+lVchKKhKlHPuBWAK0AikgIXAdFV9JGOdKcAV0Xpx4H3gl6p6X8Y6VwIHquqRrXzGWcDdQG3WoltV9ZJOxNfkt6p6dsY644D5wLOqOjVr+2uBy4D/p6rfyZg/AFgGVAHbq+oS59zZwEWquku0zoPABlU9P2uf50bnY09VXZ0x/ybgIOAzqpoZrzHG5IV/+V347l2wrpZX4gOZOXoM9+9/BLWlZdSVlJKOKktzttseicdZL2XcfMhR1PQbAL75ZfsfA0bz+vEPUJ5MUltawuEXfIYXHp9L1dB+TLtgMoOHD8jHIRasQhmhPFshVqKuUdUKYChwL/CQc24CgHNuKvA88CowHhgB3ADc7Jy7qhOfsUBVK7JeW0ygsuPLeJ2dtfwcoBo40jm3Yyvbvw+c5pzrlzHvK4QkqtNU9dfAG8AdTfOcc4dFcZxmCZQxptf4xq9gzjJYXs1fd/sUr47fmZWVVZQmGzcnUADJeJyPB1Yxd/goqgeUh5kizSogNZUDWVVeScKnGVhfz5O3/4vqVbUsen8NMx58u8cPrdAVasfyQkyiAFDVJHAnodq2dzT7VuBhVb1KVVeraq2q/h74HnCZc25sfqINnHMlwNeAawjJ0jmtrLYI+Dfw5Yx55xCOtavOAT7jnDvTOVcF3Ad8X1XnbMU+O6Smpsambdqmbbpj03WfjO+UFmFQXS2STlM9oILy+k2bl1VuqmXMmlWM3LAe8enN8xEBEQZuqvvkfSSWUamq2/hJx/Nec+w9NJ0rXlq+CkHBJlHOuVLgm9HbOc65icAE4MFWVn+I0FT+2R4Kry3HA0MIMd4NfC1KrLLdSZRgOecmEapqM7r6oaq6CjgLuIVwLmZGFaqcq6ystGmbtmmb7tj0jWdCaQIfj3PQ3PeYNvNVDlg0l6G1G9h38QIOmv8en3tvFg89eBsxYPS6ar7++ku4xQv4wjszGbN6JQPqN7Fw0BDKNtVRtX4dALWJEj519M4AlA8s5ehpe3QsngKczpVCrUQVVJ+oyGXOuYuASkLfo7NV9T/OuQOi5UuzN1DVBufcKkLzXkeMc86tzZr3LVV9qBPxNTlKVV+Lps8FZqjqKufc/cB1wBeBR7L28RjwK+fcztE29wLJDsbeKlV92jn3MHAyMHFr9mWMMbkgpxyEP3EKAhyVSjPz3pc486dPUFFWwqN7Tmby0g9YUTGQmw/8LBe+9Ay7rlrB7suXstvypdx6wBF8MHQ4AAnvOeKj99jp3guora5lt/3GAnDMV/cknogRixXGBb43KZTKU7ZCTKKmq+q1zrnBwF3A4dG/K6Pl2wGzMzeIqlbDMtZpz0JVnbA18WXPdM6NB44EjgVQ1eXOuSeA88hKolS1MUqyvk1IevbvYizZZgEHRZUpY4zpdaQkumyVwL7nHwHnHwHAuZe+TU3/chLpFPOHjGC3FR+RTCSojZeSisWYP2zk5n0M3lTHIU9fSllF/2b7LimN99hxFJtCqTxlK8QkCgBVrY7uUJvvnDse+DOwADgVeC5r9VMI/Q2f6dkomzmX0KR4j3OuqXG+HCh3zk1Q1XlZ698JvAc8r6rz892fyxhj8umdn+wBfNIMt+pOYUgySU1pGXXxBIlUkmQ8XPKGbKxpkUCZHCvMHKpwkygAVV0T3ap/HaHP0IXAY865hcBtQB1wDHAzcIOqLszYPJZ1BxxATp5aGfV7OguYDvwqa/FLhP5Pze7+U9U5zrlDaKV5cgvirRxTvaoWypAdxhgDwLD1D/HByHMYVbeBh/fef3MCBdiz80y3KegkKnIL4e67M1T1XufcEcDlwMWEcaLmEMZTujtru8MISVamadG/451zG7KWzVDVaXTNF4GBwC8yx2oCcM79ArjSOffj7I1U9eVOfs7Z0SvTZEA7uR9jjOn1xiy/k7v3vZbZo7ZtNn9FRVWeIipehdqcJ95bEcLklX0BjTE589jtD/HCS+WQMY7UoA3rufJPp+Uxql4tJ9nO3lfObfG7/q0rd+rzmVXBDnFgjDHGxLerpKKu+QMmGhOtjRxjcsmGODDtcs7dDpzexuJJqrq4J+MxxhgDlQ31bCiv2Py+JLVVI8KYLijUx75YEtWNoufSnd/uisYYY3pM/8bm9wQN25j7EbpNc4Xab8Oa84wxxhS0EXsJo6tXU5JKMnb1ClIVVj/oaV6kxasQ2DfJGGNMQSs/aBQX3Xgcm2pq6Vc5IN/hFKVCrURZEmWMMaYoWAKVP4VSecpmzXnGGGOMMV1glShjjDHG5FShVqIsiTLGGGNMTlmfKGOMMcaYLijUSpT1iTLGGGOM6QKrRBljjDEmpwrlMS/ZLIkyxhhjTE75wsyhLIkyxhhjTG5ZJcoYY4wxpgusEmWMMcYY0yWFmUXZ3XnGGGOMySkvLV+tEZFFIrJ7z0bXdVaJMsYYY0xOFWqfKKtEGWOMMSanOlqJao2ITBaRV0XkP9G/k6P514vID6Lpr4hIWkRGRO+fFJGpuTiWTJZEGWOMMSanPNLi1REiUgo8CvzYe78n8CPg0Wj+c8AR0apHAK8Bh4tICbAf8HJ3H0c2a84zeSUifwOG5TuOLUkkEsOSyeSqfMfRVX09fuj7x2Dx55fF3yl/9d4f1d07/eCSbbvanrcz0OC9fxbAe/+ciDRE818Bfh8lVAcAFwFfBpYCb3vva7c+8i2zJMrkVS7+s3Y355yqqst3HF3V1+OHvn8MFn9+Wfx9mtD684u9975ORGYB04CPgOeBnwNLgL/3RHDWnGeMMcaY3mo2UCYihwFE/5YAc6LlzwFXAc957+sJCdRZ0fycs0qUMcYYY3qTZ0UkmfH+BOCXIlIObAS+7L1viJY9B1zDJ0nTc4SmvTd6IlBLooxp36/zHcBW6uvxQ98/Bos/vyz+PsJ7P7aNRVPaWP9VMkby9N7fCNzY/ZG1TrxvranRGGOMMcZsifWJMsYYY4zpAmvOMyaLc+5Wwpgj9cAG4L9UVVtZ7yzgZmBRNGuhqp7QQ2G2qaPxR+v+mNAJE+BeVb2mR4Jsh3PudOBiYBLwXVX9VRvrHQo8ySedTOtVdb8eCXILOhp/tO45wCWEJomngO+oarpHAm07pgHAPcC+QBK4SFWfaGW9Q+kl5985NxG4DxgKrAbOUNW5WevEgV8CRxHu+PqJqv6mp2NtTQfjvxL4FrAsmvWKql7Qk3Ga5iyJMqalpwgXvkbn3LHA74Ad21j3WVX9cs+F1iEdit85dzBwEtD0nKrXnXMvquo/ei7UNr0FnAJc2oF13+2Ft393KH7n3DjgCuBThAvnU8DpwP25DrAdFwE1qjrBObcT8JJzboKqbmhl3d5y/m8HblXVB6Mk9g7g8Kx1TgMmADsRkpU3nXPPquqiHo20dR2JH+B+Vb2oZ0MzbbHmPGOyqOoTqtoYvX0VGO2c6zP/VzoR/8mEX8h1qlpHuHCf3FNxbomqvqOq7wJ5rch0VSfi/zLwmKqujKpPd9I7fgYnEy7qRNUQBT6f14i2wDk3AtgHeDia9TCwj3NueNaqJwN3qmpaVVcCjxH+kMirTsRvepk+c2EwJk8uBP6yheaVQ5xzbznn/uGcO6YnA+ugLcW/A/BBxvvFwPY9ElX3muicm+mce905d2a+g+mk3voz6ExcveH8bw8sVdUUQPTvMlrG3FvPd0fjBzjFOfcf59zTzrlW71gzPcea80zRcc7NJPwybc3Ipl9kzrlTgFOBg9tY9wngd6pa55z7FPBX59yhqvpetwedoRvjz5uOHkMHzAS2V9V1UdPYs865par6bLcE2oZujD8v2ou/E7vKy/kvYrcD06Om+s8CjzvndlXV1fkOrFhZEmWKjqru0946zrkTgOnAEaq6vI39rMqYftM59zLwaSCnSVR3xU/4K3xMxvsdgA+3PsL2deQYOrif9RnTC51zjxEG2svpRby74idPP4P24nfONcW1MiOu51vZT17Ofys+BLZzzsVVNRV1IN+Wluey6bj+Fb3PrkzlS4fiV9WPM6afcc59SOjT+GKPRms2s+Y8Y7JEnbFvAj63pQ6nzrntMqbHAPsD/8l5gO3oaPzAI8AZzrn+zrn+wBnA73sgxG7jnNvGOSfR9BBgKqFTd1/xKPBF59zwqN/aOfSOn8EjwHkAUcfyycBfs1fqLedfVVdEnzstmjUNeDPq95TpEeAc51ws6m/0RcLPIK86Gn/W75y9gbHA+z0UpmmFDbZpTBbn3EqggU/+CodQ0VntnPsN8GdV/bNz7jrgeMIt4AA3qep9PRxuCx2NP1r3SuCrhNvr71fVK3s43FY556YBPwUGE45lIzBVVd91zl0NLFPV251zFwLfBBoJlfX7VbXHRituS0fjj9Y9jzAcAsDTwIX5bg50zpUD9xLuGkwBF6vq49GyXnn+nXO7EIYIGAxUE4YIeN859yRwuapqVOH5FSHZA7hBVXvFaOAdjP8+wrATKcL36gpVfTJvQRtLoowxxhhjusKa84wxxhhjusCSKGOMMcaYLrAkyhhjjDGmCyyJMsYYY4zpAkuijDHGGGO6wJIoY0ynichYEfEiMjrHn3O+iDyQ8f4pEbl4S9uY3BCReSJyVgfX7ZHvR08QkTIRmSsiu+Q7FtP7WBJlTA6JyHgReUREPhaRDSLyoYj8SURKo+Vnici8VrZra/7p0cXp8laWvSAi9dHnrBORN0XkxNwcWe6JSDlwNXBl0zzv/ee993kfB6ot0c/mwHzHUQxyca5F5FARSWbO897XAz8jjPtlTDOWRBmTW08CHwE7A5XAFOBvhMEtu+JcYA1wtojEW1l+jfe+AhhKeBL870RkYhc/K99OB9723s/PdyCm6D0MHC4iE/IdiOldLIkyJkdEZCghebrde7/OB0u897dHf912dn+7AgcBZwLbAJ9va13vfRK4DYgDe7SyrwtF5M2seeNEJCUiY6P390SVsxoReVdETt1CbFeKyLNZ814QkR9lvN9dRP4mIqtEZLGIXC8iJVs45C8Cz7S1z4wmozOj+DaKyJMiMlhEfiIiK6IK4AUZ258VNUtdIiIfRev8PDOO9o5bRPYUkb+KyEoRWSMiz0TzZ0WrPB1VA3/TxrkaICK3RJ+xSkQeE5EdMpa/EMX0aBTDfBE5vq2TlHFM3xORJdE2PxORodE+1ovI7MyqjYgkRORyEVkQHcNzIrJ7xvISEbkp4xxe0srnHiQiL0fbzxeR74tIh/84EJETRWRWVDWdJSInZB9T1vr3Np3Tts61iCyKjuvlaL6KyOTW9pExb5GECu+2wFNAPNp2g4icCeC9X0943t4XOnp8pjhYEmVMjnjvVwP/B/xGRM4QkUmduci04jxCZeYJQoXr3LZWlNBceAHhcRyzWlnlf4FdRWTvjHlnAS947xdF718G9gYGEZrV7hWRSV0JXERGEB6S+kfCg1WnAJ8F/nsLm+0DvNuB3Z8IHEh4mOxY4HVgfvQ5XwNuzkxSCA+g3QEYH8VxHHBRxvI2j1tEtomO48Xos0YBNwB47/eKtp/qva/w3p/dRry/IDxncf8ollXADGleWTyT8PzDKsJjSu4TkQFbOAdjonjHR+fi24SEoOnRM38E7slY/weEZyUeTUjIXwKeEZGB0fJLgWOBzwDjomPd/KBkEdmN8B38KTAcOAa4kPAIoXaJyBTCd/BSQtX0h8DDIrJfR7Zv51yfD/wXMAT4A/BkxnFtaZ/LCH+YpKJ9VnjvMx/j9DbhO2nMZpZEGZNbhwIvAN8lPGB0uYj8OCuZGiciazNfhCrSZiLSj3CBujuadRdwtLTsuHtZtP0SwnP9TvTet+hb5b2vBh4nJBlE8ZyZsX+893d571d771Pe+98SHq58aBfOAYQL9izv/R3e+wbv/VLg+mh+WwYD6zuw72u892uipPUJoNF7f6f3Pum9f4rwHLJPZayfBn7gva+LmgpvJDoP0O5xfxWY572/3nu/MTqWZhW4LRGRGOGYf+S9X+q930j4buwKfDpj1d9571/x3qeBXxOSqZ22sOs64KoonlmExPlf3vvXvPcp4EFggohURet/DbjBez87qopeTXge2zHR8jOi5fO893WEJDPzGWHfBB7x3j8enafZhGRvSz/PTF8DHvXePxX9nP4C/An4ege335K7vPf/9t43EBLcOkJCuLXWExIzYzazJMqYHPLer/Le/9B7vw+hUnAxcDkZF21gofd+UOYL+FbWrk4CKggXQwhVgBVAdrVjerSPEd77z3jvZ2whvHuA06Kq1eFRfH+EcLEXkatF5P2ouWUtsBeh6tAV44ADshLFuwmVnLZUA+1WEAh9zprUZr1vmleZ8X6F97424/0iYDR06LjHAnM6EFNbhgP9gAVNM7z3Gwg/y+0z1vsoY/nGaDLzGLKtiBKuJtnnoel4m/axfVYMacJ5aIphdPQ+M4YVGfsbB0zL+nleQahqdUSzz4/Mp/k56KpFTRM+PBx2MdHPdysNJPRHNGYzS6KM6SHe+1rv/b2Eysbe7aye7TxC/6Z3RORjQqVpCPANab2DeUc8DWwi/JV+FvDbqOoAMI2QoJ0IDI4Su1m03SF+A1CeNW/bjOkPgGezksWqqBN8W94EutR82I4RWU1jYwnnE9o/7kVsuSLU3hPdVwL1hCQEABGpAEYAH3Ys/G7xYVYMMcJ5aIphafS+aXk5IcYmHwB3Z/08B3rvd+vK50fGZ3x+e98naPtcZ8YthKbbpp9vs/2KSILmx5WZiGbbnfCdNGYzS6KMyREJHZyvl9ChuiTqzHsi4ZfxS53YzyTgAOAEQvLV9Po0oZJzdFfii6oP9wPfAb5ERlMe4a/uJOGiHxORrxMqMm1RYB8R2Tc6zgtpfpG8H3Ai8nUR6RdVfMaLyFFb2OdjwJGdP7J2xYCfiEh/ERlPaKpq6vvS3nE/COwsoWP6gOjnekTG8o/ZQpKVcc6vEZFto2Tu58Bs4I1uOr6OuBe4WEQmRpXIy4AE8Jdo+QPAD0RkRxHpT2jyzEygbwNOEZHjMr7bk0TkkE58/oki8jkRiYvI5wnfwaZ+W28Skt1jo+/KCcDBWfto61x/XUT2kXCzwA+AARnHpcAREm6iKAOmA5k3N3xM6FjeLMETkUrC/7c/d/D4TJGwJMqY3Gkg/JX7R0IzwErgR8C3vfePdGI/5wEzvfczvPcfZ7z+AzwSLe+qe4BDCE2KmRfx+wgdtOcRqhKT2ELi571/gZAM/JXQjDQSeCVj+cfAYYQ77hYRmur+RKg+tOUBYK8o0TP0XbwAAAFZSURBVOlOHxCOaSHhGP9KSBKgneOOOh8fSugUvwRYDmTeuXYZcLWIVIvIHW18/vcIF/N/EZqatgG+EPVd6ik/Jdy2/zThGA4ndNJu6oN2PWEojtcI52kx4bwB4L1/h1DB/C7h572CkBh1qLnXe/9PQh+8nxG+CzcCp3vvX4uWzyd0Dv814f/OUcCjWbtp61z/GvhltN+TgWO89+uiZf9LSIRmEpoPFxN+zk1xzSEkiG9EzZRNHeWnAc977+d25PhM8ZDQZGyMMb2PiJwPHOC979BdXx3Y31mETt023k8BEpFFhJ/vg+2t24l9lgHvEBLd97prv6YwJPIdgDHGtMV7fztwe77jMMUruntxS/3gTBGz5jxjjDHGmC6w5jxjjDHGmC6wSpQxxhhjTBdYEmWMMcYY0wWWRBljjDHGdIElUcYYY4wxXWBJlDHGGGNMF1gSZYwxxhjTBf8flErGrlvhbNAAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "shap.summary_plot(shap_values, X)" ] }, { "cell_type": "code", "execution_count": 88, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAFJCAYAAABHBFN/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xu8VHW9//HXVxAv4S2VUlSQ0IzSLD+eIjXvZippp4vXDBVv5ekcy6IjppiAUh1FE1NRxEtgeioLO5pamubvlOdj5qVSRDaYIAkIKqFycf3++H63LhYze8+evfbMhv1+Ph7z2HvW97vW+qzvrJn5zPf7nTUhyzJEREREpDzrNTsAERERkXWNEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkvVudgCy9po+fXo2bNiwZochIiLSSKGWSurBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESlZyLKs2THIWmrA+Hk6eUREpNuaM3LbrthsqKWSerBERERESqYES0RERKRkSrBERERESqYES0RERKRkSrBERERESta72QGsy8zsAWBf4Gh3vy23/GPAH4A57j4wLRsMfAc4CNgceBl4Cpjk7j9LdUYD5wFvABmwALgJuNDd2/1Gn5llwD7u/nsz2w+4H/gn8BawAnga+Ckw0d3f7NzRi4iI9Fzqwep6fwNOLSw7NS0HwMx2BR4lJjr7AZsCOwGXA58trPuAu/dNdb4MfCv9rccqd+/r7psC/YHRwMnAA2bWp85tioiI9Hjqwep6PwPOMLNB7j7LzDYBPgeMA76a6kwA/s/dT8qttwq4O93WkHqsHjKzvwAGTOlMkO7+BnCvmX0WeJKYtE3qzDZFRER6KvVgdb03gB8Dp6T7xwK/A14EMLONiMOI0zqyUTNbz8z2Bz4EPFNWsO7+LLE37cCytikiItLTqAerMSYB95jZBcBpwAXAFqns3UAvYG5rZTPbHXgg3d0QeL+7z0n39zWzJcBGQB/gR+lWpheALUvepoiISI+hHqwGcPengDnESezvYfVhv8XE4cDtcvX/7O6bE3unNmD1y/L/LpVtApxLnLO1cckhbwcsKnmbIiIiPYYSrMa5lphgXe/uq1oXuvsy4EHgmI5szN2Xu/vFxG8SXlhWkOnbjHsAvy1rmyIiIj2NhggbZxrwd+L8pqKvEyesTyZOfm8hDhvuVcN2zwPuM7MJuWHEDjOzDYC9gcuAx4Eb692WiIhIT6cEq0HSt/Tuq1L2ZzMzYrL0IPE6WIuAvxAv01A1cXL3h8zsIWIv1vAOhtXLzJYSLw+xkjhZ/hbgCl0HS0REpH4hy9q9PqVIRQPGz9PJIyIi3dackdt2xWZD+1U0B0tERESkdBoiXIeki44OqFA0x90/2Oh4REREeiolWOsQJVEiIiLdgxIsqduVQx5l2LBhzQ5DRESk29EcLBEREZGSKcESERERKZkSLBEREZGSKcESERERKZkSLBEREZGSKcESERERKZkSLBEREZGS6bcIpW76LUIRka7XRb+nJ/XTbxGKiIiINIMSLBEREZGSKcESERERKZkSLBEREZGSKcESERERKVnvZgfQXZjZA8BQYAWwCmgBxrr77RXK84a6+5NmtjVwCXAosBnwGvBn4GR3fzFtY0NgFHAs0B94FbgL+I67/z3VGZj2vb27v5CLb7XlZjYaOA94A8iABcBNwIXunuXW2wY4HzgM2ApYBPwBGO/uj5rZcGAysKxwXBPdfWQHmlBEREQSJViru8jdx5hZb+AsYKqZPebuM/PlVda9BXgF+Ii7v2Rm/YjJVgZgZr2AXwHvBY4HHgW2B/4L+KOZ7enuczsY7wPufpCZBWBv4NfAbGBK2ue2wCPERO8w4GlgI+CzwL+mGABmufvgDu5bREREqlCCVYG7rzSzScBlwO7AzHZWAfgEcLS7v5S28RKxR6nVscA+wJBcwtZiZkcDfwEuBEbUGW8GPGRmfwGMlGAB3wX+CXzW3Vt73pYCN9ezHxEREamNEqwKzKwPcGa6O6PG1R4Evm9m2wH/Bzzh7qty5YcBf8wlVwC4+woz+wlwcifiXQ/YF/gQqyd1hwGTc8mViIiINIASrNWNMrNzgE2Ic61GuPsTFcrf5u6bp3+PBv4NOAm4HHjTzKYA33b3N4CtgWpDgPOAfnXEu6+ZLSEO+/UBfpRurdraZ96OaTt5X3H3qXXEJCIi0uMpwVrd2DQHawvgeuCA9He18koruvtS4GLg4tQDdihxKO5V4iTzBcQ5V5Vsm8rhnUn06xfqrF8oB/hdmoPVB/gG8CVg47RP0jb7V9lnXovmYImIiJRHl2mowN0XE+dDHWZmR9ax/nJ3/yVwH3EOF8DdwMfMbFC+bppQ/0XitwkBXgTeBIoJz2DifKqXquzvYmJCdWGu6H+Az5tZMVkTERGRLqQerCrc/WUzuxQYZ2bT26uf6k4DngSWA58E9if2agFMBU4BfmFmp/DOtwi/T7ysw+i037fM7Gbgu2Y2C5gFDCImTjfmL8FQwXnAfWY2wd3nABcAfwT+28y+TZxPtiHwGeCD7n5ere0hIiIitVMPVtsuB7YBTkz3v2NmSwu3I1LZesANxB6mxcBVwA+Il2HA3VcShw1/CdxK7I16BHgd+JfW62Al/0GcNH8PcbjvHmJv2Grzv4rc/SHgIVIvVrrsw57EXrHWbf0NOAr4aW7VQRWOa1rtzSQiIiJ5Icva6hARqW7A+Hk6eUREutickds2OwRZXailknqwREREREqmBEtERESkZEqwREREREqmBEtERESkZLpMg9TtyiGPMmzYsGaHISIi0u2oB0tERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZPqxZ6mbfuxZpOfSDxBLD6YfexYRERFpBiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiXr3YidmNkDwFBgBbAKaAHGuvvtuTpDgQtSvV7AM8AV7n5jrs5oYG93P6jCPoYDk4FlhaKJ7j6yxviWA28Bi4CHgQnu/miV48gb6u5PmtkU4HjgzbSdF9IxXG1mS3P1N0h/32xd4O590/b3BfZ19wdz+50JjHH3KbllgdhG7wW2dfelubL9gPuBv7r7BwvHehdwKHCSu08xs4HEx2MZkL/swhPu/omKDSYiIiJtakiClVzk7mPMrDdwFjDVzB5z95lmdgjwS+BiYoLyOnAEcI2ZDXL3C2rcxyx3H9yZ+ADMbABwKvAHM/uiu/+8Ur0qbnT3EWa2HnA08ThnuHvf1gpmdh3Q292HV1h/EfADM/uYu7d1nan9gUHAUuBYYFKhfBWwvpnt5e4Pp/3uAHwMmFdhe+939xfa2J+IiIjUqOFDhO6+kpgM9AZ2T4snAtPc/UJ3X+Tuy9z9NuBsYFTqZWlkjHPc/TzgJuCHqbeoo9t4y92nEROmj3Rg1UnAdsSkqS2nA3cDN6f/K7mOmCi2OgWYRkxgRUREpIs0PMEysz7AmenuDDPbGRgM3FKh+lTiFVMPblB4RbcC/YH3d3RFM+tlZscB7wa8A6v+EzgfGGdmG1SqYGZbA0cRh0SvB/Ywsz0qVJ0CHGVmm5lZL+Bk1uzpEhERkZI1MsEaZWZLiL0nY4AR7v4EsHUqn1tcwd2XAwuBfjXuY0czW1K4HdeJmFuHzLbMLRtV3EdhnS+lZS8B5wCnuPvvOrjfG4DXgH+vUn4S8Aow3d3/DDwGnFas5O4vAfcBJwCfBuan+pX8pXBcV3UwZhEREUkaOQdrbJqDtQWx1+WA9HdBKu8PPJ1fIfV2bZWr056WTszBqmS79HdRbtnYduZg3ezuIzqzU3dfZWbfAqaZ2fX5sjRceSpwi7u3Tra/HrjEzL6Rn+yeTALGA3Nou/fqg5qDJSIiUo5mzMFaDIwADjOzI4FngVlApZ6mY4jfbLu3cRGu5mhiz9ozjd6xu98FPEIcLsw7kDikerKZzTez+cCFQF8qt+E9wGbESfHTui5iERERadXIHqy3ufvLZnYpMA6YTvxW4R1m1gJcRRxGPByYAIx395bc6uuZ2YaFTS4vMz4z256YBA4Hjm7n23xd6ZvAH1j9+E4DHiQmf3njiJPdr80vdPfMzA4HNnL317owVhEREUmakmAllxO/JXhiuh7TgcTemm8Rr4M1AzjH3ScX1tufNb8F1/qNu0GF601BnKfU3jfyAL5jZiOJPWaLgP8HfMLdH6lQ79uFZce4+5017KND3P1xM7uVmOhhZv2Ik9s/5+7z83XNbDzwNzOzCtv5aw27e8bM8onkEnffrmptERERqSpkWbM6Z2RtN2D8PJ08Ij3UnJHbNjsEkWap6dJN+qkcERERkZI1c4iwYczsauKlCioZ4u7PNzIeERERWbdpiFDqNn369GzYsGHNDkNERKSRNEQoIiIi0gxKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpguNSt30W4QizaffBBRpOF1oVERERKQZlGCJiIiIlEwJloiIiEjJlGCJiIiIlEwJloiIiEjJlGCJiIiIlKx3M3duZg8AQ4EVwCqgBRjr7rfn6gwFLkj1egHPAFe4+425OqOBvd39oAr7GA5MBpYViia6+8jOxldijOe5++Bay4rLK8Q6Cxjj7j9t5xgPA84BdkuxPwWc6+4PtbWeiIiIVNcderAucve+wJbAFGCqmbUmDYcA9wP/CwwC+gHjgQlmdmEH9jHL3fsWbm0mV7XEV3KMZcjHOg34iZnt3M46WwA/BAYDWwNTgbvMbPsujVRERGQd1h0SLADcfSUwidirtntaPBGY5u4Xuvsid1/m7rcBZwOjzGxgk+PrVjEWYr2K2CO1azt1f+zuP3f3Je6+0t1/BLwOWANCFRERWSd1mwTLzPoAZ6a7M1LPy2DglgrVpxKvpHpwg8JbI760rFvF2CrF+lXicOHjHVx3N2IP2FNdEJqIiEiP0B0SrFFmtoTYazIGGOHuTxCHqwDmFldw9+XAQuJwXC12NLMlhdtxnYyPkmMsQ2usLwBHAp9z95m1rmxm/YD/Br7n7s92UYwiIiLrvKZOck/GuvsYM9sCuB44IP1dkMr7A0/nV0g9NFvl6rSnpdIE8k7GR8kxVrMCWL/C8vVT2Rqx1rMTM9sWuBe4B/jPerYhIiIiUXfowQLA3RcDI4DDzOxI4FniN+Eq9TQdA2TEhKBZ8UFjYpwNbGNmGxeWD0777rQ0T+wh4C53P8vd9SPOIiIindAderDe5u4vm9mlwDhgOnAWcIeZtRAnbb8OHA5MAMa7e0tu9fXMbMPCJpd3VXxmNt3d3zKzsmIMFcpWAI8QE7kJZjYSeBXYi5jsndHZYzKzXYD7gCnufl5ntyciIiLdqAcr53JgG+BEd78LOBD4JLEnZyEwCjjH3UcV1tufmNzkb19MZYPMbGnhNq2z8QGUGWOFsm+6+wpiwrYZceL5y8TLKpxTvB5XnUYShzj/o9A+x5ewbRERkR4pZJlGg6Q+A8bP08kj0mRzRm7b7BBEeppQS6Xu2IMlIiIislbrVnOwGs3MrgZOqFI8xN2fb2Q8XcHM9gHuqlI8zt3HNTIeERGRnkBDhFK36dOnZ8OGDWt2GCIiIo2kIUIRERGRZlCCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJdN1sKRu+qkckcbTT+OINJ2ugyUiIiLSDEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZEqwRERERErWu9kBNJKZPQAMBVYAq4AWYKy7356rMxS4INXrBTwDXOHuN+bqjAb2dveDKuxjODAZWFYomujuI2uI0YDzgL2ADYD5wP8A4939xVRnCPBdYH9go3Qc1wMT3P2tCnG8BbwBPAX8GLihSr0OxysiIiJr6lEJVnKRu48xs97AWcBUM3vM3Wea2SHAL4GLgeOB14EjgGvMbJC7X1DjPma5++COBmZmBwPTgcuBr7r7XDPbBhgB7Avcama7AQ8DtwAfAhalshuADwNfrhSHmfUFDknbPgz4XGfjFRERkcp6YoIFgLuvNLNJwGXA7sBMYCIwzd0vzFW9zcw2Bq4zsxvcfXYXhnUVMDXfc5R6rS7K1bk0LvYzc8vuNbMTgPvNbJK7/764YXdfCvzMzBYCvzOzg9393q45DBERkZ6tx87BMrM+QGuSMsPMdgYGE3uGiqYSr9x6cBfG07r/qW3U2QjYjwoxuvsDwAvAp9vaj7s/CMwDDqw/WhEREWlLT+zBGmVm5wCbEOdijXD3J8xsr1Q+t7iCuy9PPT/9atzHjma2pLDsK+5eNXkCtq62/5x3E+eFVaszj9pifAHYMne/nnhFRESkip6YYI1Nc7C2IE4MPyD9XZDK+wNP51dIvV1b5eq0p6WOOU35/f+tSp2XiZPz+1cp3xb4TQ372g64P3e/nnhFRESkih47ROjui4mTxw8zsyOBZ4FZwHEVqh8DZECXzVly9xnEeWDHtlHndeBBKsRoZp8kJk53tbUfM9ubmIj9tjPxioiISHU9sQfrbe7+spldCowjfnvvLOAOM2shTjh/HTgcmEC8TEJLbvX1zGzDwiaXdzKkrwDTzewfwJXuPs/M+gGnEL/p9xPgG8BDZnYlMIbYq7UP8VuEU939oUobNrN3EeeQXQ78wt3v6WSsIiIiUkWPTrCSy4GzgRPdfYqZHQicD3yLON9pBnCOu08urLc/MQHLa+19GmRmSwtl0929au8UgLvfm3qYzgOeTEOT84E7iQkU7v6YmX2ceB2svwIbAnOAHxK/YZjXGkcGvEm8DtY44Loq9ToUr4iIiFQWsixrdgyylhowfp5OHpEGmzNy22aHINLThVoq9dg5WCIiIiJdRUOEDWRmVwMnVCke4u7PNzIeERER6RoaIpS6aYhQpPE0RCjSdDUNEaoHS+p25ZBHGTZsWLPDEBER6XY0B0tERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZLrQqNRNFxoVKZcuIiqyVtBvEYqIiIg0gxIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpmRIsERERkZL1bq+CmT0ADAVWAKuAFmCsu9+eqzMUuCDV6wU8A1zh7jfm6owG9nb3gyrsYzgwGVhWKJro7iM7EF+rW919RK7OjsBzwH3ufkhh/THAKOCH7v613PKNgXnAZsD27v6CmY0AznH3XVKdW4Cl7n5GYZunpfbYzd0X5ZZfCuwDfMLd8/Hm190F+BvwTyADVgIzgV8Cl7n70gr18h5x9wNSnU8B5wNDiMn0vNQ2F+b294FUZ3+gL/AP4DfAJe4+q1KMIiIi0rZae7Aucve+wJbAFGCqmQ0GMLNDgPuB/wUGAf2A8cAEM7uw8uYqmuXufQu3NpOrYny524hC+anAYuAgM3tfhfWfAY43sw1zy75ITEg6zN2vBR4BrmldZmb7pziOr5ZcFQx0902I7fl14DDgETPbrEK9/LG3JlfvB34O/BDYmvjYfZGYrLXG9FHg/4BXgY8DmwAfA/4KHNrhAxcRERGghh6sPHdfaWaTgMuA3Ylv1hOBafleEeC21AN0nZnd4O6zywq4o8xsfeAk4CLgdGKS8+1CtdnA88DngVvSslOBScClde76VOAJM/sycAdwI/ANd5/RkY2kZOwhMzsSeBr4GvFY2mPAQne/NbfsyXRrdQXwkLufnlu2EJjQkRhFRERkdR2ag2VmfYAz090ZZrYzMJh3kpK8qcSrnR7cqQg770jg3cQYJwMnpaSraBIxKcLMhhB746bXu1N3XwgMBy4ntsWfUs9Wvdt7idhTeGCNqzwC9DOzyWb2GTPrny9MPWGfAH5cb0wiIiJSWa0J1igzWwK8DowBRrj7E8ShJ4C5xRXcfTmxN6RfjfvY0cyWFG7HdSS+3O3jubLTgOkp4bmJmGwdVWEbdwC7pKG104hDoStr3H9F7n4PMI04R6w4bFmPF4hDfXnPFY790rTvZ4nDfhB7pP5uZn8xs8PTsq2ICfAaj52IiIh0Tq1DhGPdfYyZbQFcDxyQ/i5I5f2Jw1dvS71dW+XqtKfF3QfXWLdifMWFZjYIOAg4AsDd/2FmdxKHCm/P13X3FWZ2E/BvwNG8k5x01uPAPinB66ztgEWFZe+rtm13/zNwMoCZvYc48f5nKYlcRJxE37/SuiIiIlK/Dg0RuvtiYk/MYWlO0LPALKBST9MxxDfwezsbZCecRuylucHM5pvZfGLCdUDrJP2CScQh0Mfd/bkGxtkuM9ua+E2/39azvrv/g5hg9QGGuPsS4P8Bx5YWpIiIiAAdnOQO4O4vp2GoccQ5SmcBd5hZC3AVcRjxcOKw1Hh3b8mtvl7hm3oAy+uKvB1pntVwYCxwZaH4IeJ8q9W+pejuM8xsXzo2bNarwjG96e5ZxyKuLB3HvwDfA14iTkyvZb0DgA8QL+/wAvEbgt8kXtbhsVTt34HfmdlVxG9+Pg9sQUyY33L3q8o4BhERkZ6mwwlWcjlwNnCiu08xswOJ11L6FvE6WDOI14uaXFhvf2ICltfagzLIzJYWyqa7e709LEcBmxKvHbXasJqZXQaMNrPvFFdy9993cD8jWHN+1Z6Ad3A7RbPNLCNee2wmcCdwqbu/WqFe/v4/3P19wMvELxicC2xOTKz+DBzq7i8CuPujZrYnsWfrEeBdxCTuXuCSTsYvIiLSY4UsK6WjRXqgAePn6eQRKdGckds2OwQRaV+opZJ+KkdERESkZPUOETaMmV0NnFCleIi7P9/IeDrLzDZgzW8CtrrX3T/byHhERESkfBoilLppiFCkXBoiFFkr1DRE2O17sKT7unLIowwbNqzZYYiIiHQ7moMlIiIiUjIlWCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUjJdaFTqpguNdi+6SKWISEPotwhFREREmkEJloiIiEjJlGCJiIiIlEwJloiIiEjJlGCJiIiIlEwJloiIiEjJejc7gGYxsweAocAKYBXQAox199tzdYYCF6R6vYBngCvc/cZcndHA3u5+UIV9DAcmA8sKRRPdfWQbsS3N3d0g/X2zdYG7903x3wf8DrgrV39jYDmwMt1/yN0/bWYZsI+7/77C/mYD782t06q/u79SLU4RERGprMcmWMlF7j7GzHoDZwFTzewxd59pZocAvwQuBo4HXgeOAK4xs0HufkGN+5jl7oM7EpS7923938yuA3q7+/AqdR8C8vVnAmPcfUpH9gmMcPdbOriOiIiIVKAhQsDdVwKTiAnn7mnxRGCau1/o7ovcfZm73wacDYwys4HNiVZERES6OyVYgJn1Ac5Md2eY2c7AYKBSj85U4lVcD25QeCIiIrKW6elDhKPM7BxgE+JcrBHu/oSZ7ZXK5xZXcPflZrYQ6FfjPnY0syWFZV9x96l1R901rjGzK3P3n3f33ZoWjYiIyFqspydYY9McrC2A64ED0t8Fqbw/8HR+hdTbtVWuTntaOjoHq0lO1xwsERGRcmiIEHD3xcAI4DAzOxJ4FpgFHFeh+jFABtzbuAhFRERkbdLTe7De5u4vm9mlwDhgOvFbhXeYWQtwFfFbhIcDE4Dx7t6SW309M9uwsMnlDQi7o/oU4nzL3btjnCIiIms1JViru5z4LcET3X2KmR0InA98i3gdrBnAOe4+ubDe/sQELO/Y9HdQ4bpWANPd/Vga7zeF+88Au6T/rzOzqwvlQ939ya4PS0REZN0Ssixrdgyylhowfp5Onm5kzshtmx2CiEhPEGqppDlYIiIiIiXTEGGTpOG4E6oUD3H35xsZj4iIiJRHQ4RSt+nTp2fDhg1rdhgiIiKNpCFCERERkWZQgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMl1oVOq2tv0WoX6rT0RESqALjYqIiIg0gxIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpWe9m7tzMHgCGAiuAVUALMNbdb8/VGQpckOr1Ap4BrnD3G3N1RgN7u/tBFfYxHJgMLCsUTXT3kZ2Nr8QYz3P3wbWWFZdXiHUWMMbdf9rOMfYHrgJ2B3YAvuTut7S1joiIiLStO/RgXeTufYEtgSnAVDNrTRoOAe4H/hcYBPQDxgMTzOzCDuxjlrv3LdzaTK5qia/kGMuQj3Ua8BMz27mddd4C7gGOA17o4vhERER6hO6QYAHg7iuBScRetd3T4onANHe/0N0Xufsyd78NOBsYZWYDmxxft4qxEOtVxN60Xdup+6K7T3T3h4k9XyIiItJJ3SbBMrM+wJnp7ozU8zIYqDRcNZV4oa+DGxTeGvGlZd0qxlYp1q8Shwsfb/T+RUREerrukGCNMrMlwOvAGGCEuz8BbJ3K5xZXcPflwELicFwtdjSzJYXbcZ2Mj5JjLENrrC8ARwKfc/eZDdy/iIiI0ORJ7slYdx9jZlsA1wMHpL8LUnl/4On8CqmHZqtcnfa0VJpA3sn4KDnGalYA61dYvn4qWyPWTu5PREREOqk79GAB4O6LgRHAYWZ2JPAs8ZtwlXqajgEy4N4mxgeNiXE2sI2ZbVxYPjjtW0RERLqZ7tCD9TZ3f9nMLgXGAdOBs4A7zKyFOGn7deBwYAIw3t1bcquvZ2YbFja5vKviM7Pp7v6WmZUVY6hQtgJ4hJjITTCzkcCrwF7EZO+MMo4rt98ArJ/ur0yT5UVERKSDuk0PVs7lwDbAie5+F3Ag8EliT85CYBRwjruPKqy3PzG5yd++mMoGmdnSwm1aZ+MDKDPGCmXfdPcVxIRtM+Ap4GVsFEz3AAAXRklEQVTgh2n7q12PqxNa97cD8ZphrwPnlbRtERGRHidkWdbsGGQtNWD8vLXq5JkzcttmhyAiImu/UEul7tiDJSIiIrJW61ZzsBrNzK4GTqhSPMTdn29kPF3BzPYB7qpSPM7dxzUyHhERkZ5AQ4RSNw0RiohID1TTEGGP7sGSzrlyyKMMGzas2WGIiIh0O5qDJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyXWhU6tbdLzSqC4uKiEgX0G8RioiIiDSDEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkvVudgBlMrMHgKHACmAV0AKMdffbc3WGAheker2AZ4Ar3P3GXJ3RwN7uflCFfQwHJgPLCkUT3X1kB+Jrdau7j8jV2RF4DrjP3Q8prD8GGAX80N2/llu+MTAP2AzY3t1fMLMRwDnuvkuqcwuw1N3PKGzztNQeu7n7otzyS4F9gE+4ez5eERERace62IN1kbv3BbYEpgBTzWwwgJkdAtwP/C8wCOgHjAcmmNmFHdjHLHfvW7i1mVwV48vdRhTKTwUWAweZ2fsqrP8McLyZbZhb9kVigtVh7n4t8AhwTesyM9s/xXG8kisREZGOWxcTLADcfSUwidhLt3taPBGY5u4Xuvsid1/m7rcBZwOjzGxgc6KNzGx94CTgImIidWqFarOBR4HP55adSjzWep0KfMLMvmxmmwE3At9w9xmd2KaIiEiPtc4mWGbWBzgz3Z1hZjsDg4FbKlSfSrwy68ENCq+aI4F3E2OcDJyUkq6iSaTky8yGEHvjpte7U3dfCAwHLie2xZ9Sz5aIiIjUYV1MsEaZ2RLgdWAMMMLdnwC2TuVziyu4+3JgIXHIsBY7mtmSwu24jsSXu308V3YaMD0lPDcRk62jKmzjDmAXM3t/WmcKsLLG/Vfk7vcA04hzxIrDliIiItIB69Qk92Ssu48xsy2A64ED0t8Fqbw/8HR+hdTbtVWuTnta3H1wZ+IrLjSzQcBBwBEA7v4PM7sTOB24PV/X3VeY2U3AvwFHAx+nHI8D+6QET0REROq0LvZgAeDui4k9MYeZ2ZHAs8AsoFJP0zFABtzbuAjXcBpxmPIGM5tvZvOJCdcBrZP0CyYRh0Afd/fnGhiniIiItGNd7MF6m7u/nC43MI44R+ks4A4zawGuIg4jHg5MAMa7e0tu9fUK39QDWN4VcaZ5VsOBscCVheKHiPOtVvuWorvPMLN9qTDk2YZeFY7pTXfPOhaxiIiItGWdTrCSy4nfEjzR3aeY2YHA+cC3iNfBmkG8XtTkwnr7ExOwvGPT30FmtrRQNt3dj6U+RwGbApflr0UFYGaXAaPN7DvFldz99x3czwjWnF+1J+Ad3I6IiIi0IWSZOi+kPgPGz+vWJ8+ckds2OwQREVn3hFoqrbNzsERERESapScMETaMmV0NnFCleIi7P9/IeERERKQ5NEQoddMQoYiI9EA1DRGqB0vqduWQRxk2bFizwxAREel2NAdLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGQhy7JmxyBrqQ022OCp5cuXv9HsONY1vXv33mrlypULmx3HukbtWj61addQu3aNEtt1YZZlh7a7vxJ2JD3Urrvu+oa7W7PjWNeYmatdy6d2LZ/atGuoXbtGo9tVQ4QiIiIiJVOCJSIiIlIyJVjSGdc2O4B1lNq1a6hdy6c27Rpq167R0HbVJHcRERGRkqkHS0RERKRk+hahrMHMdgZuBLYEFgEnuvuzhTq9gCuAQ4EMuMTdr2uvrCcroV1HA18B5qXqD7v7VxsTffdUY5seAowDdgV+6O7n5Mp0rlZQQruORufqGmps1+8AxwAr0+1cd/91KtsYuAHYI5Wd4+53Nu4Iup8S2nQKcBDQevmG2919bBmxqQdLKrkamOjuOwMTgWsq1DkeGAzsBAwFRpvZwBrKerLOtivATe6+e7r1+DcsamvTWcCpwPcrlOlcrayz7Qo6VyuppV0fAfZ09w8DJwM/MbONUtk5wGvuPhgYBlxnZn0bEHd31tk2hfjBqvVcLSW5AiVYUmBm/YCPAtPSomnAR81s60LVo4FJ7v6Wuy8A7gC+UENZj1RSu0pOrW3q7jPd/THiJ9citXdBSe0qBR1o11+7+7J09wkgEHtnIJ6vV6d6zwIOfLqLQ++2SmrTLqMES4q2B+a6+yqA9HdeWp63AzAnd//5XJ22ynqqMtoV4Bgze8LM7jGzoV0Z8Fqg1jZti87VNZXRrqBztaiedj0ReM7dX0j3db6urow2Bfi6mT1pZneY2QfKCk4Jlsja42pgR3ffjTgs8wsz6/JPYSJ10LnaSWa2L3ARcGyzY1lXVGnTUcBgd98V+Blwd5qb2WlKsKTo70D/1hMs/d02Lc97HhiQu79Drk5bZT1Vp9vV3ee7+4r0/71p+Ye6OO7urNY2bYvO1TV1ul11rlZUc7umHr9bgKPc/Zlckc7X1XW6Td19rru/lf6/CegLbFdGcEqwZDXu/hLwZ97J8I8FHkvzU/JuB041s/XSePdRwE9rKOuRymhXM+vfWsnMdgcGAs/QQ3WgTduic7WgjHbVubqmWtvVzPYEfgJ83t3/VNjM7cDpqd5OwJ7A3V0Zd3dWRpsWztVPAauAuWXEp8s0SCVnADea2fnAYuKYNWb2P8D57u7AzcDHgNavw37X3Wel/9sq68k6267jzGwP4gvAcuBL7j6/kQfQDbXbpma2N3ArsCkQzOwY4JT0NW2dq5V1tl11rlZWy2vAVcBGwDVmb/8u8Zfc/UnicOsUM5tJbNvT3P21Bh9Dd9PZNr3RzN4DvAW8CnzG3Uv54oau5C4iIiJSMg0RioiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZb0KCGET4UQHsrd3y+EMLuJITVMCGFKCOG6Erc3MISQ5e5vHUKYE0LYqoZ1zwgh3FxWLGuDEMI+IYQlzY6jJwohnNCR53nZzxVpW1c9N+p43MeHEC4qa/9KsKTHCCEE4DLggnbqnRlCeCqE8GoIYXEIwUMIR+fKZ4cQTqiw3hrLQzQjbatvoWy/EEIWQliabvNCCDeEEN7duSNtjizLFgBTab993wV8FxjdgLC6jSzLHsqybPNmx1FNCGF0COG+ZsfRE3RVW4cQHgghnFf2drta8bnRxHPxEuCrIYT+7dasgRIs6UkOAfoA91erEEI4lpggnAJsRvzZhbOJF7Crx/7AIOJF7Cr9ptiqLMv6ZlnWF9gbGApMqHNf3cFk4KQQwqZt1DkBeDLLsucaFNNqQgi9Qgh67ROR1WRZthi4i3S1/M7Si4x0idSbc14I4f7UO/NkCGG3EMKxIYSZIYRXQgjXhRB659bZIYTw3yGEF9Pt2hDCJrnycSGEWWl7z4UQ/iNXNjD1Bn0phPDXEMJrIYR7Qgjb5MI6Crgva/vqup8AHsyy7I9Z9Hr6dHVPnU1xOvGnLG6mnSdtlmWzgDuBjxTLQgi9U5scWVh+Ywhhcvr/wBDCH1Ov24IQwq0hhH7V9pfaa+/c/f1CCCtz93uHEM5NPXBLQggPhxD2aOcYngUWAge1Ue0o4N5CLP8eQng6PW7PhxAuDiH0SmU/CCH8vFB//1T3Xen+h0IIvw4hLMytv34qaz03Tgkh/BVYBvQLIRwTQng89S6+GEK4pnV7ab33hhCmp3N1Rlo/CyEMzNU5NfV2vhJCeCyEcEi1g67QvlNCCDeHECan9p2bnh+7hxD+Lx3f/SGEbXPrzA4hnB9C+H16HngIYc9ceZvnQAhh/fSYPpO2/1wI4XMh9tCeC+wX3ulRHVTlOPZN+3glPWan58r2CyGsDCEcnbb9SgjhtvzzuML26nmt2C2E8Nt0nLPS+r1y5f+S2mZpCOH3xA85+X1unM6rlhDCyyGEu0MIg6vFWCHmLUMIN6XzZn6Iz8N358pX683OnYPbVWvrEMLwdLwj03ZfCiH8V4XzeLvcdoeHEGam/68E9gG+k7ZZ8aeJQuwd+k2Iw2ELQgiLQghfDyEMSG36Wgjh0RDCB3LrdOq5kjvXJ+XO9TXOm/R/m+1TOJbVhnJLetzvJb5GdV6WZbrpVvoNmE38+ZEPAOsTf2TzOeBa4F3EHyl9CTgu1d8QmEkcOtoI2AL4H2BybpsnEHuUAnAA8DrwqVQ2EMiICcpWxJ/veBiYlFv/j8DXCnHuB8zO3f8C8AYwBjgQ2LzKsZ3Q3nJga+BN4F+B3VN8exT2vTJ3fzDx99omV2nT7wF35O73BZYC+6T7exN/m6w38F7gQWBarv4U4Lrc/QzYu414xqU2GwT0IvbqLQS2yLd5hTinA2PaODf+AXymsOxzwI7psf1IqnN6KhtC/LmVrXP1bwSuT//3AxYRE9g+QH/AgfML58ZvUrv0ScfzaeCDxA+ag4G/Ahfn9vEb4u8Sbpr28UDazsBUfhrxnP1w2sZh6fEYXOW4i+07hXgOH57WPyOt/0vij81uDPwWuLZwjs0D9kjH8W1gAbBpjefA+HScu6W23g7YLZWNJn4Aaet5vWOK+aS0j48DLwNfyB1jBlxPPD/fQ3wdGFXia8Vm6fz4DrBBWm8W8M1c+aLUNn1Se8xn9ef5VOJrxXtSnQuBp4H1Kz1XKsR8N/E83yLdfgX8qo3XgoGpXbar1tbAcGAFMJH4Gvg+YAbwn5W2kVtnZu7+A8B57TyGo9N+RvDO82AVcF/hMbgnt05nnytTiOfNZ9I2/jXFMKDKc6Na+8wsLHv7cSrjcU919iCOOPRpqx1ruTX0TVe3nnNLLzDfzN0/LD3h8m+StwGXpf8/DzxX2MYexASlV5V9/DfwvfR/64vPnrnyrwKP5e7PAIYXtrFf/gmYlh0B/Iz4Ir6KOKT4ocKx/RNYUri9xeovqt8ivjG0vmj/CbimsO8srbsYaAGupkJSl+p/gJho9Ev3TwZmtPEYHAG8lLv/9otRul81wSK++b4GfLKwzSdbj5HqCdaPgavaiGs5sF87588PgNty9/8InJ3+34SYiOyV7p8D/Law/udIL8a5c+OT7ezzLOCR9P92aZ1BufIDWf1N4yngxMI2plPlDY7KCVb+TXnjtP0v5JZ9hdXP4dnARbn7AXielHy0dQ6kukuBw6vUHU37Cda5wMOFZRcDvy6c0/nn+feBn7exzdl07LXiOODvpJ96S8tOB55J/x+f2iRfPpb0PCd+AMuAHXLl6wGvkJ4PtJFgET/kZcBOuWXvT8u2yR1TPQnWm8DGuWUjSM/x4jZy69STYP2lsOylCo/B4hKfK1PInetp2QLgyCrPjWrt01aC1enHPS3bKdXr11Y71nLTjz1LV3ox9/8y4nyjBYVlrUMHOwI7hDW/SZIRP4nPDSF8DTiV+IQOxE95U9vY5z9z24eYxLQ1NyjuMMvuJH7KIYSwC/GHQu8MIeyYpWcgsXfllvx6IfdtlRBCSLHekmXZirT4euCSEMI3sixbmpatymqc+Jxl2d9CCH8i9uRdSuxFuCG3zz2IvU4fJr5ZB2IvQj22SutOD7lvChI/3W5XeZW3bUpMFqtZ43EIce7b14m9Zb2Jny7/kKtyAzHZuAz4IjA3y7KHU9mOwF6FcycQP53nzS7s82DgfGAXYk9IL+IbDcReMIgv2K3mFLa3IzAxhHBFbllv4AVq9/b5mmXZsnjarPG8KQ6vzc6tk4UQnic9Ju2cA1sTe4RmdCC+ou2JvUV5zwH5oevi87z4PKykI68V2xPfNPPn5XNpOcS2mFMoz5+PO6a/T6T2brV+bhttaa2T3+ZzubIXqd9LWZYty92fTfvPt3oUY1xGG+ddCc+VSvus5bzoiLIe901554Nvp2gOlnQXc4if1DYv3DbMsmxuCGEv4vDG6cBWKSmZTnwDqdVjxOGmmmVZ9jTxTX0AcSigVgcSu9JPTnM05hO7o/sSP4HX6wZgeJo38HHgplzZrcResp2zLNuUypPq8/5JfMNttW3u/4Wp/KDC4/GuLMsuaWe7HyK2dTWrPQ4hhO2JQxJjiD0AmxGHSfKP7a3ATiGEjxI/yd6QK5tD/LSbj3OzLH5xIO+t3D77AHek7e6Q2mtkbp9z098dcuvn/2/d78mF/fbNsuzMNo69DANb/0mJ/A68k9S1dQ4sID6mO1XZ7ltVluf9nXfeqFoNSssb5e/AgLD6u2Q+hrkVyvMxt77571R47DbOsmxajfuH3OPAO3N9WsuWUv25BdXbul8IYePc/YG889i2fiirZ7t1K+m50lGVjqPYprD68Zf1uH+I2MO3vN7gWynBku7iTqB1Au4mIeofQvhsKt+UOFy3AMhCCIcT5wV0xB3ExKeqEMLJIYQvhHQtpzSh9Azgr1mWvdyBfZ1GnP+yC3H+1e7EJ+4NdO4bKrcSE7crgHuzLJubK9uU2N39WghhB+JchLY48OUQQp80GfXrrQXpU+DlwA9CCDsBhBD6hngdseKL+ttS4rc1cT5HNXew+iT4vsTXogXAihDCx4Ev5VfIsmwJ8HNiElZMLG8CLD12G4YQ1kuTYg9tI4Y+xHl/i7Msez2EMIQ47NG6vxeIwy2XpPOxH1D8+vtlwOgQJ6WHEMJGIYS9U69nVzo5hPDRECc/f5PYU/WrVFb1HEiP6Y+A74X4pYDW59iuqcp8Yi9ynzb2PQ3YI4RwYohfgvgX4vl8falH2LZfER+7c9O5+37iG35rDHcSz6lvhjip/6PE4XQAsix7idjzfVVIX8cPIWweQvhsKFxKpZIsy+YB9wD/ldbbAvgv4K4sy1p7aRw4Nj1ntibOF8ur1tbrEc+5jUL8ksE5xPmGZFm2kJTUh/hN2F2JveTF7dY8Wb9GZTxXOqpS+zxGTECPSM/xzwKfzJWX9bgfTHyN6jQlWNItpG7xA4k9G08T3yR+Q0xMAH5N/CbeI8Telc8T33A74tfAyhDCfm3UWUwcivpbCOGfxLk/S4hzWWqSXmCOAn6QZdn8/I3YC/eREIJ1MHYAsix7hXjcnyZeEiHvNOKcjdeIc8hub2dzZxFfjF8mznGZUii/APgF8IsQwqvEichn0PbrxsnAlBRnNTcDH05vIGRZ9rfcvpYQk4JKPQk3EI/71+lNjrT+fOLlMI4iDqksJrZRxW/BpXWWAmcSk42lxB6z4nDzccTk5QXg97zTnm+mbUwifvHghrTP54lvpOu3cexluJaYYC8GjibOqWpt7/bOgVHEx/qOVOd3vNOjdTuxB2Z+iN/0KvZUkWVZC3F+zlnECcU3E79McFtpR9eOdKyHEJP0fxCf1zcRh81bk/HDiW2zmNhWPyps5lTiF0oeCCG8Rpxb+AXi0FAtTiC239PptgQ4MVd+HvED4YvE5OPWwvrV2noOsSemhfjaczfxHGv1ZeJr0SvpeIuJ7WXEDxtLQgh/qfFY2lTGc6UOa7RPFi/r8u/E8/9l4FDixPrWODv9uIcQNiee31fXGfdqwurDlSLrttSrcW6WZZ9M9/cjJgQDmxnX2ij1erVkWRbS/a2ARwErzJ+ptO4ZxEnqX2qrXncSQvgUMQncKGvSC2eI8/zOK87/k7VfCGE48bEtuweq4brDc6UeIYSLifP/SrlYqya5S4+SZdndxE+FUrI0hDGgxrpXU9KnxK4SQvgw8ZPtk8S5HGOAn6xNbxgijbCuPFeyLPvPMrenIULp6Wazdl85vZmWECfur6veTRxmW0oc9niCOEQhIqvTc6UCDRGKiIiIlEw9WCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUrL/D/S6lzRfehE6AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "shap.summary_plot(shap_values, X, plot_type=\"bar\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Hyperparameter tunning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training speed" ] }, { "cell_type": "code", "execution_count": 89, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a92eadf3c1c044dea4c7e513a9cc0443", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 89, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoost\n", "fast_model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=150,\n", " learning_rate=0.01,\n", " boosting_type='Plain',\n", " bootstrap_type='Bernoulli',\n", " subsample=0.5,\n", " rsm=0.5,\n", " one_hot_max_size=20,\n", " leaf_estimation_iterations=2,\n", " max_ctr_complexity=1)\n", "\n", "fast_model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accuracy" ] }, { "cell_type": "code", "execution_count": 90, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e1847556d61748cea1e37bb8c30ccb1a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tunned_model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=1000,\n", " learning_rate=0.03,\n", " l2_leaf_reg=3,\n", " bagging_temperature=1,\n", " random_strength=1,\n", " one_hot_max_size=2,\n", " leaf_estimation_method='Newton'\n", ")\n", "tunned_model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=False,\n", " eval_set=(X_validation, y_validation),\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training the model after parameter tunning" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.6431867\ttotal: 103ms\tremaining: 1m 51s\n", "100:\tlearn: 0.1590533\ttotal: 8.58s\tremaining: 1m 23s\n", "200:\tlearn: 0.1517316\ttotal: 16.1s\tremaining: 1m 10s\n", "300:\tlearn: 0.1487795\ttotal: 23.5s\tremaining: 1m\n", "400:\tlearn: 0.1468414\ttotal: 31.3s\tremaining: 53.2s\n", "500:\tlearn: 0.1448880\ttotal: 39.4s\tremaining: 45.7s\n", "600:\tlearn: 0.1433889\ttotal: 48.4s\tremaining: 38.8s\n", "700:\tlearn: 0.1421200\ttotal: 58.1s\tremaining: 31.7s\n", "800:\tlearn: 0.1411938\ttotal: 1m 6s\tremaining: 23.4s\n", "900:\tlearn: 0.1401819\ttotal: 1m 15s\tremaining: 15.2s\n", "1000:\tlearn: 0.1394250\ttotal: 1m 25s\tremaining: 6.99s\n", "1082:\tlearn: 0.1388355\ttotal: 1m 32s\tremaining: 0us\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 91, "metadata": {}, "output_type": "execute_result" } ], "source": [ "best_model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=int(tunned_model.tree_count_ * 1.2),\n", ")\n", "best_model.fit(\n", " X, y,\n", " cat_features=cat_features,\n", " verbose=100\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calculate predictions for the contest" ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Predictoins:\n", "[[0.4024 0.5976]\n", " [0.015 0.985 ]\n", " [0.0101 0.9899]\n", " ...\n", " [0.0089 0.9911]\n", " [0.0496 0.9504]\n", " [0.0135 0.9865]]\n" ] } ], "source": [ "X_test = test_df.drop('id', axis=1)\n", "test_pool = Pool(data=X_test, cat_features=cat_features)\n", "contest_predictions = best_model.predict_proba(test_pool)\n", "print('Predictoins:')\n", "print(contest_predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare the submission" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [], "source": [ "f = open('submit.csv', 'w')\n", "f.write('Id,Action\\n')\n", "for idx in range(len(contest_predictions)):\n", " line = str(test_df['id'][idx]) + ',' + str(contest_predictions[idx][1]) + '\\n'\n", " f.write(line)\n", "f.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Submit your solution [here](https://www.kaggle.com/c/amazon-employee-access-challenge/submit).\n", "Good luck!!!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" }, "widgets": { "state": { "1057714ebc614324aa3ba2cf69408966": { "views": [ { "cell_index": 17 } ] }, "8381e9eed05f4a03905ae8a56d7ab4ea": { "views": [ { "cell_index": 48 } ] }, "f49684e8c5c44241bfe2c7f577f5cb41": { "views": [ { "cell_index": 53 } ] } }, "version": "1.2.0" } }, "nbformat": 4, "nbformat_minor": 2 }