{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Solving classification problems with CatBoost"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/catboost/tutorials/blob/master/events/pydata_nyc_oct_19_2018.ipynb)\n",
"\n",
"In this tutorial we will use dataset Amazon Employee Access Challenge from [Kaggle](https://www.kaggle.com) competition for our experiments. Data can be downloaded [here](https://www.kaggle.com/c/amazon-employee-access-challenge/data)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Libraries installation"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#!pip install --user --upgrade catboost\n",
"#!pip install --user --upgrade ipywidgets\n",
"#!pip install shap\n",
"#!pip install sklearn\n",
"#!pip install --upgrade numpy\n",
"#!pip install --upgrade pandas\n",
"#!jupyter nbextension enable --py widgetsnbextension"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.10.2\n",
"Python 3.5.5 :: Anaconda custom (64-bit)\r\n"
]
}
],
"source": [
"import catboost\n",
"print(catboost.__version__)\n",
"!python --version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reading the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The most simple way — read everything in pandas data frame"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import os\n",
"import numpy as np\n",
"np.set_printoptions(precision=4)\n",
"import catboost\n",
"from catboost import *\n",
"from catboost import datasets"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# import ssl\n",
"# ssl._create_default_https_context = ssl._create_unverified_context"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"(train_df, test_df) = catboost.datasets.amazon()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
" \n",
" \n",
" ACTION \n",
" RESOURCE \n",
" MGR_ID \n",
" ROLE_ROLLUP_1 \n",
" ROLE_ROLLUP_2 \n",
" ROLE_DEPTNAME \n",
" ROLE_TITLE \n",
" ROLE_FAMILY_DESC \n",
" ROLE_FAMILY \n",
" ROLE_CODE \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" 1 \n",
" 39353 \n",
" 85475 \n",
" 117961 \n",
" 118300 \n",
" 123472 \n",
" 117905 \n",
" 117906 \n",
" 290919 \n",
" 117908 \n",
" \n",
" \n",
" 1 \n",
" 1 \n",
" 17183 \n",
" 1540 \n",
" 117961 \n",
" 118343 \n",
" 123125 \n",
" 118536 \n",
" 118536 \n",
" 308574 \n",
" 118539 \n",
" \n",
" \n",
" 2 \n",
" 1 \n",
" 36724 \n",
" 14457 \n",
" 118219 \n",
" 118220 \n",
" 117884 \n",
" 117879 \n",
" 267952 \n",
" 19721 \n",
" 117880 \n",
" \n",
" \n",
" 3 \n",
" 1 \n",
" 36135 \n",
" 5396 \n",
" 117961 \n",
" 118343 \n",
" 119993 \n",
" 118321 \n",
" 240983 \n",
" 290919 \n",
" 118322 \n",
" \n",
" \n",
" 4 \n",
" 1 \n",
" 42680 \n",
" 5905 \n",
" 117929 \n",
" 117930 \n",
" 119569 \n",
" 119323 \n",
" 123932 \n",
" 19793 \n",
" 119325 \n",
" \n",
" \n",
"
\n",
"
"
],
"text/plain": [
" ACTION RESOURCE MGR_ID ROLE_ROLLUP_1 ROLE_ROLLUP_2 ROLE_DEPTNAME \\\n",
"0 1 39353 85475 117961 118300 123472 \n",
"1 1 17183 1540 117961 118343 123125 \n",
"2 1 36724 14457 118219 118220 117884 \n",
"3 1 36135 5396 117961 118343 119993 \n",
"4 1 42680 5905 117929 117930 119569 \n",
"\n",
" ROLE_TITLE ROLE_FAMILY_DESC ROLE_FAMILY ROLE_CODE \n",
"0 117905 117906 290919 117908 \n",
"1 118536 118536 308574 118539 \n",
"2 117879 267952 19721 117880 \n",
"3 118321 240983 290919 118322 \n",
"4 119323 123932 19793 119325 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing your data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Label values extraction"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"y = train_df.ACTION\n",
"X = train_df.drop('ACTION', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Categorical features declaration"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0, 1, 2, 3, 4, 5, 6, 7, 8]\n"
]
}
],
"source": [
"cat_features = list(range(0, X.shape[1]))\n",
"print(cat_features)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looking on label balance in dataset"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Labels: {0, 1}\n",
"Zero count = 1897, One count = 30872\n"
]
}
],
"source": [
"print('Labels: ', set(y))\n",
"print('Zero count = ' + str(len(y) - sum(y)) + ', One count = ' + str(sum(y)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To train model in CatBoost we need to create wrapper class for data: Pool.\n",
"This class stores the data in CatBoost internal format.\n",
"\n",
"There exists several ways to create pool. \n",
"The most simple one: create it from pandas dataframe or numpy array"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"pool1 = Pool(data=X, \n",
" label=y,\n",
" cat_features=cat_features) #Indicies of categorical columns in X"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This approach is not the most efficient, especially for big dataset: we'll need to copy everything from pandas to our internal format.\n",
"\n",
"So CatBoost could create Pool direclty from file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets look how we could load Pool from file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Firstly, lets save data frame to disk"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"dataset_dir = './amazon'\n",
"if not os.path.exists(dataset_dir):\n",
" os.makedirs(dataset_dir)\n",
"\n",
"# We will be able to work with files with/without header and with different separators.\n",
"train_df.to_csv(os.path.join(dataset_dir, 'train.tsv'), index=False, sep='\\t', header=False)\n",
"test_df.to_csv(os.path.join(dataset_dir, 'test.tsv'), index=False, sep='\\t', header=False)\n",
"\n",
"train_df.to_csv(os.path.join(dataset_dir, 'train.csv'), index=False, sep=',', header=True)\n",
"test_df.to_csv(os.path.join(dataset_dir, 'test.csv'), index=False, sep=',', header=True)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1\t39353\t85475\t117961\t118300\t123472\t117905\t117906\t290919\t117908\r\n",
"1\t17183\t1540\t117961\t118343\t123125\t118536\t118536\t308574\t118539\r\n"
]
}
],
"source": [
"!head -n2 amazon/train.tsv"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ACTION,RESOURCE,MGR_ID,ROLE_ROLLUP_1,ROLE_ROLLUP_2,ROLE_DEPTNAME,ROLE_TITLE,ROLE_FAMILY_DESC,ROLE_FAMILY,ROLE_CODE\r\n",
"1,39353,85475,117961,118300,123472,117905,117906,290919,117908\r\n",
"1,17183,1540,117961,118343,123125,118536,118536,308574,118539\r\n"
]
}
],
"source": [
"!head -n3 amazon/train.csv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we have dataset in 2 different formats:\n",
"\n",
"1) tab-separated without header\n",
"\n",
"2) comma-separated with header\n",
"\n",
"\n",
"CatBoost, like pandas, could load data from different formats, we just need to pass proper options\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before we load data, we need to set types of each column\n",
"\n",
"Also, we need to specify columns type. For this CatBoost uses special file, column description\n",
"And we have helper-function to easily do this"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"from catboost.utils import create_cd\n",
"\n",
"feature_names = dict()\n",
"for column, name in enumerate(train_df):\n",
" if column == 0:\n",
" continue\n",
" feature_names[column - 1] = name\n",
" \n",
"create_cd(\n",
" label=0, \n",
" cat_features=list(range(1, train_df.columns.shape[0])),\n",
" feature_names=feature_names,\n",
" output_path=os.path.join(dataset_dir, 'train.cd')\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\tLabel\t\r\n",
"1\tCateg\tRESOURCE\r\n",
"2\tCateg\tMGR_ID\r\n",
"3\tCateg\tROLE_ROLLUP_1\r\n",
"4\tCateg\tROLE_ROLLUP_2\r\n",
"5\tCateg\tROLE_DEPTNAME\r\n",
"6\tCateg\tROLE_TITLE\r\n",
"7\tCateg\tROLE_FAMILY_DESC\r\n",
"8\tCateg\tROLE_FAMILY\r\n",
"9\tCateg\tROLE_CODE\r\n"
]
}
],
"source": [
"!cat amazon/train.cd"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"create_cd(\n",
" label=0, \n",
" cat_features=list(range(1, train_df.columns.shape[0])),\n",
" # feature_names=feature_names,\n",
" output_path=os.path.join(dataset_dir, 'train_without_names.cd')\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\tLabel\t\r\n",
"1\tCateg\t\r\n",
"2\tCateg\t\r\n",
"3\tCateg\t\r\n",
"4\tCateg\t\r\n",
"5\tCateg\t\r\n",
"6\tCateg\t\r\n",
"7\tCateg\t\r\n",
"8\tCateg\t\r\n",
"9\tCateg\t\r\n"
]
}
],
"source": [
"!cat amazon/train_without_names.cd"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"create_cd(\n",
" label=0, \n",
" cat_features=list(range(2, train_df.columns.shape[0])),\n",
" feature_names=feature_names,\n",
" output_path=os.path.join(dataset_dir, 'train_with_num.cd')\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\tLabel\t\r\n",
"1\tNum\tRESOURCE\r\n",
"2\tCateg\tMGR_ID\r\n",
"3\tCateg\tROLE_ROLLUP_1\r\n",
"4\tCateg\tROLE_ROLLUP_2\r\n",
"5\tCateg\tROLE_DEPTNAME\r\n",
"6\tCateg\tROLE_TITLE\r\n",
"7\tCateg\tROLE_FAMILY_DESC\r\n",
"8\tCateg\tROLE_FAMILY\r\n",
"9\tCateg\tROLE_CODE\r\n"
]
}
],
"source": [
"!cat amazon/train_with_num.cd"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"? create_cd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's load pool from file now:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"pool2 = Pool(\n",
" data=os.path.join(dataset_dir, 'train.tsv'), \n",
" #delimiter=',', \n",
" column_description=os.path.join(dataset_dir, 'train.cd'),\n",
" # has_header=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loading pool from file is the fastest way to build Pool if you don't have Pool in RAM yet"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Exercices: load the same pools from csv file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, if you want maximum performance and you data is already in RAM, than in some cases we could do better, than simply passing dataframe to Pool constructor\n",
"\n",
"We have class FeaturesData that is a fast way to pass data from numpy matrices to catboost"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"? FeaturesData"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"# Fastest way to create a Pool is to create it from numpy matrix. This way should be used if you want fast predictions\n",
"# or fastest way to load the data in python.\n",
"\n",
"X_prepared = X.values.astype(str).astype(object)\n",
"# For FeaturesData class categorial features must have type str\n",
"\n",
"pool3 = Pool(\n",
" data=FeaturesData(cat_feature_data=X_prepared, cat_feature_names=list(X)),\n",
" label=y.values\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset shape\n",
"dataset 1:(32769, 9)\n",
"dataset 2:(32769, 9)\n",
"dataset 3:(32769, 9)\n",
"\n",
"\n",
"Column names\n",
"dataset 1:\n",
"['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n",
"\n",
"dataset 2:\n",
"['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n",
"\n",
"dataset 3:\n",
"['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n"
]
}
],
"source": [
"print('Dataset shape')\n",
"print('dataset 1:' + str(pool1.shape) + '\\ndataset 2:' + str(pool2.shape) + \n",
" '\\ndataset 3:' + str(pool3.shape))\n",
"\n",
"print('\\n')\n",
"print('Column names')\n",
"print('dataset 1:')\n",
"print(pool1.get_feature_names()) \n",
"print('\\ndataset 2:')\n",
"print(pool2.get_feature_names())\n",
"print('\\ndataset 3:')\n",
"print(pool3.get_feature_names())\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Split your data into train and validation"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/noxoomo/anaconda3/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2069: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.\n",
" FutureWarning)\n"
]
}
],
"source": [
"from sklearn.model_selection import train_test_split\n",
"X_train, X_validation, y_train, y_validation = train_test_split(X, y, train_size=0.8, random_state=1234)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" \n",
" RESOURCE \n",
" MGR_ID \n",
" ROLE_ROLLUP_1 \n",
" ROLE_ROLLUP_2 \n",
" ROLE_DEPTNAME \n",
" ROLE_TITLE \n",
" ROLE_FAMILY_DESC \n",
" ROLE_FAMILY \n",
" ROLE_CODE \n",
" \n",
" \n",
" \n",
" \n",
" 14926 \n",
" 74463 \n",
" 105908 \n",
" 117961 \n",
" 118225 \n",
" 129617 \n",
" 118702 \n",
" 132654 \n",
" 118704 \n",
" 118705 \n",
" \n",
" \n",
" 7940 \n",
" 17278 \n",
" 120340 \n",
" 120342 \n",
" 120343 \n",
" 119076 \n",
" 118834 \n",
" 311236 \n",
" 118424 \n",
" 118836 \n",
" \n",
" \n",
" 24768 \n",
" 79325 \n",
" 17733 \n",
" 117961 \n",
" 118300 \n",
" 119984 \n",
" 118890 \n",
" 125128 \n",
" 118398 \n",
" 118892 \n",
" \n",
" \n",
" 1633 \n",
" 5173 \n",
" 3075 \n",
" 117961 \n",
" 117962 \n",
" 120677 \n",
" 120357 \n",
" 120678 \n",
" 118424 \n",
" 120359 \n",
" \n",
" \n",
" 2743 \n",
" 15672 \n",
" 3745 \n",
" 117961 \n",
" 118300 \n",
" 118360 \n",
" 124435 \n",
" 118362 \n",
" 118363 \n",
" 124436 \n",
" \n",
" \n",
"
\n",
"
"
],
"text/plain": [
" RESOURCE MGR_ID ROLE_ROLLUP_1 ROLE_ROLLUP_2 ROLE_DEPTNAME \\\n",
"14926 74463 105908 117961 118225 129617 \n",
"7940 17278 120340 120342 120343 119076 \n",
"24768 79325 17733 117961 118300 119984 \n",
"1633 5173 3075 117961 117962 120677 \n",
"2743 15672 3745 117961 118300 118360 \n",
"\n",
" ROLE_TITLE ROLE_FAMILY_DESC ROLE_FAMILY ROLE_CODE \n",
"14926 118702 132654 118704 118705 \n",
"7940 118834 311236 118424 118836 \n",
"24768 118890 125128 118398 118892 \n",
"1633 120357 120678 118424 120359 \n",
"2743 124435 118362 118363 124436 "
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train.head()"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(26215, 9)"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(6554, 9)"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_validation.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Selecting the objective function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Possible options for binary classification:\n",
"\n",
"`Logloss`\n",
"\n",
"`CrossEntropy` for probabilities in target"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"\n",
"model = CatBoostClassifier(\n",
" iterations=5,\n",
" learning_rate=0.1,\n",
" #loss_function='Logloss',\n",
" #loss_function='CrossEntropy'\n",
")\n",
"\n",
"train_pool = Pool(data=X_train, \n",
" label=y_train, \n",
" cat_features=cat_features)\n",
"\n",
"validation_pool = Pool(data=X_validation, \n",
" label=y_validation, \n",
" cat_features=cat_features)\n",
"model.fit(\n",
" train_pool,\n",
" eval_set=validation_pool,\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model is fitted: True\n",
"Model params:\n",
"{'iterations': 5, 'learning_rate': 0.1, 'loss_function': 'Logloss'}\n"
]
}
],
"source": [
"print('Model is fitted: ' + str(model.is_fitted()))\n",
"print('Model params:')\n",
"print(model.get_params())"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = CatBoostClassifier(\n",
" iterations=5,\n",
" learning_rate=0.1,\n",
" #loss_function='Logloss',\n",
" #loss_function='CrossEntropy'\n",
")\n",
"\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model is fitted: True\n",
"Model params:\n",
"{'iterations': 5, 'learning_rate': 0.1, 'loss_function': 'Logloss'}\n"
]
}
],
"source": [
"print('Model is fitted: ' + str(model.is_fitted()))\n",
"print('Model params:')\n",
"print(model.get_params())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Stdout of the training"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.2922585\ttest: 0.2894772\tbest: 0.2894772 (0)\ttotal: 50.6ms\tremaining: 708ms\n",
"1:\tlearn: 0.2200332\ttest: 0.2193253\tbest: 0.2193253 (1)\ttotal: 103ms\tremaining: 667ms\n",
"2:\tlearn: 0.1882770\ttest: 0.1794720\tbest: 0.1794720 (2)\ttotal: 169ms\tremaining: 676ms\n",
"3:\tlearn: 0.1787690\ttest: 0.1655785\tbest: 0.1655785 (3)\ttotal: 237ms\tremaining: 652ms\n",
"4:\tlearn: 0.1748384\ttest: 0.1586419\tbest: 0.1586419 (4)\ttotal: 309ms\tremaining: 619ms\n",
"5:\tlearn: 0.1737776\ttest: 0.1564081\tbest: 0.1564081 (5)\ttotal: 420ms\tremaining: 630ms\n",
"6:\tlearn: 0.1723479\ttest: 0.1542367\tbest: 0.1542367 (6)\ttotal: 508ms\tremaining: 581ms\n",
"7:\tlearn: 0.1719611\ttest: 0.1540092\tbest: 0.1540092 (7)\ttotal: 593ms\tremaining: 518ms\n",
"8:\tlearn: 0.1718945\ttest: 0.1536590\tbest: 0.1536590 (8)\ttotal: 676ms\tremaining: 450ms\n",
"9:\tlearn: 0.1697911\ttest: 0.1524950\tbest: 0.1524950 (9)\ttotal: 738ms\tremaining: 369ms\n",
"10:\tlearn: 0.1684481\ttest: 0.1509084\tbest: 0.1509084 (10)\ttotal: 832ms\tremaining: 303ms\n",
"11:\tlearn: 0.1673863\ttest: 0.1495139\tbest: 0.1495139 (11)\ttotal: 920ms\tremaining: 230ms\n",
"12:\tlearn: 0.1642212\ttest: 0.1473859\tbest: 0.1473859 (12)\ttotal: 1s\tremaining: 154ms\n",
"13:\tlearn: 0.1631211\ttest: 0.1468169\tbest: 0.1468169 (13)\ttotal: 1.08s\tremaining: 77.5ms\n",
"14:\tlearn: 0.1628934\ttest: 0.1465499\tbest: 0.1465499 (14)\ttotal: 1.15s\tremaining: 0us\n",
"\n",
"bestTest = 0.1465498889\n",
"bestIteration = 14\n",
"\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=15,\n",
" #verbose=5,\n",
" logging_level='Verbose'\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"{ROLE_FAMILY} pr1 tb0 type0, border=1 score 71.20196708\n",
" tensor 0 is redundant, remove it and stop\n",
"0:\tlearn: 0.3020322\ttest: 0.3024549\tbest: 0.3024549 (0)\ttotal: 46.3ms\tremaining: 648ms\n",
"\n",
"{ROLE_FAMILY} pr0 tb0 type0, border=6 score 22.73090753\n",
"{ROLE_FAMILY, ROLE_CODE} pr0 tb0 type1, border=1 score 22.85737971\n",
"{ROLE_TITLE} pr0 tb0 type0, border=13 score 23.18484659\n",
"{ROLE_DEPTNAME, ROLE_FAMILY, ROLE_CODE} pr2 tb0 type0, border=10 score 23.4271605\n",
"{ROLE_ROLLUP_2} pr1 tb0 type0, border=13 score 25.42952513\n",
"{ROLE_TITLE, ROLE_FAMILY} pr2 tb0 type0, border=7 score 25.91828995\n",
"1:\tlearn: 0.2126748\ttest: 0.2068150\tbest: 0.2068150 (1)\ttotal: 137ms\tremaining: 890ms\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=11 score 9.55130831\n",
"{MGR_ID, ROLE_CODE} pr2 tb0 type0, border=14 score 12.59291657\n",
"{ROLE_FAMILY} pr0 tb0 type1, border=9 score 14.15678078\n",
"{ROLE_TITLE, ROLE_FAMILY} pr1 tb0 type0, border=6 score 14.34218082\n",
"{ROLE_ROLLUP_1, ROLE_FAMILY} pr2 tb0 type0, border=3 score 14.2932558\n",
"{ROLE_FAMILY, ROLE_CODE} pr0 tb0 type1, border=3 score 14.4073355\n",
"2:\tlearn: 0.1856771\ttest: 0.1759821\tbest: 0.1759821 (2)\ttotal: 282ms\tremaining: 1.13s\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=10 score 6.594784046\n",
"{MGR_ID, ROLE_CODE} pr2 tb0 type0, border=12 score 8.339746264\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=11 score 9.115213253\n",
"{ROLE_ROLLUP_2} pr0 tb0 type0, border=12 score 10.5940908\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr0 tb0 type0, border=5 score 10.45322949\n",
"{ROLE_TITLE} pr2 tb0 type0, border=6 score 10.66637876\n",
"3:\tlearn: 0.1756533\ttest: 0.1621181\tbest: 0.1621181 (3)\ttotal: 397ms\tremaining: 1.09s\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=10 score 4.816932629\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 5.826899508\n",
"{ROLE_ROLLUP_2} pr2 tb0 type0, border=13 score 6.455962742\n",
"{ROLE_FAMILY} pr2 tb0 type0, border=10 score 6.668192917\n",
"{ROLE_TITLE} pr2 tb0 type0, border=8 score 6.787022309\n",
"{ROLE_FAMILY, ROLE_CODE} pr2 tb0 type0, border=1 score 6.784150409\n",
" tensor 5 is redundant, remove it and stop\n",
"4:\tlearn: 0.1732514\ttest: 0.1587816\tbest: 0.1587816 (4)\ttotal: 526ms\tremaining: 1.05s\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=7 score 2.297190459\n",
"{RESOURCE} pr2 tb0 type0, border=11 score 3.692999825\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=9 score 4.566200375\n",
"{RESOURCE} pr0 tb0 type0, border=13 score 5.195022352\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr0 tb0 type0, border=10 score 5.882398347\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr1 tb0 type0, border=6 score 5.715399768\n",
"5:\tlearn: 0.1693831\ttest: 0.1534864\tbest: 0.1534864 (5)\ttotal: 633ms\tremaining: 950ms\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=9 score 2.56631672\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 3.053372671\n",
"{ROLE_FAMILY} pr0 tb0 type1, border=14 score 3.662099526\n",
" tensor 2 is redundant, remove it and stop\n",
"6:\tlearn: 0.1692429\ttest: 0.1530965\tbest: 0.1530965 (6)\ttotal: 688ms\tremaining: 786ms\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=9 score 2.662106324\n",
"{MGR_ID, ROLE_CODE} pr0 tb0 type0, border=14 score 3.271764978\n",
" tensor 1 is redundant, remove it and stop\n",
"7:\tlearn: 0.1692302\ttest: 0.1530455\tbest: 0.1530455 (7)\ttotal: 740ms\tremaining: 647ms\n",
"\n",
"{RESOURCE} pr2 tb0 type0, border=9 score 2.704934036\n",
"{ROLE_ROLLUP_2} pr1 tb0 type0, border=9 score 3.380733801\n",
"{ROLE_FAMILY_DESC} pr0 tb0 type1, border=8 score 3.485919025\n",
"{ROLE_ROLLUP_2, ROLE_DEPTNAME} pr2 tb0 type0, border=8 score 4.040211274\n",
"{ROLE_DEPTNAME} pr2 tb0 type0, border=4 score 4.662311897\n",
"{ROLE_ROLLUP_1} pr2 tb0 type0, border=10 score 4.587959116\n",
"8:\tlearn: 0.1678218\ttest: 0.1518261\tbest: 0.1518261 (8)\ttotal: 843ms\tremaining: 562ms\n",
"\n",
"{RESOURCE} pr2 tb0 type0, border=6 score 3.22558481\n",
"{MGR_ID} pr0 tb0 type0, border=6 score 3.598670764\n",
"{MGR_ID} pr2 tb0 type0, border=6 score 4.442316954\n",
"{ROLE_ROLLUP_1} pr1 tb0 type0, border=14 score 4.534285954\n",
" tensor 3 is redundant, remove it and stop\n",
"9:\tlearn: 0.1675718\ttest: 0.1514379\tbest: 0.1514379 (9)\ttotal: 900ms\tremaining: 450ms\n",
"\n",
"{RESOURCE} pr1 tb0 type0, border=8 score 1.706869104\n",
"{RESOURCE} pr2 tb0 type0, border=12 score 2.541399767\n",
"{MGR_ID} pr0 tb0 type0, border=12 score 3.042148696\n",
"{MGR_ID, ROLE_DEPTNAME} pr1 tb0 type0, border=6 score 3.056465669\n",
"{ROLE_DEPTNAME} pr0 tb0 type1, border=14 score 3.417063506\n",
" tensor 4 is redundant, remove it and stop\n",
"10:\tlearn: 0.1662480\ttest: 0.1499766\tbest: 0.1499766 (10)\ttotal: 974ms\tremaining: 354ms\n",
"\n",
"{ROLE_FAMILY_DESC} pr1 tb0 type0, border=13 score 1.188282501\n",
"{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=6 score 2.265758008\n",
"{RESOURCE} pr2 tb0 type0, border=12 score 2.620514967\n",
"{ROLE_CODE} pr0 tb0 type0, border=14 score 3.027549244\n",
" tensor 3 is redundant, remove it and stop\n",
"11:\tlearn: 0.1660153\ttest: 0.1497512\tbest: 0.1497512 (11)\ttotal: 1.04s\tremaining: 260ms\n",
"\n",
"{RESOURCE} pr2 tb0 type0, border=3 score 1.942379095\n",
"{ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 2.328950537\n",
"{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr2 tb0 type0, border=13 score 2.976624744\n",
"{RESOURCE, ROLE_ROLLUP_1} pr2 tb0 type0, border=11 score 3.247835595\n",
"{RESOURCE, ROLE_ROLLUP_1} pr2 tb0 type0, border=4 score 3.98226103\n",
"{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr0 tb0 type0, border=14 score 3.884491014\n",
" tensor 5 is redundant, remove it and stop\n",
"12:\tlearn: 0.1655821\ttest: 0.1492094\tbest: 0.1492094 (12)\ttotal: 1.14s\tremaining: 175ms\n",
"\n",
"{RESOURCE} pr2 tb0 type0, border=6 score 1.401892445\n",
"{ROLE_DEPTNAME} pr2 tb0 type0, border=5 score 2.17295077\n",
"{RESOURCE, ROLE_ROLLUP_1} pr1 tb0 type0, border=6 score 2.344410053\n",
"{RESOURCE, ROLE_ROLLUP_1, ROLE_ROLLUP_2} pr0 tb0 type0, border=6 score 2.784440779\n",
"{ROLE_CODE} pr2 tb0 type0, border=8 score 3.478854515\n",
"{ROLE_DEPTNAME, ROLE_CODE} pr2 tb0 type0, border=7 score 3.183561803\n",
"13:\tlearn: 0.1637591\ttest: 0.1478251\tbest: 0.1478251 (13)\ttotal: 1.27s\tremaining: 91.1ms\n",
"\n",
"{MGR_ID} pr2 tb0 type0, border=5 score 1.678525175\n",
"{RESOURCE} pr2 tb0 type0, border=2 score 1.913548971\n",
"{ROLE_ROLLUP_2} pr1 tb0 type0, border=4 score 2.09899603\n",
"{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr0 tb0 type0, border=14 score 2.529101972\n",
" tensor 3 is redundant, remove it and stop\n",
"14:\tlearn: 0.1637261\ttest: 0.1477449\tbest: 0.1477449 (14)\ttotal: 1.34s\tremaining: 0us\n",
"\n",
"bestTest = 0.1477448839\n",
"bestIteration = 14\n",
"\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=15,\n",
" #verbose=5,\n",
" logging_level='Info'\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Metrics calculation and graph plotting"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b78812f3160e458dbd4fdd81998054e0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=500,\n",
" random_seed=63,\n",
" learning_rate=0.5\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Eval metric, custom metrics and best trees count"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a99d9adb234d42a9a82ff7812ebd0718",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=50,\n",
" random_seed=63,\n",
" learning_rate=0.5,\n",
" eval_metric=\"Accuracy\",\n",
" use_best_model=False\n",
")\n",
"\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"50"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.tree_count_"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3237fc1e455c42079880250710ad16a5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=50,\n",
" random_seed=63,\n",
" learning_rate=0.5,\n",
" custom_loss=['AUC', 'Accuracy']\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"28"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model._tree_count"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Metric hints"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "987ef484e520494a820090a2bb88ac50",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"\n",
"model = CatBoostClassifier(\n",
" iterations=50,\n",
" random_seed=63,\n",
" learning_rate=0.5,\n",
" eval_metric='AUC:hints=skip_train~false' #default\n",
")\n",
"\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "328bd48940a24bdc807e5bc149b56072",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=50,\n",
" random_seed=63,\n",
" learning_rate=0.5,\n",
" eval_metric='AUC:hints=skip_train~false', #default\n",
" metric_period=10\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model comparison"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model1 = CatBoostClassifier(\n",
" learning_rate=0.5,\n",
" iterations=100,\n",
" train_dir='learing_rate_0.5'\n",
")\n",
"\n",
"model2 = CatBoostClassifier(\n",
" learning_rate=0.01,\n",
" iterations=100,\n",
" train_dir='learing_rate_0.01'\n",
")\n",
"\n",
"model1.fit(\n",
" X_train, y_train,\n",
" eval_set=(X_validation, y_validation),\n",
" cat_features=cat_features,\n",
" verbose=False\n",
")\n",
"model2.fit(\n",
" X_train, y_train,\n",
" eval_set=(X_validation, y_validation),\n",
" cat_features=cat_features,\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d4ec633ced854e4f99dd4731264681ec",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from catboost import MetricVisualizer\n",
"MetricVisualizer(['learing_rate_0.01', 'learing_rate_0.5']).start()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overfitting detector"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "80e70b13a7564741b84cd3c7328d04e0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model_with_early_stop = CatBoostClassifier(\n",
" iterations=200,\n",
" random_seed=63,\n",
" learning_rate=0.5,\n",
" early_stopping_rounds=20\n",
")\n",
"model_with_early_stop.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"28\n"
]
}
],
"source": [
"print(model_with_early_stop.tree_count_)"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "32fc59c9d0204b41b75e2bb13327d8be",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model_with_early_stop = CatBoostClassifier(\n",
" eval_metric='AUC',\n",
" iterations=200,\n",
" random_seed=63,\n",
" learning_rate=0.5,\n",
" early_stopping_rounds=20\n",
")\n",
"model_with_early_stop.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cross-validation"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "447c85da2d564f4890220e78ff7eb230",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from catboost import cv\n",
"\n",
"params = {}\n",
"params['loss_function'] = 'Logloss'\n",
"params['iterations'] = 80\n",
"params['custom_loss'] = 'AUC'\n",
"params['random_seed'] = 63\n",
"params['learning_rate'] = 0.5\n",
"\n",
"cv_data = cv(\n",
" params = params,\n",
" pool = Pool(X, label=y, cat_features=cat_features),\n",
" fold_count=5,\n",
" type = 'Classical',\n",
" shuffle=True,\n",
" partition_random_seed=0,\n",
" plot=True,\n",
" stratified=False,\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" \n",
" test-AUC-mean \n",
" test-AUC-std \n",
" test-Logloss-mean \n",
" test-Logloss-std \n",
" train-AUC-mean \n",
" train-AUC-std \n",
" train-Logloss-mean \n",
" train-Logloss-std \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" 0.500000 \n",
" 0.000000 \n",
" 0.302217 \n",
" 0.003871 \n",
" 0.499996 \n",
" 0.000009 \n",
" 0.302189 \n",
" 0.002418 \n",
" \n",
" \n",
" 1 \n",
" 0.625287 \n",
" 0.098486 \n",
" 0.225502 \n",
" 0.014372 \n",
" 0.618981 \n",
" 0.088396 \n",
" 0.227776 \n",
" 0.009594 \n",
" \n",
" \n",
" 2 \n",
" 0.797381 \n",
" 0.031802 \n",
" 0.182074 \n",
" 0.010177 \n",
" 0.758575 \n",
" 0.036903 \n",
" 0.190874 \n",
" 0.004790 \n",
" \n",
" \n",
" 3 \n",
" 0.818542 \n",
" 0.019081 \n",
" 0.168630 \n",
" 0.006834 \n",
" 0.778474 \n",
" 0.012547 \n",
" 0.180657 \n",
" 0.004351 \n",
" \n",
" \n",
" 4 \n",
" 0.823875 \n",
" 0.013242 \n",
" 0.163591 \n",
" 0.007106 \n",
" 0.784664 \n",
" 0.007662 \n",
" 0.176791 \n",
" 0.003464 \n",
" \n",
" \n",
" 5 \n",
" 0.842055 \n",
" 0.006845 \n",
" 0.159424 \n",
" 0.006278 \n",
" 0.801486 \n",
" 0.008265 \n",
" 0.173545 \n",
" 0.003944 \n",
" \n",
" \n",
" 6 \n",
" 0.852505 \n",
" 0.011592 \n",
" 0.155835 \n",
" 0.007250 \n",
" 0.812029 \n",
" 0.004814 \n",
" 0.170815 \n",
" 0.003318 \n",
" \n",
" \n",
" 7 \n",
" 0.859290 \n",
" 0.013055 \n",
" 0.152686 \n",
" 0.007374 \n",
" 0.821573 \n",
" 0.004150 \n",
" 0.168168 \n",
" 0.002686 \n",
" \n",
" \n",
" 8 \n",
" 0.862400 \n",
" 0.012930 \n",
" 0.151412 \n",
" 0.007750 \n",
" 0.825537 \n",
" 0.002166 \n",
" 0.166854 \n",
" 0.002284 \n",
" \n",
" \n",
" 9 \n",
" 0.864313 \n",
" 0.012127 \n",
" 0.150388 \n",
" 0.007312 \n",
" 0.827699 \n",
" 0.004355 \n",
" 0.165980 \n",
" 0.002763 \n",
" \n",
" \n",
" 10 \n",
" 0.864621 \n",
" 0.012251 \n",
" 0.149981 \n",
" 0.007268 \n",
" 0.828238 \n",
" 0.004967 \n",
" 0.165710 \n",
" 0.002916 \n",
" \n",
" \n",
" 11 \n",
" 0.869911 \n",
" 0.012121 \n",
" 0.148150 \n",
" 0.008058 \n",
" 0.832391 \n",
" 0.004098 \n",
" 0.164250 \n",
" 0.002763 \n",
" \n",
" \n",
" 12 \n",
" 0.871178 \n",
" 0.011428 \n",
" 0.147479 \n",
" 0.008175 \n",
" 0.834270 \n",
" 0.005416 \n",
" 0.163328 \n",
" 0.003049 \n",
" \n",
" \n",
" 13 \n",
" 0.871663 \n",
" 0.011281 \n",
" 0.147303 \n",
" 0.008230 \n",
" 0.834886 \n",
" 0.005530 \n",
" 0.163057 \n",
" 0.003074 \n",
" \n",
" \n",
" 14 \n",
" 0.872252 \n",
" 0.010388 \n",
" 0.147200 \n",
" 0.008247 \n",
" 0.835580 \n",
" 0.006042 \n",
" 0.162711 \n",
" 0.003282 \n",
" \n",
" \n",
" 15 \n",
" 0.872506 \n",
" 0.010518 \n",
" 0.147003 \n",
" 0.008478 \n",
" 0.836177 \n",
" 0.006505 \n",
" 0.162350 \n",
" 0.003406 \n",
" \n",
" \n",
" 16 \n",
" 0.873782 \n",
" 0.009503 \n",
" 0.146393 \n",
" 0.008212 \n",
" 0.837587 \n",
" 0.006670 \n",
" 0.161729 \n",
" 0.003425 \n",
" \n",
" \n",
" 17 \n",
" 0.875730 \n",
" 0.008438 \n",
" 0.145739 \n",
" 0.007961 \n",
" 0.840422 \n",
" 0.008338 \n",
" 0.160630 \n",
" 0.003884 \n",
" \n",
" \n",
" 18 \n",
" 0.876655 \n",
" 0.009206 \n",
" 0.145421 \n",
" 0.008027 \n",
" 0.842564 \n",
" 0.005416 \n",
" 0.159910 \n",
" 0.003345 \n",
" \n",
" \n",
" 19 \n",
" 0.876961 \n",
" 0.009529 \n",
" 0.145151 \n",
" 0.008368 \n",
" 0.843762 \n",
" 0.004223 \n",
" 0.159456 \n",
" 0.002908 \n",
" \n",
" \n",
" 20 \n",
" 0.877131 \n",
" 0.009657 \n",
" 0.145095 \n",
" 0.008361 \n",
" 0.843862 \n",
" 0.004137 \n",
" 0.159337 \n",
" 0.002843 \n",
" \n",
" \n",
" 21 \n",
" 0.877797 \n",
" 0.009627 \n",
" 0.144691 \n",
" 0.008332 \n",
" 0.845742 \n",
" 0.004977 \n",
" 0.158503 \n",
" 0.002997 \n",
" \n",
" \n",
" 22 \n",
" 0.878456 \n",
" 0.009885 \n",
" 0.144474 \n",
" 0.008387 \n",
" 0.847026 \n",
" 0.004351 \n",
" 0.157964 \n",
" 0.002731 \n",
" \n",
" \n",
" 23 \n",
" 0.878098 \n",
" 0.010434 \n",
" 0.144497 \n",
" 0.008484 \n",
" 0.848029 \n",
" 0.005031 \n",
" 0.157516 \n",
" 0.002853 \n",
" \n",
" \n",
" 24 \n",
" 0.878518 \n",
" 0.010405 \n",
" 0.144371 \n",
" 0.008634 \n",
" 0.848554 \n",
" 0.005170 \n",
" 0.157213 \n",
" 0.002953 \n",
" \n",
" \n",
" 25 \n",
" 0.879114 \n",
" 0.010706 \n",
" 0.144143 \n",
" 0.008871 \n",
" 0.849741 \n",
" 0.005065 \n",
" 0.156656 \n",
" 0.002807 \n",
" \n",
" \n",
" 26 \n",
" 0.879001 \n",
" 0.010942 \n",
" 0.144083 \n",
" 0.008958 \n",
" 0.850191 \n",
" 0.004955 \n",
" 0.156296 \n",
" 0.002907 \n",
" \n",
" \n",
" 27 \n",
" 0.879322 \n",
" 0.011028 \n",
" 0.143980 \n",
" 0.008761 \n",
" 0.850911 \n",
" 0.005407 \n",
" 0.155861 \n",
" 0.002955 \n",
" \n",
" \n",
" 28 \n",
" 0.879694 \n",
" 0.011326 \n",
" 0.143692 \n",
" 0.009107 \n",
" 0.851471 \n",
" 0.004956 \n",
" 0.155599 \n",
" 0.002789 \n",
" \n",
" \n",
" 29 \n",
" 0.879781 \n",
" 0.011045 \n",
" 0.143615 \n",
" 0.009135 \n",
" 0.851630 \n",
" 0.005069 \n",
" 0.155455 \n",
" 0.002823 \n",
" \n",
" \n",
" ... \n",
" ... \n",
" ... \n",
" ... \n",
" ... \n",
" ... \n",
" ... \n",
" ... \n",
" ... \n",
" \n",
" \n",
" 50 \n",
" 0.882512 \n",
" 0.012500 \n",
" 0.142604 \n",
" 0.010409 \n",
" 0.860230 \n",
" 0.005378 \n",
" 0.150163 \n",
" 0.003652 \n",
" \n",
" \n",
" 51 \n",
" 0.882453 \n",
" 0.012385 \n",
" 0.142621 \n",
" 0.010400 \n",
" 0.860310 \n",
" 0.005331 \n",
" 0.150098 \n",
" 0.003616 \n",
" \n",
" \n",
" 52 \n",
" 0.882385 \n",
" 0.012415 \n",
" 0.142637 \n",
" 0.010403 \n",
" 0.860454 \n",
" 0.005418 \n",
" 0.149982 \n",
" 0.003705 \n",
" \n",
" \n",
" 53 \n",
" 0.882443 \n",
" 0.012071 \n",
" 0.142524 \n",
" 0.010384 \n",
" 0.861073 \n",
" 0.005605 \n",
" 0.149685 \n",
" 0.003798 \n",
" \n",
" \n",
" 54 \n",
" 0.882297 \n",
" 0.012177 \n",
" 0.142517 \n",
" 0.010484 \n",
" 0.861248 \n",
" 0.005420 \n",
" 0.149561 \n",
" 0.003734 \n",
" \n",
" \n",
" 55 \n",
" 0.882199 \n",
" 0.012224 \n",
" 0.142556 \n",
" 0.010513 \n",
" 0.861319 \n",
" 0.005484 \n",
" 0.149484 \n",
" 0.003754 \n",
" \n",
" \n",
" 56 \n",
" 0.882194 \n",
" 0.012205 \n",
" 0.142574 \n",
" 0.010495 \n",
" 0.861369 \n",
" 0.005492 \n",
" 0.149424 \n",
" 0.003744 \n",
" \n",
" \n",
" 57 \n",
" 0.882277 \n",
" 0.012128 \n",
" 0.142493 \n",
" 0.010513 \n",
" 0.861646 \n",
" 0.005424 \n",
" 0.149269 \n",
" 0.003714 \n",
" \n",
" \n",
" 58 \n",
" 0.882321 \n",
" 0.012203 \n",
" 0.142489 \n",
" 0.010529 \n",
" 0.861708 \n",
" 0.005415 \n",
" 0.149229 \n",
" 0.003720 \n",
" \n",
" \n",
" 59 \n",
" 0.882194 \n",
" 0.012165 \n",
" 0.142555 \n",
" 0.010500 \n",
" 0.861739 \n",
" 0.005397 \n",
" 0.149171 \n",
" 0.003693 \n",
" \n",
" \n",
" 60 \n",
" 0.882152 \n",
" 0.012189 \n",
" 0.142579 \n",
" 0.010494 \n",
" 0.861742 \n",
" 0.005393 \n",
" 0.149148 \n",
" 0.003703 \n",
" \n",
" \n",
" 61 \n",
" 0.882442 \n",
" 0.012275 \n",
" 0.142514 \n",
" 0.010480 \n",
" 0.862027 \n",
" 0.005136 \n",
" 0.148943 \n",
" 0.003607 \n",
" \n",
" \n",
" 62 \n",
" 0.882531 \n",
" 0.012148 \n",
" 0.142492 \n",
" 0.010439 \n",
" 0.862121 \n",
" 0.005156 \n",
" 0.148857 \n",
" 0.003626 \n",
" \n",
" \n",
" 63 \n",
" 0.882384 \n",
" 0.012305 \n",
" 0.142450 \n",
" 0.010402 \n",
" 0.862249 \n",
" 0.005253 \n",
" 0.148737 \n",
" 0.003715 \n",
" \n",
" \n",
" 64 \n",
" 0.882327 \n",
" 0.012362 \n",
" 0.142604 \n",
" 0.010350 \n",
" 0.862404 \n",
" 0.005327 \n",
" 0.148628 \n",
" 0.003732 \n",
" \n",
" \n",
" 65 \n",
" 0.882394 \n",
" 0.012476 \n",
" 0.142554 \n",
" 0.010403 \n",
" 0.862664 \n",
" 0.005393 \n",
" 0.148464 \n",
" 0.003785 \n",
" \n",
" \n",
" 66 \n",
" 0.882718 \n",
" 0.012037 \n",
" 0.142475 \n",
" 0.010294 \n",
" 0.862964 \n",
" 0.005743 \n",
" 0.148285 \n",
" 0.003972 \n",
" \n",
" \n",
" 67 \n",
" 0.882441 \n",
" 0.012522 \n",
" 0.142524 \n",
" 0.010263 \n",
" 0.863391 \n",
" 0.006173 \n",
" 0.148076 \n",
" 0.004102 \n",
" \n",
" \n",
" 68 \n",
" 0.882410 \n",
" 0.012563 \n",
" 0.142535 \n",
" 0.010284 \n",
" 0.863487 \n",
" 0.006285 \n",
" 0.147985 \n",
" 0.004199 \n",
" \n",
" \n",
" 69 \n",
" 0.882688 \n",
" 0.012817 \n",
" 0.142429 \n",
" 0.010369 \n",
" 0.863947 \n",
" 0.006193 \n",
" 0.147717 \n",
" 0.004175 \n",
" \n",
" \n",
" 70 \n",
" 0.882825 \n",
" 0.012729 \n",
" 0.142343 \n",
" 0.010321 \n",
" 0.864080 \n",
" 0.006283 \n",
" 0.147574 \n",
" 0.004268 \n",
" \n",
" \n",
" 71 \n",
" 0.882948 \n",
" 0.012839 \n",
" 0.142289 \n",
" 0.010418 \n",
" 0.864243 \n",
" 0.006387 \n",
" 0.147433 \n",
" 0.004301 \n",
" \n",
" \n",
" 72 \n",
" 0.883091 \n",
" 0.012895 \n",
" 0.142179 \n",
" 0.010493 \n",
" 0.864703 \n",
" 0.006083 \n",
" 0.147174 \n",
" 0.004146 \n",
" \n",
" \n",
" 73 \n",
" 0.883080 \n",
" 0.012929 \n",
" 0.142213 \n",
" 0.010555 \n",
" 0.864784 \n",
" 0.006046 \n",
" 0.147111 \n",
" 0.004117 \n",
" \n",
" \n",
" 74 \n",
" 0.882921 \n",
" 0.012968 \n",
" 0.142223 \n",
" 0.010504 \n",
" 0.865053 \n",
" 0.006098 \n",
" 0.146930 \n",
" 0.004142 \n",
" \n",
" \n",
" 75 \n",
" 0.882927 \n",
" 0.012993 \n",
" 0.142192 \n",
" 0.010595 \n",
" 0.865155 \n",
" 0.006075 \n",
" 0.146842 \n",
" 0.004101 \n",
" \n",
" \n",
" 76 \n",
" 0.882942 \n",
" 0.012987 \n",
" 0.142221 \n",
" 0.010574 \n",
" 0.865304 \n",
" 0.005987 \n",
" 0.146695 \n",
" 0.004043 \n",
" \n",
" \n",
" 77 \n",
" 0.883305 \n",
" 0.012968 \n",
" 0.142100 \n",
" 0.010714 \n",
" 0.865512 \n",
" 0.006000 \n",
" 0.146537 \n",
" 0.003998 \n",
" \n",
" \n",
" 78 \n",
" 0.883523 \n",
" 0.013146 \n",
" 0.142131 \n",
" 0.010681 \n",
" 0.865617 \n",
" 0.005938 \n",
" 0.146448 \n",
" 0.003941 \n",
" \n",
" \n",
" 79 \n",
" 0.883661 \n",
" 0.013047 \n",
" 0.142058 \n",
" 0.010638 \n",
" 0.865727 \n",
" 0.005961 \n",
" 0.146341 \n",
" 0.003937 \n",
" \n",
" \n",
"
\n",
"
80 rows × 8 columns
\n",
"
"
],
"text/plain": [
" test-AUC-mean test-AUC-std test-Logloss-mean test-Logloss-std \\\n",
"0 0.500000 0.000000 0.302217 0.003871 \n",
"1 0.625287 0.098486 0.225502 0.014372 \n",
"2 0.797381 0.031802 0.182074 0.010177 \n",
"3 0.818542 0.019081 0.168630 0.006834 \n",
"4 0.823875 0.013242 0.163591 0.007106 \n",
"5 0.842055 0.006845 0.159424 0.006278 \n",
"6 0.852505 0.011592 0.155835 0.007250 \n",
"7 0.859290 0.013055 0.152686 0.007374 \n",
"8 0.862400 0.012930 0.151412 0.007750 \n",
"9 0.864313 0.012127 0.150388 0.007312 \n",
"10 0.864621 0.012251 0.149981 0.007268 \n",
"11 0.869911 0.012121 0.148150 0.008058 \n",
"12 0.871178 0.011428 0.147479 0.008175 \n",
"13 0.871663 0.011281 0.147303 0.008230 \n",
"14 0.872252 0.010388 0.147200 0.008247 \n",
"15 0.872506 0.010518 0.147003 0.008478 \n",
"16 0.873782 0.009503 0.146393 0.008212 \n",
"17 0.875730 0.008438 0.145739 0.007961 \n",
"18 0.876655 0.009206 0.145421 0.008027 \n",
"19 0.876961 0.009529 0.145151 0.008368 \n",
"20 0.877131 0.009657 0.145095 0.008361 \n",
"21 0.877797 0.009627 0.144691 0.008332 \n",
"22 0.878456 0.009885 0.144474 0.008387 \n",
"23 0.878098 0.010434 0.144497 0.008484 \n",
"24 0.878518 0.010405 0.144371 0.008634 \n",
"25 0.879114 0.010706 0.144143 0.008871 \n",
"26 0.879001 0.010942 0.144083 0.008958 \n",
"27 0.879322 0.011028 0.143980 0.008761 \n",
"28 0.879694 0.011326 0.143692 0.009107 \n",
"29 0.879781 0.011045 0.143615 0.009135 \n",
".. ... ... ... ... \n",
"50 0.882512 0.012500 0.142604 0.010409 \n",
"51 0.882453 0.012385 0.142621 0.010400 \n",
"52 0.882385 0.012415 0.142637 0.010403 \n",
"53 0.882443 0.012071 0.142524 0.010384 \n",
"54 0.882297 0.012177 0.142517 0.010484 \n",
"55 0.882199 0.012224 0.142556 0.010513 \n",
"56 0.882194 0.012205 0.142574 0.010495 \n",
"57 0.882277 0.012128 0.142493 0.010513 \n",
"58 0.882321 0.012203 0.142489 0.010529 \n",
"59 0.882194 0.012165 0.142555 0.010500 \n",
"60 0.882152 0.012189 0.142579 0.010494 \n",
"61 0.882442 0.012275 0.142514 0.010480 \n",
"62 0.882531 0.012148 0.142492 0.010439 \n",
"63 0.882384 0.012305 0.142450 0.010402 \n",
"64 0.882327 0.012362 0.142604 0.010350 \n",
"65 0.882394 0.012476 0.142554 0.010403 \n",
"66 0.882718 0.012037 0.142475 0.010294 \n",
"67 0.882441 0.012522 0.142524 0.010263 \n",
"68 0.882410 0.012563 0.142535 0.010284 \n",
"69 0.882688 0.012817 0.142429 0.010369 \n",
"70 0.882825 0.012729 0.142343 0.010321 \n",
"71 0.882948 0.012839 0.142289 0.010418 \n",
"72 0.883091 0.012895 0.142179 0.010493 \n",
"73 0.883080 0.012929 0.142213 0.010555 \n",
"74 0.882921 0.012968 0.142223 0.010504 \n",
"75 0.882927 0.012993 0.142192 0.010595 \n",
"76 0.882942 0.012987 0.142221 0.010574 \n",
"77 0.883305 0.012968 0.142100 0.010714 \n",
"78 0.883523 0.013146 0.142131 0.010681 \n",
"79 0.883661 0.013047 0.142058 0.010638 \n",
"\n",
" train-AUC-mean train-AUC-std train-Logloss-mean train-Logloss-std \n",
"0 0.499996 0.000009 0.302189 0.002418 \n",
"1 0.618981 0.088396 0.227776 0.009594 \n",
"2 0.758575 0.036903 0.190874 0.004790 \n",
"3 0.778474 0.012547 0.180657 0.004351 \n",
"4 0.784664 0.007662 0.176791 0.003464 \n",
"5 0.801486 0.008265 0.173545 0.003944 \n",
"6 0.812029 0.004814 0.170815 0.003318 \n",
"7 0.821573 0.004150 0.168168 0.002686 \n",
"8 0.825537 0.002166 0.166854 0.002284 \n",
"9 0.827699 0.004355 0.165980 0.002763 \n",
"10 0.828238 0.004967 0.165710 0.002916 \n",
"11 0.832391 0.004098 0.164250 0.002763 \n",
"12 0.834270 0.005416 0.163328 0.003049 \n",
"13 0.834886 0.005530 0.163057 0.003074 \n",
"14 0.835580 0.006042 0.162711 0.003282 \n",
"15 0.836177 0.006505 0.162350 0.003406 \n",
"16 0.837587 0.006670 0.161729 0.003425 \n",
"17 0.840422 0.008338 0.160630 0.003884 \n",
"18 0.842564 0.005416 0.159910 0.003345 \n",
"19 0.843762 0.004223 0.159456 0.002908 \n",
"20 0.843862 0.004137 0.159337 0.002843 \n",
"21 0.845742 0.004977 0.158503 0.002997 \n",
"22 0.847026 0.004351 0.157964 0.002731 \n",
"23 0.848029 0.005031 0.157516 0.002853 \n",
"24 0.848554 0.005170 0.157213 0.002953 \n",
"25 0.849741 0.005065 0.156656 0.002807 \n",
"26 0.850191 0.004955 0.156296 0.002907 \n",
"27 0.850911 0.005407 0.155861 0.002955 \n",
"28 0.851471 0.004956 0.155599 0.002789 \n",
"29 0.851630 0.005069 0.155455 0.002823 \n",
".. ... ... ... ... \n",
"50 0.860230 0.005378 0.150163 0.003652 \n",
"51 0.860310 0.005331 0.150098 0.003616 \n",
"52 0.860454 0.005418 0.149982 0.003705 \n",
"53 0.861073 0.005605 0.149685 0.003798 \n",
"54 0.861248 0.005420 0.149561 0.003734 \n",
"55 0.861319 0.005484 0.149484 0.003754 \n",
"56 0.861369 0.005492 0.149424 0.003744 \n",
"57 0.861646 0.005424 0.149269 0.003714 \n",
"58 0.861708 0.005415 0.149229 0.003720 \n",
"59 0.861739 0.005397 0.149171 0.003693 \n",
"60 0.861742 0.005393 0.149148 0.003703 \n",
"61 0.862027 0.005136 0.148943 0.003607 \n",
"62 0.862121 0.005156 0.148857 0.003626 \n",
"63 0.862249 0.005253 0.148737 0.003715 \n",
"64 0.862404 0.005327 0.148628 0.003732 \n",
"65 0.862664 0.005393 0.148464 0.003785 \n",
"66 0.862964 0.005743 0.148285 0.003972 \n",
"67 0.863391 0.006173 0.148076 0.004102 \n",
"68 0.863487 0.006285 0.147985 0.004199 \n",
"69 0.863947 0.006193 0.147717 0.004175 \n",
"70 0.864080 0.006283 0.147574 0.004268 \n",
"71 0.864243 0.006387 0.147433 0.004301 \n",
"72 0.864703 0.006083 0.147174 0.004146 \n",
"73 0.864784 0.006046 0.147111 0.004117 \n",
"74 0.865053 0.006098 0.146930 0.004142 \n",
"75 0.865155 0.006075 0.146842 0.004101 \n",
"76 0.865304 0.005987 0.146695 0.004043 \n",
"77 0.865512 0.006000 0.146537 0.003998 \n",
"78 0.865617 0.005938 0.146448 0.003941 \n",
"79 0.865727 0.005961 0.146341 0.003937 \n",
"\n",
"[80 rows x 8 columns]"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cv_data"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best validation Logloss score, not stratified: 0.1421±0.0106 on step 79\n"
]
}
],
"source": [
"best_value = np.min(cv_data['test-Logloss-mean'])\n",
"best_iter = cv_data['test-Logloss-mean'].idxmin()\n",
"\n",
"print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n",
" best_value,\n",
" cv_data['test-Logloss-std'][best_iter],\n",
" best_iter)\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f52baa02b8ad418ea07b4b50be5def76",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best validation Logloss score, not stratified: 0.1412±0.0055 on step 67\n"
]
}
],
"source": [
"cv_data = cv(\n",
" params = params,\n",
" pool = Pool(X, label=y, cat_features=cat_features),\n",
" fold_count=5,\n",
" type = 'Classical',\n",
" shuffle=True,\n",
" partition_random_seed=0,\n",
" plot=True,\n",
" stratified=True,\n",
" verbose=False\n",
")\n",
"\n",
"best_value = np.min(cv_data['test-Logloss-mean'])\n",
"best_iter = cv_data['test-Logloss-mean'].idxmin()\n",
"\n",
"print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n",
" best_value,\n",
" cv_data['test-Logloss-std'][best_iter],\n",
" best_iter)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Select decision boundary"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ba81252dc8bb4d3e858b3627977b24ba",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = CatBoostClassifier(\n",
" random_seed=63,\n",
" iterations=200,\n",
" learning_rate=0.03,\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import sklearn\n",
"from sklearn import metrics\n",
"from catboost.utils import get_roc_curve"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"? get_roc_curve"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4FNX6wPHvmw1pJBASikhvUqUIUgRpShEQCyqgFxW5KiIiwkVUsKFcERAEqaLIT7nKVbwolyogyFXp0qR3CCCdhJaQZM/vj9kkSwhhA9lMdvN+nifPTp93J8m8M+fMnCPGGJRSSqlrCbA7AKWUUrmbJgqllFKZ0kShlFIqU5oolFJKZUoThVJKqUxpolBKKZUpTRQqy0TkCRH5ye447CYipUXkvIg4cnCfZUXEiEhgTu3Tm0Rki4g0v4H19G8wB4m+R+HbRGQ/UAxIBs4DC4Dexpjzdsblj1zH+u/GmMU2xlAW2AfkM8Yk2RWHKxYDVDLG7PbyfsqSS75zXqV3FP7hfmNMOFAbqAO8bnM8N8TOq2R/uULPCj3eylOaKPyIMeYvYCFWwgBARIJFZKSIHBSRYyIySURC3eY/ICIbRCRORPaISFvX9IIi8rmIHBWRwyLyfkoRi4g8LSK/uoYnichI9zhE5EcR6ecavlVEvheREyKyT0T6uC33jojMFJHpIhIHPJ3+O7ni+NK1/gERGSwiAW5x/CYin4hIrIhsF5F70q2b2Xf4TURGi8hp4B0RqSAiP4vIKRE5KSL/EpFI1/JfAaWB/7qKm15NXwwkIstE5D3Xds+JyE8iUtgtnidd3+GUiLwpIvtF5N6MfpciEioiH7mWjxWRX91/b8ATrt/pSREZ5LZefRFZISJnXd97nIgEuc03IvKiiOwCdrmmjRGRQ66/gXUicrfb8g4RecP1t3HONb+UiCx3LbLRdTw6u5bv4Pp7Oisiv4tITbdt7ReRgSKyCbggIoHux8AV+1pXHMdEZJRr1ZR9nXXtq5H736Br3eoiskhETrvWfSOj46pukDFGf3z4B9gP3OsaLglsBsa4zf8YmA1EARHAf4EPXPPqA7FAK6yLhhJAFde8H4DJQH6gKLAaeN4172ngV9dwU+AQacWYhYBLwK2uba4D3gKCgPLAXqCNa9l3gETgQdeyoRl8vy+BH12xlwV2Aj3c4kgCXgHyAZ1d3yfKw++QBLwEBAKhQEXXsQgGimCdoD7O6Fi7xssCBgh0jS8D9gC3uba3DBjmmlcNq2iwietYjHR993uv8Xsd71q/BOAA7nLFlbLPKa591AISgKqu9eoCDV3fqSywDejrtl0DLML6ewh1TfsbEO1apz/wFxDimjcA62+qMiCu/UW7baui27bvAI4DDVwxP+U6ZsFux28DUMpt36nHFFgBdHMNhwMNMzrOGfwNRgBHXbGHuMYb2P2/6U8/tgegPzf5C7T+0c4D51z/TEuASNc8AS4AFdyWbwTscw1PBkZnsM1irpNPqNu0rsBS17D7P6kAB4GmrvFngZ9dww2Ag+m2/TrwhWv4HWB5Jt/N4Yqjmtu054FlbnEcwZWkXNNWA908/A4Hr7Vv1zIPAuvTHevrJYrBbvN7AQtcw28B37jNCwMuk0GiwEqal4BaGcxL2WfJdN+5yzW+Q19gltu4AVpe53ufSdk3sAN44BrLpU8UE4H30i2zA2jmdvyeyeDvNyVRLAfeBQpf4ztfK1F0df896U/2/2g5oX940BizWESaAV8DhYGzWFfFYcA6EUlZVrBOwGBd2c3LYHtlsK7Qj7qtF4B153AFY4wRkRlY/6zLgceB6W7buVVEzrqt4gD+5zZ+1TbdFMa6+j7gNu0A1lV2isPGdbZwm3+rh9/hin2LSFFgLHA31lVpANZJMyv+chu+iHVljCum1P0ZYy6KyKlrbKMw1pXxnqzuR0RuA0YB9bB+94FYd3Xu0n/v/sDfXTEaoIArBrD+RjKLw10Z4CkRecltWpBruxnuO50ewBBgu4jsA941xszxYL9ZiVHdAK2j8CPGmF+AaVjFGgAnsa5MqxtjIl0/BY1V8Q3WP22FDDZ1COtqvLDbegWMMdWvsetvgEdEpAzWXcT3btvZ57aNSGNMhDGmnXvYmXylk1jFM2XcppUGDruNlxC3TOCaf8TD75B+3x+4ptU0xhTAKpKRTJbPiqNYRYOAVQeBVdyTkZNAPBn/bq5nIrAd62mkAsAbXPkdwO17uOojBgKPAYWMMZFYxXcp61zrbyQjh4Ch6X7fYcaYbzLad3rGmF3GmK5YxYQfAjNFJH9m69xAjOoGaKLwPx8DrUSktjHGiVWWPdp1tYyIlBCRNq5lPwe6i8g9IhLgmlfFGHMU+An4SEQKuOZVcN2xXMUYsx44AXwGLDTGpNxBrAbiXBWYoa6K0RoicqcnX8QYkwx8CwwVkQhXIupH2h0LWCeVPiKST0QeBaoC87L6HVwisIrxzopICazyeXfHsOpZbsRM4H4RuctVufwuV5/AAXD93qYCo8R6GMDhqsAN9mA/EUAccF5EqgAveLB8EtbvL1BE3sK6o0jxGfCeiFQSS00RSUlw6Y/HFKCniDRwLZtfRNqLSIQHcSMifxORIq7vn/I3lOyKzcm1j/0c4BYR6SvWwxsRItLAk30qz2ii8DPGmBNYFcBvuiYNBHYDK8V6smgxVsUkxpjVQHdgNNZV5C+kXb0/iVVssBWr+GUmUDyTXX8D3ItV9JUSSzJwP9ZTWPuwrpQ/Awpm4Su9hFXPshf41bX9qW7zVwGVXNseCjxijEkp0snqd3gXq0I2FpgL/Cfd/A+Awa4nev6Rhe+AMWaL67vMwLq7OIdV8ZtwjVX+gVWJvAY4jXWF7cn/6z+wiv/OYZ24/32d5RcC87EeEjiAdSfjXjw0CitZ/4SVgD7HqkQHq47p/1zH4zFjzFqsOqpxWMd7Nxk8yZaJtsAWETkPjMGqd4k3xlzE+t3+5tpXQ/eVjDHnsB5CuB+rSG4X0CIL+1XXoS/cKZ8lIk9jvQDXxO5YskpEwrGumisZY/bZHY9SmdE7CqVyiIjcLyJhrnL3kVh3DPvtjUqp69NEoVTOeQCrov0IVnFZF6O39MoHaNGTUkqpTOkdhVJKqUz53At3hQsXNmXLlrU7DKWU8inr1q07aYwpciPr+lyiKFu2LGvXrrU7DKWU8ikicuD6S2VMi56UUkplShOFUkqpTGmiUEoplSlNFEoppTKliUIppVSmNFEopZTKlNcShYhMFZHjIvLnNeaLiIwVkd0isklE7vBWLEoppW6cN9+jmIbV3PCX15h/H1Z7N5WwOruZ6PpUSimVEWMgOQGS4sEkg3Fan063YZMMyYlw6bi1vDORy/GXb2q3XksUxpjlIlI2k0UeAL50NYq2UkQiRaS4q8MZpZTKXsZpnUCdieBMsn5MElw4BkmX3E68TreTbrrhi8fBEWQNp2zDmQTnDkJAPggITDtxO5PSTtzH10NoERC5ep4zGY6ugEKVwXkZzh2ChFgIjnTNT3L7dGb5aw/4byvWH8msG5brs/PN7BJc2UFKjGvaVYlCRJ4DngMoXbp0jgSnlMqipASIPw2Xz115EnS/0nUmw6UT1gk1ORHO7ITkeEDSljFO13pu6x77AyIrXHmCdybBntnW9NR1nFeul/IZn9Wuz21w/I8rxxPOZrxcipBCIA6QANenazjANXz+CESUpEb1woz9rexNhWZnosioG8gMm7I1xnwKfApQr149be5W5W3OJFfxQ4J1kk2KTyuOyGg8tajCdXWMsT7/Wg1hxVxX2InWifvQUihQOu2KOf3V8cnNUKCMa1smbVsXcqAg4PD/Mp5+ZlfWthMYBo58IIFWwsJYdwrFG7lOsgFuP46rh09vh+IN0u4gUrZz4QhEVoTAUNd01wk7wGEtE38aoqtmPC/AYR3PsCLgCLbuWoIKWsu6LyMO664kE1u3nuCPP47ytx41AXiyu6HZgFjKlRuStePkfshueM2bFwOUchsvidVOv1L+5/J5uPCXVbSQfNm6wr1wxDoBGKd1Qt84CUKirJP24V8hopRr3mU4H2NtRwJuqPghS2L3Zj4/LpMmgxzBVmIqVNntxOa4ctgkW9+nWD3rRBh3EMrcCwFBbssFuK3nOlHHn4XI8mkn5pQTdVA4hJdMu5omwG09t09HEOQLy9ZDlZtcvJjI++8vZ8SI33E4hIYNS1KxYhQiQtmykTe1bTsTxWygt4jMwKrEjtX6CZVrGWOdQI/9cWUxSvpy6L1zIKKMdbI/thbCilpFMUmXsr7Ps7sziMNpnfgcQdaVqyMYHCEQGOL6dB9PNxzgAMRaP+WG/nIsFKvrOum6fpyJUOi2K0/wKVfBAQ5rmcCQK7clYiW5wJAbP8bqhs2fv4sXX5zHvn1WcVWPHnWJjg69zlqe81qiEJFvgOZAYRGJAd4G8gEYYyYB84B2WB2wXwS6eysWpa7p9E7Y9R/AejqEXd9DaFFr+OhK6+o45eTpqfNuN8YXj1ufKduIqmoVewQEWVfV0dUhtLB1wnUmQ1QVKNEk7Yo5/y1WUnAEWSf7oAKuE75ScPhwHH37LmTmzK0A1KxZjEmT2tOoUanrrJk13nzqqet15hvgRW/tX+Vxl07D+k8gZhmcPwyIdaJNLaM3cHqbZ9tKSRIpxT63Pea6sk5f1uywimUqPgBFaruuvEMh/FYIirhu2bJSWfXii/P48ccdhIXlY8iQ5rz8ckMCA7P/9Tif649CKSDtefK4A9bjhGd2wqmtcGAxnNmR9e2Vvx+K1korfine0PoMCk+7CxBtyEDZLynJmZoMPvzwXvLlc/DRR60pXbqg1/apiULlXkkJ1ueFoxCz3EoIR36zil92/NuzbURVhZrPWpWr4SVcFZtCavl6vjDrKR6lcrnY2HgGD/6ZnTtPs2DBE4gIlSsX5rvvHvX6vjVRKHsZp1XRe3wjbPnCqhA+ts7z9QMCrQrl8h2sl5RKNoXQaKj0CESU1OIe5fOMMXz33Vb69l3A0aPncTiEDRv+ok6dm3uJLis0Uaic4UyCE5utZ9DP7oaDi627hKwoUsu6+g8pBGVawy31oVBF78SrVC6wZ89peveez4IF1hNwjRqVZNKkDtSsWSxH49BEobzryArYNQvWjrj2MgGBVkVz4gWo+bz1fH3RWlC4pvW4p1J50MiRv/Pmm0uJj08iMjKEDz+8l7///Q4CAnL+LlkThfKeRT1h0+Srp99S33qDNaQQNBgE4Tl3C62Ur7h4MZH4+CS6davJyJGtKVo0v22xaKJQ2efSKesls63TYdv0K+c1eAMqd4YiNe2JTalc7sSJC+zYcYomTaz27AYObEzz5mVp2tT+hy00Uaibc+wP+PFBSLwI8aeunh8YBr3PWC+MKaWu4nQapk5dz6uvLiIwMIDt23sTFRVKcHBgrkgSoIlC3QhjYN88WNzLal45vejqUL49VOgIt96lTx4pdQ1//nmcnj3n8NtvVkParVqV5+LFRKKisq/5jeygiUJ5LikBfukPG8ZfPa/pcKtoKaKUJgalruPChcsMGfILo0atJCnJSbFi+fn447Z07lwdyYX/P5oolGecSTA+0mqu2l29AdDgdatiWinlkUce+Y4FC3YjAr161WPo0HuIjMy9DSpqolCZi91nPd76S/+0aUXvgDZTrYrpXHj1o1RuN3BgY44dO8/Eie1p0KCk3eFclyYKlcaZDJdOwo4ZsPtHqxOb9Ko8Du3/lfOxKeWjkpKcfPLJKvbvP8uYMfcB0Lx5Wdaufc6WdyJuhCYKZZnX7epHWt0VKAsdZ1p9FyilPLJ69WGef34OGzb8BcBzz9WlevWiAD6TJEATRd5lDBxdBYd+hl8HXT0/KMJ6Ga5kM6vbRy1iUspjZ8/G88YbS5g0aS3GQJkyBRk3rl1qkvA1mijymrhDMK2q1VxGevlvgeditGMcpW7CjBl/0rfvAo4du0BgYAD9+zfizTebkj+/775LpIkirzBOOLISZjS+cro4rErpu9613n3QPheUuik//bSHY8cu0LhxKSZObM/tt+dsA37eoInC38Xuh597w965V05v/RnUeEaLlJS6SQkJSRw+fI7y5a1HxIcPb8Xdd5fmqadq+1Q9RGY0UfijczGw5EXYMzvj+a0mw+09cjYmpfzQzz/v44UX5hIQIGzc2JOgIAeFC4fRvXsdu0PLVpoo/EXsPjj8G+z5L+z89ur5tz8L9V+DyPI5H5tSfubYsfP84x+LmD59EwBVqhQmJiYu9a7C32ii8GWHlsGCp61+ozNSqgU0HwVFa+dkVEr5LafTMGXKOl57bQlnz8YTEhLI4MF3M2BAY4KC/PchEE0UvurkFvi2xdXTKzxgfTb9EKIq52xMSvm5hx76N7Nn7wCgTZsKjB/fjgoVomyOyvs0UfiqC3+lDdd+ERoOth5vVUp5zcMPV2H16sOMGdOWRx+tlisb8PMGTRS+rnRLuGec3VEo5Zdmz95BTEwcvXrdCcCTT9bi4YerEhGRt7ro1UThi5ITYc6jdkehlN86eDCWPn3m8+OPOwgOdtC2bUXKly+EiOS5JAGaKHyPccLHbm94RpS2Lxal/ExiYjJjx67i7beXceFCIhERQbz/fkvKlClod2i20kThS45vhK/cnmAq1w7unWRfPEr5kZUrY3j++Tls2nQMgEcfrcbo0W0oUaKAzZHZTxOFL3HvE6LSw9Dxe/tiUcrPvPnmUjZtOka5cpGMG9eOdu0q2R1SrqGJwlcc/h0OLrGGG70Nd71jazhK+TpjDOfOXaZAAavOYdy4+/jyy40MGtSUsLB8NkeXu2gLcL7CvZ/qGt3ti0MpP7Bjx0nuvfcrHn743xhjAKhcuTBDh96jSSIDekfhCy6ehO1fW8ON3oECZWwNRylfFR+fxAcf/I9hw37j8uVkoqND2b//LOXK+WfTG9lFE0Vul74C+5Y77YtFKR+2aNEeevWax+7dpwF45pnaDB/eiujoMJsjy/28WvQkIm1FZIeI7BaR1zKYX1pElorIehHZJCLtvBmPz/n55SuTRMlmUKaVffEo5YOMMTzzzI+0bj2d3btPU61aEZYvf5rPP39Ak4SHvHZHISIOYDzQCogB1ojIbGPMVrfFBgPfGmMmikg1YB5Q1lsx+ZSlfWH92LTx5qPhjpe1/wilskhEKFs2ktDQQN56qxn9+jXy6wb8vMGbRU/1gd3GmL0AIjIDeABwTxQGSHlIuSBwxIvx5H7HN8CGCbBrJsSfSZv+/BEIL25fXEr5mA0b/uLo0XPcd5/1iOvAgY3p1q2m1kXcIG8WPZUADrmNx7imuXsH+JuIxGDdTbyU0YZE5DkRWSsia0+cOOGNWHOHVR/A5ilXJokXjmmSUMpD584l0K/fQurW/ZSnnvqB06cvARAcHKhJ4iZ4844iozISk268KzDNGPORiDQCvhKRGsYY5xUrGfMp8ClAvXr10m/DfyQnWJ+1e0ORWnDbIxASaW9MSvkAYww//LCdPn0WEBMTR0CA8Pjjt5Mvn74BkB28mShigFJu4yW5umipB9AWwBizQkRCgMLAcS/GlfuVvgcqPWh3FEr5hAMHztK793zmzNkJQL16tzJ5cgfuuEPvxLOLN9PtGqCSiJQTkSCgC5C+E+eDwD0AIlIVCAH8uGxJKZWdjDF06vQtc+bspECBYMaNu4+VK3tokshmXrujMMYkiUhvYCHgAKYaY7aIyBBgrTFmNtAfmCIir2AVSz1tUl6TVEqpa3A6DQEBgogwcmRrJk1ay+jRbShePMLu0PySV1+4M8bMw6qkdp/2ltvwVqCxN2NQSvmPU6cu8tpriwGYMqUjAM2bl6V587I2RuX/tKZHKZXrGWP4v//bQJUq4/nss/V8+eUmYmLi7A4rz9AmPHILZzIc+d3uKJTKdbZtO8ELL8zll18OANYdxMSJ7SlZUvuJyCmaKOxmDGyZBgufSZsWoG+NKmWM4a23lvLhh7+RmOikcOEwPvqoNd261US0hYIcpYnCLs4kWPVP2PY1nNmRNj2sGJRqbltYSuUWIsLhw+dITHTy7LN3MGzYvURFhdodVp6kicIuX90BJzdfOe3Rn6F0C3viUSoXOHLkHCdPXqRmzWIADB/eih496tC4sfYNbydNFDkt/ix8VQfi9lvj4SWh+UdQ7j4I0kf7VN6UnOxk4sS1DBr0MyVKRLBhQ0+CghwULhxG4cKaJOymiSKnLX05LUkULA89dmuLsCpP++OPozz//BzWrrUabmjatAxxcQkULqxNgOcWHiUK15vVpY0xu70cj/+76GqdpHgD6LpCk4TKs+LiEnjzzZ8ZN24NTqehZMkCjB3blgcfrKKV1bnMdROFiLQHRgFBQDkRqQ28bYx5yNvB+bVGb2uSUHmWMYamTb9g48ZjOBxCv34Neeed5kREBNsdmsqAJy/cDQEaAGcBjDEbgIreDEop5d9EhFdeaUj9+iVYu/Y5PvqojSaJXMyToqdEY8zZdLeC2h7TjTi7B/YvsDsKpXLc5cvJjBq1AodDGDDAarXnySdr8be/1cTh0AYicjtPEsU2EXkMCBCRcsDLwErvhuWn/nDr2jS0sH1xKJWD/ve/A/TsOZetW08QHOzgySdrUaxYOCKCw6HFr77Ak1TeG6gLOIH/APFYyUJ5KikBFvdK6wO7wgNQrJ69MSnlZSdPXuSZZ36kadNpbN16gkqVopgz53GKFQu3OzSVRZ7cUbQxxgwEBqZMEJGHsZKGuh5nEnxRGeIOpE2r9bxWZCu/ZYxh2rQNDBiwiFOnLhEU5OD115vw2mtNCAnRJ/J9kSe/tcFcnRQGZTBNZeTYuiuTRNff4dZG9sWjVA6YPn0zp05domXLckyY0I7KlbWo1ZddM1GISBusbkpLiMgot1kFsIqhlCecSWnDfS5CPm2rRvmfixcTiY2Np3jxCESECRPasWbNEZ544nZ9J8IPZHZHcRz4E6tOYovb9HPAa94Myi/depcmCeWX5s/fxYsvzqN8+UIsWtQNEaFy5cJ6F+FHrpkojDHrgfUi8i9jTHwOxuQ/Lp2CGU3sjkIprzh8OI6+fRcyc+ZWACIigjl16pI2veGHPKmjKCEiQ4FqQEjKRGPMbV6LytcZA3M6w87v0qaV0ISh/ENyspPx49cwePDPnDt3mfz58zFkSAv69GlAYKC+E+GPPEkU04D3gZHAfUB3tI4ic8teuTJJ3P0h1H/VvniUyiZOp6FZs2n89tshAB58sApjxrSldOmCNkemvMmTRBFmjFkoIiONMXuAwSLyP28H5tNObLQ+HcHQ54L2WKf8RkCA0Lp1BQ4ejGXcuHZ07FjZ7pBUDvAkUSSI9djCHhHpCRwGino3LD/x8HxNEsqnGWP49tstBAYG0KlTNQAGDmxMv36NCA8Psjk6lVM8SRSvAOFAH2AoUBB4JtM18jJj4OSfdkeh1E3bs+c0vXrN46ef9lCkSBgtW5ajUKFQgoMDCdb2+/KU6yYKY8wq1+A5oBuAiJT0ZlA+bWF3uHTSGhat2FO+JyEhiREjfmfo0P8RH59EoUIhDB3akoIFQ66/svJLmSYKEbkTKAH8aow5KSLVsZryaAlosnB36Bf4tvmV026pb0soSt2oZcv288ILc9m+3brY6datJiNHtqZo0fw2R6bsdM1LXhH5APgX8ASwQEQGAUuBjYA+GuvuzO50SULg+cP6gp3yKcnJTnr1spJE5crR/Pzzk3z55UOaJFSmdxQPALWMMZdEJAo44hrfkTOh+RD3tpwenA0V7rcvFqWywOk0xMcnERaWD4cjgIkT27N8+QFefbUxwcHagJ+yZPaXEG+MuQRgjDktIts1SVzDoZ+tz9ItNUkon7F58zF69pxLlSrRfP75AwA0a1aWZs3K2huYynUySxTlRSSlhVgByrqNY4x52KuR+YqLx2HVP63hfBH2xqKUBy5cuMyQIb8watRKkpKc7Nt3hjNnLlGokBaVqoxllig6pRsf581AfNaK99KGW4y2Lw6lPPDf/+6gd+/5HDwYiwj06lWPoUPvITJSn2hS15ZZo4BLcjIQn+RMgg2u/FmiCRQsZ288Sl1DUpKTzp1n8p//bAOgdu1bmDy5A/Xrl7A5MuULtLbqZhiTNtxxln1xKHUdgYEBFCwYTHh4EO+914LevetrA37KY179SxGRtiKyQ0R2i0iGfViIyGMislVEtojI196Mx2sCAiFM295XucuqVTGsWhWTOj5iRCu2bXuRvn0bapJQWeLxHYWIBBtjErKwvAMYD7QCYoA1IjLbGLPVbZlKwOtAY2PMGRHxjTakjm+EVUPh9Da7I1HqKmfPxvP664uZPHkdVaoUZsOGngQFOYiO1n4i1I257mWFiNQXkc3ALtd4LRH5xINt1wd2G2P2GmMuAzOw3s1w9yww3hhzBsAYczxL0dtl3SirGfGUNp0qdLQ3HqWwGvD7+uvNVKkyjkmT1uFwBNCxY2WSk7VXAHVzPLmjGAt0AH4AMMZsFJEWHqxXAjjkNh4DNEi3zG0AIvIb4ADeMcYs8GDb9kq+bH3ePQxqvQDBBeyNR+V5u3adoleveSxevBeAxo1LMWlSB2rU8I2bdJW7eZIoAowxB9J1kJ7swXoZ9ahu0o0HApWA5lhtR/1PRGoYY85esSGR54DnAEqXLu3Brr0oOREOLLKGI0prklC2S0xMpmXLL4mJiSMqKpThw++le/c6BARk9C+oVNZ5kigOiUh9wLjqHV4CdnqwXgxQym28JFYzIOmXWWmMSQT2icgOrMSxxn0hY8ynwKcA9erVS59scs7FkzCxSNq4I59toShljEFEyJfPwdChLVm6dD/Dh99LkSLaNpPKXp48+vAC0A8oDRwDGrqmXc8aoJKIlBORIKALMDvdMj8ALQBEpDBWUdRez0K3we4f0oYDQ6FMa/tiUXnWsWPn6dZtFu+/vzx12pNP1uKLLx7QJKG8wpM7iiRjTJesbtgYkyQivYGFWPUPU40xW0RkCLDWGDPbNa+1iGzFKs4aYIw5ldV95RhnovUZWQG6b7cei1UqhzidhilT1vHaa0s4ezaeyMgQ+vZtSESE9iKkvMuTM90aV5HQv4H/GGPOebpxY8w8YF66aW9DRCFlAAAgAElEQVS5DRusu5V+nm4zVyjTSpOEylEbN/5Fz55zWbnSei+ibduKjB/fTpOEyhGe9HBXQUTuwio6eldENgAzjDEzvB5dbnL5HCzpZXcUKo9JTEzm9deX8PHHK0lONhQvHs6YMW155JFqpHvARCmv8ej1TGPM78aYPsAdQBxWh0Z5y+Hf0oYLaJtOKmcEBgawfv1fOJ2Gl16qz7ZtL/Loo9U1Sagcdd07ChEJx3pRrgtQFfgRuMvLceU+xvXSUvitcOcAe2NRfu3gwViSk52UK1cIEWHSpPbExiZQr96tdoem8ihPCtr/BP4LDDfG/M/L8eR+RWqBXs0pL0hMTGbMmFW8/fYyGjUqyaJF3RARKlWKtjs0lcd5kijKG2O0DYCr3hVUKvusWHGInj3nsmnTMQCiokK5eDGR/PmDbI5MqUwShYh8ZIzpD3wvIledJfNUD3frx8HPL9kdhfJDZ85c4rXXFvPpp38AUK5cJOPHt+O++yrZHJlSaTK7o/i36zNv92yXeOnKJFGunX2xKL+SkJBE7dqTOXgwlnz5Ahgw4C4GDWpKWJi+8a9yl8x6uFvtGqxqjLkiWbhepMsbPeAZt2atnj0IBUpde1mlsiA4OJAePeqwZMk+Jk5sT7VqRa6/klI28OTx2GcymNYjuwPJ9fLl1yShbkp8fBJvv72Ur7/enDrtjTfuZtmypzRJqFwtszqKzliPxJYTkf+4zYoAzma8lh/SenyVDRYt2kOvXvPYvfs0RYvm56GHqhAamk97mlM+IbM6itXAKaxWX8e7TT8HrPdmULnGjm9hTmdr2OhTTyrr/vrrPP36LeSbb6xOrqpXL8KkSR0IDdV6COU7Mquj2AfsAxbnXDi5zJoRacNVH7cvDuVzkpOdTJ68jjfeWEJsbAKhoYG8/XYzXnmlEUFBDrvDUypLMit6+sUY00xEznDlSwSC1Z5flNejs1tKRfZDc6G8Pu2kPJecbPjkk9XExibQrl0lxo27j3LlCtkdllI3JLOip5TuTgvnRCC5Wv5b7I5A+YBz5xJITjZERoYQFORgypT7OXbsPA8/XFXbZlI+7Zo1aW5vY5cCHMaYZKAR8DygvaMo5WKM4T//2UbVquPp339h6vQmTUrTqZO28qp8nyePXPyA1Q1qBeBLrIYBv/ZqVLmBMXBqq91RqFxu//6zdOw4g06dvuXw4XP8+ecJ4uOT7A5LqWzlSaJwuvq0fhj42BjzElDCu2HlAv97DZITXCN6RaiulJiYzIcf/kq1auOZM2cnBQoEM27cffz++zOEhGinVsq/eNQVqog8CnQDHnRN8+9n+07vhDXDreGAQChcw954VK5y8WIiDRt+xubNxwHo0qUGo0a1pnjxCJsjU8o7PEkUzwC9sJoZ3ysi5YBvvBuWzTZPSRvuvBwc/p0XVdaEheWjXr1buXgxkQkT2tO6dQW7Q1LKqzzpCvVPEekDVBSRKsBuY8xQ74dmI2ei9VmpExRvaG8synbGGL78ciMVKkTRpElpAEaPbkNQkENfnFN5gic93N0NfAUcxiqsv0VEuhljfst8TT9Qool2UpTHbdt2ghdemMsvvxygatXCbNjQk6AgBwULhtgdmlI5xpOip9FAO2PMVgARqYqVOOp5MzCl7HTpUiJDh/6P4cN/IzHRSZEiYbz+ehPy5dO2mVTe40miCEpJEgDGmG0i4t/dbiUn2h2BstGCBbt58cV57N17BoBnn72DYcPuJSoq1ObIlLKHJ4niDxGZjHUXAfAE/too4Pmj8Fk5t8diVV5z/vxlunWbxcmTF6lRoyiTJrWncePSdoellK08SRQ9gT7Aq1h1FMuBT7wZlC3WjoJf+l85rcy99sSiclRyshOn05Avn4Pw8CDGjGlLTEwcr7zSkHz5tAE/pTJNFCJyO1ABmGWMGZ4zIdkgIfbKJFHvH9BsxLWXV35j3bojPP/8HB54oDJvvtkMgMcfv93mqJTKXa5ZMycib2A13/EEsEhEMurpzj+410k8uUmTRB4QF5fAyy/Pp379z1i37ihffbWJxMTk66+oVB6U2R3FE0BNY8wFESkCzAOm5kxYOWzNh9ZnSDQU0atJf2aMYebMrbz88gKOHj2PwyH069eQd99tocVMSl1DZokiwRhzAcAYc0JE/Pe5wA3jrM/w4vbGobzq3LkEOneeyfz5uwFo0KAEkyZ1oHZtbUZeqcxklijKu/WVLUAF976zjTEPezWynBJ3AJLireFHl9gbi/Kq8PAgEhKSKVgwmGHD7uW55+oSEKAvVCp1PZklik7pxsd5MxDbzPtb2nBQAfviUF6xfPkBihcPp1KlaESEqVM7EhISSLFi4XaHppTPyKzPbP+/vD6yEg7/ag3f+SoEarMM/uLkyYu8+uoivvhiA/fcU45Fi7ohIpQpE2l3aEr5nLzdcP7qD9KG6/azLw6VbZxOw7RpGxgwYBGnT18iKMjB3XeXJjnZEBioxUxK3QivVlCLSFsR2SEiu0XktUyWe0REjIjkbPtRyZetz7uGQP5iObprlf22bDlO8+bT6NFjNqdPX+Kee8qxefMLvP12cwID/fdZDKW8zeM7ChEJNsZ43LaFiDiA8UArIAZYIyKz3duNci0XgfXm9ypPt53tbtH2DX1dbGw8DRt+zvnzlylaND+jRrXm8cdv1/6qlcoG173MEpH6IrIZ2OUaryUinjThUR+r74q9xpjLwAzggQyWew8YDsR7HnY22PEd7F+Qo7tU2c8YA0DBgiEMHNiYnj3rsn37izzxRE1NEkplE0/ux8cCHYBTAMaYjUALD9YrARxyG48hXV/bIlIHKGWMmZPZhkTkORFZKyJrT5w44cGur+PEZpjzWNp4kVo3v02Vow4fjuORR75l+vRNqdMGDbqbiRM7UKiQtvKqVHbyJFEEGGMOpJvmSVsHGV3OmdSZ1gt8o4H+GSx35UrGfGqMqWeMqVekSBEPdn0de+emDT+2FMJvvfltqhyRlORkzJiVVKkynu+/38bbby8jOdkJoHcQSnmJJ3UUh0SkPmBc9Q4vATs9WC8GKOU2XhI44jYeAdQAlrn+wW8BZotIR2PMWk+Cv3GufFW9O5Rq7t1dqWyzZs1hevacyx9/HAXgwQerMHZsWxwOrahWyps8SRQvYBU/lQaOAYtd065nDVBJRMphdaPaBXg8ZaYxJhYonDIuIsuAf3g/SbjRJ518woULlxk4cDETJqzBGChduiCffHIfHTtWtjs0pfKE6yYKY8xxrJN8lhhjkkSkN7AQcABTjTFbRGQIsNYYMzvL0WYHY+CvNbbsWt2YwMAAFi/eS0CA0K9fI95+uxn58/t3J4tK5SbXTRQiMgW3uoUUxpjnrreuMWYeVquz7tPeusayza+3vZtmDPz8EuyeZY0H5O33DXOzPXtOExkZQnR0GMHBgXz11UOEhARy++16F6hUTvOkcHcxsMT18xtQFPDNvkJPbYEN49PGq3azLxaVoYSEJN5/fzk1akxk4MDFqdPvvLOEJgmlbOJJ0dO/3cdF5Ctgkdci8qYNE9OG/74XCpazLxZ1lWXL9vPCC3PZvv0kYD3hlJzs1MpqpWx2I2Uv5YAy2R2I1106BRsnWMPl7tMkkYscP36BAQMW8eWXGwGoXDmaiRPb06KF/o6Uyg08qaM4Q1odRQBwGrhmu025VtKltOGW/tliui86efIiVauO5/TpSwQHOxg06G5efbUxwcFaf6RUbpHpf6NYLzjUwnq8FcBpUtpM8FXhJSCyvN1RKJfChcN44IHKxMTEMWFCeypWjLI7JKVUOpkmCmOMEZFZxpi6ORWQ8m8XLlxmyJBfaN/+Npo2tUowJ0xoT3CwQ9+sViqX8qSWcLWI3OH1SLxp/yL4tNT1l1Ne9d//7qBatQkMH/47vXrNxem0bk5DQgI1SSiVi13zjkJEAo0xSUAT4FkR2QNcwGrDyRhjfCN5HFgC37dOG7/lTvtiyaMOHYrl5ZcXMGvWdgDq1LmFyZM7aH/VSvmIzIqeVgN3AA/mUCzesebDtOH7v4NK6bsCV96SlORk7NhVvPXWUi5cSCQ8PIj332/Biy/W146ElPIhmSUKATDG7MmhWLzDmWR93jsRbnvE3ljymLi4BD744FcuXEikU6eqfPxxW0qWLGB3WEqpLMosURQRkWt2JG2MGeWFeLyn0G12R5AnnD0bT2hoIMHBgURFhTJ5cgeCgx20b6/HXylfldn9vwMIx2oOPKMfpVIZY/j6681UrjyO4cN/S53+8MNVNUko5eMyu6M4aowZkmOReMPO7+HQUruj8Hs7d56iV6+5LFmyD4Dlyw9ijNEnmZTyE9eto/Bpe916WI2qal8cfio+PokPP/yVf/7zVy5fTiYqKpQRI1rx9NO1NUko5UcySxT35FgU3nbvJAgvbncUfuWvv87TtOkX7Np1GoCnn67NiBGtKFw4zObIlFLZ7ZqJwhhzOicD8SqHdnKT3YoVy0+pUgUJDAxg4sT2NGtW1u6QlFJe4r8trx1aBlum2R2F33A6DVOmrKNFi3Lcdls0IsLXXz9MoUKhBAU57A5PKeVF/vvW06FlacPF6tkWhj/YuPEvGjeeSs+ec+nVay4p7UIWKxauSUKpPMB/7yhSNBwMRW63OwqfdP78Zd55Zxkff7yS5GTDrbdG0LOnJl2l8hr/TxSiV7w34ocftvPSS/OJiYkjIEB46aX6vP9+SwoUCLY7NKVUDvP/RKGy7PDhOLp0mUlCQjJ16xZn0qQO1Kt3q91hKaVs4r+J4qxvN1GV0xITkwkMDEBEKFGiAEOHtiQoyEGvXndqn9VK5XH+eQbY9jVsm24Ni39+xez0+++HqFv3U6ZP35Q6rX//u3jppQaaJJRSfpgozu6BeU+kjWuLsdd0+vQlnn/+vzRuPJXNm48zYcJafL2nW6VU9vO/oqe4A2nDnX+B6Gr2xZJLGWOYPn0T/fv/xIkTF8mXL4BXX23MoEF3a9MbSqmr+F+iSFGqBZRsancUuc6xY+fp2vV7li7dD0CzZmWYOLE9VasWsTcwpVSu5b+JQmUoMjKEo0fPU7hwGCNHtuLJJ2vpXYRSKlOaKPKARYv2cMcdxYmODiM4OJDvvnuU4sXDiY7WBvyUUtfnf5XZKtXRo+fo2vV7WreezsCBi1On16hRVJOEUspj/nVHYQws7mV3FLZLTnYyefI6Xn99CXFxCYSGBlK5crR2JqSUuiH+lSji9sOZHdZwgTK2hmKXP/44Ss+ec1iz5ggA7dtXYty4dpQtG2lzZEopX+VficKZnDbcarJ9cdhk//6z1K8/heRkQ4kSEYwdex8PPVRF7yKUUjfFq4lCRNoCYwAH8JkxZli6+f2AvwNJwAngGWPMgas25Alj4Ps21nDB8nmys6KyZSPp3r02ERHBvPtucyIitAE/pdTN81pltog4gPHAfUA1oKuIpH/7bT1QzxhTE5gJDL/hHZpkiN1rDZfvcMOb8SX795/l/vu/4Zdf9qdO+/TT+xk1qo0mCaVUtvHmHUV9YLcxZi+AiMwAHgC2pixgjFnqtvxK4G83vVdxQMsxN72Z3CwxMZlRo1bw7ru/cOlSEidPXmTFih4AWsyklMp23kwUJYBDbuMxQINMlu8BzM9ohog8BzwHULp06eyKzyf9+utBevacw5YtJwDo0qUGo0a1tjkqpZQ/82aiyOjSNsMW50Tkb0A9oFlG840xnwKfAtSrVy9Ptlp35swlBgxYxOefrwegQoVCTJjQntatK9gcmVLK33kzUcQApdzGSwJH0i8kIvcCg4BmxpgEL8bj05xOw48/7iBfvgBee60Jr7/ehNDQfHaHpZTKA7yZKNYAlUSkHHAY6AI87r6AiNQBJgNtjTHHvRiLT9q+/STlykUSHBxIdHQY//rXw5QuXZAqVQrbHZpSKg/x2lNPxpgkoDewENgGfGuM2SIiQ0Sko2uxEUA48J2IbBCR2d6Kx5dcvJjIoEFLqFlzIsOH/5Y6vXXrCpoklFI5zqvvURhj5gHz0k17y234Xm/u3xctWLCbXr3msm/fWQBOnrxoc0RKqbzOv97M9mFHjpyjb98FfPed9fTw7bcXZdKkDtx1V6nrrKmUUt6liSIX2LnzFPXqfcq5c5cJC8vHO+80o2/fhuTL57A7NKWU0kSRG1SqFMWdd5Ygf/58fPLJfZQpow34KaVyD00UNoiLS+Ctt5bSq9ed3HZbNCLC7NldyJ8/77VPpZTK/fwnUcx/0jWQe9/HM8Ywc+ZWXn55AUePnmf79pMsWGC1WqJJQimVW/lHoji2HrZ/Yw0Xb2RvLNewd+8Zeveex/z5uwFo2LAkH36oD30ppXI//0gUywekDT80x744MnD5cjIjR/7Oe+8tJz4+icjIEIYNu4dnn61LQIA24KeUyv38I1Ekut41uOtdCMldFcGHDsUyZMgvJCQk88QTt/PRR60pVizc7rCUUspjvp8o4g7B0RXWcOncUZRz5swlIiNDEBEqVIhizJi2VKwYxT33lLc7NKWUyjKvNeGRY1a+lzYcFGFfHFgN902dup6KFT9h+vRNqdOff76eJgmllM/y/URx+Zz1WaY1FK5hWxhbthynefNp9Ogxm9OnL6VWWiullK/z7aKnM7thxwxruPrTYEPvbhcvJvLee78wcuQKkpKcFC2an9Gj29C1q31JSymlspPvJoozu2DqbWnj+W/J8RB27jxFmzbT2b//LCLQs2dd/vnPeyhUKDTHY1FKKW/x3URxYHHacIM3oFTzHA+hTJmChIQEUqtWMSZN6kDDhiVzPAaVeyUmJhITE0N8fLzdoag8JCQkhJIlS5IvX/Z1bOa7iSLFbY9Ak6E5squkJCeTJq2la9caREeHERwcyIIFT1CiRAECA32/ukdlr5iYGCIiIihbtixiQ7GoynuMMZw6dYqYmBjKlSuXbdv1/bNbaM505LN69WHq15/CSy/NZ+DAtLuZMmUiNUmoDMXHxxMdHa1JQuUYESE6Ojrb72J9/47Cy2Jj4xk06GcmTFiDMVC6dEEeeKCy3WEpH6FJQuU0b/zNaaK4BmMM//73Fl55ZSF//XWewMAA+vVryFtvNdMG/JRSeYpvlpnEn4Ulvby6i40bj9G16/f89dd57rqrFH/88RwffthKk4TyKQ6Hg9q1a1OjRg3uv/9+zp49mzpvy5YttGzZkttuu41KlSrx3nvvYUxa68vz58+nXr16VK1alSpVqvCPf/zDjq+QqfXr1/P3v//d7jAy9cEHH1CxYkUqV67MwoULM1xmyZIl3HHHHdSuXZsmTZqwe7f1HtbBgwdp0aIFderUoWbNmsybZ/UsvXnzZp5++umc+grWlbMv/dStW9eYvfOMGYn1s26MyS5JSclXjL/yygIzZco6k5zszLZ9qLxj69atdodg8ufPnzr85JNPmvfff98YY8zFixdN+fLlzcKFC40xxly4cMG0bdvWjBs3zhhjzObNm0358uXNtm3bjDHGJCYmmvHjx2drbImJiTe9jUceecRs2LAhR/eZFVu2bDE1a9Y08fHxZu/evaZ8+fImKSnpquUqVaqU+vcyfvx489RTTxljjHn22WfNhAkTUrdVpkyZ1HXuuecec+DAgQz3m9HfHrDW3OB51zeLnlKueqKqwB19smWTS5fuo1eveUye3IGmTcsAMGpUm2zZtlJ85KW6iv6e97/SqFEjNm2ympb5+uuvady4Ma1btwYgLCyMcePG0bx5c1588UWGDx/OoEGDqFKlCgCBgYH06nX1Xfz58+d56aWXWLt2LSLC22+/TadOnQgPD+f8+fMAzJw5kzlz5jBt2jSefvppoqKiWL9+PbVr12bWrFls2LCByEirMc+KFSvy22+/ERAQQM+ePTl48CAAH3/8MY0bN75i3+fOnWPTpk3UqlULgNWrV9O3b18uXbpEaGgoX3zxBZUrV2batGnMnTuX+Ph4Lly4wM8//8yIESP49ttvSUhI4KGHHuLdd98F4MEHH+TQoUPEx8fz8ssv89xzz3l8fDPy448/0qVLF4KDgylXrhwVK1Zk9erVNGp0ZXcIIkJcXBwAsbGx3HrrrZlOB7j//vuZMWMGr7766k3F6AnfTBQpCt7841/Hj19gwIBFfPnlRgBGjVqRmiiU8hfJycksWbKEHj16AFaxU926da9YpkKFCpw/f564uDj+/PNP+vfvf93tvvfeexQsWJDNmzcDcObMmeuus3PnThYvXozD4cDpdDJr1iy6d+/OqlWrKFu2LMWKFePxxx/nlVdeoUmTJhw8eJA2bdqwbdu2K7azdu1aatRIawGhSpUqLF++nMDAQBYvXswbb7zB999/D8CKFSvYtGkTUVFR/PTTT+zatYvVq1djjKFjx44sX76cpk2bMnXqVKKiorh06RJ33nknnTp1Ijo6+or9vvLKKyxduvSq79WlSxdee+21K6YdPnyYhg0bpo6XLFmSw4cPX7XuZ599Rrt27QgNDaVAgQKsXLkSgHfeeYfWrVvzySefcOHCBRYvTnvisl69egwbNkwThTc5nYbPP/+DgQMXc+ZMPMHBDgYPbsqAAXfZHZryR1m48s9Oly5donbt2uzfv5+6devSqlUrwCpyvtbTMVl5ambx4sXMmDEjdbxQoULXXefRRx/F4XAA0LlzZ4YMGUL37t2ZMWMGnTt3Tt3u1q1bU9eJi4vj3LlzRESkNfx59OhRihQpkjoeGxvLU089xa5duxAREhMTU+e1atWKqKgoAH766Sd++ukn6tSpA1h3Rbt27aJp06aMHTuWWbNmAXDo0CF27dp1VaIYPXq0ZwcHrqjzSZHR8R09ejTz5s2jQYMGjBgxgn79+vHZZ5/xzTff8PTTT9O/f39WrFhBt27d+PPPPwkICKBo0aIcOXLE41huhm8min3zb271fWf4299m8fvvhwBo3boC48e3o2LFqOyITqlcIzQ0lA0bNhAbG0uHDh0YP348ffr0oXr16ixfvvyKZffu3Ut4eDgRERFUr16ddevWpRbrXMu1Eo77tPTP9OfPnz91uFGjRuzevZsTJ07www8/MHjwYACcTicrVqwgNPTazeGEhoZese0333yTFi1aMGvWLPbv30/z5s0z3Kcxhtdff53nn3/+iu0tW7aMxYsXs2LFCsLCwmjevHmG7yNk5Y6iZMmSHDp0KHU8JibmiuIjgBMnTrBx40YaNGgAWMmzbdu2AHz++ecsWLAg9VjFx8dz8uRJihYtSnx8fKbHJzv53lNPzsuwYZw1HBh2Q5soUCCYnTtPccst4cyY0YkFC57QJKH8WsGCBRk7diwjR44kMTGRJ554gl9//TW1KOPSpUv06dMntRhjwIAB/POf/2Tnzp2AdeIeNWrUVdtt3bo148aNSx1PKXoqVqwY27ZtSy1auhYR4aGHHqJfv35UrVo19eo9/XY3bNhw1bpVq1ZNfToIrDuKEiVKADBt2rRr7rNNmzZMnTo1tQ7l8OHDHD9+nNjYWAoVKkRYWBjbt29PLf5Jb/To0WzYsOGqn/RJAqBjx47MmDGDhIQE9u3bx65du6hfv/4VyxQqVIjY2NjUY71o0SKqVq0KQOnSpVmyZAkA27ZtIz4+PvUuaufOnVcUvXmTDyaK5LThxkM8Xm3hwt0kJCQBEB0dxuzZXdi+/UU6d66hL0WpPKFOnTrUqlWLGTNmEBoayo8//sj7779P5cqVuf3227nzzjvp3bs3ADVr1uTjjz+ma9euVK1alRo1anD06NGrtjl48GDOnDlDjRo1qFWrVuqV9rBhw+jQoQMtW7akePHimcbVuXNnpk+fnlrsBDB27FjWrl1LzZo1qVatGpMmTbpqvSpVqhAbG8u5c1ZXA6+++iqvv/46jRs3Jjk5+arlU7Ru3ZrHH3+cRo0acfvtt/PII49w7tw52rZtS1JSEjVr1uTNN9+8om7hRlWvXp3HHnuMatWq0bZtW8aPH59a7NauXTuOHDlCYGAgU6ZMoVOnTtSqVYuvvvqKESNGAPDRRx8xZcoUatWqRdeuXZk2bVrq+Wrp0qW0b9/+pmP0hGRUhpab1atd3aztthWiq8HTW667/KFDsfTps4AfftjOe++1YPDgpjkQpVLWFWDKlaHyjtGjRxMREZHr36XIbgkJCTRr1oxff/2VwMCraxAy+tsTkXXGmHo3sj/fu6OI3evRYklJTkaNWkHVquP54YfthIcHERWlzX8r5U9eeOEFgoOD7Q4jxx08eJBhw4ZlmCS8wfcqs5NclUuZ9D+xcmUMPXvOYePGYwB06lSVMWPaUqJEgZyIUCmVQ0JCQujWrZvdYeS4SpUqUalSpRzbn+8lCoC206BcxmVzq1bFcNddn2MMlC0bybhx99G+/W0ZLquUt2X2GKpS3uCN6gTfTBTlO0BodIaz6tcvQZs2FalT5xYGD25KWFj2dd6hVFaEhIRw6tQpbWpc5Rjj6o8iJCQkW7frm4nCza5dp3jllYWMGtWG226z/iHnzn2cgAD9x1T2KlmyJDExMZw4ccLuUFQektLDXXby2USRkJDEsGG/8sEHv5KQkExISCAzZz4GoElC5Qr58uXL1l7GlLKLV596EpG2IrJDRHaLyFVvo4hIsIj82zV/lYiU9WS7S5YeombNSbzzzi8kJCTTvXttJk3qkN3hK6WUwot3FCLiAMYDrYAYYI2IzDbGbHVbrAdwxhhTUUS6AB8Cna/eWpp9pyO5t/2PAFStWphJkzpoI35KKeVF3ryjqA/sNsbsNcZcBmYAD6Rb5gHg/1zDM4F75Dq1fmcuhhIS4uCf/2zJhg09NUkopZSXee3NbBF5BGhrjPm7a7wb0MAY09ttmT9dy8S4xve4ljmZblvPASkNw9cA/vRK0L6nMHDyukvlDXos0uixSKPHIk1lY0zE9Re7mjcrszO6M0iflTxZBmPMp8CnACKy9kZfQ/c3eizS6LFIo8cijR6LNCKy9kbX9WbRU+noAVAAAAd+SURBVAxQym28JJC+8fTUZUQkECgInPZiTEoppbLIm4liDVBJRMqJSBDQBZidbpnZwFOu4UeAn42vtVKolFJ+zmtFT8aYJBHpDSwEHMBUY8wWERmC1cn3bOBz4CsR2Y11J9HFg01/6q2YfZAeizR6LNLosUijxyLNDR8Ln2tmXCmlVM7yvWbGlVJK5ShNFEoppTKVaxOFt5r/8EUeHIt+IrJVRDaJyBIR8du3EK93LNyWe0REjIj47aORnhwLEXnM9bexRUS+zukYc4oH/yOlRWSpiKx3/Z+0syNObxORqSJy3PWOWkbzRUTGuo7TJhG5w6MNG2Ny3Q9W5fceoDwQBGwEqqVbphcwyTXcBfi33XHbeCxaAGGu4Rfy8rFwLRcBLAdWAvXsjtvGv4tKwHqgkGu8qN1x23gsPgVecA1XA/bbHbeXjkVT4A7gz2vMbwfMx3qHrSGwypPt5tY7Cq80/+GjrnssjDFLjTEXXaMr+f/27j1EyiqM4/j311XLMkKKbrRF98ysLKz+6KJFF7IScQutNqxIutDF/giDjPojsohKTStCgwpTrMQMi7AL4lYSpSV2QcWEqAiTKIvSX3+cs+20zc68u7mzs7PPBwZ2zsz7nmcOs+8z73lnnpN+s9KIirwvAB4EHgF+r2VwNVZkLG4EZtreAmD7hxrHWCtFxsJA2xKXg/nvb7oagu33qfxbtMuBF5y0AvtJOqjafus1URwCfFtyf3NuK/sc238BW4Hyqxn1bUXGotQk0ieGRlR1LCSdAhxme0ktA+sFRd4XxwDHSFohqVXSRTWLrraKjMU0YKKkzcBS4LbahFZ3uno8Aep3PYqdVv6jARR+nZImAiOAc3o0ot5TcSwk7QI8DrTUKqBeVOR9sRtp+ulc0lnmB5KG2v65h2OrtSJjcTUw1/Zjks4k/X5rqO0dPR9eXenWcbNezyii/Ee7ImOBpNHAVGCM7T9qFFutVRuLfUhFI9+VtJE0B7u4QS9oF/0fed32n7Y3AF+SEkejKTIWk4BXAGyvBAaQCgb2N4WOJx3Va6KI8h/tqo5Fnm6ZQ0oSjToPDVXGwvZW20NsN9luIl2vGWO728XQ6liR/5HXSF90QNIQ0lTU+ppGWRtFxmITMApA0vGkRNEf16hdDFybv/00Ethq+7tqG9Xl1JN7rvxHn1NwLKYDg4AF+Xr+Jttjei3oHlJwLPqFgmOxDLhQ0lpgO3CP7Z96L+qeUXAs7gaelXQnaaqlpRE/WEp6mTTVOCRfj7kf2B3A9mzS9ZlLgG+A34DrC+23AccqhBDCTlSvU08hhBDqRCSKEEIIFUWiCCGEUFEkihBCCBVFogghhFBRJIpQdyRtl/Rpya2pwnObOquU2cU+383VRz/LJS+O7cY+bpZ0bf67RdLBJY89J+mEnRznx5KGF9jmDkl7/d++Q/8ViSLUo222h5fcNtao3wm2TyYVm5ze1Y1tz7b9Qr7bAhxc8tgNttfulCjb45xFsTjvACJRhG6LRBH6hHzm8IGkT/LtrDLPOVHSR/ksZLWko3P7xJL2OZJ2rdLd+8BRedtReQ2DNbnW/565/WG1rwHyaG6bJmmKpHGkmlsv5j4H5jOBEZImS3qkJOYWSU91M86VlBR0k/S0pFVKa088kNtuJyWs5ZKW57YLJa3M47hA0qAq/YR+LhJFqEcDS6adXs1tPwAX2D4VaAaeLLPdzcATtoeTDtSbc7mGZuDs3L4dmFCl/8uANZIGAHOBZtsnkSoZTJa0P3AlcKLtYcBDpRvbXgisIn3yH257W8nDC4GxJfebgfndjPMiUpmONlNtjwCGAedIGmb7SVItn/Nsn5dLedwHjM5juQq4q0o/oZ+ryxIeod/blg+WpXYHZuQ5+e2kukUdrQSmSjoUWGT7a0mjgNOAj3N5k4GkpFPOi5K2ARtJZaiPBTbY/io/Pg+4BZhBWuviOUlvAIVLmtv+UdL6XGfn69zHirzfrsS5N6lcRekKZeMl3UT6vz6ItEDP6g7bjsztK3I/e5DGLYRORaIIfcWdwPfAyaQz4f8sSmT7JUkfApcCyyTdQCqrPM/2vQX6mFBaQFBS2fVNcm2hM0hF5q4CbgXO78JrmQ+MB9YBr9q20lG7cJykVdweBmYCYyUdAUwBTre9RdJcUuG7jgS8bfvqLsQb+rmYegp9xWDgu7x+wDWkT9P/IulIYH2ebllMmoJ5Bxgn6YD8nP1VfE3xdUCTpKPy/WuA9/Kc/mDbS0kXist98+gXUtnzchYBV5DWSJif27oUp+0/SVNII/O01b7Ar8BWSQcCF3cSSytwdttrkrSXpHJnZyH8IxJF6CtmAddJaiVNO/1a5jnNwOeSPgWOIy35uJZ0QH1L0mrgbdK0TFW2fydV11wgaQ2wA5hNOuguyft7j3S209FcYHbbxewO+90CrAUOt/1RbutynPnax2PAFNufkdbH/gJ4njSd1eYZ4E1Jy23/SPpG1su5n1bSWIXQqageG0IIoaI4owghhFBRJIoQQggVRaIIIYRQUSSKEEIIFUWiCCGEUFEkihBCCBVFogghhFDR30CkDmpVOc0WAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"eval_pool = Pool(X_validation, y_validation, cat_features=cat_features)\n",
"curve = get_roc_curve(model, eval_pool)\n",
"(fpr, tpr, thresholds) = curve\n",
"\n",
"plt.figure()\n",
"lw = 2\n",
"roc_auc = sklearn.metrics.auc(fpr, tpr)\n",
"plt.plot(fpr, tpr, color='darkorange',\n",
" lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)\n",
"\n",
"plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
"plt.xlim([0.0, 1.0])\n",
"plt.ylim([0.0, 1.05])\n",
"plt.xlabel('False Positive Rate')\n",
"plt.ylabel('True Positive Rate')\n",
"plt.title('Receiver operating characteristic')\n",
"plt.legend(loc=\"lower right\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"from catboost.utils import get_fpr_curve\n",
"from catboost.utils import get_fnr_curve"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [],
"source": [
"? get_fpr_curve"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8VfX5wPHPk5tJCCuAIkNQ2bKDA6viqHWiuHFU66xKrbM/cBVntWrr1rr3QIuK1lkUtQrWoIgCRVEZERAIOyTkJnl+f3zvTW5CxiXcc8+9N8+7r/O6Z+WcJ6d4n3zH+X5FVTHGGGMakuZ3AMYYYxKbJQpjjDGNskRhjDGmUZYojDHGNMoShTHGmEZZojDGGNMoSxTGGGMaZYnCJBwRWSQipSKyKWLZSUR6iohG7FskIhMifk5FpCR07GcR+ZuIBBq5z5kiUlnnPveFjj0Zut4eEefvJiIasT1dRMpCP7daRKaISBevnosxfrFEYRLVUaraOmJZFnGsnaq2BsYB14nIoRHHhoSO7Q+cBJzVxH1m1LnP+Ihja4Cbmvj58aH77Qa0Bu6I5pfbHiKS7vU9jIlkicIkLVWdAcwFdq/n2ELgU2DodtziKWCwiOwfRSzrgNcau5+I5IjInSKyWETWi8h/QvtGi0hRnXMXicjBofVJIvKKiDwrIhuAq0Ilrg4R5w8LlWoyQttnich8EVkrIu+KyM6h/SIifxeRlaEY5ojIVs/PmEiWKExSCn3h7QMMBL6q53g/YF9g4XbcZjNwC3BzFPHkA8c2cb87gBHAKKAD8CegKspYjgZeAdoBtwMzgOMijp8CvKKqQRE5BrgqFE8n4BPghdB5hwD7AX1C1zoJKI4yBtNCWaIwieo1EVkXWl6rc2w1rlroUWCCqk6LOPaliJQA84HpwANN3GeviPusE5G96hz/B9BDRA5r4OfvEZH1oZg6An+o7yQRScNVg/1RVX9W1UpV/UxVtzQRX9gMVX1NVatUtRR4Hlf1hogIcHJoH8D5wF9Udb6qVuCS3dBQqSII5AH9AAmdszzKGEwLZYnCJKpjVLVdaDmmzrGOqtpeVfur6j11jg3HtRWcBOwJ5AKIyL4RDdZzI86fGXGfdqo6M/JioS/yG0OL1BPnxaraFhgMtAe6NfD7dASygR+a/M3rt7TO9ivA3iKyE66EoLiSA8DOwN3h5IdLqgJ0VdUPgPuA+4FfRORhEWnTzJhMC2GJwqQcdSbjqmeuC+37JKLBeuA2XvIJoC0wtpF7foNr+L4/9Bd+XauBMmDXeo6VAK3CG6GeWp3q3qLO/dYB7wEn4qqdXtCaoaCXAufXSYA5qvpZ6GfvUdURuGq7PsCVDf1exoAlCpPabgXOE5Edt+cioeqbScD/NXHqU0BnYEw916gCHgf+FurqGxCRvUUkC/gOyBaRI0KN0dcAWVGE9jzwW1xbxfMR+x8CJorIQAARaSsiJ4TWR4rInqH7lOCSV2UU9zItmCUKk7JCf+V/RGz+Yn4BaLQuX1XLgXuAaxs45QrgG+ALXHXQbUCaqq4HLsS1ufyM+wIvauAakaYCvYFfVPXriDheDV37xVAvqW+BcBtLG+ARYC2wGNeQ7XmXXpPcxCYuMsYY0xgrURhjjGmUJQpjjDGNskRhjDGmUZYojDHGNCrpBhfr2LGj9uzZ0+8wjDEmqcyaNWu1qtZ9PycqSZcoevbsSWFhod9hGGNMUhGRxc39Wat6MsYY0yhLFMYYYxplicIYY0yjLFEYY4xplCUKY4wxjbJEYYwxplGeJQoReTw0L++3DRwXEblHRBaG5u0d7lUsxhhjms/L9yiexM2k9XQDxw/DDZHcGzcT2YOhz0Zt2ADvvw/p6RAIuCW8Xt++aI/XO9WMMcYkqdJgKd+v+Z5N5ZvYVL5pu67lWaJQ1Y9FpGcjpxwNPB2alWumiLQTkS5Nzd/7/fdwyCExDDREZPsSTeRnbi706AHdu7vPHj2gdeumY8jKgg4dID/frRtjTHOoKiMeHsH81fNjcj0/38zuSu15gItC+7ZKFCJyHnAeQHb2IPbZByoroaLCfUau17cvmuOqEAy6JRG0bu0SRnjp1Kkm8ey8s/ts29YluJ12cgnKGGMA5q6ay/zV88lJz2HIjkPIzchlGtOafT0/E0V9lT31zqKkqg8DDwMUFBTov/8d20BUoaoqdslnwwZYuhSWLHHL0qVQWtp0HKWlUFzslk2b3LI4ipfuR42CTz/d/udgjEkNHy36CIBj+x/Ls8c+C4Cc0fz6dT8TRRHQPWK7G7DMj0BEaqqRMjP9iKA2VZdswklj9WpYudIlnMWLXfJZvBg2boRly+Czz6CoCLp18ztyY0wiWLJ+CQADOw2MyfX8TBRTgfEi8iKuEXt9U+0TLYWIq1Zq2xZ22aXxc8eOhddec9VSV14JJ50EI0bEJ05jTGJaU7oGgPxW+TG5npfdY18AZgB9RaRIRM4Wkd+LyO9Dp7wF/AgsxE32fqFXsaSyBx6Avn3d+u23w8iRLlk8+yxMnQrLLfUa0+IUlxYD0CGnQ0yu52Wvp3FNHFfgIq/u31J06QJz58KHH8JDD8GUKTB5slsA0tLggANg331dW8b++ydG9ZoxxjvhEkXCJwoTP4EAHHywW374wSWLGTNcG8ZHH8G0aW4B6NoVrr4azj7bEoYxqWpt2VoA2me3j8n1LFGkmF13dW0VYcXFrrTx6afw9tuwYAFceCH8+c9w9NGwww5wySXQsaN/MRtjYmtzcDMArTOjeIErCjbWU4rLz4fjj4e//x3mzYObbnIvBK5aBY8+CjffDJ07w5YtfkdqjImVkvISAFpltIrJ9SxRtCBpaa7aafVqeP55uO46t18Vdt8dnngicV44NMY0X7hEYYnCNFt2NowbB9dfDy+/DL17w8KFcNZZsNtucO+9rsrKGJOcwokiNzM3JtcT1/koeRQUFGhhYaHfYaSUYBBefBH+8heYHxoaJhBwpYx+/eC3v609VlVeHgwdagMpGpOIgpVBMm/KJCABgtcGkdB/qCIyS1ULmnNNa8w2ZGTA6afDqafC66+7dzOmT4evv3bLSy9t/TP9+8Pdd8Ovfx33cI0xjYisdpIY/TVnicJUS0tzb3qPHQvr1sHs2fDcc66nVKSFC13J49BD4aijXNXV5ZfDjjv6E7cxpkZJ0DVkx6raCSxRmAa0awejR7ulrmDQ9Z666SZXAgF45RUoLHS9rIwx/ol1QzZYY7ZphowM1xA+Zw488ogb5nzRIjdPyIMPunVjjD+qG7IzYleisERhmm3gQDjnHDeuFMCXX7qX+Xr1gjFj4Isv/I3PmJYo1u9QgCUKEwMHHODe/L7kEjjhBNf99o03YI89XE+qJOtYZ0xSs6onk7BGjXJvf0+e7Bq/Tz7Z7b/qKjcg4Y8/+hufMS2FF43ZlihMzPXoAS+84LrVpqW50sbuu8PTT/sdmTGpz0oUJqmceKIbAv03v3HTvJ59Nrz3nt9RGZParDHbJJ1+/eDNN13jdkWFSxp77QXr1/sdmTGpyRqzTVJKT4dXX4XLLnPbn3/uEkZVlb9xGZOKrOrJJK20NLjzTje/d06OSxYdO8L991vCMCaWqhuzrerJJKujj3ZzYACsXQvjx8OkSb6GZExKsRKFSQmXXALff+/e7ga48UY47LCakWuNMc1nbRQmJYi4eS8uucQN+wHwzjswaJDrJWUv6BnTfJsrYjsXBViiMD5q0wbefde9Z5GeDpWV7n2LXr3cTHzTpvkdoTHJx6qeTEoaNcrNqHfGGS5hLF4Mt9ziqqN++cXv6IxJLuGqJ2vMNimnTRt48knYssUNAwJuOPPwgIPGmOhYicKkvLQ0N7Dgq6+67T//2U2gZIyJjiUK02IccwyccgqUlLg3uSdOdKUNY0zjSitKAcjJyInZNS1RmIT12GNuFNotW+DWW91wILfeCsuX+x2ZMYmrrKIMgOz07Jhd0xKFSVjZ2W7O7kcfha5d3cx5EydCz56uO60xZmvhRJGTbiUK00KkpblRZxcvhn/9Cw4/HMrLXY+oQYPgq6/8jtCYxFIadFVPWelZMbumJQqTFAIBlySmToUDD3T7vv0Whg+HQw+FVav8jc+YRGHDjJsWLxCAf//bvXdxxBFu37vvQu/ebjhzY1oyVbXGbGPADQHSoYObl/vaa10V1Pr1rlvtG2+4qVh//tnvKI2Jv8iG7DSJ3de7p4lCRA4VkQUislBEJtRzvIeIfCgiX4nIHBE53Mt4TGoRgRtugK+/hmOPhbIyN0FSv37QrRtMmeJ3hMbEV7jaKZYN2eBhohCRAHA/cBgwABgnIgPqnHYNMFlVhwEnAw94FY9JXSLw/PNw0UXQp49rAAc49VSbetW0LF68bAfelij2ABaq6o+qWg68CBxd5xwF2oTW2wLLPIzHpLCsLLjvPlftVF4ORx3lShgnnghr1vgdnTHxkYyJoiuwNGK7KLQv0iTgNBEpAt4C/lDfhUTkPBEpFJHCVda9xTQhEIAXX3TvW6xf797wtjm6TUuQjIlC6tlXd6aBccCTqtoNOBx4RmTrFhhVfVhVC1S1oFOnTh6EalJNq1bwz3+6aql3362Zr9uYVJaMiaII6B6x3Y2tq5bOBiYDqOoMIBvo6GFMpgUZPhz+8he3/sEH/sZiTDyEu8YmU6L4AugtIr1EJBPXWD21zjlLgIMARKQ/LlFY3ZKJmSuugLw8N/zHEUe4OS8uuww2bfI7MmNir7rXUwzfoQBIj+nVIqhqhYiMB94FAsDjqjpXRG4AClV1KnA58IiIXIqrljpT1SbCNLETCMDYsfD00/DWWzX7y8vh3ntd1ZQxqcKrqifPEgWAqr6Fa6SO3HddxPo8YB8vYzDmvvvcMB/l5bB2LVx6Kdx/vxuddto0N8OeMakgKROFMYkgLw/GjavZ7tzZvWNRVuZKG9984/YZk+yqE0V68rRRGJOQTjkFNm50b2+vXAk77FC7WsqYZJWMvZ6MSVitW8Mrr7g2DICbbvI3HmNiITzEuCUKY2Jkzz3dKLQAM2bUzNNtTLLyqteTJQrTorVtC/fc49avu67xc41JdFb1ZIxHzj3Xfc6bB5WV/sZizPawRGGMR7KzoVMnqKqCW25xDdzGJKPNFZYojPHM8OHu87rroGtX927FkiX+xmTMtrIShTEeuu8+uOACGDwYKipc4/bYsVYVZZKL9XoyxkO77QYPPACzZ8PU0IhkX37phio//ni46y5fwzMmKlaiMCYORNykR/ffDz16QFGRG6780kvhvPPcMCDGJKqkmwrVmGR24YXw00/wySdQUOD2PfII7LGH6x1lTCKyEoUxcZaWBr/6FXzxBfz3v6531Ndfu5KFMYnIEoUxPho5Ej791K1/+qlryzAm0ViiMMZnw4fDH//o1g89FJ59FoJBf2MyJlIyznBnTMq57jr3jsUvv8Dpp8OgQfDhh7B8ud+RmZZOVW2sJ2MSQYcO8PHHcOONkJkJCxbAgQfCTjvVdKs1xg9lFWUAZAWySJPYfrVbojBmGwUCcM01NfNwh40dC3Pm+BaWaeG8ap8ASxTGNFuXLvDmm1BaCjvv7MaK2ndfWL3a78hMS2SJwpgElp3tZsjLy4MNG+Cyy9w0q8bEkyUKYxLcgAE181k88wz07evGizImXrzq8QSWKIyJmfPPdwML9u3rRp496CCYNs3vqExLYSUKY5JAXp4bWPDbb+Gss1zbxcEHw4oVfkdmWgKvusaCJQpjYi493Y0LtfPObvvII13yMMZLVqIwJsmkpcGtt7r1WbPci3mPPmqN3MY7liiMSUInn+ze2g4791x44QX/4jGpzRKFMUlq9Gh4//2a7bPOgrlzfQvHpLDq2e3SfUoUIpIjIn1jfndjWoCDD4YpU2q2zz3Xplg1sedriUJEjgJmA++EtoeKiI1qY8w2GDsW3n7brc+YASNGwPr1/sZkUovfvZ4mAXsA6wBUdTbQM+aRGJPiDj3UTakKbgKkwYNdF1pjYsHvNooKVbW/fYyJgb/9zU2lmpHhXsqzxm0TK34nim9F5BQgICK9ReRe4LOYR2JMC9G/Pzz4oFt/7jl/YzGpw+9E8QdgILAFeB5YD/wxmouLyKEiskBEForIhAbOOVFE5onIXBF5PtrAjUlmxx0HWVnwwQeulLF5s98RmWTn91hPR6jq1ao6MrRcA4xp6odEJADcDxwGDADGiciAOuf0BiYC+6jqQOCSbf4NjElC7drB//2fW7/8cigosPYKs338LlFMjHJfXXsAC1X1R1UtB14Ejq5zzrnA/aq6FkBVV0ZxXWNSwqRJcNddIALz58O118KWLX5HZZKVl4kivaEDInIYcDjQVUTuiTjUBqiI4tpdgaUR20XAnnXO6RO616dAAJikqu9EcW1jkp4I/PGPsMsuMGYM3Hkn/OtfUFgIubl+R2eSTXX32PT4do9dBhQCZcCsiGUq8Jsori317NM62+lAb2A0MA54VETabXUhkfNEpFBECletWhXFrY1JHkceCWec4db/9z84+mj46Sd/YzLJx5eqJ1X9WlWfAnZT1acilinhqqImFAHdI7a74ZJP3XNeV9Wgqv4ELMAljrqxPKyqBapa0KlTpyhubUzyEIEnn3SDB+bmujkshg+Hl17yOzKTTPxuo+gpIq+Eeib9GF6i+LkvgN4i0ktEMoGTcaWRSK8BBwCISEdcVVQ01zYm5QwfDp9/7rrPrlvnBhX8z3/8jsokC797PT0BPIhrlzgAeBp4pqkfUtUKYDzwLjAfmKyqc0XkBhEJ95p6FygWkXnAh8CVqlq87b+GMalh4EA3xEefPm7797+Hr77yNyaTHLwsUYhq3WaDOieIzFLVESLyjaoOCu37RFX3jXk0USgoKNDCwkI/bm1M3BQXw157wcKFbvvYY+Gxx1y3WmPqk3tLLpuDm9k0cRO5mVv3hgh9lxc059rRlCjKRCQN+F5ExovIWKBzc25mjIlOfr5rqzjwQLc9ZYorZWza5G9cJjGpqu+DAl4CtAIuBkYApwNnxDwSY0wtPXq4ZDE11LK3ahVcfDFURNM53bQoZRVu6sSsQBZpEvtphpq8oqp+oaqbVLVIVX+nqscCy2MeiTGmXkcd5XpAicATT0BODnTq5KqnjAFv2yegiUQhInuLyPEi0jm0PTg0HpP1xTAmjk480ZUuevVyJYrVq12yuP9+vyMzicDLHk/QSKIQkduBx4HjgH+JyJ+B94HPqeddB2OMtw44AH74AT76CNq2BVUYP95NglRQUHvZf383hHkTfVVMivC6RNHgEB7AEcAwVS0Tkfa4l+UGq+r3nkRijGmSCOy3H6xZA3vvDf/9L3z5Zf3nfvwxvPYaPP20G6nWpC4/E0WpqpYBqOpaEVlgScKYxJCWBjNnwjffQHl57WNVVfDGG/CXv8DkyS653HuvO9a+PaQ39l+9SUpe9niCxhPFrnXmxu4Zua2qTQ41bozxjoibTrU+e+wBxxwDo0a5hvDwcCBdusAzz8BBB8UvTuM9P0sUdYcEv9OTCIwxnhgxws2g96c/wcaNrgF8+XI4+GDX3vH665CX53eUJhZ8SxSq+pEndzTGxM3xx7sF3MRIhx8O06fDhx+6brdnngmBABxyCOywg5+Rmu1RGvS215PVVhrTQuTkuASxYIGrkvroI7eEjR8Pp58OI0e6ai2TPHx9j8IYk3r69nU9pcaPh9NOg512cvvvuw/23BN2373hnlQmMVUninR/XrgLhN6nMMakkJ13dj2hnnkGiopc76hDD3XH5s1zM+6ttImJk4afvZ5Q1UoRGSEiok0NM2uMSUoicMIJblm0yL2nsXSpa7M47zxoFfojtXVruPRS18XWqqYSi5+9nsK+Al4XkZeBkvBOVZ3iSUTGGN/07AnvvOO61n7/PTz8cO3jN93kutjecQeccoovIZp6JEKi6AAUAwdG7FPAEoUxKWjAAPjuOze73hdfuH3LlsGjj8KGDa6L7amnurfCR4xwVVY2Q7G/vB7rqclEoaq/8+TOxpiE9qtfuSXs9tvd2FF33glXXgl33+32t2kDZ50F55zjZugz8ed7rycR6SYir4rIShH5RUT+KSLdPInGGJPQROCKK+Ddd11iGD3alTLuuguGDIFrrvE7wpYpEaqengCeB04IbZ8W2vdrTyIyxiS8Qw5xC0BhoRvu/Mkn4eaboV8/2LfORMki0K2bG6PKxJ7vJQqgk6o+oaoVoeVJwGokjTGAG9b88cdr3uw+/XTXKB657LwzHHecDXvulerusenedI+NJlGsFpHTQu9UBETkNFzjtjHGAK7EMH26G6SwR4+tl/R0N+R5z56u2srEViKUKM4CTgRW4KZAPT60zxhjqvXrB19/DYsXb73ceKMbU2rJEtdLqlUr6NwZJk6E9ev9jjz5ef3CXZNvZgPHqeoYVe2kqp1V9RhVXexJNMaYlDRhApSVwQ03QGamG6Bw1Sq49Vb3XsZ++7k3wk3zFJe6Sp4OOR08uX6jiUJVK9l6uHFjjNlm6elw7bWul9SaNXDddW5sqdJS+OQTN2PfqFG1lwkT/I46Oawvc8Wy9tntPbm+NDUyh4jcDLQFXqL2m9m+DBtWUFCghYWFftzaGBNjqrBwIVx2Gbz5Zv3nHHCAGzrksMNsdr76qCqBGwIoSvDaIOlp9T8kEZmlqgXNuUc0j31U6POGyNio/aa2McZsMxHo3RumTnXtG5s31xx7/XU3VMiHH7qlTRs46SR3rFMnl1zS093+ljz21IYtG1CU3IzcBpPE9mr0qiKSBjyoqpM9ubsxxuC+6IcOrb1v1Cg3sdJ998EDD7gqq0ceqTl+yy3uMz8fVqxouaWNdWXrAGif4021EzTdRlEFjPfs7sYY04j+/d3LfN98A//4h1tOOQU6dKiZxrW4GO65x984/VSdKDxqn4Douse+LyJXiEh3EekQXjyLyBhj6th9dzfk+XnnuXnAi4tdCeO229zx997zNz4/rS1bC0C77Hae3SPa9yguAj4GZoUWa002xvhu7Fj32ZJf4otH1VM0o8f28uzuxhizHXbZBbKyYMsW18Zx+umud9Quu0B2tt/RxUc4UbTNauvZPRosUYjInyLWT6hz7BbPIjLGmCgFAu4lPnC9pq64wg11npMDF1wAn34Kq1fXLGvX+huvFzZs2QBAm6w2nt2jsaqnkyPWJ9Y5dqgHsRhjzDb705/ghx9cI3f//jX7H3rIzafRqVPN0qGDSyIffOBfvLEWLlH41UYhDazXt13/BUQOFZEFIrJQRBp8x1JEjhcRFZFmvQxijGnZdtnFNXLPm+dm53v0UfjNb1zX2fASVlYGBx3khhJ5/nn/Yo4VX6uecC/V1bde3/ZWQuNE3Q8cBgwAxonIgHrOywMuBj5vMlpjjGlC795w9tlu7u/IaidVePllV10FEAy6KV3bt4f993dDiSQjv9+jGCIiG0RkIzA4tB7eHhTFtfcAFqrqj6paDrxI/eNG3Qj8FSjb1uCNMWZbHH88VFTAL7+4HlPp6bBuHXz8sauW+vlnvyPcduu3uHGefClRqGpAVduoap6qpofWw9sZUVy7K7A0YrsotK+aiAwDuqtqA6O8VJ93nogUikjhqlWrori1McY0rHNnmDLFlTQuu8ztKyuD7t3dm+DffgtVVf7GGK1wY3bbbH+qnrZXfe0Y1VVWoeFB/g5c3tSFVPVhVS1Q1YJOnWxyPWNMbLRtC3fe6Zbu3V311B/+AIMGuVn5IseeSlThkWP96vW0vYqA7hHb3YBlEdt5wO7AdBFZBOwFTLUGbWNMvF12mStFHHaYSx4ARUWQm+vmBE9kXs9FAd4mii+A3iLSS0Qycd1tp4YPqup6Ve2oqj1VtScwExijqgn+f4sxJhW1aQNvveXaLK6/vmb/yJGJ3di9pnQNAB1bdfTsHp4lClWtwA0o+C4wH5isqnNF5AYRGePVfY0xZntddx1Mm1az/fHHbvrW//7Xv5jqU6VVSV/1hKq+pap9VHVXVb05tO86VZ1az7mjrTRhjEkUBx7o2izOPrtm36RJvoVTr41bNqIoeZl5ns1FAR4nCmOMSXaPPuoauwHefhu6doXbb3dJxG/xGDkWLFEYY0yTzjrLdakFWLbMDRty5JGuwdvPhLG21CUKL1+2A0sUxhjTpHbtXIKYMaNm31tvuS61Q4f6N9hguETh5aRFYInCGGOiEgjAXnu5CZMGD3ZVUABz5sA55/gTk5UojDEmAeXluSHNi4rg2mvdvilTXOnikUdg6dLGfz6Wwl1jrURhjDEJatIkGBAa6rSoyE3VOnx4/O4fThRevmwHliiMMabZ0tJg7lxXojj2WLdv9WqYODE+jdzht7K9fNkOLFEYY8x2GzsW/vlPN2Q5wK23uiTy9NPe3nfVZjdIan5OfhNnbh9LFMYYEyNz59bePuMMuOQSWLLEmxLG6s2rAeiU6+1gqZYojDEmRrp0cfNdvP9+zb6773Yj0WZkwAMPxHb48lUlrkTRqZUlCmOMSRqBABx8MHz1FZxwQs3+ykq46CK44ILY3ctKFMYYk8SGDoXJk12V0/vvQ7dubv/DD8PlTc7CE51wG4U1ZhtjTJI7+GA330XY3/7mZtTbHuWV5WzYsoGABGysJ2OMSQVt29aeMS8nB554ovnXC1c75bfKJ028/Sq3RGGMMXGSk1O72umss1yyaE6PqOr2CY8bssEShTHGxNUdd7ghQMLOOgvOP3/brxPu8eR1+wRYojDGmLgbPNiNRivith95BDp2rN2O0ZQVm1YAsEPrHTyIsDZLFMYY44MuXSAYhGHD3HZxMQwaFH01VNGGIgC65XXzKMIaliiMMcYngQDMmgXPPFOzb/Lk6H72540/A9CtjSUKY4xJaSJw2mkwZIjbHj8+up+rLlFYojDGmJbh+uvd5+rVMHt20+eHE0XXNl09jMqxRGGMMQngqKNq1v/1r6bPt6onY4xpYdLSakoV11wD33zT8LnByiDLNy5HELq07uJ9bJ7fwRhjTFSOP75m/eKLGz5vxaYVKMoOrXcgI5DheVyWKIwxJkEMGAAPPujWp09veEjyeFY7gSUKY4xJKOecU7Me2W02Ujx7PIElCmOMSSjp6e6ujTJaAAAUXUlEQVTFO4Azz3TzWNS1ZP0SALrmed/jCSxRGGNMwpkypWZ91qytj3+z0rV09+vYLy7xWKIwxpgEs9tu0LmzW7/99q2Pz1rmsseILiPiEo8lCmOMSUCnnOI+33qr9v5gZZB5q+YhCEN2HBKXWCxRGGNMArr0Uve5eXPtUWV/XPsjlVpJ97bdaZXRKi6xWKIwxpgE1L17zfo999Ssh9snBnUeFLdYPE0UInKoiCwQkYUiMqGe45eJyDwRmSMi00RkZy/jMcaYZCECkya59X//u2b/nF/mADB4h8Fxi8WzRCEiAeB+4DBgADBORAbUOe0roEBVBwOvAH/1Kh5jjEk2v/qV+/zpJzd3BcDXv7jp8VIiUQB7AAtV9UdVLQdeBI6OPEFVP1TV8HTjM4H4vD1ijDFJYP/9a9YzM2HZMmVm0Uwgfj2ewNtE0RVYGrFdFNrXkLOBt+s7ICLniUihiBSuWrUqhiEaY0ziSk+HK6+s2R530Y+sLFlJp1ad2K3DbnGLw8tEIfXsq3eSPxE5DSgA6ukxDKr6sKoWqGpBp06dYhiiMcYktr/+FS680K1/vPgjAPbpsQ8i9X3FesPLRFEERLTb0w1YVvckETkYuBoYo6pbPIzHGGOS0i23hFZ6fQDA/j0OjOv9vUwUXwC9RaSXiGQCJwNTI08QkWHAP3BJYqWHsRhjTNJq2xbmzNHqRJFRFN9Eke7VhVW1QkTGA+8CAeBxVZ0rIjcAhao6FVfV1Bp4OVSMWqKqY7b1XsFgkKKiIsrKymL4GySG7OxsunXrRkaG92POG2MSV0aXBZC3HDZ15rYrBnDRCfG7t2eJAkBV3wLeqrPvuoj1g2Nxn6KiIvLy8ujZs2dc6+28pqoUFxdTVFREr169/A7HGOOjD35ypQl+OhA0vt9zKfFmdllZGfn5+SmVJABEhPz8/JQsKRljts37P77vVn46kKVLXXfZ996Lz71TIlEAKZckwlL19zLGRG9zcDPv/+ASxR75vwHcC3hPPRWf+6dMojDGmFT16ZJPKQmWMLzLcGa+24PbbnP7FyyIz/0tUcRIIBBg6NCh1cuiRYuYPn06bdu2ZdiwYfTv35/rr78eoNb+fv36ccUVV/gcvTEmkb353ZsAjN55NCIwIvRSdn2TGnnB08bsliQnJ4fZs2fX2rdo0SL23Xdf3nzzTUpKShg6dChHHnkkQPX+0tJShg0bxtixY9lnn338CN0Yk8Aqqip4/tvnARjbfywA/fvXHN+wAdq08TaGlCtRiHizbK/c3FxGjBjBDz/8UGt/Tk4OQ4cO5eeff97+mxhjUs5/lvyH1ZtX07tDb/bp7v6Y3GmnmuPjx3sfQ8olCr+UlpZWVzuNHTt2q+PFxcXMnDmTgQMH1tq/du1avv/+e/bbb794hWqMSSJT5rsJtMf2G1urc8u4ce5z2jTvY0i5qietdzQp79VX9QTwySefMGzYMNLS0pgwYQIDBw5k+vTpfPLJJwwePJgFCxYwYcIEdtxxRx+iNsYksoqqCl6a+xIAJwys/YbdJZfACy/AsmVuFrxWHk52l3KJItGE2yIa2v/dd9/xq1/9irFjxzJ06FAfIjTGJKo3v3uTlSUr6ZPfZ6thxUeOrFn//nsY4uH02Vb15LM+ffowceJEbgv3dzPGGFxp4qppVwFwQcEFW71TJQLhGusXXvA2FksUCeD3v/89H3/8MT/99JPfoRhjEsSTs59k/ur57NJ+Fy4ouKDecyoq3Gd49juviPpVqd9MBQUFWlhYWGvf/Pnz6R/ZXyzFpPrvZ4yprbyynD739mHx+sU8f+zzjBs0rt7zHn0Uzj3XrTf1VS4is1S1oDnxWInCGGMSzGNfPsbi9Yvp37E/Jw48scHz+vWrWfdySDhLFMYYk0Bmr5jN5e9dDsD1o68nkBZo8NxRo2rWv//eu5gsURhjTIIoDZZy+qunU1pRynH9j+P4Acc3en5aGhSEKpM+/NC7uCxRGGNMAlhftp4jnj+Cb1d+S5/8Pjx1zFNRjR7doYP7nDDBu9gsURhjTAL4w9t/4MNFH9KxVUemnDiF3MzcqH7unHPcZ2lpTS+oWLNEYYwxPlJVLn3nUp6Z8wyZgUw++d0nDOw8sOkfDDkh4oXtF1/0IEAsUcRMQ8OMiwhvvPFG9XlHHnkk06dPB2D06NH07duXIUOGMHLkyHqHADHGpK6FaxZyzEvHcNfnd5GRlsHjYx6nX8d+Tf9gHeHe8w89FOMAQyxRxEh4rKfw0rNnTwC6devGzTff3ODPPffcc3z99ddceOGFXHnllXGK1hjjp+LNxZz9+tn0ubcPUxdMJS8zjzfGvcGpg09t1vXOP999LlkSwyAjpNxYT3K9N1OH6p+b92LikCFDCAaDvP/++/z6179u8Ly9996b22+/vbnhGWOSQGmwlLtm3sUdM+5gTeka0tPSOXXQqdx4wI10b9u92dc95hg3SKBXsxWkXKLwS3iYcYBevXrx6quvVh+75ppruOaaaxpNFO+88w7HHHOM53EaY+JLVfli2Rc8O+dZnp3zLGvL1gIwuudoHjriIfp27Lvd98jPd59VVVBZCYGGX71olpRLFM39y397NTTMOLiRYsENOV7XqaeeSklJCZWVlXz55ZeexmiMiZ9N5Zt4Z+E73P353fxnyX+q94/oMoJr97uWMX3HRNX9NRqtW9esL1gAAwbE5LLVUi5RJKqrr76am2++mfT02o/8ueeeY8iQIUyYMIGLLrqIKVOm+BShMWZ7banYwmdLP+PleS8zee5kikuLAcjPyee3Q37LaYNPY3iX4Z7cOz8fiovhnXcsUSStQw45hGuvvZZly5ZtdSwjI4ObbrqJXXfd1QYANCZJLN+4nP+t/h/fFX/Hl8u/pHB5IfNXzae0orT6nBFdRnDiwBM5f8T5tM1u62k8gwe7t7O9GMrDEkUcXX311Rx99NH1HsvJyeHyyy/njjvu4LHHHotzZMaY+myp2MLSDUuZv2o+3xV/x7KNy/hh7Q/MWj6Log1F9f7MwE4DOarPUZy8+8kM3mFwzKqXmnLccS5RvPNO7K9tiSJGNm3atNW+0aNHM3r06OrtMWPGEDmse/h9irDLL7/cq/CMMQ1YWbKSxesWM3/1fH7e8DM/b/yZBcUL+K74O5auX4pSf7tnm6w2DOo8iN75vemX349R3Uexe+fdaZ/TPs6/gdM91Glq8+bYX9sShTEmpVRWVbJs4zJWlqxkTeka1patZW3pWtaWrWVVySqWbFjCik0rWL5xOSs2raAkWNLgtdIkjR5tetCrfS8GdR5Etzbd2ClvJ4Z3GU6/jv1Ik8R5FW3QIPe5cmXsez5ZojDGJDRVpayijNWbV/NLyS+sLFnJqpJVrNq8yq2HPleWrGTZxmWs2LSCKq2K+vp5mXns1mE3erbrSe8Ovemc25m+HfvSJ78PPdv1JDOQ6eFvFzu9etWsv/02HHlk7K6dMolCVeNWFxhPyTYDoUldVVpFWUUZFVUVVFRVUFlVWb1eUVXBlsotbA5upqS8xH0GS2pth5eSYAmbyjdRXFpMSXkJZRVl1cuWyi21tssqyiivLN/mWHfI3YEueV3okNOB9tntqz/b57Rn57Y70yWvC11ad2HH1jvSJqtNynx39OsH//sf/P3vlii2kp2dTXFxMfn5+Snzfzi4JFFcXEx2drbfoZgkUllVSWlFaa0v583BzZQGSymtKHVfyBXuC3lT+SY2lm9k45aN1Z/hc8oqytiwZQPFpcUUby5mTekaKrXSl98pM5BJx1Yd6ZzbuXrp1KpTrc/OuZ3ZKW8ndmi9Q9KUAmLtt7+Fq66CDz6ADRugTZvYXDcl5swOBoMUFRVR5uVcgD7Jzs6mW7duZGRk+B2KiZEqraI0WFr913VJeUnUn5srNjd4PJwMtlRu8Sz27PRs0tPSay0BCZCelk5mIJPczFxyM3JpldGK3Ez32Sq9Va3t3IxccjNz6ZDTgTZZbcgKZJGdnr3VkpXu9mcGMhOqLSCRbdgAbUO9cKdOhaOOqjm2PXNmp0SJIiMjg16RFXTGNFOwMljvl3BpsOav7MgqlHC1ypbKLVRUVRCsDLrPqiDBqiAbt2xkw5YN1cvasrWsK1vn+e/RKqNVvUt2enb1F3NWehatM1qTl5VHXmZe9WduZm71l3XrzNbk5+ST3yqfDjkdWuxf6smiTRv3PsWcOfDtt7UTxfbwNFGIyKHA3UAAeFRVb61zPAt4GhgBFAMnqeoiL2MyiaFKq6q/WMN10+HqkGBVsFbdd2RdeLAqWKvqJFynXV5ZXn29YFWQYGWw+lhpRWl1tUv1l3x5CeWV5dXnhuvNm1Mf3hzZ6dm1/rqu97OxY3X+cg/va5XRiqxAVkpVwZptM3KkSxRXXQUTJ8bmmp4lChEJAPcDvwaKgC9EZKqqzos47WxgraruJiInA7cBJzV23c3BzRQuc1VPdavN6vZ3TuXjqkqwKkhpsJRgVZAqraKyqpIqrXLrGrEe2l+pEV+4ob98K7Wy1vHKqsrqnw2vV1ZVVn95V39G/uVc2fCx8P3CX+7h/dvSKyWe0iStwS/k8F/ZORk51V/k4S/qrEAWGYEM0tPSyUgLfQYyaJ3ZmrZZbWmT1aZ66ZDTgUBajEdtMybk5JMh/M7uZ59BVtb2X9OzNgoR2RuYpKq/CW1PBFDVv0Sc827onBkikg6sADppI0HJTqKc70nIJs7CX6iR9dHhL9yMtAwCaYGt6sIzAhk1ddiBmp/JDGRWfzlnpGWQEcggM5BJTnoOORk51Z/hKpjWma1rnZeRlkG77HZkp2fbX+Mm6dX/Tzgx2yi6AksjtouAPRs6R1UrRGQ9kA+sjjxJRM4DzgttbmES33oScfLpSJ1nlUyCof+VUtr0yU1L6mcRY/YsatizqNHs8cy9TBT15bS6JYVozkFVHwYeBhCRwuZmxVRjz6KGPYsa9ixq2LOoISKFTZ9VPy/7nBUBkVM2dQPqDp1afU6o6qktsMbDmIwxxmwjLxPFF0BvEeklIpnAycDUOudMBc4IrR8PfNBY+4Qxxpj486zqKdTmMB54F9c99nFVnSsiNwCFqjoVeAx4RkQW4koSJ0dx6Ye9ijkJ2bOoYc+ihj2LGvYsajT7WSTdm9nGGGPiy96LN8YY0yhLFMYYYxqVsIlCRA4VkQUislBEJtRzPEtEXgod/1xEesY/yviI4llcJiLzRGSOiEwTkZ39iDMemnoWEecdLyIqIinbNTKaZyEiJ4b+bcwVkefjHWO8RPHfSA8R+VBEvgr9d3K4H3F6TUQeF5GVIlLvu2bi3BN6TnNEZHhUF1bVhFtwjd8/ALsAmcDXwIA651wIPBRaPxl4ye+4fXwWBwCtQusXtORnETovD/gYmAkU+B23j/8uegNfAe1D2539jtvHZ/EwcEFofQCwyO+4PXoW+wHDgW8bOH448DbuHba9gM+juW6ilij2ABaq6o+qWg68CBxd55yjgadC668AB0lqjr3Q5LNQ1Q9VNTxT7kzcOyupKJp/FwA3An8FUm/c+RrRPItzgftVdS2Aqq6Mc4zxEs2zUCA8O0Nbtn6nKyWo6sc0/i7a0cDT6swE2olIl6aum6iJor7hP7o2dI6qVgDh4T9STTTPItLZuL8YUlGTz0JEhgHdVfXNeAbmg2j+XfQB+ojIpyIyMzSacyqK5llMAk4TkSLgLeAP8Qkt4Wzr9wmQuPNRxGz4jxQQ9e8pIqcBBcD+nkbkn0afhYikAX8HzoxXQD6K5t9FOq76aTSulPmJiOyuqt5PiBFf0TyLccCTqnpnaMDSZ0LPIjGHMfZOs743E7VEYcN/1IjmWSAiBwNXA2NU1bspzvzV1LPIA3YHpovIIlwd7NQUbdCO9r+R11U1qKo/AQtwiSPVRPMszgYmA6jqDCAbN2BgSxPV90ldiZoobPiPGk0+i1B1yz9wSSJV66GhiWehqutVtaOq9lTVnrj2mjGq2uzB0BJYNP+NvIbr6ICIdMRVRf0Y1yjjI5pnsQQ4CEBE+uMSxaq4RpkYpgK/DfV+2gtYr6rLm/qhhKx6Uu+G/0g6UT6L24HWwMuh9vwlqjrGt6A9EuWzaBGifBbvAoeIyDygErhSVYv9i9obUT6Ly4FHRORSXFXLman4h6WIvICrauwYao/5M5ABoKoP4dpnDgcWApuB30V13RR8VsYYY2IoUauejDHGJAhLFMYYYxplicIYY0yjLFEYY4xplCUKY4wxjbJEYVoMEckXkdmhZYWI/BxaXxfqQhrr+40WkW0aSkREptf3gqCInCki98UuOmOiZ4nCtBiqWqyqQ1V1KPAQ8PfQ+lCgyaEcQiMAGNPiWKIwxgmIyCOheRveE5EcqP4L/xYR+Qj4o4h0EpF/isgXoWWf0Hn7R5RWvhKRvNB1W4vIKyLyPxF5LjzCsYgcFDrvm9AcAll1AxKR34nId6F77xOn52DMVixRGOP0xg3JPRBYBxwXcaydqu6vqncCd+NKIiND5zwaOucK4KJQCWVfoDS0fxhwCW4OhF2AfUQkG3gSOElVB+FGSLggMpjQ0M/X4xLEr0M/b4wvLFEY4/ykqrND67OAnhHHXopYPxi4T0Rm48bNaRMqPXwK/E1ELsYllorQ+f9V1aLQKKWzQ9ftG7rfd6FznsJNOBNpT2C6qq4KzbHwEsb4xOpcjXEiR9ytBHIitksi1tOAvVW1lNpuFZF/4cbRmRkazbe+66ZT/1DP9bHxdUxCsBKFMdvmPWB8eENEhoY+d1XVb1T1NqAQ6NfINf4H9BSR3ULbpwMf1Tnnc2B0qKdWBnBCrH4BY7aVJQpjts3FQEFoYvp5wO9D+y8RkW9F5Gtc+0SDswyqahlu1M6XReQbXI+rh+qcsxw3K9sM4N/Al7H+RYyJlo0ea4wxplFWojDGGNMoSxTGGGMaZYnCGGNMoyxRGGOMaZQlCmOMMY2yRGGMMaZRliiMMcY06v8B4mO7HDDPik8AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.figure()\n",
"lw = 2\n",
"(thresholds, fpr) = get_fpr_curve(curve=curve)\n",
"(thresholds, fnr) = get_fnr_curve(curve=curve)\n",
"plt.plot(thresholds, fpr, color='blue', lw=lw, label='FPR')\n",
"plt.plot(thresholds, fnr, color='green', lw=lw, label='FNR')\n",
"\n",
"plt.xlim([0.0, 1.0])\n",
"plt.ylim([0.0, 1.05])\n",
"plt.xlabel('Threshold')\n",
"plt.ylabel('Error Rate')\n",
"plt.title('FPR-FNR curves')\n",
"plt.legend(loc=\"lower left\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.4719014981770065\n",
"0.9876472944285266\n"
]
}
],
"source": [
"from catboost.utils import select_threshold\n",
"\n",
"print(select_threshold(model=model, data=eval_pool, FNR=0.01))\n",
"print(select_threshold(model=model, data=eval_pool, FPR=0.01))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Snapshotting"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [],
"source": [
"#!rm 'catboost_info/snapshot.bkp'\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.3019732\ttest: 0.3022828\tbest: 0.3022828 (0)\ttotal: 67ms\tremaining: 1.27s\n",
"1:\tlearn: 0.2238474\ttest: 0.2233617\tbest: 0.2233617 (1)\ttotal: 148ms\tremaining: 1.33s\n",
"2:\tlearn: 0.1904507\ttest: 0.1818139\tbest: 0.1818139 (2)\ttotal: 252ms\tremaining: 1.43s\n",
"3:\tlearn: 0.1802458\ttest: 0.1659696\tbest: 0.1659696 (3)\ttotal: 363ms\tremaining: 1.45s\n",
"4:\tlearn: 0.1754740\ttest: 0.1587777\tbest: 0.1587777 (4)\ttotal: 485ms\tremaining: 1.46s\n",
"5:\tlearn: 0.1703625\ttest: 0.1514527\tbest: 0.1514527 (5)\ttotal: 566ms\tremaining: 1.32s\n",
"6:\tlearn: 0.1684087\ttest: 0.1485265\tbest: 0.1485265 (6)\ttotal: 654ms\tremaining: 1.22s\n",
"7:\tlearn: 0.1672054\ttest: 0.1477918\tbest: 0.1477918 (7)\ttotal: 741ms\tremaining: 1.11s\n",
"8:\tlearn: 0.1662518\ttest: 0.1470028\tbest: 0.1470028 (8)\ttotal: 842ms\tremaining: 1.03s\n",
"9:\tlearn: 0.1656720\ttest: 0.1469340\tbest: 0.1469340 (9)\ttotal: 929ms\tremaining: 929ms\n",
"10:\tlearn: 0.1646958\ttest: 0.1458786\tbest: 0.1458786 (10)\ttotal: 1.02s\tremaining: 832ms\n",
"11:\tlearn: 0.1639292\ttest: 0.1456439\tbest: 0.1456439 (11)\ttotal: 1.1s\tremaining: 737ms\n",
"12:\tlearn: 0.1631213\ttest: 0.1453225\tbest: 0.1453225 (12)\ttotal: 1.18s\tremaining: 638ms\n",
"13:\tlearn: 0.1628037\ttest: 0.1449395\tbest: 0.1449395 (13)\ttotal: 1.25s\tremaining: 538ms\n",
"14:\tlearn: 0.1626140\ttest: 0.1450753\tbest: 0.1449395 (13)\ttotal: 1.32s\tremaining: 440ms\n",
"15:\tlearn: 0.1614957\ttest: 0.1448378\tbest: 0.1448378 (15)\ttotal: 1.42s\tremaining: 356ms\n",
"16:\tlearn: 0.1614173\ttest: 0.1448587\tbest: 0.1448378 (15)\ttotal: 1.51s\tremaining: 267ms\n",
"17:\tlearn: 0.1614154\ttest: 0.1448968\tbest: 0.1448378 (15)\ttotal: 1.61s\tremaining: 179ms\n",
"18:\tlearn: 0.1613764\ttest: 0.1448062\tbest: 0.1448062 (18)\ttotal: 1.65s\tremaining: 87ms\n",
"19:\tlearn: 0.1612640\ttest: 0.1445666\tbest: 0.1445666 (19)\ttotal: 1.72s\tremaining: 0us\n",
"\n",
"bestTest = 0.1445666012\n",
"bestIteration = 19\n",
"\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#!rm 'catboost_info/snapshot.bkp'\n",
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=20,\n",
" save_snapshot=True,\n",
" snapshot_file='snapshot.bkp',\n",
" snapshot_interval=1,\n",
" random_seed=43\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" eval_set=(X_validation, y_validation),\n",
" cat_features=cat_features,\n",
" verbose=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "902ccb0a2fef4f25b41b4697f325192a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"20:\tlearn: 0.1607445\ttest: 0.1440626\tbest: 0.1440626 (20)\ttotal: 1.82s\tremaining: 17.1s\n",
"21:\tlearn: 0.1606770\ttest: 0.1439965\tbest: 0.1439965 (21)\ttotal: 1.9s\tremaining: 15.8s\n",
"22:\tlearn: 0.1606616\ttest: 0.1440083\tbest: 0.1439965 (21)\ttotal: 1.97s\tremaining: 14.8s\n",
"23:\tlearn: 0.1606540\ttest: 0.1440366\tbest: 0.1439965 (21)\ttotal: 2.04s\tremaining: 14s\n",
"24:\tlearn: 0.1606450\ttest: 0.1440838\tbest: 0.1439965 (21)\ttotal: 2.11s\tremaining: 13.6s\n",
"25:\tlearn: 0.1604540\ttest: 0.1439186\tbest: 0.1439186 (25)\ttotal: 2.22s\tremaining: 14.3s\n",
"26:\tlearn: 0.1600134\ttest: 0.1436727\tbest: 0.1436727 (26)\ttotal: 2.33s\tremaining: 15s\n",
"27:\tlearn: 0.1600132\ttest: 0.1436712\tbest: 0.1436712 (27)\ttotal: 2.4s\tremaining: 14.5s\n",
"28:\tlearn: 0.1597968\ttest: 0.1437510\tbest: 0.1436712 (27)\ttotal: 2.49s\tremaining: 14.6s\n",
"29:\tlearn: 0.1597306\ttest: 0.1437718\tbest: 0.1436712 (27)\ttotal: 2.62s\tremaining: 15.3s\n",
"30:\tlearn: 0.1595170\ttest: 0.1437298\tbest: 0.1436712 (27)\ttotal: 2.77s\tremaining: 16.1s\n",
"31:\tlearn: 0.1593775\ttest: 0.1439295\tbest: 0.1436712 (27)\ttotal: 2.88s\tremaining: 16.2s\n",
"32:\tlearn: 0.1589895\ttest: 0.1438260\tbest: 0.1436712 (27)\ttotal: 2.99s\tremaining: 16.2s\n",
"33:\tlearn: 0.1589845\ttest: 0.1438430\tbest: 0.1436712 (27)\ttotal: 3.04s\tremaining: 15.6s\n",
"34:\tlearn: 0.1589319\ttest: 0.1438151\tbest: 0.1436712 (27)\ttotal: 3.12s\tremaining: 15.4s\n",
"35:\tlearn: 0.1588918\ttest: 0.1438180\tbest: 0.1436712 (27)\ttotal: 3.19s\tremaining: 15s\n",
"36:\tlearn: 0.1588731\ttest: 0.1438125\tbest: 0.1436712 (27)\ttotal: 3.24s\tremaining: 14.6s\n",
"37:\tlearn: 0.1588159\ttest: 0.1438478\tbest: 0.1436712 (27)\ttotal: 3.32s\tremaining: 14.4s\n",
"38:\tlearn: 0.1588010\ttest: 0.1438314\tbest: 0.1436712 (27)\ttotal: 3.4s\tremaining: 14.2s\n",
"39:\tlearn: 0.1586945\ttest: 0.1436908\tbest: 0.1436712 (27)\ttotal: 3.49s\tremaining: 14.1s\n",
"40:\tlearn: 0.1586894\ttest: 0.1436654\tbest: 0.1436654 (40)\ttotal: 3.56s\tremaining: 13.9s\n",
"41:\tlearn: 0.1582296\ttest: 0.1432243\tbest: 0.1432243 (41)\ttotal: 3.66s\tremaining: 13.9s\n",
"42:\tlearn: 0.1582208\ttest: 0.1432282\tbest: 0.1432243 (41)\ttotal: 3.71s\tremaining: 13.6s\n",
"43:\tlearn: 0.1582196\ttest: 0.1432159\tbest: 0.1432159 (43)\ttotal: 3.78s\tremaining: 13.4s\n",
"44:\tlearn: 0.1579211\ttest: 0.1430249\tbest: 0.1430249 (44)\ttotal: 3.86s\tremaining: 13.3s\n",
"45:\tlearn: 0.1578847\ttest: 0.1431223\tbest: 0.1430249 (44)\ttotal: 3.96s\tremaining: 13.2s\n",
"46:\tlearn: 0.1578482\ttest: 0.1430689\tbest: 0.1430249 (44)\ttotal: 4.05s\tremaining: 13.2s\n",
"47:\tlearn: 0.1578470\ttest: 0.1430681\tbest: 0.1430249 (44)\ttotal: 4.11s\tremaining: 13s\n",
"48:\tlearn: 0.1578462\ttest: 0.1430580\tbest: 0.1430249 (44)\ttotal: 4.17s\tremaining: 12.7s\n",
"49:\tlearn: 0.1577480\ttest: 0.1432538\tbest: 0.1430249 (44)\ttotal: 4.25s\tremaining: 12.6s\n",
"50:\tlearn: 0.1575513\ttest: 0.1433252\tbest: 0.1430249 (44)\ttotal: 4.34s\tremaining: 12.6s\n",
"51:\tlearn: 0.1571386\ttest: 0.1428501\tbest: 0.1428501 (51)\ttotal: 4.43s\tremaining: 12.5s\n",
"52:\tlearn: 0.1571009\ttest: 0.1428871\tbest: 0.1428501 (51)\ttotal: 4.52s\tremaining: 12.5s\n",
"53:\tlearn: 0.1570543\ttest: 0.1428245\tbest: 0.1428245 (53)\ttotal: 4.6s\tremaining: 12.4s\n",
"54:\tlearn: 0.1569937\ttest: 0.1428190\tbest: 0.1428190 (54)\ttotal: 4.67s\tremaining: 12.2s\n",
"55:\tlearn: 0.1568604\ttest: 0.1428687\tbest: 0.1428190 (54)\ttotal: 4.79s\tremaining: 12.3s\n",
"56:\tlearn: 0.1568583\ttest: 0.1428933\tbest: 0.1428190 (54)\ttotal: 4.84s\tremaining: 12s\n",
"57:\tlearn: 0.1566636\ttest: 0.1425799\tbest: 0.1425799 (57)\ttotal: 4.92s\tremaining: 11.9s\n",
"58:\tlearn: 0.1566105\ttest: 0.1425226\tbest: 0.1425226 (58)\ttotal: 5s\tremaining: 11.9s\n",
"59:\tlearn: 0.1565786\ttest: 0.1425208\tbest: 0.1425208 (59)\ttotal: 5.1s\tremaining: 11.8s\n",
"60:\tlearn: 0.1565691\ttest: 0.1425103\tbest: 0.1425103 (60)\ttotal: 5.2s\tremaining: 11.8s\n",
"61:\tlearn: 0.1564572\ttest: 0.1425675\tbest: 0.1425103 (60)\ttotal: 5.3s\tremaining: 11.8s\n",
"62:\tlearn: 0.1564145\ttest: 0.1426340\tbest: 0.1425103 (60)\ttotal: 5.42s\tremaining: 11.8s\n",
"63:\tlearn: 0.1555005\ttest: 0.1421151\tbest: 0.1421151 (63)\ttotal: 5.52s\tremaining: 11.7s\n",
"64:\tlearn: 0.1548624\ttest: 0.1414761\tbest: 0.1414761 (64)\ttotal: 5.63s\tremaining: 11.7s\n",
"65:\tlearn: 0.1548316\ttest: 0.1415012\tbest: 0.1414761 (64)\ttotal: 5.73s\tremaining: 11.7s\n",
"66:\tlearn: 0.1546954\ttest: 0.1414039\tbest: 0.1414039 (66)\ttotal: 5.83s\tremaining: 11.6s\n",
"67:\tlearn: 0.1545918\ttest: 0.1412461\tbest: 0.1412461 (67)\ttotal: 5.94s\tremaining: 11.6s\n",
"68:\tlearn: 0.1545238\ttest: 0.1412214\tbest: 0.1412214 (68)\ttotal: 6.05s\tremaining: 11.6s\n",
"69:\tlearn: 0.1543865\ttest: 0.1412259\tbest: 0.1412214 (68)\ttotal: 6.17s\tremaining: 11.5s\n",
"70:\tlearn: 0.1541683\ttest: 0.1414024\tbest: 0.1412214 (68)\ttotal: 6.27s\tremaining: 11.5s\n",
"71:\tlearn: 0.1540775\ttest: 0.1414143\tbest: 0.1412214 (68)\ttotal: 6.37s\tremaining: 11.4s\n",
"72:\tlearn: 0.1540220\ttest: 0.1413343\tbest: 0.1412214 (68)\ttotal: 6.45s\tremaining: 11.3s\n",
"73:\tlearn: 0.1539815\ttest: 0.1413039\tbest: 0.1412214 (68)\ttotal: 6.52s\tremaining: 11.2s\n",
"74:\tlearn: 0.1539774\ttest: 0.1413123\tbest: 0.1412214 (68)\ttotal: 6.62s\tremaining: 11.1s\n",
"75:\tlearn: 0.1539583\ttest: 0.1413918\tbest: 0.1412214 (68)\ttotal: 6.71s\tremaining: 11s\n",
"76:\tlearn: 0.1538207\ttest: 0.1412925\tbest: 0.1412214 (68)\ttotal: 6.8s\tremaining: 10.9s\n",
"77:\tlearn: 0.1537301\ttest: 0.1412290\tbest: 0.1412214 (68)\ttotal: 6.89s\tremaining: 10.9s\n",
"78:\tlearn: 0.1536745\ttest: 0.1411666\tbest: 0.1411666 (78)\ttotal: 6.99s\tremaining: 10.8s\n",
"79:\tlearn: 0.1536233\ttest: 0.1412135\tbest: 0.1411666 (78)\ttotal: 7.07s\tremaining: 10.7s\n",
"80:\tlearn: 0.1533953\ttest: 0.1413119\tbest: 0.1411666 (78)\ttotal: 7.18s\tremaining: 10.6s\n",
"81:\tlearn: 0.1533845\ttest: 0.1412750\tbest: 0.1411666 (78)\ttotal: 7.27s\tremaining: 10.6s\n",
"82:\tlearn: 0.1532740\ttest: 0.1413232\tbest: 0.1411666 (78)\ttotal: 7.35s\tremaining: 10.4s\n",
"83:\tlearn: 0.1531701\ttest: 0.1416607\tbest: 0.1411666 (78)\ttotal: 7.45s\tremaining: 10.4s\n",
"84:\tlearn: 0.1530255\ttest: 0.1416930\tbest: 0.1411666 (78)\ttotal: 7.53s\tremaining: 10.3s\n",
"85:\tlearn: 0.1530003\ttest: 0.1416376\tbest: 0.1411666 (78)\ttotal: 7.6s\tremaining: 10.2s\n",
"86:\tlearn: 0.1529827\ttest: 0.1415299\tbest: 0.1411666 (78)\ttotal: 7.7s\tremaining: 10.1s\n",
"87:\tlearn: 0.1529783\ttest: 0.1415534\tbest: 0.1411666 (78)\ttotal: 7.78s\tremaining: 9.98s\n",
"88:\tlearn: 0.1529650\ttest: 0.1415712\tbest: 0.1411666 (78)\ttotal: 7.86s\tremaining: 9.87s\n",
"89:\tlearn: 0.1529603\ttest: 0.1415611\tbest: 0.1411666 (78)\ttotal: 7.95s\tremaining: 9.79s\n",
"90:\tlearn: 0.1526304\ttest: 0.1415594\tbest: 0.1411666 (78)\ttotal: 8.03s\tremaining: 9.69s\n",
"91:\tlearn: 0.1525361\ttest: 0.1417001\tbest: 0.1411666 (78)\ttotal: 8.14s\tremaining: 9.62s\n",
"92:\tlearn: 0.1525222\ttest: 0.1416753\tbest: 0.1411666 (78)\ttotal: 8.25s\tremaining: 9.56s\n",
"93:\tlearn: 0.1523982\ttest: 0.1417682\tbest: 0.1411666 (78)\ttotal: 8.34s\tremaining: 9.48s\n",
"94:\tlearn: 0.1520970\ttest: 0.1417407\tbest: 0.1411666 (78)\ttotal: 8.44s\tremaining: 9.4s\n",
"95:\tlearn: 0.1520886\ttest: 0.1416787\tbest: 0.1411666 (78)\ttotal: 8.53s\tremaining: 9.31s\n",
"96:\tlearn: 0.1519551\ttest: 0.1416528\tbest: 0.1411666 (78)\ttotal: 8.62s\tremaining: 9.22s\n",
"97:\tlearn: 0.1518608\ttest: 0.1417604\tbest: 0.1411666 (78)\ttotal: 8.72s\tremaining: 9.15s\n",
"98:\tlearn: 0.1515918\ttest: 0.1418670\tbest: 0.1411666 (78)\ttotal: 8.81s\tremaining: 9.06s\n",
"99:\tlearn: 0.1514836\ttest: 0.1421103\tbest: 0.1411666 (78)\ttotal: 8.88s\tremaining: 8.95s\n",
"100:\tlearn: 0.1513803\ttest: 0.1424265\tbest: 0.1411666 (78)\ttotal: 8.94s\tremaining: 8.82s\n",
"101:\tlearn: 0.1513654\ttest: 0.1424137\tbest: 0.1411666 (78)\ttotal: 9s\tremaining: 8.7s\n",
"102:\tlearn: 0.1513427\ttest: 0.1423719\tbest: 0.1411666 (78)\ttotal: 9.06s\tremaining: 8.57s\n",
"103:\tlearn: 0.1511778\ttest: 0.1422654\tbest: 0.1411666 (78)\ttotal: 9.14s\tremaining: 8.48s\n",
"104:\tlearn: 0.1511117\ttest: 0.1423467\tbest: 0.1411666 (78)\ttotal: 9.24s\tremaining: 8.41s\n",
"105:\tlearn: 0.1510436\ttest: 0.1423221\tbest: 0.1411666 (78)\ttotal: 9.35s\tremaining: 8.33s\n",
"106:\tlearn: 0.1508977\ttest: 0.1423137\tbest: 0.1411666 (78)\ttotal: 9.43s\tremaining: 8.24s\n",
"107:\tlearn: 0.1508959\ttest: 0.1423254\tbest: 0.1411666 (78)\ttotal: 9.5s\tremaining: 8.13s\n",
"108:\tlearn: 0.1508424\ttest: 0.1424438\tbest: 0.1411666 (78)\ttotal: 9.59s\tremaining: 8.04s\n",
"109:\tlearn: 0.1508101\ttest: 0.1424697\tbest: 0.1411666 (78)\ttotal: 9.67s\tremaining: 7.95s\n",
"110:\tlearn: 0.1507799\ttest: 0.1424306\tbest: 0.1411666 (78)\ttotal: 9.75s\tremaining: 7.85s\n",
"111:\tlearn: 0.1507635\ttest: 0.1424197\tbest: 0.1411666 (78)\ttotal: 9.84s\tremaining: 7.76s\n",
"112:\tlearn: 0.1507610\ttest: 0.1423976\tbest: 0.1411666 (78)\ttotal: 9.92s\tremaining: 7.67s\n",
"113:\tlearn: 0.1507207\ttest: 0.1423904\tbest: 0.1411666 (78)\ttotal: 10s\tremaining: 7.58s\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"114:\tlearn: 0.1506871\ttest: 0.1423714\tbest: 0.1411666 (78)\ttotal: 10.1s\tremaining: 7.52s\n",
"115:\tlearn: 0.1506365\ttest: 0.1423830\tbest: 0.1411666 (78)\ttotal: 10.2s\tremaining: 7.43s\n",
"116:\tlearn: 0.1505736\ttest: 0.1425609\tbest: 0.1411666 (78)\ttotal: 10.3s\tremaining: 7.35s\n",
"117:\tlearn: 0.1505321\ttest: 0.1425929\tbest: 0.1411666 (78)\ttotal: 10.4s\tremaining: 7.29s\n",
"118:\tlearn: 0.1505028\ttest: 0.1425452\tbest: 0.1411666 (78)\ttotal: 10.5s\tremaining: 7.21s\n",
"119:\tlearn: 0.1497712\ttest: 0.1424179\tbest: 0.1411666 (78)\ttotal: 10.7s\tremaining: 7.17s\n",
"120:\tlearn: 0.1497006\ttest: 0.1423512\tbest: 0.1411666 (78)\ttotal: 10.8s\tremaining: 7.11s\n",
"121:\tlearn: 0.1496638\ttest: 0.1423660\tbest: 0.1411666 (78)\ttotal: 11s\tremaining: 7.1s\n",
"122:\tlearn: 0.1496092\ttest: 0.1423799\tbest: 0.1411666 (78)\ttotal: 11.1s\tremaining: 7.04s\n",
"123:\tlearn: 0.1495761\ttest: 0.1422939\tbest: 0.1411666 (78)\ttotal: 11.2s\tremaining: 6.95s\n",
"124:\tlearn: 0.1495648\ttest: 0.1423047\tbest: 0.1411666 (78)\ttotal: 11.3s\tremaining: 6.86s\n",
"125:\tlearn: 0.1493551\ttest: 0.1421014\tbest: 0.1411666 (78)\ttotal: 11.5s\tremaining: 6.8s\n",
"126:\tlearn: 0.1492764\ttest: 0.1420590\tbest: 0.1411666 (78)\ttotal: 11.6s\tremaining: 6.72s\n",
"127:\tlearn: 0.1490840\ttest: 0.1417298\tbest: 0.1411666 (78)\ttotal: 11.7s\tremaining: 6.65s\n",
"128:\tlearn: 0.1490683\ttest: 0.1417569\tbest: 0.1411666 (78)\ttotal: 11.8s\tremaining: 6.58s\n",
"129:\tlearn: 0.1490611\ttest: 0.1417673\tbest: 0.1411666 (78)\ttotal: 12s\tremaining: 6.53s\n",
"130:\tlearn: 0.1490453\ttest: 0.1417709\tbest: 0.1411666 (78)\ttotal: 12.1s\tremaining: 6.45s\n",
"131:\tlearn: 0.1490231\ttest: 0.1417575\tbest: 0.1411666 (78)\ttotal: 12.3s\tremaining: 6.39s\n",
"132:\tlearn: 0.1489009\ttest: 0.1419888\tbest: 0.1411666 (78)\ttotal: 12.3s\tremaining: 6.29s\n",
"133:\tlearn: 0.1486744\ttest: 0.1420246\tbest: 0.1411666 (78)\ttotal: 12.4s\tremaining: 6.2s\n",
"134:\tlearn: 0.1485575\ttest: 0.1421380\tbest: 0.1411666 (78)\ttotal: 12.5s\tremaining: 6.11s\n",
"135:\tlearn: 0.1485306\ttest: 0.1421969\tbest: 0.1411666 (78)\ttotal: 12.6s\tremaining: 6.01s\n",
"136:\tlearn: 0.1483855\ttest: 0.1420489\tbest: 0.1411666 (78)\ttotal: 12.7s\tremaining: 5.91s\n",
"137:\tlearn: 0.1483428\ttest: 0.1421026\tbest: 0.1411666 (78)\ttotal: 12.8s\tremaining: 5.83s\n",
"138:\tlearn: 0.1483134\ttest: 0.1420898\tbest: 0.1411666 (78)\ttotal: 12.9s\tremaining: 5.72s\n",
"139:\tlearn: 0.1483052\ttest: 0.1420878\tbest: 0.1411666 (78)\ttotal: 13s\tremaining: 5.63s\n",
"140:\tlearn: 0.1482157\ttest: 0.1420241\tbest: 0.1411666 (78)\ttotal: 13.1s\tremaining: 5.53s\n",
"141:\tlearn: 0.1481112\ttest: 0.1421268\tbest: 0.1411666 (78)\ttotal: 13.2s\tremaining: 5.45s\n",
"142:\tlearn: 0.1479973\ttest: 0.1422019\tbest: 0.1411666 (78)\ttotal: 13.3s\tremaining: 5.36s\n",
"143:\tlearn: 0.1479948\ttest: 0.1422188\tbest: 0.1411666 (78)\ttotal: 13.4s\tremaining: 5.28s\n",
"144:\tlearn: 0.1479705\ttest: 0.1421976\tbest: 0.1411666 (78)\ttotal: 13.6s\tremaining: 5.22s\n",
"145:\tlearn: 0.1479037\ttest: 0.1423085\tbest: 0.1411666 (78)\ttotal: 13.8s\tremaining: 5.16s\n",
"146:\tlearn: 0.1478758\ttest: 0.1424303\tbest: 0.1411666 (78)\ttotal: 13.9s\tremaining: 5.08s\n",
"147:\tlearn: 0.1478342\ttest: 0.1424301\tbest: 0.1411666 (78)\ttotal: 14s\tremaining: 4.98s\n",
"148:\tlearn: 0.1478086\ttest: 0.1423590\tbest: 0.1411666 (78)\ttotal: 14.1s\tremaining: 4.89s\n",
"149:\tlearn: 0.1477729\ttest: 0.1422865\tbest: 0.1411666 (78)\ttotal: 14.2s\tremaining: 4.8s\n",
"150:\tlearn: 0.1474847\ttest: 0.1422271\tbest: 0.1411666 (78)\ttotal: 14.3s\tremaining: 4.71s\n",
"151:\tlearn: 0.1474169\ttest: 0.1423728\tbest: 0.1411666 (78)\ttotal: 14.4s\tremaining: 4.63s\n",
"152:\tlearn: 0.1472865\ttest: 0.1422631\tbest: 0.1411666 (78)\ttotal: 14.6s\tremaining: 4.54s\n",
"153:\tlearn: 0.1472423\ttest: 0.1422256\tbest: 0.1411666 (78)\ttotal: 14.7s\tremaining: 4.46s\n",
"154:\tlearn: 0.1471948\ttest: 0.1421481\tbest: 0.1411666 (78)\ttotal: 14.8s\tremaining: 4.37s\n",
"155:\tlearn: 0.1468854\ttest: 0.1419873\tbest: 0.1411666 (78)\ttotal: 14.9s\tremaining: 4.28s\n",
"156:\tlearn: 0.1468790\ttest: 0.1420101\tbest: 0.1411666 (78)\ttotal: 15s\tremaining: 4.18s\n",
"157:\tlearn: 0.1468687\ttest: 0.1419619\tbest: 0.1411666 (78)\ttotal: 15.1s\tremaining: 4.09s\n",
"158:\tlearn: 0.1468658\ttest: 0.1419551\tbest: 0.1411666 (78)\ttotal: 15.2s\tremaining: 3.99s\n",
"159:\tlearn: 0.1468288\ttest: 0.1420261\tbest: 0.1411666 (78)\ttotal: 15.3s\tremaining: 3.89s\n",
"160:\tlearn: 0.1467160\ttest: 0.1419321\tbest: 0.1411666 (78)\ttotal: 15.4s\tremaining: 3.8s\n",
"161:\tlearn: 0.1467084\ttest: 0.1419409\tbest: 0.1411666 (78)\ttotal: 15.5s\tremaining: 3.7s\n",
"162:\tlearn: 0.1464073\ttest: 0.1416654\tbest: 0.1411666 (78)\ttotal: 15.7s\tremaining: 3.61s\n",
"163:\tlearn: 0.1462285\ttest: 0.1416156\tbest: 0.1411666 (78)\ttotal: 15.8s\tremaining: 3.52s\n",
"164:\tlearn: 0.1460339\ttest: 0.1416086\tbest: 0.1411666 (78)\ttotal: 15.9s\tremaining: 3.42s\n",
"165:\tlearn: 0.1460218\ttest: 0.1416287\tbest: 0.1411666 (78)\ttotal: 16s\tremaining: 3.33s\n",
"166:\tlearn: 0.1460092\ttest: 0.1416437\tbest: 0.1411666 (78)\ttotal: 16.1s\tremaining: 3.23s\n",
"167:\tlearn: 0.1456841\ttest: 0.1418764\tbest: 0.1411666 (78)\ttotal: 16.2s\tremaining: 3.13s\n",
"168:\tlearn: 0.1456011\ttest: 0.1419543\tbest: 0.1411666 (78)\ttotal: 16.3s\tremaining: 3.04s\n",
"169:\tlearn: 0.1455905\ttest: 0.1420247\tbest: 0.1411666 (78)\ttotal: 16.4s\tremaining: 2.94s\n",
"170:\tlearn: 0.1455581\ttest: 0.1420612\tbest: 0.1411666 (78)\ttotal: 16.5s\tremaining: 2.85s\n",
"171:\tlearn: 0.1455441\ttest: 0.1420924\tbest: 0.1411666 (78)\ttotal: 16.6s\tremaining: 2.75s\n",
"172:\tlearn: 0.1454990\ttest: 0.1421376\tbest: 0.1411666 (78)\ttotal: 16.7s\tremaining: 2.65s\n",
"173:\tlearn: 0.1453813\ttest: 0.1420980\tbest: 0.1411666 (78)\ttotal: 16.9s\tremaining: 2.56s\n",
"174:\tlearn: 0.1451034\ttest: 0.1420903\tbest: 0.1411666 (78)\ttotal: 17s\tremaining: 2.46s\n",
"175:\tlearn: 0.1450984\ttest: 0.1421000\tbest: 0.1411666 (78)\ttotal: 17s\tremaining: 2.36s\n",
"176:\tlearn: 0.1449863\ttest: 0.1421289\tbest: 0.1411666 (78)\ttotal: 17.1s\tremaining: 2.26s\n",
"177:\tlearn: 0.1447876\ttest: 0.1420145\tbest: 0.1411666 (78)\ttotal: 17.2s\tremaining: 2.16s\n",
"178:\tlearn: 0.1447701\ttest: 0.1420666\tbest: 0.1411666 (78)\ttotal: 17.3s\tremaining: 2.06s\n",
"179:\tlearn: 0.1446955\ttest: 0.1420109\tbest: 0.1411666 (78)\ttotal: 17.4s\tremaining: 1.96s\n",
"180:\tlearn: 0.1446544\ttest: 0.1420329\tbest: 0.1411666 (78)\ttotal: 17.5s\tremaining: 1.86s\n",
"181:\tlearn: 0.1445431\ttest: 0.1421508\tbest: 0.1411666 (78)\ttotal: 17.6s\tremaining: 1.76s\n",
"182:\tlearn: 0.1442690\ttest: 0.1421298\tbest: 0.1411666 (78)\ttotal: 17.7s\tremaining: 1.67s\n",
"183:\tlearn: 0.1441832\ttest: 0.1421534\tbest: 0.1411666 (78)\ttotal: 17.8s\tremaining: 1.57s\n",
"184:\tlearn: 0.1440706\ttest: 0.1423811\tbest: 0.1411666 (78)\ttotal: 17.9s\tremaining: 1.47s\n",
"185:\tlearn: 0.1439856\ttest: 0.1425963\tbest: 0.1411666 (78)\ttotal: 18s\tremaining: 1.37s\n",
"186:\tlearn: 0.1439761\ttest: 0.1426506\tbest: 0.1411666 (78)\ttotal: 18.1s\tremaining: 1.28s\n",
"187:\tlearn: 0.1439714\ttest: 0.1426176\tbest: 0.1411666 (78)\ttotal: 18.2s\tremaining: 1.18s\n",
"188:\tlearn: 0.1439492\ttest: 0.1426109\tbest: 0.1411666 (78)\ttotal: 18.4s\tremaining: 1.08s\n",
"189:\tlearn: 0.1439454\ttest: 0.1426164\tbest: 0.1411666 (78)\ttotal: 18.4s\tremaining: 984ms\n",
"190:\tlearn: 0.1439413\ttest: 0.1426552\tbest: 0.1411666 (78)\ttotal: 18.5s\tremaining: 885ms\n",
"191:\tlearn: 0.1438359\ttest: 0.1427087\tbest: 0.1411666 (78)\ttotal: 18.6s\tremaining: 787ms\n",
"192:\tlearn: 0.1438287\ttest: 0.1427033\tbest: 0.1411666 (78)\ttotal: 18.7s\tremaining: 689ms\n",
"193:\tlearn: 0.1437879\ttest: 0.1426357\tbest: 0.1411666 (78)\ttotal: 18.8s\tremaining: 590ms\n",
"194:\tlearn: 0.1437860\ttest: 0.1426468\tbest: 0.1411666 (78)\ttotal: 18.9s\tremaining: 492ms\n",
"195:\tlearn: 0.1437606\ttest: 0.1426516\tbest: 0.1411666 (78)\ttotal: 19s\tremaining: 393ms\n",
"196:\tlearn: 0.1437591\ttest: 0.1426400\tbest: 0.1411666 (78)\ttotal: 19.1s\tremaining: 294ms\n",
"197:\tlearn: 0.1437577\ttest: 0.1426297\tbest: 0.1411666 (78)\ttotal: 19.2s\tremaining: 196ms\n",
"198:\tlearn: 0.1437447\ttest: 0.1426317\tbest: 0.1411666 (78)\ttotal: 19.3s\tremaining: 98ms\n",
"199:\tlearn: 0.1437230\ttest: 0.1426660\tbest: 0.1411666 (78)\ttotal: 19.4s\tremaining: 0us\n",
"\n",
"bestTest = 0.1411666436\n",
"bestIteration = 78\n",
"\n",
"Shrink model to first 79 iterations.\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = CatBoostClassifier(\n",
" iterations=200,\n",
" save_snapshot=True,\n",
" snapshot_file='snapshot.bkp',\n",
" snapshot_interval=1,\n",
" random_seed=43\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" eval_set=(X_validation, y_validation),\n",
" cat_features=cat_features,\n",
" verbose=True,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model predictions"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"? model.predict_proba"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[0.0688 0.9312]\n",
" [0.008 0.992 ]\n",
" [0.0068 0.9932]\n",
" ...\n",
" [0.0129 0.9871]\n",
" [0.0184 0.9816]\n",
" [0.0273 0.9727]]\n"
]
}
],
"source": [
"print(model.predict_proba(data=X_validation))"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1. 1. 1. ... 1. 1. 1.]\n"
]
}
],
"source": [
"print(model.predict(data=X_validation))"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2.6056 4.8177 4.9809 ... 4.3367 3.9791 3.5714]\n"
]
}
],
"source": [
"raw_pred = model.predict(data=X_validation, prediction_type='RawFormulaVal')\n",
"print(raw_pred)"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.9312 0.992 0.9932 ... 0.9871 0.9816 0.9727]\n"
]
}
],
"source": [
"import math\n",
"def sigmoid(x):\n",
" return 1 / (1 + math.exp(-x))\n",
"probabilities = [sigmoid(x) for x in raw_pred]\n",
"print(np.array(probabilities))"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[0.0688 0.9312]\n",
" [0.008 0.992 ]\n",
" [0.0068 0.9932]\n",
" ...\n",
" [0.0129 0.9871]\n",
" [0.0184 0.9816]\n",
" [0.0273 0.9727]]\n"
]
}
],
"source": [
"X_prepared = X_validation.values.astype(str).astype(object)\n",
"# For FeaturesData class categorial features must have type str\n",
"\n",
"fast_predictions = model.predict_proba(data=FeaturesData(cat_feature_data=X_prepared, \n",
" cat_feature_names=list(X_validation)))\n",
"\n",
"print(fast_predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Staged prediction"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 0, predictions:\n",
"[[0.3065 0.6935]\n",
" [0.2317 0.7683]\n",
" [0.2317 0.7683]\n",
" ...\n",
" [0.2317 0.7683]\n",
" [0.2317 0.7683]\n",
" [0.2746 0.7254]]\n",
"Iteration 1, predictions:\n",
"[[0.3002 0.6998]\n",
" [0.1762 0.8238]\n",
" [0.1422 0.8578]\n",
" ...\n",
" [0.1422 0.8578]\n",
" [0.1916 0.8084]\n",
" [0.2116 0.7884]]\n",
"Iteration 2, predictions:\n",
"[[0.3307 0.6693]\n",
" [0.13 0.87 ]\n",
" [0.1038 0.8962]\n",
" ...\n",
" [0.1454 0.8546]\n",
" [0.1956 0.8044]\n",
" [0.1938 0.8062]]\n"
]
}
],
"source": [
"predictions_gen = model.staged_predict_proba(data=X_validation, ntree_start=2, ntree_end=8, eval_period=2)\n",
"for iteration, predictions in enumerate(predictions_gen):\n",
" print('Iteration ' + str(iteration) + ', predictions:')\n",
" print(predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Solving MultiClassification problem"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1536e93770b14eb48d913a0209269c87",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 69,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoostClassifier\n",
"model = CatBoostClassifier(\n",
" iterations=150,\n",
" random_seed=43,\n",
" loss_function='MultiClass'\n",
" #loss_function='MultiClassOneVsAll'\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" eval_set=(X_validation, y_validation),\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Metric evaluation on a new dataset"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.6569860\ttotal: 53.2ms\tremaining: 10.6s\n",
"50:\tlearn: 0.1950260\ttotal: 3.33s\tremaining: 9.74s\n",
"100:\tlearn: 0.1700584\ttotal: 6.72s\tremaining: 6.58s\n",
"150:\tlearn: 0.1641016\ttotal: 10.6s\tremaining: 3.42s\n",
"199:\tlearn: 0.1604074\ttotal: 14.1s\tremaining: 0us\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 70,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = CatBoostClassifier(\n",
" random_seed=63,\n",
" iterations=200,\n",
" learning_rate=0.03,\n",
")\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" verbose=50\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"metrics = model.eval_metrics(data=pool1, \n",
" metrics=['Logloss','AUC'],\n",
" ntree_start=0,\n",
" ntree_end=0, \n",
" eval_period=1,\n",
" plot=True)"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"AUC values:\n",
"[0.4998 0.538 0.5504 0.5888 0.6503 0.6487 0.6487 0.6601 0.6601 0.6612\n",
" 0.6614 0.67 0.6699 0.6697 0.6697 0.6698 0.6698 0.6698 0.6698 0.7265\n",
" 0.7338 0.7376 0.7478 0.7478 0.7506 0.7591 0.7642 0.7877 0.8148 0.8265\n",
" 0.8353 0.8459 0.8533 0.8564 0.8573 0.8748 0.8874 0.893 0.8948 0.898\n",
" 0.9003 0.9052 0.9122 0.9159 0.92 0.9204 0.9209 0.9246 0.9262 0.9266\n",
" 0.9279 0.9303 0.9311 0.9312 0.9327 0.9329 0.9337 0.9341 0.9341 0.9349\n",
" 0.9354 0.9365 0.9391 0.9411 0.9424 0.9435 0.9446 0.9458 0.9463 0.9471\n",
" 0.9478 0.9481 0.9482 0.9483 0.9486 0.9497 0.951 0.9514 0.9515 0.9519\n",
" 0.9522 0.9526 0.953 0.9537 0.9543 0.9544 0.9545 0.9548 0.9548 0.9548\n",
" 0.9548 0.9548 0.955 0.9551 0.9553 0.9554 0.9557 0.9557 0.9557 0.9557\n",
" 0.956 0.9566 0.9566 0.9569 0.9568 0.957 0.9569 0.9572 0.9573 0.9575\n",
" 0.9576 0.9576 0.9577 0.958 0.9587 0.9594 0.9595 0.9594 0.9601 0.9609\n",
" 0.9609 0.9608 0.9608 0.9612 0.9616 0.9618 0.9622 0.9623 0.9623 0.9625\n",
" 0.9625 0.9628 0.9629 0.9629 0.9629 0.963 0.9631 0.9632 0.9635 0.9636\n",
" 0.9637 0.9638 0.9638 0.964 0.9641 0.9643 0.9645 0.9645 0.9645 0.9647\n",
" 0.9647 0.9647 0.9647 0.9648 0.9648 0.9648 0.965 0.9652 0.9652 0.9653\n",
" 0.9653 0.9654 0.9655 0.9655 0.9655 0.9656 0.9656 0.9657 0.9657 0.9657\n",
" 0.9658 0.9658 0.9659 0.9659 0.966 0.966 0.966 0.966 0.966 0.966\n",
" 0.966 0.966 0.9661 0.9662 0.9663 0.9663 0.9663 0.9663 0.9663 0.9663\n",
" 0.9665 0.9664 0.9666 0.9666 0.9666 0.9666 0.9666 0.9667 0.9667 0.9666]\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "79673ab90b80409f9098024a23d609ec",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"print('AUC values:')\n",
"print(np.array(metrics['AUC']))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Saving the model"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 73,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"my_best_model = CatBoostClassifier(iterations=10)\n",
"my_best_model.fit(\n",
" X_train, y_train,\n",
" eval_set=(X_validation, y_validation),\n",
" cat_features=cat_features,\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {},
"outputs": [],
"source": [
"my_best_model.save_model('catboost_model.bin')"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'iterations': 10, 'loss_function': 'Logloss', 'logging_level': 'Silent'}\n",
"17379826207872\n"
]
}
],
"source": [
"my_best_model.load_model('catboost_model.bin')\n",
"print(my_best_model.get_params())\n",
"print(my_best_model.random_seed_)"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"my_best_model.save_model('catboost_model.json', format='json')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feature importances"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.6569860\ttotal: 56.9ms\tremaining: 11.3s\n",
"50:\tlearn: 0.1950260\ttotal: 4.06s\tremaining: 11.9s\n",
"100:\tlearn: 0.1700584\ttotal: 7.18s\tremaining: 7.04s\n",
"150:\tlearn: 0.1641016\ttotal: 10.8s\tremaining: 3.5s\n",
"199:\tlearn: 0.1604074\ttotal: 14.3s\tremaining: 0us\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = CatBoostClassifier(\n",
" random_seed=63,\n",
" iterations=200,\n",
" learning_rate=0.03)\n",
"\n",
"model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" verbose=50\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {},
"outputs": [],
"source": [
"fstrs = model.get_feature_importance(prettified=True)"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{b'MGR_ID': 18.398056223683028,\n",
" b'RESOURCE': 21.27625707750949,\n",
" b'ROLE_CODE': 11.943230282191124,\n",
" b'ROLE_DEPTNAME': 15.252806962601875,\n",
" b'ROLE_FAMILY': 2.4789173515872176,\n",
" b'ROLE_FAMILY_DESC': 9.984073192415533,\n",
" b'ROLE_ROLLUP_1': 2.6278918788673633,\n",
" b'ROLE_ROLLUP_2': 13.582536028250486,\n",
" b'ROLE_TITLE': 4.456231002893895}"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{feature_name : value for feature_name, value in fstrs}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shap values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"https://github.com/slundberg/shap"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [],
"source": [
"def object_predictions(model, obj):\n",
" print('Probability of class 1 = {:.4f}'.format(model.predict_proba([obj])[0][1]))\n",
" print('Formula raw prediction = {:.4f}'.format(model.predict([obj], prediction_type='RawFormulaVal')[0]))\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"The model has complex ctrs, so the SHAP values will be calculated approximately.\n"
]
}
],
"source": [
"import shap\n",
"explainer = shap.TreeExplainer(model)\n",
"shap_values = explainer.shap_values(Pool(X, y, cat_features=cat_features))\n"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3.345740996864643"
]
},
"execution_count": 82,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"explainer.expected_value"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Probability of class 1 = 0.9798\n",
"Formula raw prediction = 3.8820\n"
]
}
],
"source": [
"object_predictions(model, X.iloc[3,:])"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" Visualization omitted, Javascript library not loaded! \n",
" Have you run `initjs()` in this notebook? If this notebook was from another\n",
" user you must also trust this notebook (File -> Trust notebook). If you are viewing\n",
" this notebook on github the Javascript has been stripped for security. If you are using\n",
" JupyterLab this error is because a JupyterLab extension has not yet been written.\n",
"
\n",
" "
],
"text/plain": [
""
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"shap.initjs()\n",
"shap.force_plot(explainer.expected_value, shap_values[3,:], X.iloc[3,:])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above explanation shows features each contributing to push the model output from the base value (the average model output over the training dataset we passed) to the model output. Features pushing the prediction higher are shown in red, those pushing the prediction lower are in blue"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Probability of class 1 = 0.6404\n",
"Formula raw prediction = 0.5772\n"
]
}
],
"source": [
"object_predictions(model, X.iloc[91,:])"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" Visualization omitted, Javascript library not loaded! \n",
" Have you run `initjs()` in this notebook? If this notebook was from another\n",
" user you must also trust this notebook (File -> Trust notebook). If you are viewing\n",
" this notebook on github the Javascript has been stripped for security. If you are using\n",
" JupyterLab this error is because a JupyterLab extension has not yet been written.\n",
"
\n",
" "
],
"text/plain": [
""
]
},
"execution_count": 86,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import shap\n",
"shap.initjs()\n",
"shap.force_plot(explainer.expected_value, shap_values[91,:], X.iloc[91,:])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get an overview of which features are most important for a model we can plot the SHAP values of every feature for every sample. The plot below sorts features by the sum of SHAP value magnitudes over all samples, and uses SHAP values to show the distribution of the impacts each feature has on the model output. The color represents the feature value (red high, blue low)."
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAFZCAYAAAC4x3ouAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XecXFd58PHfuWX6zO7O9tVqtatVlyxL1rXcccfGxg1MdQwGm+QlIQTCG8qLwUBICIlDCRhCMG6AKQ5Vxr3LtlyuZFm9S6vV9jq93Pb+MaPVqtjYktaSlvP9fEaaueWcc++dnfvMc889IzzPQ5IkSZIkSXpzlGPdAEmSJEmSpBORDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSZIOgwyiJEmSJEmSDoMMoiRJkiRJkg6DDKIkSZIkSTouCCF2CSEWHDDNFEKcJ4T4mhDifW+gjK8IIW6duFbuo70VlUiSJEmSJB0Jz/O+fKzbcCCZiZIkSZIk6bgnhLhLCPGJ8vMKIcRvhBCbhBCPCyHuOSD7NEUI8UB5/p+EEKGJaJPMREmSJEmSdDz5XyFEftzrWYdY5svAiOd5c4QQcWAl8Jtx8w3gVCABPAxcB/z4aDdUBlHSseYd6wZIkjR5LVu2DIArrrjiGLfkhCEmpNTY+w/+rE/+8rXqutbzvHVjDRLCPMQy5wN/D+B53rAQ4vcHzH/Y87zR8vovAu2H0+w/RwZRkiRJkiRNKO8QsdkRRmuC1/8SPj6T5QDBI6vu0GSfKEmSJEmSJpQnDn4coSeBDwMIIaqAq464xMMggyhJkiRJkiaUhzjocYS+BtQJIdYDPwOeo9T/6S0lL+dJkiRJkjSh3mjmyfO81kNMM8pPnxo3OQN8wPO8vBAiBjwL3F1e/isHrL/f66NJBlGSJEmSJE2wo95fvQp4UAihAgHgXs/zHjvalfw5MoiSJEmSJOmE4nleP7DkWLdDBlGSJEmSJE2oo9CR/LgkgyhJkiRJkibUUehIflySQZQkSZIkSRNKZqIkSZIkSZIOg8xESZIkSZIkHQZPyCBKkiRJkiTpTZusP5IqgyhJkiRJkibUZM1EyZ99kSRJkiRJOgwyEyVJkiRJZX3Deb70yUfpq4xjCcG0/l4+9rULOWV2xbFu2gltsmaiZBAlSZIkSWVf+eSjDFRVE3RsAsCrrTP47I+24HNc6hND3HnXZce6iSekydonSl7OkyRJkqSydY1T8Ts2UPq1t6pCjt7KagajMbKBMK++2nNsG3iC8oQ46DEZyCBKkiRJksqmJEdxx41plPX5x/4fDYX5+D3dDIwWjlXzTljeIR6TgQyiJEmSJKns3B2b2F5VzXAwRFe0goFwdGxe2uen1irynq+vx/MmSxggHQkZREmSJElSWdrvx1UUdsTr6IvEcIXAAxwh6I9E2VpTT9i2+OUTnce6qSeUyXo5T3YslyRJkv5iff5HG7Cf3MQZu7eR1AP4AgFA4AIIgee65DUdhEABLCFIBEKse7EXLmw5to0/gciffZEkSZKkE5RrO3z5/9xPd0Eh4FoEbQfhuKTDEb743GP4HJtHZi3g5WkzyPl8+FwH4XkUdR1PUQFwPA/VEwSLBUYrY/uVn88W+MGyDn621iavqBRVjfMri/z4H+Yei8097sgfIJYkSZKkE5BruXzgb54gHa5DC3ic3LObKYkRXEXBSSV4vnUGg+Eot592HgBByyKvaSieh6vs3+ulMpdm5mAvDUvmMtIxxK2ffpgNtU20JofZXttIvaLi4jEQjfFYNsKpX1jNIzfPpyqsH4MtP35M1kyUkJ3jpGNMvgElSXrTPM/j1U1D7BnK8sMHh8il88zNjtChhmkfHSLguayvn0LO9cgGQ+QCAU7p3s2F2zagUOrjlNH9eELgCEHaH+CXi88YK98CLti+kWXzF4MoBVK6bRMt5GgZHsRnW7SNDtMfiuApCnsqqpg70MPZOzcTtiw84P45C1neNpuCpvHfV1Vw0UlVx2RfvUkTEu0MNd500Gd9dc/tJ3xkJTNRE8gwjKeAc4H3mab563HTTwNeADpM02wtT5sBfAm4CKgEhoF1wI9N0/xteZmvADcDeUrBxwBwD/BV0zT/bDBiGIYHnGOa5rOGYZwHPAlkAJfSZ8Ym4DfAbaZpynt4pTFrugrc+WKCsE/hia1Z0okCV68x6Y9VsLGuiayuU5XNMHOgl/WNU6nI59AdmwW9XYwGQ3RU1VBQVWb197Czph5L03A9CDgWBVXFEQoKkPH5Sfn8TB8eJO33010RxxUCzbbxuw4+u8iHzeeYPjxAR0U1K9pmsquqhgu2bWDWQC/3LDmLVCCI8DyW7NnJeds2UZ3LsKm2kfvnncxwMExlIYfftpkyOsxIIIjR3UFB1fntQoOColKfTuKoKsJzyep+HEWhfaif97z6MmGrwMbaBhKBEJFiAd11yOg+nm6fQ1HVyWsatqJQ0HTO6NjGNetW8sK0dnqjlbSMDCFwUTxBQVXZVl3H5vomovkcOd1Hxh8g5ffTPDrMtOEBZg710xutoDdaia2qjPqDdMRrcBWFqSODTB0doS9aQV7TeNe6lbSMDvOnmfNQFJWVza0UdJ2mxChG5w5S/iA74zUIz6WoafRFK7FUjWg+i89xyGs6toBPrHiSqlyGZ1tnMnOgl6dmzKOzooqMPwCA8DwQEM+kcYVCTvfxjk1rqMsk2V1Zjc+yWN4+BxtoSo0yv68bV1FYMW0GfZEY4UKeD6x+ke6KKlY3tZDXNFTXoy6T5LxtG1g2dxGrm9twhcDn2ISLBWxFxRKCeD7LBds3EsvleL5lOkWfH92x0R2basdBdR3au3ppUwTba+qZ17WbilyWnxtn4SHwPI/ZA71jd1Opnofu2ATtUsBjj8s4uUC0WODxWQuIZzMonktlPsdNK57ka2+/hu019WiuQyoQZGdNHUWtlGVqH+ojZFmlfQVcsmU9j85aQKhY5OO/G2HDvEp09YSPGw7LZM1EySBq4m0EPgb8ety0j5WnhwAMwzgJeBb4LXAesAPQy8+vK0/f6ynTNC8yDEMAZwMPA7uAuw6jbY5pmpFyGwLAOcC3gfcYhnGuaZrFwyhTmmTSBZebftFHMu+OTTO6dtNTUcXy9jmlCZ7H9atW8PPFZ9A+PIDulpbtj1bQXVH69u3zPHbWNIAi0B0HxXNRgJDjUFTAVlUiVpGAbZMIhtlTWQXlO3iKmkZKUbnlkd9xg/ksAFndx7KTloCisHz6HNY0NJPxB9DK2fVVzW38zYtPEynk+da5l2JrOjGrSNCyCNo2PRVVzB7oIRkI8svFZ5AIhgAYisaIFAo0JUcIOjY4cP3K56nOZQBY1NPJ2sap+BwHAN3JEbAdhFDwFR1G/UFUz+Pmx5exua6Rl1ra+Ui5zR6Q1f1srq3nwfmLx/ZnU2KEinyOeDaN5nks7u5kTeNU1jdOHbucFM1nmZoYYVttA4lgBLV8UtI8j3uMs/if++4gatv87qSTCZTbtjteg+J5xAo5iroPPK90PAT4bYtI+YQ/GAjyb3/6NZdtXgPA6R3b+dXJSxF4ZAPBsSAKz8NnFUtlAe/Y9Crn7toKwNyBXn6zYAk5n4/RYIiLdmxiSioBQO36V/if08/nfWteZlN9E+sap45tu6PCzngtubmLWNPciiJAwaOo6eR1fSwouXLDappSo6R9fgr+AJ4QFBRfaS84DtvidSzs6uSOpeeS8Qd4aPZCQlZxbP/5LYuQte+7oQdECnlE+T0Wz2awFAEIXCFIlt8Ped3H5RtX86XH/kikkOe751zCLY/9gdaRQR6euYDvNlwyVubGhilctnktavk9WLoDTWDpOrpt84WHRrj18jh/iSZrnyg5xMHE+y2w2DCM6QCGYUSBdwN3jlvmO8DLpml+xDTNraZpOqZp5k3TfMg0zesPVahpmp5pmsuB9YBxpI0s1/cocA2wGPjwkZb5RqRSKfn8OH/e2Z/cL4ACqM8k6Yvt+y0xzXWJFgvYqjoWQAGk9p589xL7/hv/4aN4pXU8KPVDEWIsgAIQ5TzrnP7usWkhq0hNptROS9NI+YMHfaA1J4bZXVU9likAsMudhIUHAdsi5QuMBVBQOvFprrPf9+ZIcd/J1+c4KO6+xK8CBO3i2HbZmsrUxDDRYoEtNQ00JUf323zVcxkM7Rt7CKColr7P7j35xnNZEsHQfv1xLFUbG0nbV/5/b5kg2FrbQEHXOfBqjKWpqHvbKwRuebY27jgVNI324f6x17XZNL3RGKrr7ZehQQjUcTnv2kx6v7r8tk1FPkvGH8Bv7fsOFnBsArZFbSbFrqqa/dYRgKsodFXE9z/meCCUsev9lfksAHlN3+/2eFcIcppOX7SSjnjNvoCvPG/8Nib8IUYCIXqiFfRGYuiuMzbf59gI1z2oDxRAXSpJpJCno7KaG19+htN3b6chleDDq55jYVfH2HKK66E6DjXpJHWpBOdu30Tr0ECpfNdlfV8pIDwe/q5f6/lE8RAHPSYDGURNvDzwc+DG8usPAE8DPQCGYQQpXfL7xZsp1DAMxTCM84EFwOaj1VjTNLcCK4ELj1aZrycajcrnx/nz2c0VnDsjyHirmts4c+dWRDn4sVWVV5paaB/sYzRQWtYD2gf70ced8IOFwthoxU75BOcBVjmwKaoqo8FQ6VJOIT8231EU8Dx+v2DJWFnd0Qq6YqUsV0NylCs3rMIqnwA9QHcdnm2dxez+XuJ7T/aeNxaAhIoFuqOV1GVSTB/cF0AorktO1/dr32Mz5uGWn2+qaaCoqmMndw9I+UonbkcIfJbNzqoadsRrSwM3VtdSVEvbZwsFWygs7Okkms8BpX42sXKAkCmPjr2tuo7W4QF8tjXW7qBVpDdaClyT/kDp0hpgKQr1qQQXbN1AZS5DTSY5NuK25jjEMxmy5cwRnke4fAzy2r5tDBfy3Lvo9LF98MjM+cwZ7CXpDxDJ58fqEp5LUVXZG36tmDadXDlAHQ6GwfPYWtNAqFhgZ7xurB2baxpI+QP8Yf4pLNmzi/GKqoriupzesRVt3HvFQyDcfWOHv9jSDkA8m6ainBXE83BK4RYCWNU0Dd0uleEqCsq4QHH6UD+KgNFQmLUNU9lc20hzYqS0bZ6Hrag0pNMEiwVq0ilah/oJFov4rSJTR4dY1zCFB+aeTLC4f4J+QW8Xum0TyeeIFgsUdR+nde7k1D27qM5lWLpnB3geNekEnzq79Hd1PPxdv9bzieKJgx+Tgbyc99b4MfCIYRi3AH8N3ALs7WEYB1Sga+/ChmEsAp4qvwwAs03T3Pt151zDMEaBIOADflh+HE17gOqjXKZ0glKE4AfvreOVPQXiIYWtAxbL1ocQrSrXb9xEB0Fcn0a4OoA/XSQnFAYCAQLFIgOBIM39fWxuaMJv27ieS8PIEH2RCiwhKCoCVyh4nkc4l0UXCrFUippsill9XeyoqGY0FCLrD1CZTbOqoZm/veo6alIJ9lTU0DLQy7a6RhSrSFpRuXDDK6xrmIrmuUQLBVY0t/F883SmDvcTy6boCcfIazrRbIpooUBeUdhZEeecTWsItM7EURRChTy2olLUdWwUIvkcm2rqqci0MhQMMhqOkVN1LFXD5znkdB/9/iCKItCLBXKahuu4fOGSa3jHpjWc0rGdZbNPwnNcYvkcw5EYddk0l6xbSU9dPeGQykjGZSASJR8No6U8NjVPYcaePZy1dSPpQICEP0BfNEoslSJQDr5CVp7aeICOrMvMvi6++PZrGAhHMXZuZTgSoytWQctAH2d27aInUkFPRZxIPsueqmpGAkEEgqpsmow/QEDX+NVCA3NKC1NGR/AVCzSlErQO9ePaFq7rlvoF5TJYmk7rQD+D0QhpVeeX8xdTncsiHJdEIEDzyBCq49AfCvPSlBbyms7m2gYahwbojFQwomnUJIZJ+4KkAwFimRRLujqY39cNjsvz09pLwwoIhWAhj6so5DSd7RWV2K2ziRTzpFDw5/LM6e8mahVY3jqLaDbDaDBEbTqJ8Dw8AVWpJEVVx+9YtCWGgdKo45trG/CEoCKX4+Tcbgqqxm1nX4yCR2MyQdAuErIsqvI51tdP4cuXvpuZg31cvX4Vg+EIfZEY9ekkK6dMozscI1AskgwE6ais5uUp07hm3UoAiorK9ngtU0cGyQeDXDIr9Fp/ZpPeZMk8HUgGUW8B0zTXGYbRQanjeD3wEKWMFMAI4ADN45ZfDVQahtEMdLJ/fv7pcp8oH/AZ4HpKfauSR7HJe+uVJABURWC0lLIt02t8XDI3DNQBJx3Tdr213n4Y67ztqLfi0I74in7ZjKNUzpt1NgDvOQolJVIFbvvn5VTlMyy8eCYjxc2s/3k/52/eye1L38aq5jYsXSeSz/KJq69j2sgQqWCYkVApk9aUGBnrD9Udq2I0FAZgfUMzRVXl4s1rufim/8u1K58nEYrQPtjHvz9wH9d/8K/ZUVPPLxefQV+0gsVdHaxsbqUrWoEQgmf/ZcFR2LoT12QZofxAMoh66/wP8BPga6ZpOoZR+tAzTTNrGMYzwPuB299oYeVO398wDONS4KvAp49GI8t3CS4B7j4a5UmSJL2VKqJ+/t+/XzT2etmy7cy7sYnHf6DydyueYCAU4WsXXUU6GCSv6SiOQyYYwlFUcrpOd7SC2myG5sTwQeOvdFbWcP+8xTQkR8lOb+L2/zgHgB9dC+9Z/RL3z17AhsZm1jY0s7GuCUsoPPiFGdTE/G/hHjg+TdaxbGQQ9db5BaXszspDzPtHYLlhGHcA/wrspHSJ76w3UO7NwGOGYXxn3CW/N80wDD+lr4PfBl5FBlGSJE0iX3jwprHnF42bPv+WrVSnU2QiodJPu7guA+EoiUCQhV27SPkDDIUjaK6L5jrkdT/BYoHWln39iP7mf28A4COAW+5jJYQYu/NPmrxkx/K3SPnut8dM0xw5xLzVlPLxOvAMkAK2U/qbvAZ4zeCofIfeckrZqDdLNQwjbRhGklJH968DPwPOleNESZL0l+CZz7aR8euEClk0q8CMgV6ihSyDkSijoQjvffVF2of6CdoWiucRtAponstpJ9ccsjxFCBRFkQHUQcQhHic+OWK5dKzJN6AkSRNm2bJlAFxxxRVvaPkbP/owJ/V18fv5p9ATidGQTlHUNIqqSsoXYFpimLbBXr7+T4upm984kU0/ViYkuuma+vGDPuundP7whI+kZCZKkiRJkspqM0ku37Cau371Y65d+zL9sQpGQ2Gy/gC663DN6hdLQ2NMzgBqwkzWcaJkn6hJxDCM9cC0Q8zqME1z/lvdHkmSpBPNB1e9wMyh0rhhzYnR/ebVppNsPc/g/ptPPRZNO6FNlnGhDiSDqElEBkqSJElHpi6zb/Tua9at5L/OeTsIURpQM5Pmm994I/f7SAeaLJmnA8nLeZIkSZJU9sD8k8dGZH+6fQ4FVSWczTC9q4Nbv7QInyZPm4dDjlguSZIkSZPcRx/9O5770wXced8O+kJhPt6S5Z/+aj4gE/1HYrJmomQQJUmSJEnjnHX5XM66fO6xbsakIkcslyRJkiRJOgyTdSwbGURJkiRJkjSxZCZKkiRJko4/ruPymwu/xeZoHVduXoPPsSm6HuqtHwLfsW6dBDITJUmSJEnHpcd+/DIPT5vHVx//I7pburfOA4Y/9RO8712KUCc+C7Lj3hU8e8cqOmrihC5ZyNL5VZyztGnC6z1RyD5RkiRJknQcev75LuJZi/X1U8jpPub1dVORzxJwXbI9RcLN/gmt/97zv8ODMxbQdfJp1GfS1D2+nf6HivwkUsFdP7l4QuuWji054IUkSZJ0QutTA5za08mL02awpqmFP85fjO44ZP1+iv6Jv543qgbwOQ5TkwnClkXGH2Q4FCXv8/GC2T3h9Z8IvEM8JgMZREmSJEknNN2ySAaCY6+zPj+DoTD9oQghtzjh9b/aNLXUcdrzyKsqeU3DVRTyisLmhzdPeP0nAk+Igx6TgbycJ0mSJJ3QapJJpg/2010RB6AulcBsbuPknt30785C/b7LeXbvMD33reDuHj+Fjf3MG+hlbn833bFK/jD3ZJbu3Iqt6+yqquHqf1jKhuW7eXFjiqnJESpyWYZCUZKKQAn5COeKBAoFds1eQPtgPzHXQXNdXCHojsaY1d/LlDkNx2q3HFfkYJuSJEmSdBwq+Hy0D/bgd2wcVWXa8CBPt82iM1bFA521ZAbDrF79PHOXPU13dS2NqQTXd2znXy94J7lAkB019QjPpTabIR+OsKeiikQwxK9+tI7egJ8GPEajFfRUVbOnspqr15pc8tJa0v4APZEo79y6HkcIlrfNpiNegwLEczlUAfVu4VjvnuPCZPmZlwPJIEqSJEk67jiPrWH1tx5md08WLaTzjic/i+Y79CmrK1bFK02t9FRVk/b5eaVhKrtq6vDZNv/vqYfQXYdb33YJmxafjqWVyrAVhYu2b2RXvBYBCNclZNtYmk5tJk1e92ErCrXFIqpX6sETtG3es/oFqnI5bj/tPJoTw8zp6yaj+3i+bSaeUKjMZRkNhvA5Nif17GFgZXKsna/sSnHXFx6jPp3CUgQd8VoKmsqd/30RvrfgDsJjSWaiJEmSJGmCWUWH+8+9lfO3buCUYoElQFLX+eHlt7O7spr87Cl8/MYFzGuLja0TzmXYWVNPVteZmhgm5QuS9PnZ2TiVtN/P93//M87dsZnfLDptbJ3OyjjNo8Njp/bxp/h9nYU9OqqqqcxlOX33dnK6H3+xyO9PWlIqo6qawWCErN9PtFjKOAVti6TncdbOLVy+aQ0/aL6UC4CC5fDTzz/CYEUVO2sb6I9EGQ1FALj2k8/RruT5j2+dj6arE7Fbj73JGUMhPG+y9JE/MoZhPAWcAViAA+wE/sU0zfsOMX+8M0zTXGsYRi3wb8ClQAWQAlYDHzVNs6dcRgD4IvABYAqQBB4EvmSaZmd5mdZy3VNN09wzrn37TTcM4yvAzUCe0o0OA8A9wFdN0/TGrdcIfBm4DKgBhoAXgG+aprnSMIwbgDuA7AHbdZtpmp97E7vwcMk3oHTU2LaLpimMjGTZtX0Yn1+jMqRR1RCBZI6+rYNUzazF1nXiQRXH70NVxNhgyp7rIhQFcYSdXj3P26+MTLZIQAHHsvBFQmOjNxfyFj6/hmvbFEez+EJ+cukctqqjeR7+iiDZZJaeVV0Mbuyhcck0ambU4mkqPbuHyDy7hcYaH+qUegpVEZRUjmhDjHBLnKLlYRUstj26kaf/sJWEDbZQqC+kaQtBTtWwdwyS9vvZGq/FUVQ0x6EhnSDj8xMvpumtqmGOkiXeWo3vzFn0v7qbbNKm6eRG5nlpAhURqi+cy0jGIR7xIbTSvitaDh2vdoMCen0FuucRqwzi8wmGejKsfrUfJZnGF4/S3F7F9OYIu7rTdO5O8fv7tvDuFc9yXsc2AJa3zWJFSzs+x6E7WkFftAJLCNKBANXpFKlgmFguy4L+Ls7euYWle3aV9q2qsmLaDJbNXURDOsn6uiaEgIw/gOK6fPCVFSybs4ioVaAplcDzPDTXRQA5TWdTXSOuELgC/u2B+whbRVzg7iVnsbl+ytix3VFVw8zBPmYO9e89+Lzc2MIFOzczc6gPs7EZCxVXVfEJD79jk9V8/Pi083CFIGhbVOayLNqzk201DXz08nrOmx8nHA8R0uGZ7Uk++pvMfu+v81rgv6+tI+8oRAMKytHtpD0h4c6GmZ856LN+3tb/POFDK5mJ2t8/m6b5dcMwNOATwL2GYbximua28fNfY92fAQlgsWma/YZh1FEKqDwAwzBU4E9AA3AdsBKYCvwn8KJhGKeaptn1Jtv7lGmaFxmGIYCzgYeBXcBd5TqbgJcoBXOXAZuAIHAN8K5yGwB2mKY5403Wfcx1/PIlvv1gP9FslpBnc9oVc+iIVrHp92uYOjRIVWOMy79yMZXNlXjpPMW/+jb285vorKpm87svYu3ONF7BJnbyVP7hlnPYuD3Jd+7eTL7gcuO1bZwdyJH/q+9g947yy4VLeXjOQurifr75Twv5+bLdPGsO0NYc4XN/PYdoWD/Wu+M1pYayfPcfH2JjoIqG1CifWv4Imuvw8rsv4Z23vJ3EjT+kf2MfD8xcwEAowkgoTLhY5Kbkdmbf9/e423tJfPA7bPRCrJ4yjbyusz1eSy4U4rqPLuSiU6vZ9swOnrntOVRN4Up9FP8TqxFNcdyRNEJV4bs38a2OAKs2jNDW10vr8AAA3bFKGt4+n6suaOJ/fvgqo0NZ6kdHUXSFCz9/IWecHOfz317Hqb95mFO6OuiuruH7p57Hou4O3rVuFY4Q/ODMC0j7A+Q0nZzuY0pimJbECLai0Jgc5ZIt69geDHH70nPpiNcyZXQLl29YzeMz5+IJhfbhAZoSI5y9YzP/euGVjIZCNI0Oc+G2jbzcMp2G5ChtI4PcveRs2kYGqEsniebzdMYqEYqCC5zZsY0ZQ/30RWLcceo55HUfUxNDRAoFovkcswb6OKW7A4BH22bx0yVn4XNsCpqO6ro0JUeozOeIFAv0hWP0RStQPZfm0WHqU6Psiteiuw7DK7N0V/QiPKjKZWhKjuAhEF4PZ+/cwtz+bm459x0MhSKkAwGEB/Fsml1Ns6nOJJnb10Pe0diacNkTq6A9XOBjLz7NyuZp/OKUM7CFYFskQjyT5ppVz1OVy3D3krMZGO4n+HI3W2vrqUmnuPRnf6A+ncQWCs9Nm84dp76Ns3Zt47pVK8hpGqOhMO3Dg7jAw7MW8L2zL8YVgsXdu0gGwmPbFikWWCcUeqIV7IrXoHgeRT3CC22zOK9jG0P+IC9NaWUkEGT1lFYEHjP6e1EUQcBx2FVb6rRd0HVGkiMs6ukce9/7HIdFPZ1sr6lnJBShIZNkZ2Ud7NPfAAAgAElEQVQ1c3u78NsW6+qmUJ8aZUtdI2fs3MpAJMpIKEJPtIJUIAhCoAA+u9RZHEoZqqZUgpda2qko5MlrGrviNXRWVVPUdIJWgemDfXz18T8yNTGMoyi0D/Xx0ffcyEVbN5Dx+ekOVNFRWc2vf3obPsfmXy+6khemzaA+nWRqYoTvPebn2XvWsLF+CjvitYf8uZSndsOcb/XvN01X4NGP1TOt6vg8rU/WPlFyiINDME3TBn5MKchc9AZXOxO4yzTN/nIZ/aZp3mOaZm95/geAc4CrTNN80TRN2zTNncD7gDTw1SNor2ea5nJgPWCMm/U1IANcY5rmetM0HdM006Zp/tQ0zS8ebn3HA8/zuOc3O1Bsh6Bt4TkeL/x+I8/fv4Vp/f0ojktizygr7ngJAOcnj8Iz69Fsh7aBfoKPrMKfyaF6HunVu3nppR5+9KvtDI4USWdtbrt3G9bn70HtHMBvWVy/6nmi+Rz9wwX+655tPPZ8H/miy8YdSX7/2JuNfd9af/re82z3xRgNhfnkc49Rl0kRz2U551f3Y37rCayXtvFs83Syuo/+aAVFTWckFOanoRa483GKN99LcTDNS9PaKWoaiufRkEqgF4rc+fNNeK7HE//5NPlEntiOLvzLXoRMAW9rDwym8PoTPPKfz/DiuhEsF1pGBlE9D9XzaE6MsO7ZDu66az2DgzlsT9AVq6QykeTXt73MMysH8a/YyJkd2wnYNtP7erlg+0au2rCaSLHAQCRG1h9AgfJJrMjMoQGCtkVDKsFVG14haFs0pRJcueEVALoq4/RHY7QPDzJjeAAB9FRUsbOmjve/+gIjoQjpQJBnp88ir/s4o2M7P1l6Ln7HpjqbwVFURkNhfJ6HpyhMSY4yr7+nNE5QYoTzt2+kITVKyLJwFYVEKMyswd5SvxvgbTu3MHVkiILuAyEoahphyyJaLGCpKn2xShACR1EZiMQYjMSIWEV8jkNPrApPKCieR+vIIMIr9zQRgmenz+YP808hHQiSDgTwhIKjKOyM16I7Dgt79hBwbFxFwVYUpiZHuHLDK2iey5Mz542drDXP491rTaYkS9vwsRefpjE1yqa6RhxF5ar1q2hKJVA9D5/rYOzeha2ofPSlZwjaFvF8junDgwhABd6xZR3z+7sJWUXSgRAK4AmFzqoaBsJRcrrO7qrq0slICAQeT86YR04o9MQq6YzXMBKJMSU1Sl+0klXlwLa3onLsPZ7xB1CEoC+y7xKfIxTyqkZPtAKAmQN9tI4MoXserqphaxojoRCLu3ezo7aeRDCEpaoUdH2/2+9tVaUrtq+uzbUNrGlo5unps1neNhtL0wkXCzzZPodVU1q5eOt6WkeHSvvHcajI5zln51ZcIXhh2gzWNTbz9Yd+w6KeTub19/D9394DnkdO15k92Et/tJJosYDRueNN/Z1bLnz+weE3tc5byUMc9JgMZBB1CIZh+ICPl19ueYOrPQP8h2EYf20YxuJy5mm8y4AXx2W1ADBN0wJ+BbzjCNqrGIZxPrAAGD8oyWXAfeU6jkupVOqwn3vj/uWAqXtZRbs8+bWXE+XZ+13a9sCx7XGvS3/2AJa1/+50y+sdybZM6PNxm6WUv1EDKN6+bTnUmC2eEOB6OLZ90F4eW7q83/buu9f6WHy9XgMC76DD6FEuFxAHrKyUDhYA7iHavfc47Q1a9luvLGDbB63rCYHiMbbu+LVdIQ5qx4H1ja/nwFY5+9UlSt/Kx5W3t21i3LYd2GaPfcdpb/C0X6meh+44h2gf6I491jl6b1m2UMiXO1lX5vZdzXeEIDDuPS7w0F1nrC7fuDrEIbb/UBTPxRMHdy4+VH8kEDQnhumM19AZryGvlwbL1FyXeDZNxudHAEFr3/hPPttC8TwemzGfh2fOZ1t1Lc+0zeKzl7+PRDCEcF3Sfj/NyZF9bQKihQIK4LcsWkeGiOayDIciDAdDY8tV5LI0pEYBcIFIocDcgR7U8qW/BV0dzOnv4aS+LmoyKTbXNmIp+06teU1jNBgk5Q/gKgoCiBXyY/PDxQKK53L+1g08NnM+Iasw1r43a+8hPpLPjYniiYMfk4EMovb3RcMwRoEc8HXgJtM01xw4f/xj3Lz3Ubqk9xHgeWDIMIzvlPtBAdQCr5Wy6AbqDqO9545r7xPAncAPx81/vTrHaztwuwzD+OBhtOdNi0ajh/VcCMH172zB0jRymo5QBEuvnMPSS2eyq6YWVxFEGqKc/ddnAKDeeDHemXNwFIVd1bWkL1hMPhjEEYLgwqksXdrAx97bTmVMJxhQ+Zv3txP45g24jXEsVeXni88gGQhRXenjUzfM5byldWiaYGZrhKsvnHJE2zLRzy/7+9OZXkhSkc3ygzMvJOkPkNV9PHPNJZz+TxejndLGWbu3EbAsatNJdMcmlsvyV8md8JELCH7jevzVEZZ07kR1HVxK3/iLPp0PXzcLRVU4/9NvwxfSGWlpJP8OA/w6oq0OYiGIR3j7p85iydwKVAG7q6pxhMBF0BWrYs6Z0/jQh+dRWeVHwaMpOUoiGuXd/8fgbafUkDltDi9NbcVWFHbX1PL4jLn8cd5icppOU2KEUKGAB/itIjndx67KGjxgJBji4ZnzcYRgKBjmgdkLqcqmWdqxnVguQ2dFFbsqq/GA+mSC9oE+fnnyUuKZNNFchjN3bUFzHJ5rncmHzGcZCQYZCobB84jmc7ieh3Bd9sSq2FJTjysEvZEYT7TPpTdSQVFVwfOoSSXoj8QQnocLPN02i46qWvyWheY4Y2VZioLmurSMDKLbNn6rSHUmRWNiGI/Sh3V9chQ8D0tV2ROrQnVdwMNnW1y66VWu2PAKdakEsVwWzXEIWkVO6u7EFgr94dL7wQPyms7W6lpuO+si1tc1cdqubSiOg+eVLrHefvp5jARC2IrCsrknMxoMM2OwD82xeWzmPPKaVg7EBM+1zURzHe5ceg45TWc4EGR9XWP5GMMjM+extmEqOd1PoJgHz0NzbNoH+6jMZfA5Did176aoKDhCMBoM4XkerqLQnNg/s5L1+TilqwPheSzo2UNjcoTaVIKF3Z0IoKMyztz+burSKYK2hSNg6ugwmlf68rCwe/dYMBzPpvFbFqFigQ++soKr16/iRvNZlnTuIBEMs7mmnuaRAT71zIPUZtKEC3k21TRw0db1nL9tE9e++jKffPoh2kYGyfgD9EZivDplGp+56jpufO+NAKR9Pv40bxGb6prYWls/th13nHo2BVXFEYJ/O/+d2IrKfQuXsidWyWkd2wHYUlPPm6Er8M3L4kf8uTFRJmsmSnYsLyt3HH+s3CeqCvgJkDNN87oD57+BsnyU+kP9FPiuaZpfNgzjXkqdws85xPL/DNxommaTYRhTgD3A9PLlvr3LzKSUFWswTbOv3LH87HKfKB/wGeB64HTTNJPldbqBO0zTvPl12noDcPMx7BMl34DSEfM8D9eFfMFGAHs6RrGyefKjBRxNo63Gz66iIBpW8RxBpZsnPiVOUVfxA6qm4gkPVdewbBfXdgkEtLGMmKIoY53FxzqNe95+/VUcp5ytEcrYKcL1ACHI54vYOZuiZROtCuEr36pftFxcBFoyiZcpMqroOBkL4VfRBGiWBbrKOnMPieVbaXz7PBrnNqLkciS2D9HRl6Uh6GGjkxnNMe20FnK6j4jPI+sLUlHI05m02HP7cnZ3Z5gy2M9oKMyeuTOozyTYnNHRVIWiECzs6WQoFGV7XQO2UGjr28O0TJLB9mm0t8fIZIr0FxQKDXFq7SxN9REWNel4S2dTTFk0tFRANo/l9yNch5HOYbJDGV7udqkVOc68cAYpB7o7k/S9sJNQMUOfCLDwjBbCMT+b1g9ANMSqO1+iZdsu/Aiu3vAKLze3saG+CUsIhkIRKgt5tlbGidg2tqqS0XRUoD45wgU7tjBnsHfsmLwwtY11TS1jr0/t2EZDOkUiEKQhmeDfzr+MU/fs5N3rVo0ts6Klnaenz2ZzbQNdsSoevf0/xrJve2JVfPXiq/HKmSbNsSkqCiORGBtrGyno+/pGzuzrpn24nwXdnfz2JIPeWBVzBrrpqoyT8gfxANV1SQeCaI7N1OEhpiSGiUY0PvnpJVRX+REIntg6xBceGZcVB+oD8L1rqmiu1KkOK/i0o3pH34REN6/O+exBn/Unb/r3Ez6SOj57oB1jpmmOGIZxE7DdMIyrTNP8w5tcvwj80TCMx9jXp+oh4HbDMKabpjl2sbvcif29lO7SA+gBCsAMSnfj7TWDUv+m/XsT7qvvG4ZhXEqpb9Wny7MeAK41DOOrx/MlPUk6UkIIVBXCodJJbNbc2oOWOXgKHOpnaX26CuXbzEW5bA7x/4EdflV1/ImsNG9vqj8U8kPo4Nr8vvI6NZVQ89rp6LNba+HaxftNq5/fzKzXWH68WuCUM6a9gSWPwN6dGwuVTyoqjbNKnb7bxy0WBOriITj54FG8p8wsZV4uuKCNr/39g9zw+2X8dPEZ+G2LuX1dNI0O88c5C+mdN52v/ftFhEL7fhPva9f8HEdVCdr7/8SLo+w7Jh6wurGFyzavJWhZ/GnuyfTFKljf0My71q0aixwGwxFcIfjBb+9ma3U9lqKOBVG1mSSesu+4W6rGSCiMBwTs4lgQVZtO0jg6xEgwTOyi+Tz52SVYtsv1f7uTYKFAVvcRKeSpSybI+/w4rksNRT7zrimc8o45+23D+0+p5/2nvIFjIB0TMoh6DaZpDhuG8S3gXw3DWPbnli8v+wtgLVAE3gacD3yjvMi9wI3AHwzDuJF9d+f9B6UhEb5Srtc1DOOnwNcMw9gB7ACmUwqO7h4/fMEh3Aw8ZhjGd0zT7ABuAV4E/tcwjM9TymQFgCuB+a+XoZIkSToWhBDc8v3LSN1yDjXvuo3I8DDDVVUsfOXr/K3/0Kesc7eu4/4Fp/LA3JO58cWn0V2XlD/AnaecRcwuECoW6YnE+O/f/RS/Y1NUVS7esp5IMceTM+bz+Iy5nNq5E7O5lbUNzVy4bQOxYoHN9Y14isLSPaXvsytaZpQzkACly5WRYp5kIERzYoSAZaG5Lp974n6Wdu7ge2dcQF9LKwC6pvDNzy7iu198iq6qOG0DfRR9Pmyh8PXvv51YPHTIbZssJuslBxlEvb7vUsrqfKj8+kvlYGS895umeT+lL513Ai2U3i9dwK2UhjDANE27nCm6Gfgl0ERpnKiHgKV7x4kq+xSlsZ0eofTltJ9S5/N/fr3Gmqa53DCM5ZQCrhtM0+wyDONUSsHUI0Cc0jhRKyiNabXXdMMw0gcUt8w0zQ+8Xn2SJEkTJVob5R3LD/y4PbSYZbO0czsvtM7kGxdeQTSfo6DpVOezbKqppzqX4X3T4d6Fp3Lm7m2MhiJsqa3n9F3buGjrRjTPZSQQQnNsPvvknwg6Np0VcVY2TWPF1HaSuo+dNfUMRGOlsnUdn20j8GgZHWZtw1SSgSBzBnrRXBcF0F2Hk/p7iKv7cqDTZlTzrV+9e4L22PFtsvzg8IFknyjpWJNvQEmSjsjvl36Ds3dtZTAcJR0IorgufzhpCT7LouXqCJU1KldccQWe55Fau4e1y7fyp1eT5JIeVfksVbk0Od1PLJ+lYXSEkXCYzoo4V/7jGax5eBubtyRR7CIzhwfwPJdVTa2EC3lsn4+C7kN4LkVfgLzuI55J0ZBKYKkqum1TvaCej3zzsmO9i96MCYl2Xpn7uYM+6xdv/OYJH1nJTJQkSZJ0Qsv6/YwGQ9TkMtTmMnjAvN499IejBDI+qAkCpUuFsYVTOWvhVM56g2UvOm/mQdM+fMDrmz7yEMPBMEv2dKB7zljfqIKuE+wcPPwNm0Qmy914B5JDHEiSJEkntI21jWyvqd9v3Kmm5CiLujooRg91+8DRZQlIhML83DiTnljF2HRXCGa/c/6E138ikONESZIkSdJxqKuhgQdnLsAdNy2r+VjQ143VceDPgh59tU6BBd27uXqtSevQIOFCHhdY0ziVxR9cMuH1nwgm6zhRMoiSJEmSTmhve3sbLaPD/Oj089lSU8/zLe2c1LOb7mgF1a0T32vl1ruv4oYPzSEdDtJRUYE7u4n3/MvF3PeDt0143SeKyZqJkn2iJEmSpBPaDVe38s17fUzv7mN101TO27aJjKIS/6d3sqMq95a0YfGlc1h86Zw/v+BfqMmSeTqQDKIkSZKkE97nfn39oWcs+7PD/Elvgck6xIEMoiRJkiRJmlCTdSwbGURJkiRJkjSxZCZKkiRJkv5y/O3fPkx8Tz++xihf/tHVx7o5J7TJmomSd+dJkiRJ0gE+/qE/Yec8+mvqWK9U8dEPyb5V0sFkECVJkiRJ43zkpocIOBa2UKjOpGhOjDBYUX2sm3VC84Q46DEZyMt5kiRJkjSOV3TIqRqnd24nZFkA6LZ9jFt1Ypusl/NkECVJkiRJ41ieS4XjEi4WUV0XTwiaEiPHulkntMmSeTqQvJwnSZIkSeN4QmE0EERzXRRA9Tz8js1nb37uWDfthOUd4jEZyEyUJEmSJI0jPJfWxPB+Y2wrwNrk5MymvBVkJkqSJEmSJrk1q3tYMDRAR6ySvFrKM3jAtnjt5EmfHAOT9QeIZSZKkiRJmlSyBYuvfukpoiGdBYs8VPWNn7C/f+vLzFAUIo6NOXUa0UKBs3dsZkFfFxsamiaw1ZPbZPnB4QPJIEqSJEk6oa3dNsInvr0BoekErDzTUwk0IEeeh3aoLDrHfd31Pc9jsHeUO3+wGheFoqZz1q5tOIrCprpGnp4xl4s2r8NS9bdmgyalyRlFCc+b+PykYRhPAWcAFuAAO4F/MU3zvnHLnAHcUl5OBTYD/2Wa5t3jlvkKcLZpmhcdoo4bgDuA7AGzbjNN83NvsH1FwAWGgOeA75imufI1tmO8M0zTXGsYxl3AdUChXM6e8jb8t2EY6XHL+8v/F/ZOME0zUi7/XOBc0zSfGVfvNuDrpmneNW6aoLSPGoAm0zTT4+adBzwJbDBNc/4B2/ogcCnwEdM07zIMo5XS8ciyf7J6jWmaZx5yhx1dMkEu/eV4ZQe8soO0ovKx5RZdldUorkvb8AC12TRtgwN4nsva5lbSmo6jaWi2zSmJXk7dvIVYPo0rFCxV4/mW6aybMYu4sBmsr6HCLtJ+cj1FVaPvqc3M2LSVUc3Pqimt+FyXtuE+6nJpNlfXE83l6IlUsrmugVgui8+2yasatfkcolig0raY0d/LQDRKTvOxuX4KarFIfT7FYCCCIjxaRkYYDoXZXN+E57n4HIe8puFoGrrjEM7nCDgOBVUDzyNUtOgLh9GEQHcdKrMZRv0B5gz0MBKO0B2pxOe5LNm9nc6qGnZX1+AhyKsqyVAU3bFRHQef52F7Ho6u43dsLCHwPA+/4+ApCp4QxFNJ4vkcYbtAMhBkV7QKv2NjazqOoqA5NrplYwtBxHHwORaVuSwZf4CM7ucjK58dO+WnfX5ebJmOz7ZZMW0Gt/a+yNCGPpoyCdRPXcnS6w0UdVL1jJmQaOfJU7560Gf9+atuOeEjq7cyE/XPpml+3TAMDfgEcK9hGK+YprnNMIy3A38EvkEpCMkB7wR+ZBjGdNM0b3mDdewwTXPGkbQPwDCMacDHgBcMw3ivaZq/O9Ryr+Fu0zRvMgxDAd5HaTu3mKYZ2buAYRi3A5ppmjccYv0h4FbDME4zTfP1AozzgelAGvgA8OMD5juAbhjGWaZpPleutwU4Deg+RHmzTdPc8zr1SZJ0JG57AL5wDwBfvOID9Da1oApByLLor6pmV30Tu2vq+cxTD/DQwlMpqipZTSNk2zxSVY1XcPjUs48igKQ/wI6aen4Xq0F1HK55YRWRYoGO9XGemT6bT7y8moW9e3CEIB2MsK6hmbkDfZzUs4drV5vUZDNYisL/fecHeHLGXPK6TlHTCOfznLVrK56AbU3NtA328cv5pyA8j08vf5jLNq3BBW4780KenzaTXfEaXFUFIFAsELYt8DxsVaM7Xks8lwUh8ID+Kj+OouCWB1rsr6zCUVT6qmtRHIdEKAxCsL2+ib9f/gjJcITZg31kdR9Ptc8hEwiWfn/N81Bcl8p8jmghj+46FDUdPI8+f5SGdIJ3bFtPyLLoqKxmRUs7CgLHdaCcNChqOvghYFukyocn4w/icx00xyGj64QtCwHYioKtqiQDAVwhGFzfz7XrVqIAPf/vdm57YCt/94sPTLZA6qibLH2gDvSWH3XTNG1KJ3wNWFSefBvwC9M0v2qa5pBpmlnTNH8NfBr4Yjlb8la2scM0zZuBe4DvlbM+b7YM1zTNX1AKiha/iVV/DDRTCoxez98ADwE/LT8/lP/P3nmHx1FdC/x3Z2abVqsuWe6WewUMl2J6CxCIIZBGAiE8YhIgPSGk8QJppLwkkEryEgjJS4BUSExC6L3EvgZsDO69qHdt35n7/piVLMlykbEt2b6/75tPd28598zuaOfsuWfO/TW+MdjNh4H78I1Ug8FwMPnpP3uKy0aN69mQNREOkwj6zun6WDEby6uY2NKABQS07vmSvnDl6z23oaJ0itn12wnlsmghKMz4Tu1NZRVUxTs5qs7/PWRrnTeKBGrMREJujopEHICA5/H+115CAKF8IsnyRBeRnO9o10KwfNQ4BFCQSXPhymWAf9O49I1X8Cyrx4ACSAVDdC+a2doj5OZ6zlEAQdf1n9DK17mWP9azLFLBYE99zrZ5YPZxTGppRABLxtaQCIV3bGArBJ5t01IQJZjL+gZRvr44k2Lu9s09CTLHtzUzubmRoOf29On+K+h7A3QtCw0kg0F+L0/lT0efQF1hjDdGjMZD0BAtIpTJMG/Lup5xI7s6GLt5C3Vrm3f1qRvyaLHzcThw0I0oKWUQuD7/crWUciowGfj9AN3vxf//e9tBUq8/9wOjgWmDHSiltKWUHwDKADWIoXHgq8BtUsrQQB2klJXAO/GXL+8CjpNSHjdA13uAd0opi6WUNnANO3ushpTOzk5TNuUjozy2oue147o9Zby+8Tql8S6aCmIAuHkvDkBDrKinjysE8UCQjO1g6R3jY+kUHaFwz1NlAM0FUcA3kHKWhdfLI7CtqNRXIW9cJAN9Y34KU0k0kHICtIUjPfX1hUV95gWwvB2SPSFwLavH89OtM7sIH7F612vNuLZmMvlzSDu7WDARgrTTV18tINWvf/9z6p6jf66icDbjv99C+J6uZII1ldXY2iPj2KQCQUZ1tlIbK9kh2wnQEI1hR3fIGTbX2z6WDxTm6by3zleklDcCMfyYogVKqWVSylPy7dv6D1BKZaSUTUDVXs5RI6Vs61d3g1Lq3n3UuXt5q/emSd3n0VvPkl4vPyilfDf+ctom4MNKqWcGOe9vgE/lj+8N0P5fQDuwUCmVlVK+CnyEfh4ppVSDlPJx4Mq8LnVKqdeklAPN+YaUsvd3yr1KqRsGqfegicVipmzKR0b5j5+H934PvWYbX3riH3z73IvJWjYViS4Crks4l+P4LetJBIII7aG1xsnlQHtUdHWysaScEyybnG2xrqySB2bNJZzNUJJMsHj0eEZ0dZCxLHKWxa9POJ0z16+kKxji4WlHEUslqGmuZ0NpJYtHT2BqUz11hcXcftr5CM8jlMtieS4ugtrCIsa1NVOeiDOxsY72cIS1ldXcfP5lXLXkRbqCIZ6YPIPiZIKaxnq2lZShhWBkewvN0RgCyFoWkxrrsLWmPRIlks1gh8K4kSgFqQRB10VoTWtBFMvTZCyB0Jq0E2B0eyuTmht4fvxkxna0MbqtlbZwge898ly0sPwlQq3pCIaZ3lRHWyRK2rFpLyzi1VHjKE0mKEynWDFiFFuKywi6OUKuS2E6RXVHG8lAkI1llUSyGaLpNNFMmmtfepL7jp3Huspqqro6iGVSgO8pO3PdKtaVtbChuIQlI8ci0BSnktRGCznm42cwYkzl8LjG9kPZMDgOphH1rXxMVCm+9+Ts/N/GfPtoYGXvAXmvVUWvPntiw1uIiRqIMfm/vX2139pDTNT/KaUWvJVJlVKulPIm4D4p5V292/JLi9cCv1dKdQe43wV8R0r5ud4B5nl+BXwX34janRdqlomJMhgOIKWF8NjXEfhffmfvpuvFA9b6z9OEgKPyx97wgZ7Szg79dww4YkafV+/r8+o8AObv1cwz9txlQMbSfXa5nMvFt7xGcWM9IWBcewttkQLWV1UTTqWZ0FTHlkiUs9avZuXI0XQFQxRkM5DL8dSk6eRshwtWvc7yWJSz55TwzGvttAcCJFxNJJ0ikMkwurmOLifIX+ZIGsMRgll/q5feeEIwpq2F1cVlXPf49Yi8527qPp7hkcjhsnzXn4Oe4kAp1SqlXACsk1Jegh9Qvh7/f/2Jft0vx/e4PnZwtezhffgeslUHe2Kl1MNSykX4S3u9OQd/+fOa/HIh+J9jIf57+L/9+j8K/AI/EP3KA6exwWAw7F8cx+Zf3xooUqGbmfm/J/ap/eSla/igeoHtsWKWllbwqYuSzJ9/Ll/eiznPueEFRre34FkWGduhIt5FWSKOGj2O0Z2tPQaUYXAcLst3/RmSPFFKqRYp5Q+B24CF+E/rPSil3AD8HD/w+SLgDuC7SqkNvYZbUspwP5GZ/amflHIssAC4GnjfHp6SO5B8HniZvuf3EeBZ+v9A9N/Lj9LPiFJKaSnlRUBEKXXgF74NBoNhiPnxAzuey1m4cOGgxhaHPIrjXUxsbWZ9uf+E4SujxrF81Diq2prRWhtDah84XLd9Gcpkmz/Cf/ruqny+onPwvS434eeJWg3cqJS6u9+4s9j56bLu/5iJ/fIxgR83tKcn3QD+W0r5BXzPVzPwInCyUmrRAP2+2K/ucqXUQ3sxx6BQSi2VUt6Pb8whpazCDyh/l1KqrndfKeV3gRVygIAnpdSbezHdqn4xUW1KqTG77G0wGAyHIX+7/TQu/q848zat5XcnnLajQWtaIiZ2yMwCB8QAACAASURBVNCXg5Js02DYDeYCNBgMB4xuT9T8+XsXyQXwnmse5arXF/P3OZJtJWUARNIpXGHx95+esofRhzwHxGX07xNu2+m7/oJFXz7k3VNm2xeDwWAwGHqRcT3eGDmWlRXVHFO3hYztUJBKcvu97xxq1Q5ZDtdfy0eEESWl/AW7DqqeqZTafDD1MRgMBsPwJSwEKcfhhK0bcLSmIJsl4QRMLNRbwMREHcIopa4DrhtqPQwGg8Ew/HFtm6Tt4PQKd+nO5G7YNw5XT5TZ7MdgMBgMhl586EPTCeeydOeV10Bnry1uDIPncM1Ybowog8FgMBh6Mf/0MWwcO4pOO4CTzbAxGuPMk0qHWq1DmsN177wjYjnPYDAYDIbB8NsfnzXUKhxWHC6ep/4YI8pgMBgMBsMB5XANLDfLeQaDwWAwGAz7gPFEGQwGg8GwC17+z1Z+9r9v4toOLoKI7fG/PzqTYNAEmh/qCCFmAO8GqrXWHxNCTAeCWutleyvDGFEGg8FgMADv+cRzOOkMTiZHLKj55GeP4c5frcRzAgAEtCblWiy44Unu/sU5OI5ZzNlbhttynhDiPcDPgL8BHwA+BhQC3wHO3Vs55gowGAwGwxHPFQseY1J9HWO6OhmRSdIkQlx9TyOeZZGyHSytydk2NpC1HT5z49NDrfIhhR7gGGK+Dpyntb4OerJZLAWOHowQ44kyGAwGwxHPqM423LzHSQDTmxuIujlSTgDHc+kKhUEIcDSRTJr2Lnf3Ag19GG6eKKAK32iCHTbdoO0744kyGAwGwxHN5255iYwd2Onu2RkK0xUKkw4EfQMKQAhcyxoOnpRDimHoiVoCfLBf3eXAosEIMZ4og8FgMBzRzH3qZVojUR6ZdhQj4p0kAwHeGDGKrG0TTacZ3dm+o7PWeK7LYZr26IAxDD1RnwQeFUJ8GIgKIR4BpgLnDUaIMaIMBoPBcEQTcl1emDgNB01drJh15VVMamlgUnMDnhA0R6IkgyEAwtkMYc/DLOYNjmHgeeqD1npl/mm8dwAPAVuAh7TWXYORY4wog8FgMBxWfOxjj7G0cAQVXR0UpSJ0hMP898srmNTcgBCQtSyOn13CzdfPoaU1xWOTZ5IKBAEIeC6VXe1Mam4AwNKagmyGrlCYSCZDNJdFA5nh51kZ1gxDTxRa6wTwp7ciwxhRBoPBYDjkWb14EzfdtYHyZJL6oiI8y6K+uIQR8S2Eu7Ksq6zmtfGTAKhpqmfp0hY+9NnnqNhaTwWahuIde+MlnQA5y8LxPMDf5y0eCFKcSqABFzEsjYLhzfB6v4QQz7ELB5nW+vS9lWOMKIPBYDAMW7TWpDMuX7ljKWvXd1DgeRRmUqwvq6IlWkjWCWBrj8JUEqcgRmkmS2OshLq8UVRbVMKV6gVGdrazLhQGoL6omIDnEmlvo62wiM8+92/WVIxg0dhJTG+s5XfHnowaPZ5Z9dvJWRYbSysY09EGwiLlWIRzWSKux92/eZ1r/mvOUL49hwzDcMPhX/d7XQ18GPj9YIQIrYdupVJK+TQwD8ji52nYAHxLKfXnXn3mAbfk+9nAKuDHSqnf9upzK3CqUmqnBFlSyquBu4FEv6afKaW+8Fb124863qyUmry3bf3rB9B1PfBNpdRf93COFwI3AkfldV8OfFkp9dzuxu1HhttSucGwd6Qy8Lm74fkVMG8afO9qeGEFLFkH0TD88t/QEYdYAcweD196N8id/sXhi7+FP78A1aW47zsN9/Fl2BvrsLpS5LRGl8Zw5k3FetfJZE6bzaPP1cLrG6l+6XVWFpSwaMJkarpaOfbJF3mjvJqVVaOIB4LMqN+G3LKB9kgB0xrr2Fxcxh/nnkR5vAvXtilMJUg7AQKuRyIQ4NUxEwhmcxTk0pQn45TFu2iKRNlaWk4yEGRMWwtTG+vYVF7JiI42zl+9nGdqpvFG9RgquzrYUlJK1glQnExy7LYNrKkYSdpxSASDNERjxINhAp5LLJHgmLoteEKwobQCAIHA1h6uJfCEhaehNJWgrKuDt695g/JEHDVmAn89+njiwRAIQTCXo6yrnZXVYwnksqSdAKPbW0kEQzQWFpFxdvgIzl21nPpYEasrRzKmvZVQLovwXIrSKQKex1G1W/jcMw9jATkhuPCaz+JoTdBzIb+c1/tRdtt1cbSH1pAT4AoLGyiO2hw9p4JIxCYSdpg1q5xp08oO5FV4IDgg5s79p9+x03f95c9+eliZVkKIycBvtNan7e2Y4eCJ+oZS6ptSSgf4OHCvlPJVpdRaKeV5wD+AbwNXAEn8ILBfSiknKqVu2cs51g9koLxV/QD2o477g966fhb4o5RyplJq9W7GlAI/AZ4CuoBrgYellDOUUlsOvMoGwyHKZd/2DSiADfW+AbWpced+XWmobYWnXoeXvwdTR+9o++Yf4ecP++XGDqzXNwH+rxmAAJBpiZNbX0/w/55m4fUf5Mk6wQ8W3kc4l0UCtaefz9WLnuXeY+fx9OSZPaL/UzOVmfW1zF+xFAuY1lRPRyTK01NmUB8r7qNiynbYVlJOYTrFxNZG2goKaYoWsam4jNZoIZ5lIbdt5IVJ08jZDhvLq5ja3MCmiiqwLRqLSwhqTTibwXNsnpoyi+aCQt/w0Jp4MNQTc+RaNtUJP3bXs206QhHCbg6AjGXz2qhxFKWSzGmsZVbdVqbmY5POX/MGaypH8FLNVL+v45AJBIlk0iSCIYQQbC8uJZzLEcmkGdPWQnmii0QwSCoYpK6oBNAEcxmqO9sJuC6pQBDPslg6ejz3yNM4fst6Hpk2m6xtM76lkaxt0xaK7JQLSOR/+2kBlrB2nGdXlpdequ3p9/C/N/HFL0gmTSrZ9XV0hDAMPVEDsQ3fobDXDJs8UUqpHPArfMPumHz1z4D7lFJfU0o1K6USSqk/AZ8BviKlnDDE+g0rHfvp+nP87+Ld+pqVUn9QSj2glGpTSuWUUnfiG4LyIKhKZ2enKZvyoVl+bQN9GMiA6k3OhSXr+shxH1rcp4tgZzeAoz1cy/+qDixZw4SWRsK5bE97dVcnpakkK6tG7TSltkTPl7wAJrQ10Zlf0upN2M0RcF3fy9NrXlt7ePm5K7s6ydk7fncvrx7jJ6DM0/tm4uu3I69Spte4tkgBbj6eqDwR7zFIAIKeSyiXoy1SgAZEv5WSwnS618lpuoIhgrncjhxOQCCX5fgtG5jRWEtVvJMJrc0E3BwF2QzasihOp4jkcjhaU5BJQ36Ou048g/dfeQOrqkYyu2E7kVyWonSK4nSKTK9PRXgeVn5M/8+q/2vP06xd56dHGDbX7R7KRwpCiGv6HR8H/gm8PBg5w8aIklIGgevzL1dLKacCkxl4ffJe/Ov1bQdJvZ30y9cNKx27yev6MfylvaV76N5/7FFAOf6y3gEnFouZsikfmuXTZ9GHmePA2s3P7XAA5k3vI8e+8sw+XTTg9RuWFfkAZ8fCPXMOaypH0JE3XjxgS3EJ9dFC5JZ+Rp3WBHPZHoNFA2sqqihN9o9sgJQTIGPbxNLJnjpXCHKWje36D/NvLyqhsqvD19t1OX/VMt8IydNbb1dYhHIZACzP6yN3WmMtdt4I6QyF+6znxwNBUoEABZk0FvDK6PG0RgoAWFNexerKamKpJJFMhpJkgrJEnIxtE8wblQKY3FRPyOuXgEDTIyec3WGAWvkxGcumvjBGSTLBihGjaSgs6unjeC4WmqwQ5PCf3hM7xPafpg+BgMXMGf5y3rC5bvdQPlBoxE7HEPPBfscFwIv4++jtNcNhOe8rUsobgRj+TX+BUmqZlPKUfPu2/gOUUhkpZRN+2va9oUZK2dav7gal1L37ql++rXI/6rg/6NY1A6wF3tW97Lg3SCmrgL8A31NKrTlAOhoMhwd/+Bz84EF45FU4ew58/jLfO/Xaeqgqhv/5GzR1QnkMTpgCH38HTOj3dfDxd4Blwd2PQ80I9AfOQD/yKu6GekRLJzlXI0YUEzh9Frz9OC45agIVSxpZct4oSl5+k1XhEtLVo/j7yZMY/6+XeMebr/LGiFE0RQqZ0lQHluBPcyTTG+tYV1rBUxOnU5ROMamxjuJEF13hCLbn4WKRsiysnAuex/jWJiY31NEYK2JdaRVbSst8mXVbGRWO4Gp4rXosKQTBdJryRBebSstJOwEmNdUzd9smXhkzgWgmzYbSSuqKS7Bcj4p4B6WdHQSzGZKBIA2RKNFMmrZwlK5QkPZQAZFUkrHtrSScABrNnSeeRcTNkrQdcgiKkwmytkMslaQ9GGJMRxtFyQSJUJhoJk00k+7jmfKA5kgU2/UIkaU+VkxNSyMC31AsSCU5qrGWteWVdOQNLRdBYTqFpT3iwRAegoKc74/KWpbvidKg0XhATlhEbI+RVQUcN7eKSMQmELCYPq2MMWMOvIFyKDDclvO01mftDznDwYj6Vj6OpxS4Czg7/7fbNz4aWNl7QN7TUtGrz57Y8BZionalH/tZx12RxQ+N6E8g37aTrvsyiZRyFPAY8CjwpX2RYTAcUdgW3HSZf3RzwhT/AHjnSXsn54YL/QPfM2JdumNcsF9XCzjt+Co4vgouP4q5vRu/cAq7Yy7w7r3TaFjyj0c28eAf3yTjCYoyKRKRKNPrt9NcWERTpICxHa24lo2VD/juDIbJOAFqY8VknACRbIaSzgThbJa2UIRz1yxnRt1WfnLa+awZMZpoNkO7EwCtmdJcD0LgCZtwNosW9HjPAlqTsmyE9rjtqycwflzxHjQ3AMPB84QQYuLe9NNar99bmcPBiAJAKdUqpVwArJNSXoIfrL0e37X2RL/ul+N7Th8bKv2UUn8H1hwEHTcCI6WUBUqp3n74yfm53zL5uK0ngAeUUjfuD5kGg8GwP7n4/PFcfP74XbbnXI87r7yX9TpCbXEZqWCIloJCEIKKeAcl8S4sIXA8j5wleGbCNB6Xx1GfdahpaWRkpICuUAQPCHg7FidtNL1DswT+F3sAbQyoQTBMPFFr8T++3Wmj2fFsxx4ZNkYUgFKqRUr5Q+A2YCH+03APSik34AdKJ4GLgDuA7yqlegcBWFLK/hGTmQOln5RyoVLKk1LuLx3FAG1Z/M0Q1wB3SCm/AHQApwALgOve6jlJKacDjwP3KKVufqvyDAaDYShwbItP3HclAD+4fiEjX32F5ydMJW47dIXDNBQUEkvGqeloIykEsy6dzXUfmEEymeOOdy/n0teXsLa8imdrppAWFiHdNzqt+86bth1yloXVP+7KsFuGgydKa73f48CHlRGV50f4T7ZdpZS6R0p5DvBV4CZ863A1cKNS6u5+487CN2B68/7834lSyv774SxUSr2fwdOjH77h8fD+0nGAti8ppb4jpbwI+C5+sHchvnfqxv75qvaRL+AvR35aSvnpXvUfVUr9YT/INxgMhoPK5+6cD/hLBAsXLgS6mD9/4BCYSMRBWDaLxtYQzmU5un47Qmvaw2Fc2wHt3/49IehyArjCwnUcfxNiw95zmGZ4H9JkmwYDJtmmwWA4gPhGFMyfP3+XfT525T/oCkWoSsZ76lK2TSoQJJzL9jzBlwwEiAdCaEvg5LL88Rf7JTZ5uHFArJ3fnv3Tnb7rP/Tkx4fMshJCOMANwBn48cs9ugxm25dhk+LAYDAYDIah4KizanaK2YnksjjujnQGfvZyv154Gtvrn4zCsDu0EDsdQ8ztwEeBZ4HjgL/iP03/5GCEDMflvIOGlPIXwJW7aJ6plNp8MPU5EEgpTwMe3kXzbUqp2w6mPgaDwTDc+OiH5/DPV9pwtKYs6W8y3B6J+oZS3n/SHRNla03QzfUJPjfsmWG45HAZME9rvVkI8TWt9Y+EEI8AvwRu3VshR7QRpZS6jv0QnD2cye+BVzjUehgMBsNwZoLXyTNjJlOYTiOAis52hOdRmUrgeK6fdT3vPbE8j4LwkHtSDimGgeepPwVA99ZmSSFEgdZ6pRBi7u4G9ccs5xkMBoPhiOfHd17IVbEWQukUZR2tVDk5fnHzMWS0pr6wiIzjp+vzAK09bv3qvKFV+BBDD3AMMSuA4/NlBdwqhLiZAZJn744j2hNlMBgMBkM3n7vxJD7Xr+6/b53HF761JJ/ZWFNRYvPN/z6F4qKd9yA07Jph6In6FND9iOVngTvxdyb5yGCEGCPKYDAYDIZdMGlCCX/51TlDrYZhP6O1XtyrvAY4d1/kmOU8g8FgMBgMB5ThtpwnhFgqhPi8EGLsW5FjjCiDwWAwGAwHGDHAMaTcih8TtUII8YwQ4qNCiLLBCjHLeQaDwWA44mmO53jPlxQdsSJAE+3q4pkfnTjUah02DLeYKK31A8ADQogYfrqD9wM/FEI8obW+eG/lGE+UwWAwGI54LrllGQXa48pXXuKcdSsJWRbnfvqloVbrsGG4Led1o7XuBO7FDyx/GbhwMOONJ8pgMBgMRzyj29u462+/JZLLkrYdbj33EpaNGEkqkyUcDAy1eoc8w80TJYQQwNn4WyxeCmzCN6auHowc44kyGAwGwxHD4jdbeN8tS/jDwjV96k/dtIZIzk9kEHJznLN+BR0FhRz7jfVDoeZhxzD0RG0HfpH/e4rWeq7W+n+01lv2MK4PxhNlMBgMhiOCsz/9MhurRqHDVbz6msuvn1Q8dbsEoCFW0qdvc0EhCIuCbHooVD3sGG6eKOCdWuv/vFUhxhNlMBgMhsMerTUbK0cSdF0q4nGKMylaYjE+e+sLANQWFfGTk89hyZjx/G3Ocbw4YTLJgINnG1/D/mC4eaL2hwEFxhNlMBgMhiOA2TevRESjFKeSvvdA+5sJL/JiPPjQepojBSwbMYp/zTwax3WpixaRs23SgSCvrWvjmEkle5rCsBuGoSdqv2A8UQaDwWA4rHlhWwGWbXPZssWE83FP4N8AXdvhZ0800xaN8erYiWRsh61FpTQXFdMRiZIOhrj211uHTvnDhOHmidpfGE+UwWAwGA4LLv7JWuq3JhjV2UZLQSEJ22beNk1rBI5zN/A///ozD804mu+e9Q5cIUgGAiSDQZKOQ3Gii85QmEgug6V73eKFIB0MDt1JHS4cpp4oY0QZDAbDYUQ8meHXj9dywawIlSOKuOeBjbR05Djm2HLOmVlCSSyM2MMNrT2RZfH6DuZNKSYa2v+3CdfTeK5LIODLXrW5k1AAtPB4alkXZ06PEA5YfHthHSXJODctmEUsGuYrP13Ko5s0WSCmM0wIeaxNOliBAIWpJGsqqymIRFhdEMXxPDTwxKTpYNlc+roC4B0rlnLequXcI0/ljtPPJ+kEcYVF1rIpSiWJZLJsKynvo2/Sdvj1E1u4+oxR2La1x/fPsDN66DOU74QQ4m3A5UCV1nq+EEICRVrrJ/dahtaHi1Ntz0gpnwbmAVn83Zs3AN9SSv25V595wC35fjawCvixUuq3vfrcCpyqlNppw0Ip5dXA3UCiX9PPlFJf2AsdJXAzcAoQAuqAfwHfVUrV5vvMBL4OnAVE8udxF3CHUsobQA8PSAHLgT8Av9lFv0Hrux84ci5Ag2E3fP3bi3m4I4IWgnPWLOfCN5eycPZxZG2btnCY5SPH0lxQSDibJZrNUJqI8+1//ZHiVIpPXfwBaotL0UJw8sY1TGhpYnushC2l5ZQm4wTcHIvGTqIzHEZoEGhszyOUTVOQzdEVDpMTFiE3hwDOWfU6L02YSmNhIVOaGhjT3kJxKklNSwM/n3c2HeECcraNFhYlXR3Mrd1KRzjM1uJy0o5DzrJwtSYbDKGBSNb37tjaw/I8skJgC4GtNcXJBNe+/BQ/OPNCkk4ALQRaCITWRNMpRnS2kwoGSQaCJAMBWiOFOJ7LmLYWxrS3kHKCvH3161R2dfLkpOk8PmUWtt5xy/YAW3tE02nuu/dORne00REKc9Xl17Js5Dhc2watKU4liabTdIbDdIYjfT8crSlNxolkMrSEI5SnkriWhYtgUksjxakE1R1tpAJBGgpjPD9hCtmA772yXJcx7a0ct3UDNQ31/OfDl/G795bjWMPPqMhzQBT78YV37/Rd/8l/XTNkb4IQ4hPAp4BfA1/SWhcLIWYBv9Jan7y3co5ET9Q3lFLflFI6wMeBe6WUryql1kopzwP+AXwbuAJIAu8AfimlnKiUumUv51ivlJo8WMWklG8DFgI/Aj6mlNompRwJLADOAO6XUh4FvAD8HpgNNOfbfgMcDXxoID2klIXAeXnZFwLveqv6GgyG/cOLz23lkfYQ2vbDVJ+cMpuQ61Ka8n/bLBs5hqbCItCaaDaDBbQXRLnnhDP40d//QEEui6M1SdvhH7OPA0BozfSG7RRnUqRtm5mNtSwZNQ7PtgGBa9sE0pqmwljPUouNJuS6PDr9aJLBIIXpJBPamgGIh8KUJRL86i/3cNmHPom2fF2Prt+OhWZzSQXasnC0RngeyXCYnO3guDm0ZeECLjYAjpvD8TwA2iMF3HbuJTsCj4Xw7+JCEA9HiGcz1MeKAf8Xl2tZuJbFhNZmitNJjtm2ginNDQC8e/kS1pZXUVtUgmvZPbLioTAd4QjvuuoTzGzYTllXJ6+OqenzGQityVkWwVwO23P98T2Ngo5QhEguR9jzyDh+8s1xrU2MiHcA0BYt5PjN63l08iyyzo7knJ5ts7msgs1lFXzy2UfYvmQr368O8sUzi/b5ejkU0cPPZvw0cI7WeqMQotthsBKYNhghR2xguVIqB/wK35A8Jl/9M+A+pdTXlFLNSqmEUupPwGeAr0gpJxxgtX4O3KuU+oJSaltez1ql1DeUUvfn+/zQr1bX59sySqnHgCuBq6SUpw4kWCnVpZT6G75xeFneYBtyOjs7TdmUj/hyc2MCLXZ8HfuemJ6XZPOP2Qut+3xpt0aiAHTkPScZZ8fvYi0EuV6GQMBzCXguvfGE6BOr4uV18Lq9JP18B54QTGus6+OqcDwXT1g9RtUORF5nBmCHBM3un9yyBlotyccz+fN7PdXbikpIBENoYSGAaCZFLJ3Eyhs+9bFiXhg3iZGdbYxvbqS6vZX/fuxBvv/Q/Zy4aS2W9rC1R0VnB3O3bNilTr3PvTdZ2ybo5XYZ/9MRjlCe7GJjc6qnbqivvf7lA4VG7HQMMTGgO7Fm90UWADKDEXLEGlFSyiBwff7lainlVGAyvoenP/fi/9cfMMOj1/z37qZPBDiTAXRUSj0NbAXevrt5lFLP4mdoPWfftd1/xGIxUzblI7587gU1VCU6IW8wzK7dQk1zA93mQXVHG7broi2LZN5Qsj2XaxY9A8C0hlo0EMple2RYnofd6yafsh0ytoPItwut/aW2TD6ZpNYE3ByW5xHOZvCA9nCEjSXlCK0Z3dbCuWve5K9zJEJ7PfNsKS7D8Vyqutp9MUDWssBzEVqTte0dtlh+TE4IcsLK97WZ2lDbM5ZeRpMLjGxvJZpOYeVjnBCCQC5HaSIOwJtVI2kNFwDw2JRZJIIhPMsiks0Q8DwcrRnR1YHtuWgBjtbcfeKZtBZEmbt9E1Ob6hnd0cYVr/2H47ZuJB4MkbMdVlWNxOo20LSmJBknnEmTDARwXBe0pjZWTNr2DdVuHePBEMHsjicAu89nXEsTY1ubWD5uAl88q/QtXzMHqnyg0GLnY4h5Fvhiv7pPAk8NRsiRuJz3FSnljfhWaBZYoJRaJqU8Jd++rf8ApVRGStkEVO3lHDVSyrZ+dTcopXZpIAGVu5q/F2X4cVq76rOdvdNxK9A7cnJf9DUYDPuJSEGAP3//BB56chsrWj0uuHwu4v44zSXVPJmOMW7TFsa11NMRjpC1A9QEUpR0dZLO5Xh0dA2rS8pI57I42SzFmSxuMEg4laCyvY1AKkVdrJhN5RVY6SQjujpIBUMk7QApJ4DtuhR0deJZFkJ7dDkOJ25YR31pGasqq1lVVMr2QJBjgU9c+B4Wj6mhsK2VeN77taUgSspzydoB2oUArbFyWQpTCQKuJhsMEEylOK52C8mgQ0O0hKZwiGwwgpPNMqWhluZYMaFEnLQTwAayAjwEtmXxRtVIJjfWUhpPosaNx85mcbTH4pHjKEt20RyN8cCsY4mmUwQ8r8dzJXq50QT4hg87PG+eZbG1pJzGaIyRne0EPBcPCOZyZB2Ho7dvoSTRxdOTZzC2uRHcHI1FJRQkEyTQRFNxElrzn+qxRJMJgtpfdvUKQ0SjNlfNDrKi0WVlbY65K1/nVKuLyb/+MK+MCRENHnn+i2HgeerPJ4CFQohrgZgQYhXQAcwfjJAj0Yj6Vj4mqhQ/GPvs/N/GfPto/HXRHvJeq4peffbEhn2IMeo9/4pd9GnB/3E2ehfto4An9mKuMfS1tvdFX4PBsB8JBGwuPX8cl3ZXzLmMufiRrzBn4EHfPwPwgx33L7P2u0Q4dhf1x+yivjcz99hj+bY47/r5FkrcLLbrkrIdbM+PH0vYDgHX9Q0orXsMqVgqmTeu4O8zjmHJ2Ilc8uYrjOpoY2txGfcdfSIC6CiI8p9vz96rsxyYXX1lHzkMA89Tf+qB4/PHePylvUVaa2+3o/pxJBpRACilWqWUC4B1UspL8APK1+Pv6NzfELkc39P82AHUZ7WUci3wfuDxXfRJSimfzet4V+82KeXp+MbRw7ubJx8zNQrY60c4DQaDYbgze3SUVd+a3qdu3boWvvLjV2jCoSswCixBKJcj4HmUJuJ0hCN856yL6AwEqYh3ML2hlrHtrQCMb2vmqLqtvDpqHIlQaChO6bBiOHmihBA20AWUaK0XAYv2VdYRa0QBKKVapJQ/BG7Dfyru48CDUsoN+EHeSeAi4A78FAO9Iw0tKWW4n8hBBaQNwA3AQillPfBTpdR2KWUV8GH8J+j+CHwOeE5K+VPgm/jeqdPwn867Vyn13ECCpZRR/JiuHwF/V0o9+hZ1NRgMhmHNpEllXHFuEoAvLgYQNOWf9KstLmVcSyOnrnmD5RNrSGsL3S+SPpzNMLqjlXBl9CBrfhgyjHJraa1dCq2j1wAAIABJREFUIcRq/LCW7W9F1hFtROX5Ef7Td1cppe6RUp4DfBW4CT/+aDVwo1Lq7n7jzsI3snrz/vzfiVLKrn5tC5VS72c3KKUey3uKbgZezy8j1gEP4RtJKKVelVKehJ8n6k0gDGwCfoL/5F5vuvXQQBo/T9Rt+HkxBuo3KH0NBoPhUOG1r05i6nf7bt9ia836kjIe/t4JzP3SG7RGotS0NTOyo42tJWUsHlPDxIbtPPKdo4ZI68OHYZgQ8A/AQ0KIH+HHCfd6/sEk2zQcOpgL0GAwHDAWLlwIwPz585lx61pSoTCeZRHI5ShKdLHkmzMQQvCRq/5JIlJASSpJ0gmwvGoU9cXFRJJJVnxtyhCfxUHlgLiMvnfJ/+30XX/T3z84lMk2d5XDQmutJ+6tHOOJMhgMBsMRgfpyDSd9YzWe5RBKp7jzihE9W7hkHYeSlL+4EMllmdjaSGs0Sm6n/FeGfWG4/VrWWtfsudeeMUbUQURK+Qv8pJgDMVMptflg6mMwGAxHEtGgzevfmDFgW0cwRDS5Y/ercC5LLBHnvKMLD5Z6hzW7S6h6KGOMqIOIUuo64Lqh1sNgMBgMfamPFuI6ASoScbQQZCybKbVb+PZtFwy1aocFw80TJYTYwi7U0lqP21s5xogyGAwGwxHPLRdVcONTKVZWjSScy6G1x9JvTN/zQMOhSv9VoZH4adnuH6DvLjFGlMFgMBiOeN529gReOcPj63cuozBi8/kP7yLBqWGfGG7LeVrrZ/rXCSGeBv6N/9T+XmGMKIPBYDAYANu2+NrH9yaDumGwDLflvF2QBgYVcG6MKIPBYDAYDAeU4eaJEkJ8vV9VAXAhe9j1oz/GiDIYDAbDEUN7PE3Q84jEIkOtyhHG8DKigLH9XsfxE1b/32CEGCPKYDAYDIc1//gjPHLvgywZP5GTNq8n5QR4buJUPlCV4ss3mNing8EwXM77kta6rn+lEKIaf6eQvcJkETMYDAbDYcuiJ3JcuvI1Hpt1DNqyqSsqZnxbMze8+CR/22rxnQUP9unvepqvXvsgt1z8W274wINsrO2/I5ZhX9BC7HQMMat3Uf/mYIQYT5TBYDAYDlu8DZ3ccsG7KUynuOSNV7h0+SvUF8Z4euI0Ll+2mP+MqeHOz/yT62+/CIBrP/wIbQUVbDpqOp5lcd13ljOnbhtJx+H4i2fwofdNHeIzOjQZhp6onaw4IUQR4A1GiDGiDAaDwXDYUtnRQWVXB5967lHetuYNLGB6Yy0Zx+Hu409nRfVo5i56FoDW9gxj2ltZWT0GR3to1+OMDWsoS/mZzOv+r5WfdKb5xAKzBDhYhoHnCeiTZDMihOi/S0g5cN9g5BkjymAwGAyHJbde8RdmpxNsLSnn6+ddyvlr3uhpK0kmebFmKpab46VxkwB4/tFVvDp6AsXxLpLhCK6GsmQc8gZARTLO8ifXgTGiBs0w8kRdie+F+hfwwV71GqjXWq8ajDBjRBkMBoPhsEJrzcUfe5501XjWRUpojxTQHingHnkqV6vniQeC3H76+QCUx+M0RWPcf9uTPLw2R5mA1VWjsLSmurONqq4OGmLFAFR3tpO2zW1zXxgunqjuJJtCiAqtdWJP/feEuRoMBoPBcNjgdSY56+bX2ThuIgjBqhGjetpuOf8yfnzq2xCeR8YJ8N7XXuady1/hE/PfT93Tq+mcPJMtZVW4lv/MVWukkMquDsa0t2JpjZPLsrGkdKhO7ZBmGHmiANBaJ4QQxwCnARX0ipHSWn91b+UYI8pgMBgMhwVf++IT/NUZjVdc3LMERz8PSFukgFv//QCz67ZSmk4CcMaG1Tw9aSaJULjHgAJoihZy5RXXccLm9WRtm/GtTWTc4WYOHCIME09UN0KIjwC3A48Cb8dPsnke8PfByDFGlMFgMBgOSRo6M3z/Z6/zTCO0lJRBdBKF2TTHbdvEyxOmkAiG+vQPZTPc/o97OXbbJi5YcCNfeOohpjXWMa25gUktTTxfMwXPcmiJFqKBeCiEFhaLxk3CQjOyo43t0cKhOVnD/uYm4AKt9XNCiFat9aVCiLcDlw9GiDGiDAaDYT/guR7ZdI7Ohi5KRheRbEkSrYjipnMECoK7HZtLZUknsoQscIUg40Fza4ryugbSZcUIx6IrFiWnLYLC4/7n68m1JCgTWZq2tlMeEayOVjBuxRqseILxy9fTEo7w56NPAAThbAYn59JYVMxNz/ybRCjMX+cchwdkLJsAmvqiEiKZNAXZDK6wWVVehSc0p6xfRVNRCS2BMO2RCIGATaWTJtOSojIZZ3NVFTXba0nbDptKKxE6x/i2FlrCBWwuryLa1cWx9Vup7mqntjDGiurRzKzbzoTWJpKhCK+PGE3YdUk7Dh2BAOlAiOqONkZ2tPNmZSWZUAG29kDD6RtWc8qWddRFi1hVWU11vAMvVkzdMSeBEAjt0eHYPDt5Bho4etsm3qgeTS4fx+QJi7nbNhPMuYzoaOO5mqlsLKlg/srXuPmCd+NaNuFcltPWvcnf5xzfE8fjeC7TGmpJCovmSITTPqOIZFIUZJNUpZKktMXZ61cRC7i0HjWNbeNGs6rLory2kY9eNJrq7duI17VTcsPbiYfC1HfmGFvioIUglcyxtC5DTUWQtaua6MwJTp1eRHkuTYMTpn57F8fMLGHjigbqU3CiF8c9fgqtSc3YUgchBF1pl0xWE3QE0ZCFGGZeHwA9/DKWV2mtn8uXPSGEpbV+WAjxh8EI2aMRJaV8GpgHZAEX2AB8Syn151595gG35PvZwCrgx0qp3/bqcytwqlLq3AHmuBq4G+gf5PUzpdQXBqFfN/crpRb06lMDrAMeV0qd12/8N4GvAD9RSn2yV30BsB0oBsYqpbZKKRcANyqlpuf7/B7oUkpd10/mR/Lvx1FKqeZe9T/EX389WSnVW9/eY6cDK/BT0GsgB6wF/gHcrpTqGqBfbxYppc7O9zkf+CowEz+x6vb8e/O1XvPNyPc5CygE6oEngO8opdYPpKPBYOhL7Zv1/OPmf5NJ7vi3Fp5H1M0htGbyBdM59fPnDDh22X1LeOXXL5MTFstGjmFm3TZ+Oe8scpbNp597gvlvvson33kFi8dNQgMVnR18/tl/M7KznVWV1fzhtPOxUyCSmkDZdE7pWM1s1yUFrB41nqzjgNbM2b6Z7z76AAHP5UtnXsDm0gpsz+OClUv509x5vjJa5yND/BteJJ1ibDxOVSbNfXPn9SzJlDTVcfHqN3C0Jr5tCydt3cCZ61dRFyvmysuv5bWaqdiuS0Br2ssreKqikqqONs5f9TrveP4JNpdW8IkXn+D7p5/PxxY/x0Url9JcEOWGSz/E2ctfoTCd4r65JzF/w2rOXLeCJybP5L65J/Hg3BOJejnOWr+KGS0N3HnimSweNxEAy/MQ+El+tBAIrXnf0kV8q2J+jxHlWhb1hUV89fxLmdzSwIJFz1KYTvG3WcfiWjahXJZwLktztIjzVr3OI9OPYnRbC2WdHWyoHMHaymqE1thuDkv7b9OWXI7iZJILnvgHZckEGzdt40PvW0A2EERXRXn5Py6nr49z/UvP0/KbZ7n99PNZOOtYbNelNBlHC0F7KMyU5gZimTQa+JPl0BUO8+UnFnLWupU0Rgv54ruvYW3lCJycg/v0ZjQQDNtceXQB/6sSaCGIZDJMK4KfXjOeESWB/XyVvzX0sLOh2CqEmKC13oifePMSIUQTkBmMkL3NWP4NpVQhfg6Fe4B7pZSTAaSU5wFPAS8BE4Eq4LvAHVLKrw0sbkDWK6UK+x27NaD669frWNCv/VqgFThXSjlpgPGrgCuklOFede/FNzoGjVLqf4FFwC+766SUZ+X1uGJXBlQ/JiilYvjv52fxN0ZcJKUsHqBf73PvNqCmAQ8APwEq8T+79+IbZN06HQssBjqAk4AYcCJ+xtYLBn3iBsMRysu/U30MKADH8xDaj59Z+++VNK2s32lcNpHhlbte9vtrjylNDfzqpDPJOAE8y+LHp76N5ydMYXH+EXwBnLlhFSM72wGY1lhH2M3l7R7/LvXO5a9goVk6apxvQOXbpjQ3UJDLEvA8GguLfIPDsnh86uwdCglB7xyEyVCYR6bPRm7diKV3xAIdt3UjTv71jKY6zlzvPxVe3dnOgsX+j/us7ZAMhnr0aigqYXtJGfFQmHesWEo8EGTJ2BouWrkUgPJEnKuWvEDOtinMpChOJfngKy8ytr2VReMm0R6JkgwEuevEM2kLR3i9egzPT5pO2glga43Vo7lACwvPsnlkyizioR1f62WJLj59yRUsGz2ef86cyy3nXUrA8whojXBdwjn/M9SWRWE6xfL/+RKnr1/FuupRuLYNWqOFwLVstGWhhUU6ECQeCvHy+MkAjGlvIesEej4v13a4esmLBD0XW2s+8fxjABRkM9hakwgGKUkniWXSPWPCXo6Z9ds5a91KACrjXVz7n6cByDm+BwshSKc97lLxHo9ZMhhkU0uW+17o+e0+bNCInY4h5nvAjHz568DvgSeBwdgtg9v2RSmVA36F78E6Jl/9M+A+pdTXlFLNSqmEUupPwGeAr0gpJwxmjv2NlDIA/BfwDXxj6doBum0ElgDv7lV3Lf657ivXAidLKT+UN3x+C3xOKbWrVPMDopTKKqWeAy4BqoFP7mFINxJoUkrdr5TK5Y/XlVK9XZU/Bp5TSn1UKbVRKaWVUk1KqTuUUj8fjJ77Smdnpymb8iFfDoQH+NXf7x7hRAI7jY0nE9gBu+d1zrIIuG7P63A2Q2H+5tpNwtm1h8ETFjnLl1cZ7+zTNqqjrafce45oJt1j7AG+N6oXQdelLlaM12uJqCO0Y/PeZD992sPdbf0CsLVHOJslmsmwsayCoJsjKyy8Xm9UezjC5KZ6HM+jNLnDyS70zsHcm0vKd7T3qu9dfmbKzD5jmgqL2FRW0fN6Q1klAKsrq9lQXoHwdiSr3lJazpXv/yj/mnF0j9zd3fYL0ykAbK376OsBycCO9yiTN2y7ewhNz2fWjQYSwb5LwJ2hMP3RgNVbqfzcDrkd4wZ5PR8otNj5GEq01vdorR/Olx8GSoFSrfWdg5EzKCNKShkErs+/XC2lnApMxrfg+nMv/jX3tsHMcQC4BCjD1/Fu4L/yhlV/fkXewJJSzsT3qi3c10mVUk3A1cCP8N+LV/Ieqn2V14Dv8Rt4TWBnFgFVUsq7pZQXSylH927MG3YnA4Na/93fxGIxUzblQ758+vXzqJ5e2ecu6zoOVjRE4cgiTrjhVErGl+00tqS8hDNuPo9gaQHxUIjG4iJuevZhJjY3MKa1mW/++6/IrRu57LXFWJ6HBp6eOI3nJ0yhNlbMQ9OPIh4I4uIvHwZyWe6Rp9JUEGV8SxM1TfWUJroY19JI2rLJWRauEBy3ZQNCawJujmsWPcuXH3uQaDoFaNAegVwOx80xsr2FUDbLMzVT/XPTGrTmqckzWFY9ho0l5Tw5aQa/OPFM1pRX8c/pR/HzeWejtSaSyfgytSaYy/K2VW9w4uZ1NBUU8MCsufx1juSUjWv45jnzWVUxgkVjarA8lxVVI3muZirVHW28NnIsAO9d+jKxZIKCTJobXnic4mSCcDaN5XkgBLrbaNF65z07+hlgvfMVzarbynfOvJBHp83h5E3rOX7LerKWRca2aQsX0BEpoCLe0SOjPN5JcSKOk8v21AVy/pJtXayI2lgxf5l9HOFMmpwQZC2LwnSSv884hs3FZbSHIvz3eZeC1sSDIdK2QziTJh4IUlcYI2tZuAg6AiE2F5fyk1POYVNJGU9NnMZPTjkXtKais51R7S0ATB4R4gfzy4nZmgAeI0WWc2bFuObcHWkdBns9HyiGoScKIUS5EOKDQoibtNYZoEgIMWYwMvY2sPwrUsob8Zd7ssACpdQyKeUp+fZt/QcopTJSyib85ai9oUZK2dav7gal1L2D0K+bC5RSL+fLHwEWKqWapJS/A24D3gn8uZ+MB4Gf5pfBPoK/bJnjLaCUelRKeR/wPmB/bLi0dQA566SUvb8l7lZKfVYptUZKeRK+5+oOYIKUcgVwk1Lqn+zIi7HTZ2cwGAZH0YgY77n9kn0aO+6UiXzglIl96i7rKR0PwP/kjx34XwPvAb7TT57WM9G502lri1Ndn6axJcObLzSyfUQ5t5eczuSGBq5Y/CwXvfkqz9ZM5XdHnUBHQZRAJk1lKs6M2u1sLS5lXGsTG0sriJaFKI2FOI8mxoyOcuV5YxhbHqQjNYpYyMKxLdY0JHhpQ5KQrfkBOcZWR9nUnGVDXZzyQmhd1kDVySUcf/1FZMaUc6ljs/jp8Wz41ypeKyznudHvobmygumBFMfoLlrjFi1VJbwk5lH9xhpEPMtZK5fRGCtmedUoFo8eT21JGZMaa8k6AeLBIPFg2DeogLGtzazuzg8lBOVdHTRHY9Q0N3D22jdZOHMuFV2d/Pb+X/PItDm8Y8VSTti0ji9d9F5aI1F/6a77/RQWkxtrmdrcwHGb13PHKW/jM8cI5kwrxvI8jj1qBC8ua6HsQx9k3NQyrhCCKwD3/9u78/i4yurx458zM0naJmm6t0ChC6VAWYWnQmVfrMgiIiIUEFDZFPSrLxH4irIXBBXBn/AFkZ0vqIiCRVAWAQFZPBYqfKF0p7SFrmmbNmmSmXl+fzw3ZTJJmqWZTDJz3q/XvHrnbnPuzTT35DzPfW4qTSwmUSfvvQiXAfhjp74dDgh/0bfoTJzhqD1ynwRtrXxXnrKJyCHAo4ACBxCa93YCLgKO6+h+OppETVfVa51zg4G7gMOjf1dGy7cDZmduEFWthmWs056Fqjqhg+u2Gl/2TOdc03fvWABVXe6cewI4j6wkSlUboyTr24Rv+/5djCXbLOCgqDK1tUYD2Y3dO7a1b1V9C/g6gHNuJKGz+x+jRHE1oRq8XWvbGmP6JhFBSuIMGT6Qz4TWKo4/uPX/5ke3OrdjjzQZVvFJc9MuoyrYZVTzW/933gHCJQA4qOXnH/eV3TnuK7u3mN9Sa91YP7GpLslVP32VR1Oj6Oc9n5vzDvstns9lR5/EppJShm6sYdqbr/GHvT7NwmEjeXjgIH72+EOMr17JxtLSzX2yFg4eSjoWI55OM3rdGj4YPAxEOObdt/jOy8/gY8IvDpzKDmtX842TDm0Ww4F7D2sRVzzeqYaegtcbKk9ZbgZO9t4/JyLV0bzXgU93ZiedGuJAVaujO9TmO+eOJ9wxtgA4lXBHV6ZTCBfpZzrzGd3sXEK15Z6Mak05UO6cm6Cq87LWv5Nwx9vzqjo/3/25MjnnhhPuoLu5K9tHCeQVhObYSar6pHPun8A0Wm+ONcaYXq9f/wTXX34Q1wN/eruGH/+2kcd235dLn59BXWkZjbE4/7P/4dRHfYxGbFhPQ0kJc0Zsy7xhaaYsmktZKsn/DR/FV/Vl7tzvEE6c9S/c0kXUlpQyZu0a5g8dzsvjJjJzmx0YtX5dfg+4j+ptlShgrPe+KW9pyg8a6GRe1OlxolR1TXSr/nWEPkMXAo855xYCtwF1wDGEi/0NqrowY/NY1h1wTUF3u6jf01nAdOBXWYtfIvR/anb3n6rOcc4dQueauOKtHFO9qnbLsLbRcXyaUGpcQegM3pHtDifcefBnQjNgJfADwpAIb0ar/RfwonPuNsIdlYsJnetOBdI91bncGGO6wwl7VHLCHrvx1XOf4fkxE5m0ajkPTD5wcwIFsPOKjzZ3Bk7HYqysGMgbo8fy2jbbc/7Mf/L592Zx6bEnM27NSn743AwWDxrKjYcdQyKd4lNLP2BxRfYN0qZjel0W9a6IfM57/7eMeUcCb3dmJ10dbPMWwt13Z6jqvc65IwhjDV1MGCdqDmE8pbuztjuMkGRlmhb9O945tyFr2QxVnUbXfBEYSBhbqVkTmHPuF8CVzrkfZ2+kqi938nPOjl6ZJhPaWbfGoqh6liIMS/AEcJOqrm9lvcz3y1V1R2ANoVP/D4FBhOTpLUJ/sY8AVPXfzrnJhGa+NwhVuhWE6mF2VwtjjOkTHvj1Z/nWN/7CkoFV1JZ8kkCJ91TV1RIN8YQHbjnwSD4or+Kxnzl++eVqHt17fxoTCeaM2IZvnPwN9lr6IeOrQ4+JslSKqrrsy5TpiN7yAOIM3weeEJG/AP1F5A5CX6hOdW4U38qto8b0IPsCGmNy4r+uVp5PDmTdgHIS6TSnvPUan/5wITWlZbwydidmj9iGgxfMpi6W4MZHTuKDFXWc8KvFn4wt5T1u8QKG1tVSWV9HLJ3mvaHD+cMdh+f3wHIrJ9nO9097vMXv+p//7/F5zaxEZFvgdGAM8CHwoPd+SWf2YT3fjDHGFKRbLneMXb6Ufg31xNMpasr68c8xE7jlwM/y3IRdOXTOO/RLJnl99BgARgwq4/Pvzdo8ztOo9Wup7j+ADaVlfFxRRW0sztdO2yWfh9RneZEWr3wQkVGbY/J+mff+Ru/9Bd77n3Q2gYI+8Ow859zthEyxNZNUdXFPxrO1nHNltLzDrskzqnpCT8ZjjDGFzMeEYRs3sGTwUB7ffd9m40bV9uvPH3bek9MmhSEN+pfGeH/YCK6b8Vs+HDyUJVVDaEwkeH/QcAbXbeC7Vx3GHrsObeujTN8wh9DVBwAR+aP3/ktbWH+Len0SFT2X7vx2V+wjVLWe8Iw6Y4wxOXbwbrW89m6MhkSCNf3L8TEh7j3JWIy3h4/i/H3KuOCLYzav/4f/OZT/fDiZf/xuHkNra7j40v0pK+31l8perxf128gugR26NTuzb4YxxpiCtcuYFDPfT7Fd9Wq2X7uGhkQiDCa6ZgX3/GASQ8e0rCztuX05N1+0Vx6iLVy9qGN5t+ZzlkQZY4wpaKdO3cBxx2UPQr1zXmIpVr2oEpUQkcP4pCKV/R7v/d87vLNuDs4YY4wxppleVIlaQXiObpPVWe894Uk7HWJJlDHGGGOKgvd+bHfuz5IoY4wxReeBVz/k6r81ECPN+1fulO9wCl4vqkR1KxsnyhhjTFF56J9Lue7ZJIlEHIknmHj1gnyHVPA80uJVCCyJMsYYU1R+9PQmiCojIkIsFmPcFXPzHFVh89LyVQisOc8YY0xREZEw6GaUSHkgXVqW36AKXKFUnrJZJcoYY0xR8T5NKkqkvPfUx+xSmGu+lVchsEqUMcaYoiISoyGeCBfyWAy8R1LJfIdV0KxjuTHGGFMA4slGBB8SKAARYr5QaiO9U6FWoiyJMsYYUxC+c8bjnH3u0+2u17+xgZJkRuXJe0qSqRxGZrxIi1chsOY8Y4wxfVZ9Q5Ky0gSPTLmRA+Jx7pl8MIddNJNFVUOYItVMa+UReMmyfpSmksQaIBmL0S/ZyKZYvOeDN32eJVHGGGP6nCn/rVSXVhED+tVv4vpNtZx78jmb77hLpJLMTA3iX6+VcdP+8wFYtrKG46+fQ1VZKcsrBpJMlACQlhipgmlgMj1JfJG2AzvnXgCmAI1AClgITFfVRzLWmQJcEa0XB94Hfqmq92WscyVwoKoe2cpnnEV4Jk9t1qJbVfWSLcS2IeNt03239U0zVLUiiv9Z4EXgqYz1BwANQFOt+iVV/bxzzgMHqerLrXzeImBUxjZNtlPVdW3F2U2K8wtojOmw8x74kFfmb0LSaYasX8/a8nIGNtSztryS+kSCtMTwsRhpIC0CIkg6Tf/GBvCe0mQjey9bzILho9gUT7Cu/wDqowQqnk5Tmmxkk8SYOqmCyw6vYszgkvwecH7lpJ3tnG883eJ3/Z13Te3zbXrFXom6RlWvdc4lgAuBh5xzb6rqPOfcVODPwPXAaUAdcCxwh3NuvKpe0cHPWKCqEzoTlKpWNE07534DJFT1rDbWfQnIXH8ecK2q3tuZzwTOVtUHO7mNMcbk1OduWsDSGo9IDOIxqgcOpL60jFIgKTEa4wligKTTJPCkJEYqFqN/YyP9ozvudlq1nFmjx1LZUM/Gsn4hgYoqVqlYjJj37LR2Ne+9vo7D5tXz8vkj2baq2C+P3cvGiSpgqpoE7iQklXtHs28FHlbVq1R1tarWqurvge8BlznnxuYnWmOMKR7LappfftOxOMl4go2l/dhUWgpECZRPE/eeklSKbdZVk8oY+2nJoCEk0mkAGuItk6NUPE4yHmNTPE7Kw+Oz63J6TMWoUEcstyQKcM6VAt+M3s5xzk0EJgCtVWYeIpQ7P9tD4RW0mpoam7Zpm7bpNqdLYtC824lH0mkaShIIkI7mNRGB6vIKUiI0Rp3FqzbVkYya+2LeE0+nw4jl3lOSTlGSSlGaTDIg2Qh4dh+SbBZDMU3nSqE+O6/Y+0TtR+hrVEnoG3WBqt7lnDsAeBnYVVVnt7LtckLfqOkd6BN1F5D9Df2Wqj7UwThbbc5r6hOlqtdmzW+1Oa8DfaKGE85Bk8WqumdHYtxKxfkFNMZ0SHVtksN/9gGN6XDZHbp+LasrKtlU2o90PI73nphPh0qTyOYxiFKxOCWpFJX1tUjKc8I7/+JvO+/JysoqUrEYHoinUpQ3NlBVV0dNWRkbBg/iR1MHM23v8vwedH7lJLv52jnPtvhdf8+dR/b5TKrYG32nR32iBhOSncOjf1dGy7cDmiVRUdVqWMY67VnY2T5ReXKe9YkyxvQ2gwckePPyHVvMH3/Zu5SVluElBqkUjSUJmgYpSMdiIEJjTNgYi/Pjnd/l9oZd2XblMso2buCjEaMQhIZ4goZ4gtrSEpaXV7Lo4tE9e3BFpFAqT9msOQ9Q1WrgbOBo59zxwFxgAXBqK6ufQvhD55mei9AYY0ymBdMnsU+/DSQa6zllN6FyUx396+vxaY+kUsQaGiivqeGGyXOoGJTgxZ/vx8P3Hcvf/udgypJJ1vUfQG1ZGWvLK1hRWRWa+EzOFGqfqGKvRG2mqmucczcB1wEzCHfrPeacWwjcRrg77xjgZuAGVV2YsXnMOdcva5cNPRB2Z5VmxZlW1d4YpzHGtOt/L/lDBnx7AAAgAElEQVRkJM1BMoeHZjZSmkoy/cQqpu4bqkozZixssd2mRKLZiNlpiZHwNmJ5bhVI1pTFkqjmbiHcfXeGqt7rnDsCuBy4mDBO1BzgIlW9O2u7wwhJVqZp0b/js8Z9ApihqtPoec9lvX8f2CWa/o1z7vas5VNU9e3ch2WMMVvnuydN5LsndWzdkatXsrFff1KxOHjPiA3rWN6/qPtB5VyhVJ6yFW3HctNr2BfQGJMzM2bMAOC4447bPO+A7/+bq/7+BD897GhGrV/LgYvm8tODj2LO5ePzFWZvkpN056vnPd/id/0DdxzW51Mrq0QZY4wpKisGDuLhfabww78/wdKBg5l+xHEkraCQU4XywOFslkTlSdR0dnobiyep6uKejMcYY4qFpNO8OH4Xnt9xV7wI4j2pVPZTr4xpnyVReaKq5wPn5zsOY4wpNjGgIRYjHg24mYrFkEarROVSoZ5dG+LAGGNMURkcT+KBZDxOKh6nrLGBRVdYf6hc8iItXoXAkihjjDFF5dXLd2VgzToq6moZsmE9j55ale+QCp5v5VUIrDnPGGNM0Xn7ut3yHUJRKZTKUzZLoowxxhiTU4VSecpmSZQxxhhjcssqUcYYY0zf05CEyZe+TXkqydpECW9dv3u+Qyo6hVqJso7lxhhjCtrPXtuBfT9ewvjqVey7Yhl7XfZevkMqOoV6d55VoowxxhS0wZvqeHncRGpLSumXbKRqY/bjTE2ueXsAsTHGGNP3LK+sIhWL0S+VJC1CQyKe75CKTqE+gNiSKGOMMQVtU0kpHiEtIN7TEC/Jd0hFxypRxhhjTB+UlBipeFR98h4Re05eTyvUSpR1LDfGGFPQ0rGMS50IUiCdmk3+WSXKGGNMwbr437tAwpP2HkRIpJKFe799L2bNecYYY0wfcfQPX2MpFfj+5ZQkkzT0HwCAAGNXr8xvcKZgWBJljDGmoOx18SySA4bhYzHEe+pLSzcvS8bj1JWUbmFrkwuFMi5UtrwmUc65F4ApQCOQAhYC01X1kYx1pgBXROvFgfeBX6rqfRnrXAkcqKpHtvIZZwF3A7VZi25V1Uu2Nr5ujPFHqjqho8uy57cS6wLgWlV9tJ1j3A64Ddgb2AH4qqo+uKVtjDGmq7z3PPfCEv705/ls2JhqvixrumLTJg6e9x7vbbM9c4aP4qS3XiMGvDtyW6o21bHvkkWUN9SzsqKSucNGsXDQUN4cPZbhEiOxrpryxnrGr1rBn/ZwmQFQ3X8Ai3Y8n7Fzb4NYDN5cAHc/C9sPg+9+AUqtvtDdCrUFtTd0LL9GVSuAocC9wEPOuabEYCrwPPAqMB4YAdwA3Oycu6oTn7FAVSuyXltMoDoSXzfH2B0yY30Y+J1zbmI726SBp4FTgSU5js8YU+Se/8dSHnh4TosECkJTW9Mr5j3n/OsfLByxDX/cazLvbLs9Nxz5BQ74YB6nvvU6p8x6gykfLmDP5UvZffkyxq1dzcdVgxmQSrJuwAAWDxnGeyO349mJu1OaSoL3xNIp+jU24JYsZPaA4XDID2F1DXzhWrjv73Dt7+HKh3v6lBSFQh2xvDckUQCoahK4k1Ad2zuafSvwsKpepaqrVbVWVX8PfA+4zDk3Ns/x9aoYs2K9jVAV26OddT9S1VtV9RVCBatH1dTU2LRN23QRTS9Z2rHRwmPpNIPralkwdPjmebWlZSwePJShG2sYVvvJfqrqQkPDuqjfUzL2yWCa9SUl1JaWgQhpiRHzno2l/egnkF60HJasgnUZDRX/t7jXnKt8TOeKR1q8CkGvSaKcc6XAN6O3c6IKygSgtaalhwh/rHy2h8JrEV80r1fF2CSK9QJC096snv78zqisrLRpm7bpIpr+tBtJR4oQ6Xic94eN5JD5s4ml0wCMW72CScuXMXvENnwwaOjmdZdXDgzLV60gKcLQ2g2btymvr8MDZY2NDGhsAGDZwMGUNtQTO+kA2GU0fGp82JEInHxQrzlX+ZjOFS8tX4WgNzT8XuacuwioJFz0z1bV/zjnDoiWL83eQFUbnHOrCE1nHTHOObc2a963VPWhrsYXLWv6E6k7YuwOTbE2APOAE1V1Xg9+vjHGbNEuEwfzk6un8NIry3j+paVs3Nh84Ms0of9MGrh7n88wYdVyvvPiUywZNBS3aC637ncI/95uLCM3rGOvZR8ycmMNqwZUUJ9IsOPKj3ljxChG1G1keXklgzbVsqp/OYlUioQPSRUi1PTvz2cumQpfi7qoPnk5vPgObDsU9h7Xk6ejaBRK5Slbb0iipqvqtc65wcBdwOHRv033oG4HzM7cIKq0DMtYpz0LW+u0vZXx0c0xtqURaO0ZBSXRshaxbuXnGWNMTo0aOYCTvjSBk77U+V/Lx7az/EJgr2/+g/GrlrO6YiCJZJJEVukrGYt9kkABlPeDox0mdwql8pSt1zTnqWo1cDZwtHPueGAu4Q6zU1tZ/RTCHyvP5DE+6JkYFwHbOOcGZM2fEH22McaYDLP+52Aev/VArjuuHJ9I4H3ze8N6vPOnKdg+Ub2hErWZqq5xzt0EXAfMIPxR8ZhzbiGho3QdcAxwM3CDqi7M2DzmnOuXtcuGXMXnnJuhqmnnXHfFKK0sawTeICRrNzvnLgHWAwcQErrzu+O4Mj5XgJLofTLqoG6MMX3SwQfuCDP+Q7qiknjGfCmQC3ifUqCnvNdUojLcAmwDnKGqTwFHAAcTKjKrgMuAi1T1sqztDiMkMJmvr0TLxjvnNmS9unof6+b4ALozxlaW/UBVGwlJWRXwDrAG+H/R/puNV7UVmj5vB8KYWnXAj7pp38YYkzc3HvhBi+t3oTYt9WaFWomS7DKnMT3MvoDGmJyZMWMGl8ycRMJ7krEY5Q31bEiU8N6VXe0mW/Bykt187vv/bvG7/m8/37fPZ1K9qjnPGGOM6W7br13DjtWr8IQMYWnFQEK3UtNTCrX6V9RJlHPuduD0NhZPUtXFPRlPLjjnDgKeamPxdap6XU/GY4wxPW34xjCYZNN1fIe1a/IXTJEqlOa7bEWdRKnq+XRT5+zeSlVfAiryHYcxxuRLI5BIJUnGEwyo30RNWVm+Qyo6hfKYl2y9sWO5McYY023eHjaSupJSJq5YSkM8zitDhre/kelWvpVXISjqSpQxxpjCd8OUBfz3C9vxyvbjoHYd7/50cr5DKjqFWomyJMoYY0zBe+fn++Q7hKJWKJWnbJZEGWOMMSa3rBJljDHGGNN5hVqJso7lxhhjClpDTSNzx1xI9fCvs2zb8/jnHS/mO6Si40VavAqBJVHGGGMK2rZXvcaSwcO4c8ohvDx+IpU3PpbvkIpOoT72xZrzjDHGFLR/bz+WeaNGgwjLqoZQlyhlj3wHVWRsxHJjjDGmD1o4fJtmHZtnjxiVx2iKU6FUnrJZc54xxpiCVV+XZuSG9c3mlTU05CkaU2isEmWMMaZgrb9xFi6Z5pkB5dSVljFu5Qri6VS+wyo6hdqcZ5UoY4wxBen+T11NRWk/ypKNHLjgfbatXo2PwfxhI/IdWtGxjuXGGGNMH7Fk4DSmAQk8a/v15+qpJ2weq6isoT6foRWlQq1EWRJljDGmzxv74/dJ9OuHFyGZSjGHpguc8NqYnRBACIM+NiYSpNNpYjFrjOk5hZlF2TfIGGNMn3XnF3/FjUf9hgOWLaY0maIklWLSxx+xoaQUgI0lpfxz7I6bq1ACpGNxnpt6Q95iLka+lVchKKhKlHPuBWAK0AikgIXAdFV9JGOdKcAV0Xpx4H3gl6p6X8Y6VwIHquqRrXzGWcDdQG3WoltV9ZJOxNfkt6p6dsY644D5wLOqOjVr+2uBy4D/p6rfyZg/AFgGVAHbq+oS59zZwEWquku0zoPABlU9P2uf50bnY09VXZ0x/ybgIOAzqpoZrzHG5IV/+V347l2wrpZX4gOZOXoM9+9/BLWlZdSVlJKOKktzttseicdZL2XcfMhR1PQbAL75ZfsfA0bz+vEPUJ5MUltawuEXfIYXHp9L1dB+TLtgMoOHD8jHIRasQhmhPFshVqKuUdUKYChwL/CQc24CgHNuKvA88CowHhgB3ADc7Jy7qhOfsUBVK7JeW0ygsuPLeJ2dtfwcoBo40jm3Yyvbvw+c5pzrlzHvK4QkqtNU9dfAG8AdTfOcc4dFcZxmCZQxptf4xq9gzjJYXs1fd/sUr47fmZWVVZQmGzcnUADJeJyPB1Yxd/goqgeUh5kizSogNZUDWVVeScKnGVhfz5O3/4vqVbUsen8NMx58u8cPrdAVasfyQkyiAFDVJHAnodq2dzT7VuBhVb1KVVeraq2q/h74HnCZc25sfqINnHMlwNeAawjJ0jmtrLYI+Dfw5Yx55xCOtavOAT7jnDvTOVcF3Ad8X1XnbMU+O6Smpsambdqmbbpj03WfjO+UFmFQXS2STlM9oILy+k2bl1VuqmXMmlWM3LAe8enN8xEBEQZuqvvkfSSWUamq2/hJx/Nec+w9NJ0rXlq+CkHBJlHOuVLgm9HbOc65icAE4MFWVn+I0FT+2R4Kry3HA0MIMd4NfC1KrLLdSZRgOecmEapqM7r6oaq6CjgLuIVwLmZGFaqcq6ystGmbtmmb7tj0jWdCaQIfj3PQ3PeYNvNVDlg0l6G1G9h38QIOmv8en3tvFg89eBsxYPS6ar7++ku4xQv4wjszGbN6JQPqN7Fw0BDKNtVRtX4dALWJEj519M4AlA8s5ehpe3QsngKczpVCrUQVVJ+oyGXOuYuASkLfo7NV9T/OuQOi5UuzN1DVBufcKkLzXkeMc86tzZr3LVV9qBPxNTlKVV+Lps8FZqjqKufc/cB1wBeBR7L28RjwK+fcztE29wLJDsbeKlV92jn3MHAyMHFr9mWMMbkgpxyEP3EKAhyVSjPz3pc486dPUFFWwqN7Tmby0g9YUTGQmw/8LBe+9Ay7rlrB7suXstvypdx6wBF8MHQ4AAnvOeKj99jp3guora5lt/3GAnDMV/cknogRixXGBb43KZTKU7ZCTKKmq+q1zrnBwF3A4dG/K6Pl2wGzMzeIqlbDMtZpz0JVnbA18WXPdM6NB44EjgVQ1eXOuSeA88hKolS1MUqyvk1IevbvYizZZgEHRZUpY4zpdaQkumyVwL7nHwHnHwHAuZe+TU3/chLpFPOHjGC3FR+RTCSojZeSisWYP2zk5n0M3lTHIU9fSllF/2b7LimN99hxFJtCqTxlK8QkCgBVrY7uUJvvnDse+DOwADgVeC5r9VMI/Q2f6dkomzmX0KR4j3OuqXG+HCh3zk1Q1XlZ698JvAc8r6rz892fyxhj8umdn+wBfNIMt+pOYUgySU1pGXXxBIlUkmQ8XPKGbKxpkUCZHCvMHKpwkygAVV0T3ap/HaHP0IXAY865hcBtQB1wDHAzcIOqLszYPJZ1BxxATp5aGfV7OguYDvwqa/FLhP5Pze7+U9U5zrlDaKV5cgvirRxTvaoWypAdxhgDwLD1D/HByHMYVbeBh/fef3MCBdiz80y3KegkKnIL4e67M1T1XufcEcDlwMWEcaLmEMZTujtru8MISVamadG/451zG7KWzVDVaXTNF4GBwC8yx2oCcM79ArjSOffj7I1U9eVOfs7Z0SvTZEA7uR9jjOn1xiy/k7v3vZbZo7ZtNn9FRVWeIipehdqcJ95bEcLklX0BjTE589jtD/HCS+WQMY7UoA3rufJPp+Uxql4tJ9nO3lfObfG7/q0rd+rzmVXBDnFgjDHGxLerpKKu+QMmGhOtjRxjcsmGODDtcs7dDpzexuJJqrq4J+MxxhgDlQ31bCiv2Py+JLVVI8KYLijUx75YEtWNoufSnd/uisYYY3pM/8bm9wQN25j7EbpNc4Xab8Oa84wxxhS0EXsJo6tXU5JKMnb1ClIVVj/oaV6kxasQ2DfJGGNMQSs/aBQX3Xgcm2pq6Vc5IN/hFKVCrURZEmWMMaYoWAKVP4VSecpmzXnGGGOMMV1glShjjDHG5FShVqIsiTLGGGNMTlmfKGOMMcaYLijUSpT1iTLGGGOM6QKrRBljjDEmpwrlMS/ZLIkyxhhjTE75wsyhLIkyxhhjTG5ZJcoYY4wxpgusEmWMMcYY0yWFmUXZ3XnGGGOMySkvLV+tEZFFIrJ7z0bXdVaJMsYYY0xOFWqfKKtEGWOMMSanOlqJao2ITBaRV0XkP9G/k6P514vID6Lpr4hIWkRGRO+fFJGpuTiWTJZEGWOMMSanPNLi1REiUgo8CvzYe78n8CPg0Wj+c8AR0apHAK8Bh4tICbAf8HJ3H0c2a84zeSUifwOG5TuOLUkkEsOSyeSqfMfRVX09fuj7x2Dx55fF3yl/9d4f1d07/eCSbbvanrcz0OC9fxbAe/+ciDRE818Bfh8lVAcAFwFfBpYCb3vva7c+8i2zJMrkVS7+s3Y355yqqst3HF3V1+OHvn8MFn9+Wfx9mtD684u9975ORGYB04CPgOeBnwNLgL/3RHDWnGeMMcaY3mo2UCYihwFE/5YAc6LlzwFXAc957+sJCdRZ0fycs0qUMcYYY3qTZ0UkmfH+BOCXIlIObAS+7L1viJY9B1zDJ0nTc4SmvTd6IlBLooxp36/zHcBW6uvxQ98/Bos/vyz+PsJ7P7aNRVPaWP9VMkby9N7fCNzY/ZG1TrxvranRGGOMMcZsifWJMsYYY4zpAmvOMyaLc+5Wwpgj9cAG4L9UVVtZ7yzgZmBRNGuhqp7QQ2G2qaPxR+v+mNAJE+BeVb2mR4Jsh3PudOBiYBLwXVX9VRvrHQo8ySedTOtVdb8eCXILOhp/tO45wCWEJomngO+oarpHAm07pgHAPcC+QBK4SFWfaGW9Q+kl5985NxG4DxgKrAbOUNW5WevEgV8CRxHu+PqJqv6mp2NtTQfjvxL4FrAsmvWKql7Qk3Ga5iyJMqalpwgXvkbn3LHA74Ad21j3WVX9cs+F1iEdit85dzBwEtD0nKrXnXMvquo/ei7UNr0FnAJc2oF13+2Ft393KH7n3DjgCuBThAvnU8DpwP25DrAdFwE1qjrBObcT8JJzboKqbmhl3d5y/m8HblXVB6Mk9g7g8Kx1TgMmADsRkpU3nXPPquqiHo20dR2JH+B+Vb2oZ0MzbbHmPGOyqOoTqtoYvX0VGO2c6zP/VzoR/8mEX8h1qlpHuHCf3FNxbomqvqOq7wJ5rch0VSfi/zLwmKqujKpPd9I7fgYnEy7qRNUQBT6f14i2wDk3AtgHeDia9TCwj3NueNaqJwN3qmpaVVcCjxH+kMirTsRvepk+c2EwJk8uBP6yheaVQ5xzbznn/uGcO6YnA+ugLcW/A/BBxvvFwPY9ElX3muicm+mce905d2a+g+mk3voz6ExcveH8bw8sVdUUQPTvMlrG3FvPd0fjBzjFOfcf59zTzrlW71gzPcea80zRcc7NJPwybc3Ipl9kzrlTgFOBg9tY9wngd6pa55z7FPBX59yhqvpetwedoRvjz5uOHkMHzAS2V9V1UdPYs865par6bLcE2oZujD8v2ou/E7vKy/kvYrcD06Om+s8CjzvndlXV1fkOrFhZEmWKjqru0946zrkTgOnAEaq6vI39rMqYftM59zLwaSCnSVR3xU/4K3xMxvsdgA+3PsL2deQYOrif9RnTC51zjxEG2svpRby74idPP4P24nfONcW1MiOu51vZT17Ofys+BLZzzsVVNRV1IN+Wluey6bj+Fb3PrkzlS4fiV9WPM6afcc59SOjT+GKPRms2s+Y8Y7JEnbFvAj63pQ6nzrntMqbHAPsD/8l5gO3oaPzAI8AZzrn+zrn+wBnA73sgxG7jnNvGOSfR9BBgKqFTd1/xKPBF59zwqN/aOfSOn8EjwHkAUcfyycBfs1fqLedfVVdEnzstmjUNeDPq95TpEeAc51ws6m/0RcLPIK86Gn/W75y9gbHA+z0UpmmFDbZpTBbn3EqggU/+CodQ0VntnPsN8GdV/bNz7jrgeMIt4AA3qep9PRxuCx2NP1r3SuCrhNvr71fVK3s43FY556YBPwUGE45lIzBVVd91zl0NLFPV251zFwLfBBoJlfX7VbXHRituS0fjj9Y9jzAcAsDTwIX5bg50zpUD9xLuGkwBF6vq49GyXnn+nXO7EIYIGAxUE4YIeN859yRwuapqVOH5FSHZA7hBVXvFaOAdjP8+wrATKcL36gpVfTJvQRtLoowxxhhjusKa84wxxhhjusCSKGOMMcaYLrAkyhhjjDGmCyyJMsYYY4zpAkuijDHGGGO6wJIoY0ynichYEfEiMjrHn3O+iDyQ8f4pEbl4S9uY3BCReSJyVgfX7ZHvR08QkTIRmSsiu+Q7FtP7WBJlTA6JyHgReUREPhaRDSLyoYj8SURKo+Vnici8VrZra/7p0cXp8laWvSAi9dHnrBORN0XkxNwcWe6JSDlwNXBl0zzv/ee993kfB6ot0c/mwHzHUQxyca5F5FARSWbO897XAz8jjPtlTDOWRBmTW08CHwE7A5XAFOBvhMEtu+JcYA1wtojEW1l+jfe+AhhKeBL870RkYhc/K99OB9723s/PdyCm6D0MHC4iE/IdiOldLIkyJkdEZCghebrde7/OB0u897dHf912dn+7AgcBZwLbAJ9va13vfRK4DYgDe7SyrwtF5M2seeNEJCUiY6P390SVsxoReVdETt1CbFeKyLNZ814QkR9lvN9dRP4mIqtEZLGIXC8iJVs45C8Cz7S1z4wmozOj+DaKyJMiMlhEfiIiK6IK4AUZ258VNUtdIiIfRev8PDOO9o5bRPYUkb+KyEoRWSMiz0TzZ0WrPB1VA3/TxrkaICK3RJ+xSkQeE5EdMpa/EMX0aBTDfBE5vq2TlHFM3xORJdE2PxORodE+1ovI7MyqjYgkRORyEVkQHcNzIrJ7xvISEbkp4xxe0srnHiQiL0fbzxeR74tIh/84EJETRWRWVDWdJSInZB9T1vr3Np3Tts61iCyKjuvlaL6KyOTW9pExb5GECu+2wFNAPNp2g4icCeC9X0943t4XOnp8pjhYEmVMjnjvVwP/B/xGRM4QkUmduci04jxCZeYJQoXr3LZWlNBceAHhcRyzWlnlf4FdRWTvjHlnAS947xdF718G9gYGEZrV7hWRSV0JXERGEB6S+kfCg1WnAJ8F/nsLm+0DvNuB3Z8IHEh4mOxY4HVgfvQ5XwNuzkxSCA+g3QEYH8VxHHBRxvI2j1tEtomO48Xos0YBNwB47/eKtp/qva/w3p/dRry/IDxncf8ollXADGleWTyT8PzDKsJjSu4TkQFbOAdjonjHR+fi24SEoOnRM38E7slY/weEZyUeTUjIXwKeEZGB0fJLgWOBzwDjomPd/KBkEdmN8B38KTAcOAa4kPAIoXaJyBTCd/BSQtX0h8DDIrJfR7Zv51yfD/wXMAT4A/BkxnFtaZ/LCH+YpKJ9VnjvMx/j9DbhO2nMZpZEGZNbhwIvAN8lPGB0uYj8OCuZGiciazNfhCrSZiLSj3CBujuadRdwtLTsuHtZtP0SwnP9TvTet+hb5b2vBh4nJBlE8ZyZsX+893d571d771Pe+98SHq58aBfOAYQL9izv/R3e+wbv/VLg+mh+WwYD6zuw72u892uipPUJoNF7f6f3Pum9f4rwHLJPZayfBn7gva+LmgpvJDoP0O5xfxWY572/3nu/MTqWZhW4LRGRGOGYf+S9X+q930j4buwKfDpj1d9571/x3qeBXxOSqZ22sOs64KoonlmExPlf3vvXvPcp4EFggohURet/DbjBez87qopeTXge2zHR8jOi5fO893WEJDPzGWHfBB7x3j8enafZhGRvSz/PTF8DHvXePxX9nP4C/An4ege335K7vPf/9t43EBLcOkJCuLXWExIzYzazJMqYHPLer/Le/9B7vw+hUnAxcDkZF21gofd+UOYL+FbWrk4CKggXQwhVgBVAdrVjerSPEd77z3jvZ2whvHuA06Kq1eFRfH+EcLEXkatF5P2ouWUtsBeh6tAV44ADshLFuwmVnLZUA+1WEAh9zprUZr1vmleZ8X6F97424/0iYDR06LjHAnM6EFNbhgP9gAVNM7z3Gwg/y+0z1vsoY/nGaDLzGLKtiBKuJtnnoel4m/axfVYMacJ5aIphdPQ+M4YVGfsbB0zL+nleQahqdUSzz4/Mp/k56KpFTRM+PBx2MdHPdysNJPRHNGYzS6KM6SHe+1rv/b2Eysbe7aye7TxC/6Z3RORjQqVpCPANab2DeUc8DWwi/JV+FvDbqOoAMI2QoJ0IDI4Su1m03SF+A1CeNW/bjOkPgGezksWqqBN8W94EutR82I4RWU1jYwnnE9o/7kVsuSLU3hPdVwL1hCQEABGpAEYAH3Ys/G7xYVYMMcJ5aIphafS+aXk5IcYmHwB3Z/08B3rvd+vK50fGZ3x+e98naPtcZ8YthKbbpp9vs/2KSILmx5WZiGbbnfCdNGYzS6KMyREJHZyvl9ChuiTqzHsi4ZfxS53YzyTgAOAEQvLV9Po0oZJzdFfii6oP9wPfAb5ERlMe4a/uJOGiHxORrxMqMm1RYB8R2Tc6zgtpfpG8H3Ai8nUR6RdVfMaLyFFb2OdjwJGdP7J2xYCfiEh/ERlPaKpq6vvS3nE/COwsoWP6gOjnekTG8o/ZQpKVcc6vEZFto2Tu58Bs4I1uOr6OuBe4WEQmRpXIy4AE8Jdo+QPAD0RkRxHpT2jyzEygbwNOEZHjMr7bk0TkkE58/oki8jkRiYvI5wnfwaZ+W28Skt1jo+/KCcDBWfto61x/XUT2kXCzwA+AARnHpcAREm6iKAOmA5k3N3xM6FjeLMETkUrC/7c/d/D4TJGwJMqY3Gkg/JX7R0IzwErgR8C3vfePdGI/5wEzvfczvPcfZ7z+AzwSLe+qe4BDCE2KmRfx+wgdtOcRqhKT2ELi571/gZAM/JXQjDQSeCVj+cfAYYQ77hYRmur+RKg+tOUBYK8o0TP0XbwAAAFZSURBVOlOHxCOaSHhGP9KSBKgneOOOh8fSugUvwRYDmTeuXYZcLWIVIvIHW18/vcIF/N/EZqatgG+EPVd6ik/Jdy2/zThGA4ndNJu6oN2PWEojtcI52kx4bwB4L1/h1DB/C7h572CkBh1qLnXe/9PQh+8nxG+CzcCp3vvX4uWzyd0Dv814f/OUcCjWbtp61z/GvhltN+TgWO89+uiZf9LSIRmEpoPFxN+zk1xzSEkiG9EzZRNHeWnAc977+d25PhM8ZDQZGyMMb2PiJwPHOC979BdXx3Y31mETt023k8BEpFFhJ/vg+2t24l9lgHvEBLd97prv6YwJPIdgDHGtMV7fztwe77jMMUruntxS/3gTBGz5jxjjDHGmC6w5jxjjDHGmC6wSpQxxhhjTBdYEmWMMcYY0wWWRBljjDHGdIElUcYYY4wxXWBJlDHGGGNMF1gSZYwxxhjTBf8flErGrlvhbNAAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"shap.summary_plot(shap_values, X)"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAFJCAYAAABHBFN/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xu8VHW9//HXVxAv4S2VUlSQ0IzSLD+eIjXvZippp4vXDBVv5ekcy6IjppiAUh1FE1NRxEtgeioLO5pamubvlOdj5qVSRDaYIAkIKqFycf3++H63LhYze8+evfbMhv1+Ph7z2HvW97vW+qzvrJn5zPf7nTUhyzJEREREpDzrNTsAERERkXWNEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkvVudgCy9po+fXo2bNiwZochIiLSSKGWSurBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESlZyLKs2THIWmrA+Hk6eUREpNuaM3LbrthsqKWSerBERERESqYES0RERKRkSrBERERESqYES0RERKRkSrBERERESta72QGsy8zsAWBf4Gh3vy23/GPAH4A57j4wLRsMfAc4CNgceBl4Cpjk7j9LdUYD5wFvABmwALgJuNDd2/1Gn5llwD7u/nsz2w+4H/gn8BawAnga+Ckw0d3f7NzRi4iI9Fzqwep6fwNOLSw7NS0HwMx2BR4lJjr7AZsCOwGXA58trPuAu/dNdb4MfCv9rccqd+/r7psC/YHRwMnAA2bWp85tioiI9Hjqwep6PwPOMLNB7j7LzDYBPgeMA76a6kwA/s/dT8qttwq4O93WkHqsHjKzvwAGTOlMkO7+BnCvmX0WeJKYtE3qzDZFRER6KvVgdb03gB8Dp6T7xwK/A14EMLONiMOI0zqyUTNbz8z2Bz4EPFNWsO7+LLE37cCytikiItLTqAerMSYB95jZBcBpwAXAFqns3UAvYG5rZTPbHXgg3d0QeL+7z0n39zWzJcBGQB/gR+lWpheALUvepoiISI+hHqwGcPengDnESezvYfVhv8XE4cDtcvX/7O6bE3unNmD1y/L/LpVtApxLnLO1cckhbwcsKnmbIiIiPYYSrMa5lphgXe/uq1oXuvsy4EHgmI5szN2Xu/vFxG8SXlhWkOnbjHsAvy1rmyIiIj2NhggbZxrwd+L8pqKvEyesTyZOfm8hDhvuVcN2zwPuM7MJuWHEDjOzDYC9gcuAx4Eb692WiIhIT6cEq0HSt/Tuq1L2ZzMzYrL0IPE6WIuAvxAv01A1cXL3h8zsIWIv1vAOhtXLzJYSLw+xkjhZ/hbgCl0HS0REpH4hy9q9PqVIRQPGz9PJIyIi3dackdt2xWZD+1U0B0tERESkdBoiXIeki44OqFA0x90/2Oh4REREeiolWOsQJVEiIiLdgxIsqduVQx5l2LBhzQ5DRESk29EcLBEREZGSKcESERERKZkSLBEREZGSKcESERERKZkSLBEREZGSKcESERERKZkSLBEREZGS6bcIpW76LUIRka7XRb+nJ/XTbxGKiIiINIMSLBEREZGSKcESERERKZkSLBEREZGSKcESERERKVnvZgfQXZjZA8BQYAWwCmgBxrr77RXK84a6+5NmtjVwCXAosBnwGvBn4GR3fzFtY0NgFHAs0B94FbgL+I67/z3VGZj2vb27v5CLb7XlZjYaOA94A8iABcBNwIXunuXW2wY4HzgM2ApYBPwBGO/uj5rZcGAysKxwXBPdfWQHmlBEREQSJViru8jdx5hZb+AsYKqZPebuM/PlVda9BXgF+Ii7v2Rm/YjJVgZgZr2AXwHvBY4HHgW2B/4L+KOZ7enuczsY7wPufpCZBWBv4NfAbGBK2ue2wCPERO8w4GlgI+CzwL+mGABmufvgDu5bREREqlCCVYG7rzSzScBlwO7AzHZWAfgEcLS7v5S28RKxR6nVscA+wJBcwtZiZkcDfwEuBEbUGW8GPGRmfwGMlGAB3wX+CXzW3Vt73pYCN9ezHxEREamNEqwKzKwPcGa6O6PG1R4Evm9m2wH/Bzzh7qty5YcBf8wlVwC4+woz+wlwcifiXQ/YF/gQqyd1hwGTc8mViIiINIASrNWNMrNzgE2Ic61GuPsTFcrf5u6bp3+PBv4NOAm4HHjTzKYA33b3N4CtgWpDgPOAfnXEu6+ZLSEO+/UBfpRurdraZ96OaTt5X3H3qXXEJCIi0uMpwVrd2DQHawvgeuCA9He18koruvtS4GLg4tQDdihxKO5V4iTzBcQ5V5Vsm8rhnUn06xfqrF8oB/hdmoPVB/gG8CVg47RP0jb7V9lnXovmYImIiJRHl2mowN0XE+dDHWZmR9ax/nJ3/yVwH3EOF8DdwMfMbFC+bppQ/0XitwkBXgTeBIoJz2DifKqXquzvYmJCdWGu6H+Az5tZMVkTERGRLqQerCrc/WUzuxQYZ2bT26uf6k4DngSWA58E9if2agFMBU4BfmFmp/DOtwi/T7ysw+i037fM7Gbgu2Y2C5gFDCImTjfmL8FQwXnAfWY2wd3nABcAfwT+28y+TZxPtiHwGeCD7n5ere0hIiIitVMPVtsuB7YBTkz3v2NmSwu3I1LZesANxB6mxcBVwA+Il2HA3VcShw1/CdxK7I16BHgd+JfW62Al/0GcNH8PcbjvHmJv2Grzv4rc/SHgIVIvVrrsw57EXrHWbf0NOAr4aW7VQRWOa1rtzSQiIiJ5Icva6hARqW7A+Hk6eUREutickds2OwRZXailknqwREREREqmBEtERESkZEqwREREREqmBEtERESkZLpMg9TtyiGPMmzYsGaHISIi0u2oB0tERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZPqxZ6mbfuxZpOfSDxBLD6YfexYRERFpBiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiXr3YidmNkDwFBgBbAKaAHGuvvtuTpDgQtSvV7AM8AV7n5jrs5oYG93P6jCPoYDk4FlhaKJ7j6yxviWA28Bi4CHgQnu/miV48gb6u5PmtkU4HjgzbSdF9IxXG1mS3P1N0h/32xd4O590/b3BfZ19wdz+50JjHH3KbllgdhG7wW2dfelubL9gPuBv7r7BwvHehdwKHCSu08xs4HEx2MZkL/swhPu/omKDSYiIiJtakiClVzk7mPMrDdwFjDVzB5z95lmdgjwS+BiYoLyOnAEcI2ZDXL3C2rcxyx3H9yZ+ADMbABwKvAHM/uiu/+8Ur0qbnT3EWa2HnA08ThnuHvf1gpmdh3Q292HV1h/EfADM/uYu7d1nan9gUHAUuBYYFKhfBWwvpnt5e4Pp/3uAHwMmFdhe+939xfa2J+IiIjUqOFDhO6+kpgM9AZ2T4snAtPc/UJ3X+Tuy9z9NuBsYFTqZWlkjHPc/TzgJuCHqbeoo9t4y92nEROmj3Rg1UnAdsSkqS2nA3cDN6f/K7mOmCi2OgWYRkxgRUREpIs0PMEysz7AmenuDDPbGRgM3FKh+lTiFVMPblB4RbcC/YH3d3RFM+tlZscB7wa8A6v+EzgfGGdmG1SqYGZbA0cRh0SvB/Ywsz0qVJ0CHGVmm5lZL+Bk1uzpEhERkZI1MsEaZWZLiL0nY4AR7v4EsHUqn1tcwd2XAwuBfjXuY0czW1K4HdeJmFuHzLbMLRtV3EdhnS+lZS8B5wCnuPvvOrjfG4DXgH+vUn4S8Aow3d3/DDwGnFas5O4vAfcBJwCfBuan+pX8pXBcV3UwZhEREUkaOQdrbJqDtQWx1+WA9HdBKu8PPJ1fIfV2bZWr056WTszBqmS79HdRbtnYduZg3ezuIzqzU3dfZWbfAqaZ2fX5sjRceSpwi7u3Tra/HrjEzL6Rn+yeTALGA3Nou/fqg5qDJSIiUo5mzMFaDIwADjOzI4FngVlApZ6mY4jfbLu3cRGu5mhiz9ozjd6xu98FPEIcLsw7kDikerKZzTez+cCFQF8qt+E9wGbESfHTui5iERERadXIHqy3ufvLZnYpMA6YTvxW4R1m1gJcRRxGPByYAIx395bc6uuZ2YaFTS4vMz4z256YBA4Hjm7n23xd6ZvAH1j9+E4DHiQmf3njiJPdr80vdPfMzA4HNnL317owVhEREUmakmAllxO/JXhiuh7TgcTemm8Rr4M1AzjH3ScX1tufNb8F1/qNu0GF601BnKfU3jfyAL5jZiOJPWaLgP8HfMLdH6lQ79uFZce4+5017KND3P1xM7uVmOhhZv2Ik9s/5+7z83XNbDzwNzOzCtv5aw27e8bM8onkEnffrmptERERqSpkWbM6Z2RtN2D8PJ08Ij3UnJHbNjsEkWap6dJN+qkcERERkZI1c4iwYczsauKlCioZ4u7PNzIeERERWbdpiFDqNn369GzYsGHNDkNERKSRNEQoIiIi0gxKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpguNSt30W4QizaffBBRpOF1oVERERKQZlGCJiIiIlEwJloiIiEjJlGCJiIiIlEwJloiIiEjJlGCJiIiIlKx3M3duZg8AQ4EVwCqgBRjr7rfn6gwFLkj1egHPAFe4+425OqOBvd39oAr7GA5MBpYViia6+8jOxldijOe5++Bay4rLK8Q6Cxjj7j9t5xgPA84BdkuxPwWc6+4PtbWeiIiIVNcderAucve+wJbAFGCqmbUmDYcA9wP/CwwC+gHjgQlmdmEH9jHL3fsWbm0mV7XEV3KMZcjHOg34iZnt3M46WwA/BAYDWwNTgbvMbPsujVRERGQd1h0SLADcfSUwidirtntaPBGY5u4Xuvsid1/m7rcBZwOjzGxgk+PrVjEWYr2K2CO1azt1f+zuP3f3Je6+0t1/BLwOWANCFRERWSd1mwTLzPoAZ6a7M1LPy2DglgrVpxKvpHpwg8JbI760rFvF2CrF+lXicOHjHVx3N2IP2FNdEJqIiEiP0B0SrFFmtoTYazIGGOHuTxCHqwDmFldw9+XAQuJwXC12NLMlhdtxnYyPkmMsQ2usLwBHAp9z95m1rmxm/YD/Br7n7s92UYwiIiLrvKZOck/GuvsYM9sCuB44IP1dkMr7A0/nV0g9NFvl6rSnpdIE8k7GR8kxVrMCWL/C8vVT2Rqx1rMTM9sWuBe4B/jPerYhIiIiUXfowQLA3RcDI4DDzOxI4FniN+Eq9TQdA2TEhKBZ8UFjYpwNbGNmGxeWD0777rQ0T+wh4C53P8vd9SPOIiIindAderDe5u4vm9mlwDhgOnAWcIeZtRAnbb8OHA5MAMa7e0tu9fXMbMPCJpd3VXxmNt3d3zKzsmIMFcpWAI8QE7kJZjYSeBXYi5jsndHZYzKzXYD7gCnufl5ntyciIiLdqAcr53JgG+BEd78LOBD4JLEnZyEwCjjH3UcV1tufmNzkb19MZYPMbGnhNq2z8QGUGWOFsm+6+wpiwrYZceL5y8TLKpxTvB5XnUYShzj/o9A+x5ewbRERkR4pZJlGg6Q+A8bP08kj0mRzRm7b7BBEeppQS6Xu2IMlIiIislbrVnOwGs3MrgZOqFI8xN2fb2Q8XcHM9gHuqlI8zt3HNTIeERGRnkBDhFK36dOnZ8OGDWt2GCIiIo2kIUIRERGRZlCCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJdN1sKRu+qkckcbTT+OINJ2ugyUiIiLSDEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZEqwRERERErWu9kBNJKZPQAMBVYAq4AWYKy7356rMxS4INXrBTwDXOHuN+bqjAb2dveDKuxjODAZWFYomujuI2uI0YDzgL2ADYD5wP8A4939xVRnCPBdYH9go3Qc1wMT3P2tCnG8BbwBPAX8GLihSr0OxysiIiJr6lEJVnKRu48xs97AWcBUM3vM3Wea2SHAL4GLgeOB14EjgGvMbJC7X1DjPma5++COBmZmBwPTgcuBr7r7XDPbBhgB7Avcama7AQ8DtwAfAhalshuADwNfrhSHmfUFDknbPgz4XGfjFRERkcp6YoIFgLuvNLNJwGXA7sBMYCIwzd0vzFW9zcw2Bq4zsxvcfXYXhnUVMDXfc5R6rS7K1bk0LvYzc8vuNbMTgPvNbJK7/764YXdfCvzMzBYCvzOzg9393q45DBERkZ6tx87BMrM+QGuSMsPMdgYGE3uGiqYSr9x6cBfG07r/qW3U2QjYjwoxuvsDwAvAp9vaj7s/CMwDDqw/WhEREWlLT+zBGmVm5wCbEOdijXD3J8xsr1Q+t7iCuy9PPT/9atzHjma2pLDsK+5eNXkCtq62/5x3E+eFVaszj9pifAHYMne/nnhFRESkip6YYI1Nc7C2IE4MPyD9XZDK+wNP51dIvV1b5eq0p6WOOU35/f+tSp2XiZPz+1cp3xb4TQ372g64P3e/nnhFRESkih47ROjui4mTxw8zsyOBZ4FZwHEVqh8DZECXzVly9xnEeWDHtlHndeBBKsRoZp8kJk53tbUfM9ubmIj9tjPxioiISHU9sQfrbe7+spldCowjfnvvLOAOM2shTjh/HTgcmEC8TEJLbvX1zGzDwiaXdzKkrwDTzewfwJXuPs/M+gGnEL/p9xPgG8BDZnYlMIbYq7UP8VuEU939oUobNrN3EeeQXQ78wt3v6WSsIiIiUkWPTrCSy4GzgRPdfYqZHQicD3yLON9pBnCOu08urLc/MQHLa+19GmRmSwtl0929au8UgLvfm3qYzgOeTEOT84E7iQkU7v6YmX2ceB2svwIbAnOAHxK/YZjXGkcGvEm8DtY44Loq9ToUr4iIiFQWsixrdgyylhowfp5OHpEGmzNy22aHINLThVoq9dg5WCIiIiJdRUOEDWRmVwMnVCke4u7PNzIeERER6RoaIpS6aYhQpPE0RCjSdDUNEaoHS+p25ZBHGTZsWLPDEBER6XY0B0tERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZLrQqNRNFxoVKZcuIiqyVtBvEYqIiIg0gxIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpmRIsERERkZL1bq+CmT0ADAVWAKuAFmCsu9+eqzMUuCDV6wU8A1zh7jfm6owG9nb3gyrsYzgwGVhWKJro7iM7EF+rW919RK7OjsBzwH3ufkhh/THAKOCH7v613PKNgXnAZsD27v6CmY0AznH3XVKdW4Cl7n5GYZunpfbYzd0X5ZZfCuwDfMLd8/Hm190F+BvwTyADVgIzgV8Cl7n70gr18h5x9wNSnU8B5wNDiMn0vNQ2F+b294FUZ3+gL/AP4DfAJe4+q1KMIiIi0rZae7Aucve+wJbAFGCqmQ0GMLNDgPuB/wUGAf2A8cAEM7uw8uYqmuXufQu3NpOrYny524hC+anAYuAgM3tfhfWfAY43sw1zy75ITEg6zN2vBR4BrmldZmb7pziOr5ZcFQx0902I7fl14DDgETPbrEK9/LG3JlfvB34O/BDYmvjYfZGYrLXG9FHg/4BXgY8DmwAfA/4KHNrhAxcRERGghh6sPHdfaWaTgMuA3Ylv1hOBafleEeC21AN0nZnd4O6zywq4o8xsfeAk4CLgdGKS8+1CtdnA88DngVvSslOBScClde76VOAJM/sycAdwI/ANd5/RkY2kZOwhMzsSeBr4GvFY2mPAQne/NbfsyXRrdQXwkLufnlu2EJjQkRhFRERkdR2ag2VmfYAz090ZZrYzMJh3kpK8qcSrnR7cqQg770jg3cQYJwMnpaSraBIxKcLMhhB746bXu1N3XwgMBy4ntsWfUs9Wvdt7idhTeGCNqzwC9DOzyWb2GTPrny9MPWGfAH5cb0wiIiJSWa0J1igzWwK8DowBRrj7E8ShJ4C5xRXcfTmxN6RfjfvY0cyWFG7HdSS+3O3jubLTgOkp4bmJmGwdVWEbdwC7pKG104hDoStr3H9F7n4PMI04R6w4bFmPF4hDfXnPFY790rTvZ4nDfhB7pP5uZn8xs8PTsq2ICfAaj52IiIh0Tq1DhGPdfYyZbQFcDxyQ/i5I5f2Jw1dvS71dW+XqtKfF3QfXWLdifMWFZjYIOAg4AsDd/2FmdxKHCm/P13X3FWZ2E/BvwNG8k5x01uPAPinB66ztgEWFZe+rtm13/zNwMoCZvYc48f5nKYlcRJxE37/SuiIiIlK/Dg0RuvtiYk/MYWlO0LPALKBST9MxxDfwezsbZCecRuylucHM5pvZfGLCdUDrJP2CScQh0Mfd/bkGxtkuM9ua+E2/39azvrv/g5hg9QGGuPsS4P8Bx5YWpIiIiAAdnOQO4O4vp2GoccQ5SmcBd5hZC3AVcRjxcOKw1Hh3b8mtvl7hm3oAy+uKvB1pntVwYCxwZaH4IeJ8q9W+pejuM8xsXzo2bNarwjG96e5ZxyKuLB3HvwDfA14iTkyvZb0DgA8QL+/wAvEbgt8kXtbhsVTt34HfmdlVxG9+Pg9sQUyY33L3q8o4BhERkZ6mwwlWcjlwNnCiu08xswOJ11L6FvE6WDOI14uaXFhvf2ICltfagzLIzJYWyqa7e709LEcBmxKvHbXasJqZXQaMNrPvFFdy9993cD8jWHN+1Z6Ad3A7RbPNLCNee2wmcCdwqbu/WqFe/v4/3P19wMvELxicC2xOTKz+DBzq7i8CuPujZrYnsWfrEeBdxCTuXuCSTsYvIiLSY4UsK6WjRXqgAePn6eQRKdGckds2OwQRaV+opZJ+KkdERESkZPUOETaMmV0NnFCleIi7P9/IeDrLzDZgzW8CtrrX3T/byHhERESkfBoilLppiFCkXBoiFFkr1DRE2O17sKT7unLIowwbNqzZYYiIiHQ7moMlIiIiUjIlWCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUjJdaFTqpguNdi+6SKWISEPotwhFREREmkEJloiIiEjJlGCJiIiIlEwJloiIiEjJlGCJiIiIlEwJloiIiEjJejc7gGYxsweAocAKYBXQAox199tzdYYCF6R6vYBngCvc/cZcndHA3u5+UIV9DAcmA8sKRRPdfWQbsS3N3d0g/X2zdYG7903x3wf8DrgrV39jYDmwMt1/yN0/bWYZsI+7/77C/mYD782t06q/u79SLU4RERGprMcmWMlF7j7GzHoDZwFTzewxd59pZocAvwQuBo4HXgeOAK4xs0HufkGN+5jl7oM7EpS7923938yuA3q7+/AqdR8C8vVnAmPcfUpH9gmMcPdbOriOiIiIVKAhQsDdVwKTiAnn7mnxRGCau1/o7ovcfZm73wacDYwys4HNiVZERES6OyVYgJn1Ac5Md2eY2c7AYKBSj85U4lVcD25QeCIiIrKW6elDhKPM7BxgE+JcrBHu/oSZ7ZXK5xZXcPflZrYQ6FfjPnY0syWFZV9x96l1R901rjGzK3P3n3f33ZoWjYiIyFqspydYY9McrC2A64ED0t8Fqbw/8HR+hdTbtVWuTntaOjoHq0lO1xwsERGRcmiIEHD3xcAI4DAzOxJ4FpgFHFeh+jFABtzbuAhFRERkbdLTe7De5u4vm9mlwDhgOvFbhXeYWQtwFfFbhIcDE4Dx7t6SW309M9uwsMnlDQi7o/oU4nzL3btjnCIiIms1JViru5z4LcET3X2KmR0InA98i3gdrBnAOe4+ubDe/sQELO/Y9HdQ4bpWANPd/Vga7zeF+88Au6T/rzOzqwvlQ939ya4PS0REZN0Ssixrdgyylhowfp5Onm5kzshtmx2CiEhPEGqppDlYIiIiIiXTEGGTpOG4E6oUD3H35xsZj4iIiJRHQ4RSt+nTp2fDhg1rdhgiIiKNpCFCERERkWZQgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMl1oVOq2tv0WoX6rT0RESqALjYqIiIg0gxIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpWe9m7tzMHgCGAiuAVUALMNbdb8/VGQpckOr1Ap4BrnD3G3N1RgN7u/tBFfYxHJgMLCsUTXT3kZ2Nr8QYz3P3wbWWFZdXiHUWMMbdf9rOMfYHrgJ2B3YAvuTut7S1joiIiLStO/RgXeTufYEtgSnAVDNrTRoOAe4H/hcYBPQDxgMTzOzCDuxjlrv3LdzaTK5qia/kGMuQj3Ua8BMz27mddd4C7gGOA17o4vhERER6hO6QYAHg7iuBScRetd3T4onANHe/0N0Xufsyd78NOBsYZWYDmxxft4qxEOtVxN60Xdup+6K7T3T3h4k9XyIiItJJ3SbBMrM+wJnp7ozU8zIYqDRcNZV4oa+DGxTeGvGlZd0qxlYp1q8Shwsfb/T+RUREerrukGCNMrMlwOvAGGCEuz8BbJ3K5xZXcPflwELicFwtdjSzJYXbcZ2Mj5JjLENrrC8ARwKfc/eZDdy/iIiI0ORJ7slYdx9jZlsA1wMHpL8LUnl/4On8CqmHZqtcnfa0VJpA3sn4KDnGalYA61dYvn4qWyPWTu5PREREOqk79GAB4O6LgRHAYWZ2JPAs8ZtwlXqajgEy4N4mxgeNiXE2sI2ZbVxYPjjtW0RERLqZ7tCD9TZ3f9nMLgXGAdOBs4A7zKyFOGn7deBwYAIw3t1bcquvZ2YbFja5vKviM7Pp7v6WmZUVY6hQtgJ4hJjITTCzkcCrwF7EZO+MMo4rt98ArJ/ur0yT5UVERKSDuk0PVs7lwDbAie5+F3Ag8EliT85CYBRwjruPKqy3PzG5yd++mMoGmdnSwm1aZ+MDKDPGCmXfdPcVxIRtM+Ap4GVsFEz3AAAXRklEQVTgh2n7q12PqxNa97cD8ZphrwPnlbRtERGRHidkWdbsGGQtNWD8vLXq5JkzcttmhyAiImu/UEul7tiDJSIiIrJW61ZzsBrNzK4GTqhSPMTdn29kPF3BzPYB7qpSPM7dxzUyHhERkZ5AQ4RSNw0RiohID1TTEGGP7sGSzrlyyKMMGzas2WGIiIh0O5qDJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyXWhU6tbdLzSqC4uKiEgX0G8RioiIiDSDEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkvVudgBlMrMHgKHACmAV0AKMdffbc3WGAheker2AZ4Ar3P3GXJ3RwN7uflCFfQwHJgPLCkUT3X1kB+Jrdau7j8jV2RF4DrjP3Q8prD8GGAX80N2/llu+MTAP2AzY3t1fMLMRwDnuvkuqcwuw1N3PKGzztNQeu7n7otzyS4F9gE+4ez5eERERace62IN1kbv3BbYEpgBTzWwwgJkdAtwP/C8wCOgHjAcmmNmFHdjHLHfvW7i1mVwV48vdRhTKTwUWAweZ2fsqrP8McLyZbZhb9kVigtVh7n4t8AhwTesyM9s/xXG8kisREZGOWxcTLADcfSUwidhLt3taPBGY5u4Xuvsid1/m7rcBZwOjzGxgc6KNzGx94CTgImIidWqFarOBR4HP55adSjzWep0KfMLMvmxmmwE3At9w9xmd2KaIiEiPtc4mWGbWBzgz3Z1hZjsDg4FbKlSfSrwy68ENCq+aI4F3E2OcDJyUkq6iSaTky8yGEHvjpte7U3dfCAwHLie2xZ9Sz5aIiIjUYV1MsEaZ2RLgdWAMMMLdnwC2TuVziyu4+3JgIXHIsBY7mtmSwu24jsSXu308V3YaMD0lPDcRk62jKmzjDmAXM3t/WmcKsLLG/Vfk7vcA04hzxIrDliIiItIB69Qk92Ssu48xsy2A64ED0t8Fqbw/8HR+hdTbtVWuTnta3H1wZ+IrLjSzQcBBwBEA7v4PM7sTOB24PV/X3VeY2U3AvwFHAx+nHI8D+6QET0REROq0LvZgAeDui4k9MYeZ2ZHAs8AsoFJP0zFABtzbuAjXcBpxmPIGM5tvZvOJCdcBrZP0CyYRh0Afd/fnGhiniIiItGNd7MF6m7u/nC43MI44R+ks4A4zawGuIg4jHg5MAMa7e0tu9fUK39QDWN4VcaZ5VsOBscCVheKHiPOtVvuWorvPMLN9qTDk2YZeFY7pTXfPOhaxiIiItGWdTrCSy4nfEjzR3aeY2YHA+cC3iNfBmkG8XtTkwnr7ExOwvGPT30FmtrRQNt3dj6U+RwGbApflr0UFYGaXAaPN7DvFldz99x3czwjWnF+1J+Ad3I6IiIi0IWSZOi+kPgPGz+vWJ8+ckds2OwQREVn3hFoqrbNzsERERESapScMETaMmV0NnFCleIi7P9/IeERERKQ5NEQoddMQoYiI9EA1DRGqB0vqduWQRxk2bFizwxAREel2NAdLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGQhy7JmxyBrqQ022OCp5cuXv9HsONY1vXv33mrlypULmx3HukbtWj61addQu3aNEtt1YZZlh7a7vxJ2JD3Urrvu+oa7W7PjWNeYmatdy6d2LZ/atGuoXbtGo9tVQ4QiIiIiJVOCJSIiIlIyJVjSGdc2O4B1lNq1a6hdy6c27Rpq167R0HbVJHcRERGRkqkHS0RERKRk+hahrMHMdgZuBLYEFgEnuvuzhTq9gCuAQ4EMuMTdr2uvrCcroV1HA18B5qXqD7v7VxsTffdUY5seAowDdgV+6O7n5Mp0rlZQQruORufqGmps1+8AxwAr0+1cd/91KtsYuAHYI5Wd4+53Nu4Iup8S2nQKcBDQevmG2919bBmxqQdLKrkamOjuOwMTgWsq1DkeGAzsBAwFRpvZwBrKerLOtivATe6+e7r1+DcsamvTWcCpwPcrlOlcrayz7Qo6VyuppV0fAfZ09w8DJwM/MbONUtk5wGvuPhgYBlxnZn0bEHd31tk2hfjBqvVcLSW5AiVYUmBm/YCPAtPSomnAR81s60LVo4FJ7v6Wuy8A7gC+UENZj1RSu0pOrW3q7jPd/THiJ9citXdBSe0qBR1o11+7+7J09wkgEHtnIJ6vV6d6zwIOfLqLQ++2SmrTLqMES4q2B+a6+yqA9HdeWp63AzAnd//5XJ22ynqqMtoV4Bgze8LM7jGzoV0Z8Fqg1jZti87VNZXRrqBztaiedj0ReM7dX0j3db6urow2Bfi6mT1pZneY2QfKCk4Jlsja42pgR3ffjTgs8wsz6/JPYSJ10LnaSWa2L3ARcGyzY1lXVGnTUcBgd98V+Blwd5qb2WlKsKTo70D/1hMs/d02Lc97HhiQu79Drk5bZT1Vp9vV3ee7+4r0/71p+Ye6OO7urNY2bYvO1TV1ul11rlZUc7umHr9bgKPc/Zlckc7X1XW6Td19rru/lf6/CegLbFdGcEqwZDXu/hLwZ97J8I8FHkvzU/JuB041s/XSePdRwE9rKOuRymhXM+vfWsnMdgcGAs/QQ3WgTduic7WgjHbVubqmWtvVzPYEfgJ83t3/VNjM7cDpqd5OwJ7A3V0Zd3dWRpsWztVPAauAuWXEp8s0SCVnADea2fnAYuKYNWb2P8D57u7AzcDHgNavw37X3Wel/9sq68k6267jzGwP4gvAcuBL7j6/kQfQDbXbpma2N3ArsCkQzOwY4JT0NW2dq5V1tl11rlZWy2vAVcBGwDVmb/8u8Zfc/UnicOsUM5tJbNvT3P21Bh9Dd9PZNr3RzN4DvAW8CnzG3Uv54oau5C4iIiJSMg0RioiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZb0KCGET4UQHsrd3y+EMLuJITVMCGFKCOG6Erc3MISQ5e5vHUKYE0LYqoZ1zwgh3FxWLGuDEMI+IYQlzY6jJwohnNCR53nZzxVpW1c9N+p43MeHEC4qa/9KsKTHCCEE4DLggnbqnRlCeCqE8GoIYXEIwUMIR+fKZ4cQTqiw3hrLQzQjbatvoWy/EEIWQliabvNCCDeEEN7duSNtjizLFgBTab993wV8FxjdgLC6jSzLHsqybPNmx1FNCGF0COG+ZsfRE3RVW4cQHgghnFf2drta8bnRxHPxEuCrIYT+7dasgRIs6UkOAfoA91erEEI4lpggnAJsRvzZhbOJF7Crx/7AIOJF7Cr9ptiqLMv6ZlnWF9gbGApMqHNf3cFk4KQQwqZt1DkBeDLLsucaFNNqQgi9Qgh67ROR1WRZthi4i3S1/M7Si4x0idSbc14I4f7UO/NkCGG3EMKxIYSZIYRXQgjXhRB659bZIYTw3yGEF9Pt2hDCJrnycSGEWWl7z4UQ/iNXNjD1Bn0phPDXEMJrIYR7Qgjb5MI6Crgva/vqup8AHsyy7I9Z9Hr6dHVPnU1xOvGnLG6mnSdtlmWzgDuBjxTLQgi9U5scWVh+Ywhhcvr/wBDCH1Ov24IQwq0hhH7V9pfaa+/c/f1CCCtz93uHEM5NPXBLQggPhxD2aOcYngUWAge1Ue0o4N5CLP8eQng6PW7PhxAuDiH0SmU/CCH8vFB//1T3Xen+h0IIvw4hLMytv34qaz03Tgkh/BVYBvQLIRwTQng89S6+GEK4pnV7ab33hhCmp3N1Rlo/CyEMzNU5NfV2vhJCeCyEcEi1g67QvlNCCDeHECan9p2bnh+7hxD+Lx3f/SGEbXPrzA4hnB9C+H16HngIYc9ceZvnQAhh/fSYPpO2/1wI4XMh9tCeC+wX3ulRHVTlOPZN+3glPWan58r2CyGsDCEcnbb9SgjhtvzzuML26nmt2C2E8Nt0nLPS+r1y5f+S2mZpCOH3xA85+X1unM6rlhDCyyGEu0MIg6vFWCHmLUMIN6XzZn6Iz8N358pX683OnYPbVWvrEMLwdLwj03ZfCiH8V4XzeLvcdoeHEGam/68E9gG+k7ZZ8aeJQuwd+k2Iw2ELQgiLQghfDyEMSG36Wgjh0RDCB3LrdOq5kjvXJ+XO9TXOm/R/m+1TOJbVhnJLetzvJb5GdV6WZbrpVvoNmE38+ZEPAOsTf2TzOeBa4F3EHyl9CTgu1d8QmEkcOtoI2AL4H2BybpsnEHuUAnAA8DrwqVQ2EMiICcpWxJ/veBiYlFv/j8DXCnHuB8zO3f8C8AYwBjgQ2LzKsZ3Q3nJga+BN4F+B3VN8exT2vTJ3fzDx99omV2nT7wF35O73BZYC+6T7exN/m6w38F7gQWBarv4U4Lrc/QzYu414xqU2GwT0IvbqLQS2yLd5hTinA2PaODf+AXymsOxzwI7psf1IqnN6KhtC/LmVrXP1bwSuT//3AxYRE9g+QH/AgfML58ZvUrv0ScfzaeCDxA+ag4G/Ahfn9vEb4u8Sbpr28UDazsBUfhrxnP1w2sZh6fEYXOW4i+07hXgOH57WPyOt/0vij81uDPwWuLZwjs0D9kjH8W1gAbBpjefA+HScu6W23g7YLZWNJn4Aaet5vWOK+aS0j48DLwNfyB1jBlxPPD/fQ3wdGFXia8Vm6fz4DrBBWm8W8M1c+aLUNn1Se8xn9ef5VOJrxXtSnQuBp4H1Kz1XKsR8N/E83yLdfgX8qo3XgoGpXbar1tbAcGAFMJH4Gvg+YAbwn5W2kVtnZu7+A8B57TyGo9N+RvDO82AVcF/hMbgnt05nnytTiOfNZ9I2/jXFMKDKc6Na+8wsLHv7cSrjcU919iCOOPRpqx1ruTX0TVe3nnNLLzDfzN0/LD3h8m+StwGXpf8/DzxX2MYexASlV5V9/DfwvfR/64vPnrnyrwKP5e7PAIYXtrFf/gmYlh0B/Iz4Ir6KOKT4ocKx/RNYUri9xeovqt8ivjG0vmj/CbimsO8srbsYaAGupkJSl+p/gJho9Ev3TwZmtPEYHAG8lLv/9otRul81wSK++b4GfLKwzSdbj5HqCdaPgavaiGs5sF87588PgNty9/8InJ3+34SYiOyV7p8D/Law/udIL8a5c+OT7ezzLOCR9P92aZ1BufIDWf1N4yngxMI2plPlDY7KCVb+TXnjtP0v5JZ9hdXP4dnARbn7AXielHy0dQ6kukuBw6vUHU37Cda5wMOFZRcDvy6c0/nn+feBn7exzdl07LXiOODvpJ96S8tOB55J/x+f2iRfPpb0PCd+AMuAHXLl6wGvkJ4PtJFgET/kZcBOuWXvT8u2yR1TPQnWm8DGuWUjSM/x4jZy69STYP2lsOylCo/B4hKfK1PInetp2QLgyCrPjWrt01aC1enHPS3bKdXr11Y71nLTjz1LV3ox9/8y4nyjBYVlrUMHOwI7hDW/SZIRP4nPDSF8DTiV+IQOxE95U9vY5z9z24eYxLQ1NyjuMMvuJH7KIYSwC/GHQu8MIeyYpWcgsXfllvx6IfdtlRBCSLHekmXZirT4euCSEMI3sixbmpatymqc+Jxl2d9CCH8i9uRdSuxFuCG3zz2IvU4fJr5ZB2IvQj22SutOD7lvChI/3W5XeZW3bUpMFqtZ43EIce7b14m9Zb2Jny7/kKtyAzHZuAz4IjA3y7KHU9mOwF6FcycQP53nzS7s82DgfGAXYk9IL+IbDcReMIgv2K3mFLa3IzAxhHBFbllv4AVq9/b5mmXZsnjarPG8KQ6vzc6tk4UQnic9Ju2cA1sTe4RmdCC+ou2JvUV5zwH5oevi87z4PKykI68V2xPfNPPn5XNpOcS2mFMoz5+PO6a/T6T2brV+bhttaa2T3+ZzubIXqd9LWZYty92fTfvPt3oUY1xGG+ddCc+VSvus5bzoiLIe901554Nvp2gOlnQXc4if1DYv3DbMsmxuCGEv4vDG6cBWKSmZTnwDqdVjxOGmmmVZ9jTxTX0AcSigVgcSu9JPTnM05hO7o/sSP4HX6wZgeJo38HHgplzZrcResp2zLNuUypPq8/5JfMNttW3u/4Wp/KDC4/GuLMsuaWe7HyK2dTWrPQ4hhO2JQxJjiD0AmxGHSfKP7a3ATiGEjxI/yd6QK5tD/LSbj3OzLH5xIO+t3D77AHek7e6Q2mtkbp9z098dcuvn/2/d78mF/fbNsuzMNo69DANb/0mJ/A68k9S1dQ4sID6mO1XZ7ltVluf9nXfeqFoNSssb5e/AgLD6u2Q+hrkVyvMxt77571R47DbOsmxajfuH3OPAO3N9WsuWUv25BdXbul8IYePc/YG889i2fiirZ7t1K+m50lGVjqPYprD68Zf1uH+I2MO3vN7gWynBku7iTqB1Au4mIeofQvhsKt+UOFy3AMhCCIcT5wV0xB3ExKeqEMLJIYQvhHQtpzSh9Azgr1mWvdyBfZ1GnP+yC3H+1e7EJ+4NdO4bKrcSE7crgHuzLJubK9uU2N39WghhB+JchLY48OUQQp80GfXrrQXpU+DlwA9CCDsBhBD6hngdseKL+ttS4rc1cT5HNXew+iT4vsTXogXAihDCx4Ev5VfIsmwJ8HNiElZMLG8CLD12G4YQ1kuTYg9tI4Y+xHl/i7Msez2EMIQ47NG6vxeIwy2XpPOxH1D8+vtlwOgQJ6WHEMJGIYS9U69nVzo5hPDRECc/f5PYU/WrVFb1HEiP6Y+A74X4pYDW59iuqcp8Yi9ynzb2PQ3YI4RwYohfgvgX4vl8falH2LZfER+7c9O5+37iG35rDHcSz6lvhjip/6PE4XQAsix7idjzfVVIX8cPIWweQvhsKFxKpZIsy+YB9wD/ldbbAvgv4K4sy1p7aRw4Nj1ntibOF8ur1tbrEc+5jUL8ksE5xPmGZFm2kJTUh/hN2F2JveTF7dY8Wb9GZTxXOqpS+zxGTECPSM/xzwKfzJWX9bgfTHyN6jQlWNItpG7xA4k9G08T3yR+Q0xMAH5N/CbeI8Telc8T33A74tfAyhDCfm3UWUwcivpbCOGfxLk/S4hzWWqSXmCOAn6QZdn8/I3YC/eREIJ1MHYAsix7hXjcnyZeEiHvNOKcjdeIc8hub2dzZxFfjF8mznGZUii/APgF8IsQwqvEichn0PbrxsnAlBRnNTcDH05vIGRZ9rfcvpYQk4JKPQk3EI/71+lNjrT+fOLlMI4iDqksJrZRxW/BpXWWAmcSk42lxB6z4nDzccTk5QXg97zTnm+mbUwifvHghrTP54lvpOu3cexluJaYYC8GjibOqWpt7/bOgVHEx/qOVOd3vNOjdTuxB2Z+iN/0KvZUkWVZC3F+zlnECcU3E79McFtpR9eOdKyHEJP0fxCf1zcRh81bk/HDiW2zmNhWPyps5lTiF0oeCCG8Rpxb+AXi0FAtTiC239PptgQ4MVd+HvED4YvE5OPWwvrV2noOsSemhfjaczfxHGv1ZeJr0SvpeIuJ7WXEDxtLQgh/qfFY2lTGc6UOa7RPFi/r8u/E8/9l4FDixPrWODv9uIcQNiee31fXGfdqwurDlSLrttSrcW6WZZ9M9/cjJgQDmxnX2ij1erVkWRbS/a2ARwErzJ+ptO4ZxEnqX2qrXncSQvgUMQncKGvSC2eI8/zOK87/k7VfCGE48bEtuweq4brDc6UeIYSLifP/SrlYqya5S4+SZdndxE+FUrI0hDGgxrpXU9KnxK4SQvgw8ZPtk8S5HGOAn6xNbxgijbCuPFeyLPvPMrenIULp6Wazdl85vZmWECfur6veTRxmW0oc9niCOEQhIqvTc6UCDRGKiIiIlEw9WCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUjIlWCIiIiIlU4IlIiIiUrL/D/S6lzRfehE6AAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"shap.summary_plot(shap_values, X, plot_type=\"bar\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Hyperparameter tunning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Training speed"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a92eadf3c1c044dea4c7e513a9cc0443",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 89,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from catboost import CatBoost\n",
"fast_model = CatBoostClassifier(\n",
" random_seed=63,\n",
" iterations=150,\n",
" learning_rate=0.01,\n",
" boosting_type='Plain',\n",
" bootstrap_type='Bernoulli',\n",
" subsample=0.5,\n",
" rsm=0.5,\n",
" one_hot_max_size=20,\n",
" leaf_estimation_iterations=2,\n",
" max_ctr_complexity=1)\n",
"\n",
"fast_model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" verbose=False,\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Accuracy"
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e1847556d61748cea1e37bb8c30ccb1a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 90,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tunned_model = CatBoostClassifier(\n",
" random_seed=63,\n",
" iterations=1000,\n",
" learning_rate=0.03,\n",
" l2_leaf_reg=3,\n",
" bagging_temperature=1,\n",
" random_strength=1,\n",
" one_hot_max_size=2,\n",
" leaf_estimation_method='Newton'\n",
")\n",
"tunned_model.fit(\n",
" X_train, y_train,\n",
" cat_features=cat_features,\n",
" verbose=False,\n",
" eval_set=(X_validation, y_validation),\n",
" plot=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training the model after parameter tunning"
]
},
{
"cell_type": "code",
"execution_count": 91,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.6431867\ttotal: 103ms\tremaining: 1m 51s\n",
"100:\tlearn: 0.1590533\ttotal: 8.58s\tremaining: 1m 23s\n",
"200:\tlearn: 0.1517316\ttotal: 16.1s\tremaining: 1m 10s\n",
"300:\tlearn: 0.1487795\ttotal: 23.5s\tremaining: 1m\n",
"400:\tlearn: 0.1468414\ttotal: 31.3s\tremaining: 53.2s\n",
"500:\tlearn: 0.1448880\ttotal: 39.4s\tremaining: 45.7s\n",
"600:\tlearn: 0.1433889\ttotal: 48.4s\tremaining: 38.8s\n",
"700:\tlearn: 0.1421200\ttotal: 58.1s\tremaining: 31.7s\n",
"800:\tlearn: 0.1411938\ttotal: 1m 6s\tremaining: 23.4s\n",
"900:\tlearn: 0.1401819\ttotal: 1m 15s\tremaining: 15.2s\n",
"1000:\tlearn: 0.1394250\ttotal: 1m 25s\tremaining: 6.99s\n",
"1082:\tlearn: 0.1388355\ttotal: 1m 32s\tremaining: 0us\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 91,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"best_model = CatBoostClassifier(\n",
" random_seed=63,\n",
" iterations=int(tunned_model.tree_count_ * 1.2),\n",
")\n",
"best_model.fit(\n",
" X, y,\n",
" cat_features=cat_features,\n",
" verbose=100\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculate predictions for the contest"
]
},
{
"cell_type": "code",
"execution_count": 92,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predictoins:\n",
"[[0.4024 0.5976]\n",
" [0.015 0.985 ]\n",
" [0.0101 0.9899]\n",
" ...\n",
" [0.0089 0.9911]\n",
" [0.0496 0.9504]\n",
" [0.0135 0.9865]]\n"
]
}
],
"source": [
"X_test = test_df.drop('id', axis=1)\n",
"test_pool = Pool(data=X_test, cat_features=cat_features)\n",
"contest_predictions = best_model.predict_proba(test_pool)\n",
"print('Predictoins:')\n",
"print(contest_predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare the submission"
]
},
{
"cell_type": "code",
"execution_count": 93,
"metadata": {},
"outputs": [],
"source": [
"f = open('submit.csv', 'w')\n",
"f.write('Id,Action\\n')\n",
"for idx in range(len(contest_predictions)):\n",
" line = str(test_df['id'][idx]) + ',' + str(contest_predictions[idx][1]) + '\\n'\n",
" f.write(line)\n",
"f.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Submit your solution [here](https://www.kaggle.com/c/amazon-employee-access-challenge/submit).\n",
"Good luck!!!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
},
"widgets": {
"state": {
"1057714ebc614324aa3ba2cf69408966": {
"views": [
{
"cell_index": 17
}
]
},
"8381e9eed05f4a03905ae8a56d7ab4ea": {
"views": [
{
"cell_index": 48
}
]
},
"f49684e8c5c44241bfe2c7f577f5cb41": {
"views": [
{
"cell_index": 53
}
]
}
},
"version": "1.2.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}