{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aec0fde7",
   "metadata": {},
   "source": [
    "# T0. trainer 和 evaluator 的基本使用\n",
    "\n",
    "  1   trainer 和 evaluator 的基本关系\n",
    " \n",
    "    1.1   trainer 和 evaluater 的初始化\n",
    "\n",
    "    1.2   driver 的含义与使用要求\n",
    "\n",
    "    1.3   trainer 内部初始化 evaluater\n",
    "\n",
    "  2   使用 trainer 训练模型\n",
    "\n",
    "    2.1   argmax 模型实例\n",
    "\n",
    "    2.2   trainer 的参数匹配\n",
    "\n",
    "    2.3   trainer 的实际使用 \n",
    "\n",
    "  3   使用 evaluator 评测模型\n",
    " \n",
    "    3.1   trainer 外部初始化的 evaluator\n",
    "\n",
    "    3.2   trainer 内部初始化的 evaluator "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09ea669a",
   "metadata": {},
   "source": [
    "## 1. trainer 和 evaluator 的基本关系\n",
    "\n",
    "### 1.1  trainer 和 evaluator 的初始化\n",
    "\n",
    "在`fastNLP 0.8`中，**`Trainer`模块和`Evaluator`模块分别表示“训练器”和“评测器”**\n",
    "\n",
    "  对应于之前的`fastNLP`版本中的`Trainer`模块和`Tester`模块，其定义方法如下所示\n",
    "\n",
    "在`fastNLP 0.8`中，需要注意，在同个`python`脚本中先使用`Trainer`训练，然后使用`Evaluator`评测\n",
    "\n",
    "  非常关键的问题在于**如何正确设置二者的`driver`**。这就引入了另一个问题：什么是 `driver`？\n",
    "\n",
    "\n",
    "```python\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    train_dataloader=train_dataloader,\n",
    "    optimizers=optimizer,\n",
    "\t...\n",
    "\tdriver=\"torch\",\n",
    "\tdevice=0,\n",
    "\t...\n",
    ")\n",
    "...\n",
    "evaluator = Evaluator(\n",
    "    model=model,\n",
    "    dataloaders=evaluate_dataloader,\n",
    "    metrics={'acc': Accuracy()} \n",
    "    ...\n",
    "    driver=trainer.driver,\n",
    "\tdevice=None,\n",
    "    ...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c11fe1a",
   "metadata": {},
   "source": [
    "### 1.2  driver 的含义与使用要求\n",
    "\n",
    "在`fastNLP 0.8`中，**`driver`**这一概念被用来表示**控制具体训练的各个步骤的最终执行部分**\n",
    "\n",
    "  例如神经网络前向、后向传播的具体执行、网络参数的优化和数据在设备间的迁移等\n",
    "\n",
    "在`fastNLP 0.8`中，**`Trainer`和`Evaluator`都依赖于具体的`driver`来完成整体的工作流程**\n",
    "\n",
    "  具体`driver`与`Trainer`以及`Evaluator`之间的关系请参考`fastNLP 0.8`的框架设计\n",
    "\n",
    "注：在同一脚本中，`Trainer`和`Evaluator`使用的`driver`应当保持一致\n",
    "\n",
    "  一个不能违背的原则在于：**不要将多卡的`driver`前使用单卡的`driver`**（？？？），这样使用可能会带来很多意想不到的错误。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cac4a1a",
   "metadata": {},
   "source": [
    "### 1.3 Trainer 内部初始化 Evaluator\n",
    "\n",
    "在`fastNLP 0.8`中，如果在**初始化`Trainer`时**，**传入参数`evaluator_dataloaders`和`metrics`**\n",
    "\n",
    "  则在`Trainer`内部，也会初始化单独的`Evaluator`来帮助训练过程中对验证集的评测\n",
    "\n",
    "```python\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    train_dataloader=train_dataloader,\n",
    "    optimizers=optimizer,\n",
    "\t...\n",
    "\tdriver=\"torch\",\n",
    "\tdevice=0,\n",
    "\t...\n",
    "    evaluate_dataloaders=evaluate_dataloader,\n",
    "    metrics={'acc': Accuracy()},\n",
    "\t...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c9c7dda",
   "metadata": {},
   "source": [
    "## 2. 使用 trainer 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524ac200",
   "metadata": {},
   "source": [
    "### 2.1 argmax 模型实例\n",
    "\n",
    "本节将通过训练`argmax`模型，简单介绍如何`Trainer`模块的使用方式\n",
    "\n",
    "  使用`pytorch`定义`argmax`模型，输入一组固定维度的向量，输出其中数值最大的数的索引\n",
    "\n",
    "  除了添加`pytorch`要求的`forward`方法外，还需要添加 **`train_step`** 和 **`evaluate_step`** 这两个方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5314482b",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class ArgMaxModel(nn.Module):\n",
    "    def __init__(self, num_labels, feature_dimension):\n",
    "        super(ArgMaxModel, self).__init__()\n",
    "        self.num_labels = num_labels\n",
    "\n",
    "        self.linear1 = nn.Linear(in_features=feature_dimension, out_features=10)\n",
    "        self.ac1 = nn.ReLU()\n",
    "        self.linear2 = nn.Linear(in_features=10, out_features=10)\n",
    "        self.ac2 = nn.ReLU()\n",
    "        self.output = nn.Linear(in_features=10, out_features=num_labels)\n",
    "        self.loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.ac1(self.linear1(x))\n",
    "        x = self.ac2(self.linear2(x))\n",
    "        x = self.output(x)\n",
    "        return x\n",
    "\n",
    "    def train_step(self, x, y):\n",
    "        x = self(x)\n",
    "        return {\"loss\": self.loss_fn(x, y)}\n",
    "\n",
    "    def evaluate_step(self, x, y):\n",
    "        x = self(x)\n",
    "        x = torch.max(x, dim=-1)[1]\n",
    "        return {\"pred\": x, \"target\": y}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca897322",
   "metadata": {},
   "source": [
    "在`fastNLP 0.8`中，**函数`train_step`是`Trainer`中参数`train_fn`的默认值**\n",
    "\n",
    "  由于，在`Trainer`训练时，**`Trainer`通过参数`_train_fn_`对应的模型方法获得当前数据批次的损失值**\n",
    "\n",
    "  因此，在`Trainer`训练时，`Trainer`首先会寻找模型是否定义了`train_step`这一方法\n",
    "\n",
    "    如果没有找到，那么`Trainer`会默认使用模型的`forward`函数来进行训练的前向传播过程\n",
    "\n",
    "注：在`fastNLP 0.8`中，`Trainer`要求模型通过`train_step`来返回一个字典，将损失值作为`loss`的键值\n",
    "\n",
    "  此外，这里也可以通过传入`Trainer`的参数`output_mapping`来实现高度化的定制，具体请见这一note（？？？）\n",
    "\n",
    "同样，在`fastNLP 0.8`中，**函数`evaluate_step`是`Evaluator`中参数`evaluate_fn`的默认值**\n",
    "\n",
    "  在`Evaluator`测试时，**`Evaluator`通过参数`evaluate_fn`对应的模型方法获得当前数据批次的评测结果**\n",
    "\n",
    "  从用户角度，模型通过`evaluate_step`方法来返回一个字典，内容与传入`Evaluator`的`metrics`一致\n",
    "\n",
    "<!-- &emsp; 从模块角度，`fastNLP 0.8`会匹配该字典的键值和一个`metric`的更新函数的函数签名，自动地将`metric`所需要的内容传给该`metric`，也就是我们会自动进行“**参数匹配**”。 -->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb3272eb",
   "metadata": {},
   "source": [
    "### 2.2 trainer 的参数匹配\n",
    "\n",
    "`fastNLP 0.8`中的参数匹配涉及到两个方面，一是在模型训练或者评测的前向传播过程中，如果从`dataloader`中出来一个`batch`的数据是一个字典，那么我们会查看模型的`train_step`和`evaluate_step`方法的参数签名，然后对于每一个参数，我们会根据其名字从 batch 这一字典中选择出对应的数据传入进去。例如在接下来的定义`Dataset`的部分，注意`ArgMaxDatset`的`__getitem__`方法，您可以通过在`Trainer`和`Evaluator`中设置参数 `model_wo_auto_param_call`来关闭这一行为。当您关闭了这一行为后，我们会将`batch`直接传给您的`train_step`、`evaluate_step`或者 `forward`函数。\n",
    "\n",
    "二是在传入`Trainer`或者`Evaluator metrics`后，我们会在需要评测的时间点主动调用`metrics`来对`evaluate_dataloaders`进行评测，这一功能主要就是通过对`metrics`的`update`方法和一个`batch`的数据进行参数评测实现的。首先需要明确的是一个 metric 的计算通常分为 `update` 和 `get_metric`两步，其中`update`表示更新一个`batch`的评测数据，`get_metric` 表示根据已经得到的评测数据计算出最终的评测值，例如对于 `Accuracy`来说，其在`update`的时候会更新一个`batch`计算正确的数量 right_num 和计算错误的数量 total_num，最终在 `get_metric` 时返回评测值`right_num / total_num`。\n",
    "\n",
    "因为`fastNLP 0.8`的`metrics`是自动计算的（只需要传给`Trainer`或者`Evaluator`），因此其一定依赖于参数匹配。对于从`evaluate_dataloader`中生成的一个`batch`的数据，我们会查看传给 `Trainer`（最终是传给`Evaluator`）和`Evaluator`的每一个`metric`，然后查看其`update`函数的函数签名，然后根据每一个参数的名字从`batch`字典中选择出对应的数据传入进去。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f62b7bb1",
   "metadata": {},
   "source": [
    "### 2.3 trainer的实际使用\n",
    "\n",
    "接下来我们创建用于训练的 dataset，其接受三个参数：数据维度、数据量和随机数种子，生成指定数量的维度为 `feature_dimension` 向量，而每一个向量的标签就是该向量中最大值的索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fe612e61",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "\n",
    "class ArgMaxDatset(Dataset):\n",
    "    def __init__(self, feature_dimension, data_num=1000, seed=0):\n",
    "        self.num_labels = feature_dimension\n",
    "        self.feature_dimension = feature_dimension\n",
    "        self.data_num = data_num\n",
    "        self.seed = seed\n",
    "\n",
    "        g = torch.Generator()\n",
    "        g.manual_seed(1000)\n",
    "        self.x = torch.randint(low=-100, high=100, size=[data_num, feature_dimension], generator=g).float()\n",
    "        self.y = torch.max(self.x, dim=-1)[1]\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.data_num\n",
    "\n",
    "    def __getitem__(self, item):\n",
    "        return {\"x\": self.x[item], \"y\": self.y[item]}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb96332",
   "metadata": {},
   "source": [
    "现在准备好数据和模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "76172ef8",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "train_dataset = ArgMaxDatset(feature_dimension=10, data_num=1000)\n",
    "evaluate_dataset = ArgMaxDatset(feature_dimension=10, data_num=100)\n",
    "\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)\n",
    "evaluate_dataloader = DataLoader(evaluate_dataset, batch_size=8)\n",
    "\n",
    "# num_labels 设置为 10，与 feature_dimension 保持一致，因为我们是预测十个位置中哪一个的概率最大。\n",
    "model = ArgMaxModel(num_labels=10, feature_dimension=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e7d25ee",
   "metadata": {},
   "source": [
    "将优化器也定义好。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dc28a2d9",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from torch.optim import SGD\n",
    "\n",
    "optimizer = SGD(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f1fba81",
   "metadata": {},
   "source": [
    "现在万事俱备，开始使用 Trainer 进行训练！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b51b7a2d",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['__annotations__',\n",
       " '__class__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " '_check_callback_called_legality',\n",
       " '_check_train_batch_loop_legality',\n",
       " '_custom_callbacks',\n",
       " '_driver',\n",
       " '_evaluate_dataloaders',\n",
       " '_fetch_matched_fn_callbacks',\n",
       " '_set_num_eval_batch_per_dl',\n",
       " '_train_batch_loop',\n",
       " '_train_dataloader',\n",
       " '_train_step',\n",
       " '_train_step_signature_fn',\n",
       " 'accumulation_steps',\n",
       " 'add_callback_fn',\n",
       " 'backward',\n",
       " 'batch_idx_in_epoch',\n",
       " 'batch_step_fn',\n",
       " 'callback_manager',\n",
       " 'check_batch_step_fn',\n",
       " 'cur_epoch_idx',\n",
       " 'data_device',\n",
       " 'dataloader',\n",
       " 'device',\n",
       " 'driver',\n",
       " 'driver_name',\n",
       " 'epoch_validate',\n",
       " 'evaluate_batch_step_fn',\n",
       " 'evaluate_dataloaders',\n",
       " 'evaluate_every',\n",
       " 'evaluate_fn',\n",
       " 'evaluator',\n",
       " 'extract_loss_from_outputs',\n",
       " 'fp16',\n",
       " 'get_no_sync_context',\n",
       " 'global_forward_batches',\n",
       " 'has_checked_train_batch_loop',\n",
       " 'input_mapping',\n",
       " 'kwargs',\n",
       " 'larger_better',\n",
       " 'load',\n",
       " 'load_model',\n",
       " 'marker',\n",
       " 'metrics',\n",
       " 'model',\n",
       " 'model_device',\n",
       " 'monitor',\n",
       " 'move_data_to_device',\n",
       " 'n_epochs',\n",
       " 'num_batches_per_epoch',\n",
       " 'on',\n",
       " 'on_after_backward',\n",
       " 'on_after_optimizers_step',\n",
       " 'on_after_trainer_initialized',\n",
       " 'on_after_zero_grad',\n",
       " 'on_before_backward',\n",
       " 'on_before_optimizers_step',\n",
       " 'on_before_zero_grad',\n",
       " 'on_exception',\n",
       " 'on_fetch_data_begin',\n",
       " 'on_fetch_data_end',\n",
       " 'on_load_checkpoint',\n",
       " 'on_load_model',\n",
       " 'on_sanity_check_begin',\n",
       " 'on_sanity_check_end',\n",
       " 'on_save_checkpoint',\n",
       " 'on_save_model',\n",
       " 'on_train_batch_begin',\n",
       " 'on_train_batch_end',\n",
       " 'on_train_begin',\n",
       " 'on_train_end',\n",
       " 'on_train_epoch_begin',\n",
       " 'on_train_epoch_end',\n",
       " 'on_validate_begin',\n",
       " 'on_validate_end',\n",
       " 'optimizers',\n",
       " 'output_mapping',\n",
       " 'run',\n",
       " 'save',\n",
       " 'save_model',\n",
       " 'set_grad_to_none',\n",
       " 'state',\n",
       " 'step',\n",
       " 'step_validate',\n",
       " 'total_batches',\n",
       " 'train_batch_loop',\n",
       " 'train_dataloader',\n",
       " 'train_fn',\n",
       " 'train_step',\n",
       " 'trainer_state',\n",
       " 'zero_grad']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from fastNLP import Trainer\n",
    "\n",
    "# 定义一个 Trainer\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    driver=\"torch\",    # 使用 pytorch 进行训练\n",
    "    device=0,          # 使用 GPU：0\n",
    "    train_dataloader=train_dataloader,\n",
    "    optimizers=optimizer,\n",
    "    n_epochs=10,       # 训练 40 个 epoch\n",
    "    progress_bar=\"rich\"\n",
    ")\n",
    "dir(trainer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f8fe9c32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FullArgSpec(args=['self', 'num_train_batch_per_epoch', 'num_eval_batch_per_dl', 'num_eval_sanity_batch', 'resume_from', 'resume_training', 'catch_KeyboardInterrupt'], varargs=None, varkw=None, defaults=(-1, -1, 2, None, True, None), kwonlyargs=[], kwonlydefaults=None, annotations={'num_train_batch_per_epoch': <class 'int'>, 'num_eval_batch_per_dl': <class 'int'>, 'num_eval_sanity_batch': <class 'int'>, 'resume_from': <class 'str'>, 'resume_training': <class 'bool'>})\n"
     ]
    }
   ],
   "source": [
    "import inspect \n",
    "\n",
    "print(inspect.getfullargspec(trainer.run))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e202d6e",
   "metadata": {},
   "source": [
    "没有问题，那么开始真正的训练！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ba047ead",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb8ca6cf",
   "metadata": {},
   "source": [
    "## 3. 使用 evaluator 评测模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16c5fa4",
   "metadata": {},
   "source": [
    "模型训练好了我们开始使用 Evaluator 进行评测，查看效果怎么样吧。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1c6b6b36",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from fastNLP import Evaluator\n",
    "from fastNLP.core.metrics import Accuracy\n",
    "\n",
    "evaluator = Evaluator(\n",
    "    model=model,\n",
    "    driver=trainer.driver,  # 使用 trainer 已经启动的 driver；\n",
    "    device=None,\n",
    "    dataloaders=evaluate_dataloader,\n",
    "    metrics={'acc': Accuracy()}  # 注意这里一定得是一个字典；\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "257061df",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__annotations__',\n",
       " '__class__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " '_dist_sampler',\n",
       " '_evaluate_batch_loop',\n",
       " '_evaluate_step',\n",
       " '_evaluate_step_signature_fn',\n",
       " '_metric_wrapper',\n",
       " '_metrics',\n",
       " 'dataloaders',\n",
       " 'device',\n",
       " 'driver',\n",
       " 'evaluate_batch_loop',\n",
       " 'evaluate_batch_step_fn',\n",
       " 'evaluate_fn',\n",
       " 'evaluate_step',\n",
       " 'finally_progress_bar',\n",
       " 'get_dataloader_metric',\n",
       " 'input_mapping',\n",
       " 'metrics',\n",
       " 'metrics_wrapper',\n",
       " 'model',\n",
       " 'model_use_eval_mode',\n",
       " 'move_data_to_device',\n",
       " 'output_mapping',\n",
       " 'progress_bar',\n",
       " 'remove_progress_bar',\n",
       " 'reset',\n",
       " 'run',\n",
       " 'separator',\n",
       " 'start_progress_bar',\n",
       " 'update',\n",
       " 'update_progress_bar',\n",
       " 'verbose']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(evaluator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f7cb0165",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'acc#acc'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3</span><span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\u001b[32m'acc#acc'\u001b[0m: \u001b[1;36m0.3\u001b[0m\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'acc#acc': 0.3}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9f68fa",
   "metadata": {},
   "source": [
    "## 4. 在 trainer 中加入 metric 来自动评测；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca97c9a4",
   "metadata": {},
   "source": [
    "现在我们尝试在训练过程中进行评测。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "183c7d19",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# 重新定义一个 Trainer\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    driver=trainer.driver,  # 因为我们是在同一脚本中，因此这里的 driver 同样需要重用；\n",
    "    train_dataloader=train_dataloader,\n",
    "    evaluate_dataloaders=evaluate_dataloader,\n",
    "    metrics={'acc': Accuracy()},\n",
    "    optimizers=optimizer,\n",
    "    n_epochs=10,  # 训练 40 个 epoch；\n",
    "    evaluate_every=-1,  # 表示每一个 epoch 的结束会进行 evaluate；\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "714cc404",
   "metadata": {},
   "source": [
    "再次训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2e4daa2c",
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "eabda5eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = Evaluator(\n",
    "    model=model,\n",
    "    driver=trainer.driver,  # 使用 trainer 已经启动的 driver；\n",
    "    dataloaders=evaluate_dataloader,\n",
    "    metrics={'acc': Accuracy()}  # 注意这里一定得是一个字典；\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a310d157",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'acc#acc'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.5</span><span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\u001b[32m'acc#acc'\u001b[0m: \u001b[1;36m0.5\u001b[0m\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'acc#acc': 0.5}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ef78f0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}