hummingbird
/
fastNLP

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fdd7ff16",
   "metadata": {},
   "source": [
    "# T5. trainer 和 evaluator 的深入介绍\n",
    "\n",
    "&emsp; 1 &ensp; fastNLP 中 driver 的补充介绍\n",
    " \n",
    "&emsp; &emsp; 1.1 &ensp; trainer 和 driver 的构想 \n",
    "\n",
    "&emsp; &emsp; 1.2 &ensp; device 与 多卡训练\n",
    "\n",
    "&emsp; 2 &ensp; fastNLP 中的更多 metric 类型\n",
    "\n",
    "&emsp; &emsp; 2.1 &ensp; 预定义的 metric 类型\n",
    "\n",
    "&emsp; &emsp; 2.2 &ensp; 自定义的 metric 类型\n",
    "\n",
    "&emsp; 3 &ensp; fastNLP 中 trainer 的补充介绍\n",
    "\n",
    "&emsp; &emsp; 3.1 &ensp; trainer 的内部结构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08752c5a",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 1. fastNLP 中 driver 的补充介绍\n",
    "\n",
    "### 1.1  trainer 和 driver 的构想\n",
    "\n",
    "在`fastNLP 0.8`中，模型训练最关键的模块便是**训练模块`trainer`、评测模块`evaluator`、驱动模块`driver`**，\n",
    "\n",
    "&emsp; 在`tutorial 0`中，已经简单介绍过上述三个模块：**`driver`用来控制训练评测中的`model`的最终运行**\n",
    "\n",
    "&emsp; &emsp; **`evaluator`封装评测的`metric`**，**`trainer`封装训练的`optimizer`**，**也可以包括`evaluator`**\n",
    "\n",
    "之所以做出上述的划分，其根本目的在于要**达成对于多个`python`学习框架**，**例如`pytorch`、`paddle`、`jittor`的兼容**\n",
    "\n",
    "&emsp; 对于训练环节，其伪代码如下方左边紫色一栏所示，由于**不同框架对模型、损失、张量的定义各有不同**，所以将训练环节\n",
    "\n",
    "&emsp; &emsp; 划分为**框架无关的循环控制、批量分发部分**，**由`trainer`模块负责**实现，对应的伪代码如下方中间蓝色一栏所示\n",
    "\n",
    "&emsp; &emsp; 以及**随框架不同的模型调用、数值优化部分**，**由`driver`模块负责**实现，对应的伪代码如下方右边红色一栏所示\n",
    "\n",
    "| <div align=\"center\">训练过程</div> | <div align=\"center\">框架无关 对应`trainer`</div> | <div align=\"center\">框架相关 对应`driver`</div> |\n",
    "|:--|:--|:--|\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;\">try:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;\">try:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:20px;\">for epoch in 1:n_eoochs:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:20px;\">for epoch in 1:n_eoochs:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:40px;\">for step in 1:total_steps:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:40px;\">for step in 1:total_steps:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:60px;\">batch = fetch_batch()</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:60px;\">batch = fetch_batch()</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:60px;\">loss = model.forward(batch)&emsp;</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:60px;\">loss = model.forward(batch)&emsp;</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:60px;\">loss.backward()</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:60px;\">loss.backward()</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:60px;\">model.clear_grad()</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:60px;\">model.clear_grad()</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:60px;\">model.update()</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:60px;\">model.update()</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:40px;\">if need_save:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:40px;\">if need_save:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:60px;\">model.save()</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:60px;\">model.save()</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;\">except:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;\">except:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:20px;\">process_exception()</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:20px;\">process_exception()</div> |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e55f07b",
   "metadata": {},
   "source": [
    "&emsp; 对于评测环节，其伪代码如下方左边紫色一栏所示，同样由于不同框架对模型、损失、张量的定义各有不同，所以将评测环节\n",
    "\n",
    "&emsp; &emsp; 划分为**框架无关的循环控制、分发汇总部分**，**由`evaluator`模块负责**实现，对应的伪代码如下方中间蓝色一栏所示\n",
    "\n",
    "&emsp; &emsp; 以及**随框架不同的模型调用、评测计算部分**，同样**由`driver`模块负责**实现，对应的伪代码如下方右边红色一栏所示\n",
    "\n",
    "| <div align=\"center\">评测过程</div> | <div align=\"center\">框架无关 对应`evaluator`</div> | <div align=\"center\">框架相关 对应`driver`</div> |\n",
    "|:--|:--|:--|\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;\">try:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;\">try:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:20px;\">model.set_eval()</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:20px;\">model.set_eval()</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:20px;\">for step in 1:total_steps:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:20px;\">for step in 1:total_steps:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:40px;\">batch = fetch_batch()</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:40px;\">batch = fetch_batch()</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:40px;\">outputs = model.evaluate(batch)&emsp;</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:40px;\">outputs = model.evaluate(batch)&emsp;</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:40px;\">metric.compute(batch, outputs)</div> |  | <div style=\"font-family:Consolas;font-weight:bold;color:red;text-indent:40px;\">metric.compute(batch, outputs)</div> |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:20px;\">results = metric.get_metric()</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:20px;\">results = metric.get_metric()</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;\">except:</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;\">except:</div> |  |\n",
    "| <div style=\"font-family:Consolas;font-weight:bold;color:purple;text-indent:20px;\">process_exception()</div> | <div style=\"font-family:Consolas;font-weight:bold;color:blue;text-indent:20px;\">process_exception()</div> |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ba11c6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "由此，从程序员的角度，`fastNLP v0.8`**通过一个`driver`让基于`pytorch`、`paddle`、`jittor`框架的模型**\n",
    "\n",
    "&emsp; &emsp; **都能在相同的`trainer`和`evaluator`上运行**，这也**是`fastNLP v0.8`相比于之前版本的一大亮点**\n",
    "\n",
    "&emsp; 而从`driver`的角度，`fastNLP v0.8`通过定义一个`driver`基类，**将所有张量转化为`numpy.tensor`**\n",
    "\n",
    "&emsp; &emsp; 并由此泛化出`torch_driver`、`paddle_driver`、`jittor_driver`三个子类，从而实现了\n",
    "\n",
    "&emsp; &emsp; 对`pytorch`、`paddle`、`jittor`的兼容，有关后两者的实践请参考接下来的`tutorial-6`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab1cea7d",
   "metadata": {},
   "source": [
    "### 1.2  device 与 多卡训练\n",
    "\n",
    "**`fastNLP v0.8`支持多卡训练**，实现方法则是**通过将`trainer`中的`device`设置为对应显卡的序号列表**\n",
    "\n",
    "&emsp; 由单卡切换成多卡，无论是数据、模型还是评测都会面临一定的调整，`fastNLP v0.8`保证：\n",
    "\n",
    "&emsp; &emsp; 数据拆分时，不同卡之间相互协调，所有数据都可以被训练，且不会使用到相同的数据\n",
    "\n",
    "&emsp; &emsp; 模型训练时，模型之间需要交换梯度；评测计算时，每张卡先各自计算，再汇总结果\n",
    "\n",
    "&emsp; 例如，在评测计算运行`get_metric`函数时，`fastNLP v0.8`将自动按照`self.right`和`self.total`\n",
    "\n",
    "&emsp; &emsp; 指定的**`aggregate_method`方法**，默认为`sum`，将每张卡上结果汇总起来，因此最终\n",
    "\n",
    "&emsp; &emsp; 在调用`get_metric`方法时，`Accuracy`类能够返回全部的统计结果，代码如下\n",
    "    \n",
    "```python\n",
    "trainer = Trainer(\n",
    "        model=model,                                # model 基于 pytorch 实现 \n",
    "        train_dataloader=train_dataloader,\n",
    "        optimizers=optimizer,\n",
    "        ...\n",
    "        driver='torch',                             # driver 使用 torch_driver \n",
    "        device=[0, 1],                              # gpu 选择 cuda:0 + cuda:1\n",
    "        ...\n",
    "        evaluate_dataloaders=evaluate_dataloader,\n",
    "        metrics={'acc': Accuracy()},\n",
    "        ...\n",
    "    )\n",
    "\n",
    "class Accuracy(Metric):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.register_element(name='total', value=0, aggregate_method='sum')\n",
    "        self.register_element(name='right', value=0, aggregate_method='sum')\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e0a210",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "注：`fastNLP v0.8`中要求`jupyter`不能多卡，仅能单卡，故在所有`tutorial`中均不作相关演示"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d19220c",
   "metadata": {},
   "source": [
    "## 2. fastNLP 中的更多 metric 类型\n",
    "\n",
    "### 2.1  预定义的 metric 类型\n",
    "\n",
    "在`fastNLP 0.8`中，除了前几篇`tutorial`中经常见到的**正确率`Accuracy`**，还有其他**预定义的评测标准`metric`**\n",
    "\n",
    "&emsp; 包括**所有`metric`的基类`Metric`**、适配`Transformers`中相关模型的正确率`TransformersAccuracy`\n",
    "\n",
    "&emsp; &emsp; **适用于分类语境下的`F1`值`ClassifyFPreRecMetric`**（其中也包括召回率`Pre`、精确率`Rec`\n",
    "\n",
    "&emsp; &emsp; **适用于抽取语境下的`F1`值`SpanFPreRecMetric`**；相关基本信息内容见下表，之后是详细分析\n",
    "\n",
    "| <div align=\"center\">代码名称</div> | <div align=\"center\">简要介绍</div> | <div align=\"center\">代码路径</div> |\n",
    "|:--|:--|:--|\n",
    "| `Metric` | 定义`metrics`时继承的基类 | `/core/metrics/metric.py` |\n",
    "| `Accuracy` | 正确率，最为常用 | `/core/metrics/accuracy.py` |\n",
    "| `TransformersAccuracy` | 正确率，为了兼容`Transformers`中相关模型 | `/core/metrics/accuracy.py` |\n",
    "| `ClassifyFPreRecMetric` | 召回率、精确率、F1值，适用于**分类问题** | `/core/metrics/classify_f1_pre_rec_metric.py` |\n",
    "| `SpanFPreRecMetric` | 召回率、精确率、F1值，适用于**抽取问题** | `/core/metrics/span_f1_pre_rec_metric.py` |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc083a3",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "&emsp; 如`tutorial-0`中所述，所有的`metric`都包含`get_metric`和`update`函数，其中\n",
    "\n",
    "&emsp; &emsp; **`update`函数更新单个`batch`的统计量**，**`get_metric`函数返回最终结果**，并打印显示\n",
    "\n",
    "\n",
    "### 2.1.1  Accuracy 与 TransformersAccuracy\n",
    "\n",
    "`Accuracy`，正确率，预测正确的数据`right_num`在总数据`total_num`，中的占比（公式就不用列了\n",
    "\n",
    "&emsp; `get_metric`函数打印格式为 **`{\"acc#xx\": float, 'total#xx': float, 'correct#xx': float}`**\n",
    "\n",
    "&emsp; 一般在初始化时不需要传参，`fastNLP`会根据`update`函数的传入参数确定对应后台框架`backend`\n",
    "\n",
    "&emsp; **`update`函数的参数包括`pred`、`target`、`seq_len`**，**后者用来标记批次中每笔数据的长度**\n",
    "\n",
    "`TransformersAccuracy`，继承自`Accuracy`，只是为了兼容`Transformers`框架中相关模型\n",
    "\n",
    "&emsp; 在`update`函数中，将`Transformers`框架输出的`attention_mask`参数转化为`seq_len`参数\n",
    "\n",
    "\n",
    "### 2.1.2  ClassifyFPreRecMetric 与 SpanFPreRecMetric\n",
    "\n",
    "`ClassifyFPreRecMetric`，分类评价，`SpanFPreRecMetric`，抽取评价，后者在`tutorial-4`中已出现\n",
    "\n",
    "&emsp; 两者的相同之处在于：**第一**，**都包括召回率/查全率`Rec`**、**精确率/查准率`Pre`**、**`F1`值**这三个指标\n",
    "\n",
    "&emsp; &emsp; `get_metric`函数打印格式为 **`{\"f#xx\": float, 'pre#xx': float, 'rec#xx': float}`**\n",
    "\n",
    "&emsp; &emsp; 三者的计算公式如下，其中`beta`默认为`1`，即`F1`值是召回率`Rec`和精确率`Pre`的调和平均数\n",
    "\n",
    "$$\\text{召回率}\\ Rec=\\dfrac{\\text{正确预测为正例的数量}}{\\text{所有本来是正例的数量}}\\qquad \\text{精确率}\\ Pre=\\dfrac{\\text{正确预测为正例的数量}}{\\text{所有预测为正例的数量}}$$\n",
    "\n",
    "$$F_{beta} = \\frac{(1 + {beta}^{2})*(Pre*Rec)}{({beta}^{2}*Pre + Rec)}$$\n",
    "\n",
    "&emsp; **第二**，可以通过参数`only_gross`为`False`，要求返回所有类别的`Rec-Pre-F1`，同时`F1`值又根据参数`f_type`又分为\n",
    "\n",
    "&emsp; &emsp; **`micro F1`**（**直接统计所有类别的`Rec-Pre-F1`**）、**`macro F1`**（**统计各类别的`Rec-Pre-F1`再算术平均**）\n",
    "\n",
    "&emsp; **第三**，两者在初始化时还可以**传入基于`fastNLP.Vocabulary`的`tag_vocab`参数记录数据集中的标签序号**\n",
    "\n",
    "&emsp; &emsp; **与标签名称之间的映射**，通过字符串列表`ignore_labels`参数，指定若干标签不用于`Rec-Pre-F1`的计算\n",
    "\n",
    "两者的不同之处在于：`ClassifyFPreRecMetric`针对简单的分类问题，每个分类标签之间彼此独立，不构成标签对\n",
    "\n",
    "&emsp; &emsp; **`SpanFPreRecMetric`针对更复杂的抽取问题**，**规定标签`B-xx`和`I-xx`或`B-xx`和`E-xx`构成标签对**\n",
    "\n",
    "&emsp; 在计算`Rec-Pre-F1`时，`ClassifyFPreRecMetric`只需要考虑标签本身是否正确这就足够了，但是\n",
    "\n",
    "&emsp; &emsp; 对于`SpanFPreRecMetric`，需要保证**标签符合规则且覆盖的区间与正确结果重合才算正确**\n",
    "\n",
    "&emsp; &emsp; 因此回到`tutorial-4`中`CoNLL-2003`的`NER`任务，如果评测方法选择`ClassifyFPreRecMetric`\n",
    "\n",
    "&emsp; &emsp; &emsp; 或者`Accuracy`，会发现虽然评测结果显示很高，这是因为选择的评测方法要求太低\n",
    "\n",
    "&emsp; &emsp; 最后通过`CoNLL-2003`的词性标注`POS`任务简单演示下`ClassifyFPreRecMetric`相关的使用\n",
    "\n",
    "```python\n",
    "from fastNLP import Vocabulary\n",
    "from fastNLP import ClassifyFPreRecMetric\n",
    "\n",
    "tag_vocab = Vocabulary(padding=None, unknown=None)            # 记录序号与标签之间的映射\n",
    "tag_vocab.add_word_lst(['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', \n",
    "                        'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', \n",
    "                        'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', \n",
    "                        'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', \n",
    "                        'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP+', 'WRB', ])  # CoNLL-2003 中的 pos_tags\n",
    "ignore_labels = ['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', ]\n",
    "\n",
    "FPreRec = ClassifyFPreRecMetric(tag_vocab=tag_vocab,          \n",
    "                                ignore_labels=ignore_labels,  # 表示评测/优化中不考虑上述标签的正误/损失\n",
    "                                only_gross=True,              # 默认为 True 表示输出所有类别的综合统计结果\n",
    "                                f_type='micro')               # 默认为 'micro' 表示统计所有类别的 Rec-Pre-F1\n",
    "metrics = {'F1': FPreRec}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a22f522",
   "metadata": {},
   "source": [
    "### 2.2  自定义的 metric 类型\n",
    "\n",
    "如上文所述，`Metric`作为所有`metric`的基类，`Accuracy`等都是其子类，同样地，对于**自定义的`metric`类型**\n",
    "\n",
    "&emsp; &emsp; 也**需要继承自`Metric`类**，同时**内部自定义好`__init__`、`update`和`get_metric`函数**\n",
    "\n",
    "&emsp; 在`__init__`函数中，根据需求定义评测时需要用到的变量，此处沿用`Accuracy`中的`total_num`和`right_num`\n",
    "\n",
    "&emsp; 在`update`函数中，根据需求定义评测变量的更新方式，需要注意的是如`tutorial-0`中所述，**`update`的参数名**\n",
    "\n",
    "&emsp; &emsp; **需要待评估模型在`evaluate_step`中的输出名称一致**，由此**和数据集中对应字段名称一致**，即**参数匹配**\n",
    "\n",
    "&emsp; &emsp; 在`fastNLP v0.8`中，`update`函数的默认输入参数：`pred`，对应预测值；`target`，对应真实值\n",
    "\n",
    "&emsp; &emsp; 此处刻意调整为：`pred`，对应预测值，和模型输出一致；`true`，对应真实值，数据集字段需要调整\n",
    "\n",
    "&emsp; 在`get_metric`函数中，根据需求定义评测指标最终的计算，此处直接计算准确率，该函数必须返回一个字典\n",
    "\n",
    "&emsp; &emsp; 其中，字串`'prefix'`表示该`metric`的名称，会对应显示到`trainer`的`progress bar`中\n",
    "\n",
    "根据上述要求，这里简单定义了一个名为`MyMetric`的评测模块，用于分类问题的评测，以此展开一个实例展示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08a872e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastNLP import Metric\n",
    "\n",
    "class MyMetric(Metric):\n",
    "\n",
    "    def __init__(self):\n",
    "        MyMetric.__init__(self)\n",
    "        self.total_num = 0\n",
    "        self.right_num = 0\n",
    "\n",
    "    def update(self, pred, true):\n",
    "        self.total_num += target.size(0)\n",
    "        self.right_num += target.eq(pred).sum().item()\n",
    "\n",
    "    def get_metric(self, reset=True):\n",
    "        acc = self.acc_count / self.total_num\n",
    "        if reset:\n",
    "            self.total_num = 0\n",
    "            self.right_num = 0\n",
    "        return {'prefix': acc}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af3f8c63",
   "metadata": {},
   "source": [
    "&emsp; 模型使用方面，此处仍然使用`tutorial-4`中介绍过的预定义`CNNText`模型，实现`SST-2`二分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fd210c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('..')\n",
    "\n",
    "from fastNLP.models.torch import CNNText\n",
    "\n",
    "model = CNNText(embed=(len(vocab), 100), num_classes=2, dropout=0.1)\n",
    "\n",
    "from torch.optim import AdamW\n",
    "\n",
    "optimizers = AdamW(params=model.parameters(), lr=5e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0155f447",
   "metadata": {},
   "source": [
    "&emsp; 数据使用方面，此处仍然使用`datasets`模块中的`load_dataset`函数，加载`SST-2`二分类数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ad81ac7",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "sst2data = load_dataset('glue', 'sst2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9d81760",
   "metadata": {},
   "source": [
    "接着是数据预处理，需要注意的是，由于`MyMetric`的`update`函数中，输入参数名称为`pred`和`true`\n",
    "\n",
    "&emsp; 对应地，需要将数据集中表示预测目标的字段，调整为`true`（预定义的`metric`，应调整为`target`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfb28b1b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from fastNLP import DataSet\n",
    "\n",
    "dataset = DataSet.from_pandas(sst2data['train'].to_pandas())[:6000]\n",
    "\n",
    "dataset.apply_more(lambda ins:{'words': ins['sentence'].lower().split(), 'true': ins['label']}, \n",
    "                   progress_bar=\"tqdm\")\n",
    "dataset.delete_field('sentence')\n",
    "dataset.delete_field('label')\n",
    "dataset.delete_field('idx')\n",
    "\n",
    "from fastNLP import Vocabulary\n",
    "\n",
    "vocab = Vocabulary()\n",
    "vocab.from_dataset(dataset, field_name='words')\n",
    "vocab.index_dataset(dataset, field_name='words')\n",
    "\n",
    "train_dataset, evaluate_dataset = dataset.split(ratio=0.85)\n",
    "\n",
    "from fastNLP import prepare_torch_dataloader\n",
    "\n",
    "train_dataloader = prepare_torch_dataloader(train_dataset, batch_size=16, shuffle=True)\n",
    "evaluate_dataloader = prepare_torch_dataloader(evaluate_dataset, batch_size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e21df35",
   "metadata": {},
   "source": [
    "然后就是初始化`trainer`实例，其中`metrics`变量输入的键值对，字串`'suffix'`和之前定义的字串`'prefix'`\n",
    "\n",
    "&emsp; 将拼接在一起显示到`trainer`的`progress bar`中，故完整的输出形式为`{'prefix#suffix': float}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "926a9c50",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastNLP import Trainer\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    driver='torch',\n",
    "    device=0,  # 'cuda'\n",
    "    n_epochs=10,\n",
    "    optimizers=optimizers,\n",
    "    train_dataloader=train_dataloader,\n",
    "    evaluate_dataloaders=evaluate_dataloader,\n",
    "    metrics={'suffix': MyMetric()}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e723b87",
   "metadata": {},
   "source": [
    "## 3. fastNLP 中 trainer 的补充介绍\n",
    "\n",
    "### 3.1  trainer 的内部结构\n",
    "\n",
    "在`tutorial-0`中，我们已经介绍了`trainer`的基本使用，从`tutorial-1`到`tutorial-4`，我们也已经\n",
    "\n",
    "&emsp; 展示了很多关于`trainer`的使用案例，以下我们先补充介绍训练模块`trainer`的一些内部结构\n",
    "\n",
    "\n",
    "\n",
    "'accumulation_steps', 'add_callback_fn', 'backward', 'batch_idx_in_epoch', 'batch_step_fn',\n",
    "'callback_manager', 'check_batch_step_fn', 'cur_epoch_idx', 'data_device', 'dataloader',\n",
    "'device', 'driver', 'driver_name', 'epoch_evaluate', 'evaluate_batch_step_fn', 'evaluate_dataloaders',\n",
    "'evaluate_every', 'evaluate_fn', 'evaluator', 'extract_loss_from_outputs', 'fp16',\n",
    "'get_no_sync_context', 'global_forward_batches', 'has_checked_train_batch_loop',\n",
    "'input_mapping', 'kwargs', 'larger_better', 'load_checkpoint', 'load_model', 'marker',\n",
    "'metrics', 'model', 'model_device', 'monitor', 'move_data_to_device', 'n_epochs', 'num_batches_per_epoch',\n",
    "'on', 'on_after_backward', 'on_after_optimizers_step', 'on_after_trainer_initialized',\n",
    "'on_after_zero_grad', 'on_before_backward', 'on_before_optimizers_step', 'on_before_zero_grad',\n",
    "'on_evaluate_begin', 'on_evaluate_end', 'on_exception', 'on_fetch_data_begin', 'on_fetch_data_end',\n",
    "'on_load_checkpoint', 'on_load_model', 'on_sanity_check_begin', 'on_sanity_check_end',\n",
    "'on_save_checkpoint', 'on_save_model', 'on_train_batch_begin', 'on_train_batch_end',\n",
    "'on_train_begin', 'on_train_end', 'on_train_epoch_begin', 'on_train_epoch_end',\n",
    "'optimizers', 'output_mapping', 'progress_bar', 'run', 'run_evaluate',\n",
    "'save_checkpoint', 'save_model', 'start_batch_idx_in_epoch', 'state',\n",
    "'step', 'step_evaluate', 'total_batches', 'train_batch_loop', 'train_dataloader', 'train_fn', 'train_step',\n",
    "'trainer_state', 'zero_grad'\n",
    "\n",
    "&emsp; run(num_train_batch_per_epoch: int = -1, num_eval_batch_per_dl: int = -1, num_eval_sanity_batch: int = 2, resume_from: str = None, resume_training: bool = True, catch_KeyboardInterrupt=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c348864c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43be274f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}