{ "cells": [ { "cell_type": "markdown", "id": "fdd7ff16", "metadata": {}, "source": [ "# T5. trainer 和 evaluator 的深入介绍\n", "\n", "  1   fastNLP 中 driver 的补充介绍\n", " \n", "    1.1   trainer 和 driver 的构想 \n", "\n", "    1.2   device 与 多卡训练\n", "\n", "  2   fastNLP 中的更多 metric 类型\n", "\n", "    2.1   预定义的 metric 类型\n", "\n", "    2.2   自定义的 metric 类型\n", "\n", "  3   fastNLP 中 trainer 的补充介绍\n", "\n", "    3.1   trainer 的内部结构" ] }, { "cell_type": "markdown", "id": "08752c5a", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## 1. fastNLP 中 driver 的补充介绍\n", "\n", "### 1.1 trainer 和 driver 的构想\n", "\n", "在`fastNLP 0.8`中，模型训练最关键的模块便是**训练模块`trainer`、评测模块`evaluator`、驱动模块`driver`**，\n", "\n", "  在`tutorial 0`中，已经简单介绍过上述三个模块：**`driver`用来控制训练评测中的`model`的最终运行**\n", "\n", "    **`evaluator`封装评测的`metric`**，**`trainer`封装训练的`optimizer`**，**也可以包括`evaluator`**\n", "\n", "之所以做出上述的划分，其根本目的在于要**达成对于多个`python`学习框架**，**例如`pytorch`、`paddle`、`jittor`的兼容**\n", "\n", "  对于训练环节，其伪代码如下方左边紫色一栏所示，由于**不同框架对模型、损失、张量的定义各有不同**，所以将训练环节\n", "\n", "    划分为**框架无关的循环控制、批量分发部分**，**由`trainer`模块负责**实现，对应的伪代码如下方中间蓝色一栏所示\n", "\n", "    以及**随框架不同的模型调用、数值优化部分**，**由`driver`模块负责**实现，对应的伪代码如下方右边红色一栏所示\n", "\n", "|
训练过程
|
框架无关 对应`trainer`
|
框架相关 对应`driver`
|\n", "|:--|:--|:--|\n", "|
try:
|
try:
| |\n", "|
for epoch in 1:n_eoochs:
|
for epoch in 1:n_eoochs:
| |\n", "|
for step in 1:total_steps:
|
for step in 1:total_steps:
| |\n", "|
batch = fetch_batch()
|
batch = fetch_batch()
| |\n", "|
loss = model.forward(batch) 
| |
loss = model.forward(batch) 
|\n", "|
loss.backward()
| |
loss.backward()
|\n", "|
model.clear_grad()
| |
model.clear_grad()
|\n", "|
model.update()
| |
model.update()
|\n", "|
if need_save:
|
if need_save:
| |\n", "|
model.save()
| |
model.save()
|\n", "|
except:
|
except:
| |\n", "|
process_exception()
|
process_exception()
| |" ] }, { "cell_type": "markdown", "id": "3e55f07b", "metadata": {}, "source": [ "  对于评测环节，其伪代码如下方左边紫色一栏所示，同样由于不同框架对模型、损失、张量的定义各有不同，所以将评测环节\n", "\n", "    划分为**框架无关的循环控制、分发汇总部分**，**由`evaluator`模块负责**实现，对应的伪代码如下方中间蓝色一栏所示\n", "\n", "    以及**随框架不同的模型调用、评测计算部分**，同样**由`driver`模块负责**实现，对应的伪代码如下方右边红色一栏所示\n", "\n", "|
评测过程
|
框架无关 对应`evaluator`
|
框架相关 对应`driver`
|\n", "|:--|:--|:--|\n", "|
try:
|
try:
| |\n", "|
model.set_eval()
|
model.set_eval()
| |\n", "|
for step in 1:total_steps:
|
for step in 1:total_steps:
| |\n", "|
batch = fetch_batch()
|
batch = fetch_batch()
| |\n", "|
outputs = model.evaluate(batch) 
| |
outputs = model.evaluate(batch) 
|\n", "|
metric.compute(batch, outputs)
| |
metric.compute(batch, outputs)
|\n", "|
results = metric.get_metric()
|
results = metric.get_metric()
| |\n", "|
except:
|
except:
| |\n", "|
process_exception()
|
process_exception()
| |" ] }, { "cell_type": "markdown", "id": "94ba11c6", "metadata": { "pycharm": { "name": "#%%\n" } }, "source": [ "由此，从程序员的角度，`fastNLP v0.8`**通过一个`driver`让基于`pytorch`、`paddle`、`jittor`框架的模型**\n", "\n", "    **都能在相同的`trainer`和`evaluator`上运行**，这也**是`fastNLP v0.8`相比于之前版本的一大亮点**\n", "\n", "  而从`driver`的角度，`fastNLP v0.8`通过定义一个`driver`基类，**将所有张量转化为`numpy.tensor`**\n", "\n", "    并由此泛化出`torch_driver`、`paddle_driver`、`jittor_driver`三个子类，从而实现了\n", "\n", "    对`pytorch`、`paddle`、`jittor`的兼容，有关后两者的实践请参考接下来的`tutorial-6`" ] }, { "cell_type": "markdown", "id": "ab1cea7d", "metadata": {}, "source": [ "### 1.2 device 与 多卡训练\n", "\n", "**`fastNLP v0.8`支持多卡训练**，实现方法则是**通过将`trainer`中的`device`设置为对应显卡的序号列表**\n", "\n", "  由单卡切换成多卡，无论是数据、模型还是评测都会面临一定的调整，`fastNLP v0.8`保证：\n", "\n", "    数据拆分时，不同卡之间相互协调，所有数据都可以被训练，且不会使用到相同的数据\n", "\n", "    模型训练时，模型之间需要交换梯度；评测计算时，每张卡先各自计算，再汇总结果\n", "\n", "  例如，在评测计算运行`get_metric`函数时，`fastNLP v0.8`将自动按照`self.right`和`self.total`\n", "\n", "    指定的**`aggregate_method`方法**，默认为`sum`，将每张卡上结果汇总起来，因此最终\n", "\n", "    在调用`get_metric`方法时，`Accuracy`类能够返回全部的统计结果，代码如下\n", " \n", "```python\n", "trainer = Trainer(\n", " model=model, # model 基于 pytorch 实现 \n", " train_dataloader=train_dataloader,\n", " optimizers=optimizer,\n", " ...\n", " driver='torch', # driver 使用 torch_driver \n", " device=[0, 1], # gpu 选择 cuda:0 + cuda:1\n", " ...\n", " evaluate_dataloaders=evaluate_dataloader,\n", " metrics={'acc': Accuracy()},\n", " ...\n", " )\n", "\n", "class Accuracy(Metric):\n", " def __init__(self):\n", " super().__init__()\n", " self.register_element(name='total', value=0, aggregate_method='sum')\n", " self.register_element(name='right', value=0, aggregate_method='sum')\n", "```\n" ] }, { "cell_type": "markdown", "id": "e2e0a210", "metadata": { "pycharm": { "name": "#%%\n" } }, "source": [ "注：`fastNLP v0.8`中要求`jupyter`不能多卡，仅能单卡，故在所有`tutorial`中均不作相关演示" ] }, { "cell_type": "markdown", "id": "8d19220c", "metadata": {}, "source": [ "## 2. fastNLP 中的更多 metric 类型\n", "\n", "### 2.1 预定义的 metric 类型\n", "\n", "在`fastNLP 0.8`中，除了前几篇`tutorial`中经常见到的**正确率`Accuracy`**，还有其他**预定义的评测标准`metric`**\n", "\n", "  包括**所有`metric`的基类`Metric`**、适配`Transformers`中相关模型的正确率`TransformersAccuracy`\n", "\n", "    **适用于分类语境下的`F1`值`ClassifyFPreRecMetric`**（其中也包括召回率`Pre`、精确率`Rec`\n", "\n", "    **适用于抽取语境下的`F1`值`SpanFPreRecMetric`**；相关基本信息内容见下表，之后是详细分析\n", "\n", "|
代码名称
|
简要介绍
|
代码路径
|\n", "|:--|:--|:--|\n", "| `Metric` | 定义`metrics`时继承的基类 | `/core/metrics/metric.py` |\n", "| `Accuracy` | 正确率，最为常用 | `/core/metrics/accuracy.py` |\n", "| `TransformersAccuracy` | 正确率，为了兼容`Transformers`中相关模型 | `/core/metrics/accuracy.py` |\n", "| `ClassifyFPreRecMetric` | 召回率、精确率、F1值，适用于**分类问题** | `/core/metrics/classify_f1_pre_rec_metric.py` |\n", "| `SpanFPreRecMetric` | 召回率、精确率、F1值，适用于**抽取问题** | `/core/metrics/span_f1_pre_rec_metric.py` |" ] }, { "cell_type": "markdown", "id": "fdc083a3", "metadata": { "pycharm": { "name": "#%%\n" } }, "source": [ "  如`tutorial-0`中所述，所有的`metric`都包含`get_metric`和`update`函数，其中\n", "\n", "    **`update`函数更新单个`batch`的统计量**，**`get_metric`函数返回最终结果**，并打印显示\n", "\n", "\n", "### 2.1.1 Accuracy 与 TransformersAccuracy\n", "\n", "`Accuracy`，正确率，预测正确的数据`right_num`在总数据`total_num`，中的占比（公式就不用列了\n", "\n", "  `get_metric`函数打印格式为 **`{\"acc#xx\": float, 'total#xx': float, 'correct#xx': float}`**\n", "\n", "  一般在初始化时不需要传参，`fastNLP`会根据`update`函数的传入参数确定对应后台框架`backend`\n", "\n", "  **`update`函数的参数包括`pred`、`target`、`seq_len`**，**后者用来标记批次中每笔数据的长度**\n", "\n", "`TransformersAccuracy`，继承自`Accuracy`，只是为了兼容`Transformers`框架中相关模型\n", "\n", "  在`update`函数中，将`Transformers`框架输出的`attention_mask`参数转化为`seq_len`参数\n", "\n", "\n", "### 2.1.2 ClassifyFPreRecMetric 与 SpanFPreRecMetric\n", "\n", "`ClassifyFPreRecMetric`，分类评价，`SpanFPreRecMetric`，抽取评价，后者在`tutorial-4`中已出现\n", "\n", "  两者的相同之处在于：**第一**，**都包括召回率/查全率`Rec`**、**精确率/查准率`Pre`**、**`F1`值**这三个指标\n", "\n", "    `get_metric`函数打印格式为 **`{\"f#xx\": float, 'pre#xx': float, 'rec#xx': float}`**\n", "\n", "    三者的计算公式如下，其中`beta`默认为`1`，即`F1`值是召回率`Rec`和精确率`Pre`的调和平均数\n", "\n", "$$\\text{召回率}\\ Rec=\\dfrac{\\text{正确预测为正例的数量}}{\\text{所有本来是正例的数量}}\\qquad \\text{精确率}\\ Pre=\\dfrac{\\text{正确预测为正例的数量}}{\\text{所有预测为正例的数量}}$$\n", "\n", "$$F_{beta} = \\frac{(1 + {beta}^{2})*(Pre*Rec)}{({beta}^{2}*Pre + Rec)}$$\n", "\n", "  **第二**，可以通过参数`only_gross`为`False`，要求返回所有类别的`Rec-Pre-F1`，同时`F1`值又根据参数`f_type`又分为\n", "\n", "    **`micro F1`**（**直接统计所有类别的`Rec-Pre-F1`**）、**`macro F1`**（**统计各类别的`Rec-Pre-F1`再算术平均**）\n", "\n", "  **第三**，两者在初始化时还可以**传入基于`fastNLP.Vocabulary`的`tag_vocab`参数记录数据集中的标签序号**\n", "\n", "    **与标签名称之间的映射**，通过字符串列表`ignore_labels`参数，指定若干标签不用于`Rec-Pre-F1`的计算\n", "\n", "两者的不同之处在于：`ClassifyFPreRecMetric`针对简单的分类问题，每个分类标签之间彼此独立，不构成标签对\n", "\n", "    **`SpanFPreRecMetric`针对更复杂的抽取问题**，**规定标签`B-xx`和`I-xx`或`B-xx`和`E-xx`构成标签对**\n", "\n", "  在计算`Rec-Pre-F1`时，`ClassifyFPreRecMetric`只需要考虑标签本身是否正确这就足够了，但是\n", "\n", "    对于`SpanFPreRecMetric`，需要保证**标签符合规则且覆盖的区间与正确结果重合才算正确**\n", "\n", "    因此回到`tutorial-4`中`CoNLL-2003`的`NER`任务，如果评测方法选择`ClassifyFPreRecMetric`\n", "\n", "      或者`Accuracy`，会发现虽然评测结果显示很高，这是因为选择的评测方法要求太低\n", "\n", "    最后通过`CoNLL-2003`的词性标注`POS`任务简单演示下`ClassifyFPreRecMetric`相关的使用\n", "\n", "```python\n", "from fastNLP import Vocabulary\n", "from fastNLP import ClassifyFPreRecMetric\n", "\n", "tag_vocab = Vocabulary(padding=None, unknown=None) # 记录序号与标签之间的映射\n", "tag_vocab.add_word_lst(['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', \n", " 'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', \n", " 'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', \n", " 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', \n", " 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP+', 'WRB', ]) # CoNLL-2003 中的 pos_tags\n", "ignore_labels = ['\"', \"''\", '#', '$', '(', ')', ',', '.', ':', '``', ]\n", "\n", "FPreRec = ClassifyFPreRecMetric(tag_vocab=tag_vocab, \n", " ignore_labels=ignore_labels, # 表示评测/优化中不考虑上述标签的正误/损失\n", " only_gross=True, # 默认为 True 表示输出所有类别的综合统计结果\n", " f_type='micro') # 默认为 'micro' 表示统计所有类别的 Rec-Pre-F1\n", "metrics = {'F1': FPreRec}\n", "```" ] }, { "cell_type": "markdown", "id": "8a22f522", "metadata": {}, "source": [ "### 2.2 自定义的 metric 类型\n", "\n", "如上文所述，`Metric`作为所有`metric`的基类，`Accuracy`等都是其子类，同样地，对于**自定义的`metric`类型**\n", "\n", "    也**需要继承自`Metric`类**，同时**内部自定义好`__init__`、`update`和`get_metric`函数**\n", "\n", "  在`__init__`函数中，根据需求定义评测时需要用到的变量，此处沿用`Accuracy`中的`total_num`和`right_num`\n", "\n", "  在`update`函数中，根据需求定义评测变量的更新方式，需要注意的是如`tutorial-0`中所述，**`update`的参数名**\n", "\n", "    **需要待评估模型在`evaluate_step`中的输出名称一致**，由此**和数据集中对应字段名称一致**，即**参数匹配**\n", "\n", "    在`fastNLP v0.8`中，`update`函数的默认输入参数：`pred`，对应预测值；`target`，对应真实值\n", "\n", "    此处刻意调整为：`pred`，对应预测值，和模型输出一致；`true`，对应真实值，数据集字段需要调整\n", "\n", "  在`get_metric`函数中，根据需求定义评测指标最终的计算，此处直接计算准确率，该函数必须返回一个字典\n", "\n", "    其中，字串`'prefix'`表示该`metric`的名称，会对应显示到`trainer`的`progress bar`中\n", "\n", "根据上述要求，这里简单定义了一个名为`MyMetric`的评测模块，用于分类问题的评测，以此展开一个实例展示" ] }, { "cell_type": "code", "execution_count": null, "id": "08a872e9", "metadata": {}, "outputs": [], "source": [ "from fastNLP import Metric\n", "\n", "class MyMetric(Metric):\n", "\n", " def __init__(self):\n", " MyMetric.__init__(self)\n", " self.total_num = 0\n", " self.right_num = 0\n", "\n", " def update(self, pred, true):\n", " self.total_num += target.size(0)\n", " self.right_num += target.eq(pred).sum().item()\n", "\n", " def get_metric(self, reset=True):\n", " acc = self.acc_count / self.total_num\n", " if reset:\n", " self.total_num = 0\n", " self.right_num = 0\n", " return {'prefix': acc}" ] }, { "cell_type": "markdown", "id": "af3f8c63", "metadata": {}, "source": [ "  模型使用方面，此处仍然使用`tutorial-4`中介绍过的预定义`CNNText`模型，实现`SST-2`二分类" ] }, { "cell_type": "code", "execution_count": null, "id": "2fd210c5", "metadata": {}, "outputs": [], "source": [ "import sys\n", "sys.path.append('..')\n", "\n", "from fastNLP.models.torch import CNNText\n", "\n", "model = CNNText(embed=(len(vocab), 100), num_classes=2, dropout=0.1)\n", "\n", "from torch.optim import AdamW\n", "\n", "optimizers = AdamW(params=model.parameters(), lr=5e-4)" ] }, { "cell_type": "markdown", "id": "0155f447", "metadata": {}, "source": [ "  数据使用方面，此处仍然使用`datasets`模块中的`load_dataset`函数，加载`SST-2`二分类数据集" ] }, { "cell_type": "code", "execution_count": null, "id": "5ad81ac7", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "sst2data = load_dataset('glue', 'sst2')" ] }, { "cell_type": "markdown", "id": "e9d81760", "metadata": {}, "source": [ "接着是数据预处理，需要注意的是，由于`MyMetric`的`update`函数中，输入参数名称为`pred`和`true`\n", "\n", "  对应地，需要将数据集中表示预测目标的字段，调整为`true`（预定义的`metric`，应调整为`target`" ] }, { "cell_type": "code", "execution_count": null, "id": "cfb28b1b", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from fastNLP import DataSet\n", "\n", "dataset = DataSet.from_pandas(sst2data['train'].to_pandas())[:6000]\n", "\n", "dataset.apply_more(lambda ins:{'words': ins['sentence'].lower().split(), 'true': ins['label']}, \n", " progress_bar=\"tqdm\")\n", "dataset.delete_field('sentence')\n", "dataset.delete_field('label')\n", "dataset.delete_field('idx')\n", "\n", "from fastNLP import Vocabulary\n", "\n", "vocab = Vocabulary()\n", "vocab.from_dataset(dataset, field_name='words')\n", "vocab.index_dataset(dataset, field_name='words')\n", "\n", "train_dataset, evaluate_dataset = dataset.split(ratio=0.85)\n", "\n", "from fastNLP import prepare_torch_dataloader\n", "\n", "train_dataloader = prepare_torch_dataloader(train_dataset, batch_size=16, shuffle=True)\n", "evaluate_dataloader = prepare_torch_dataloader(evaluate_dataset, batch_size=16)" ] }, { "cell_type": "markdown", "id": "1e21df35", "metadata": {}, "source": [ "然后就是初始化`trainer`实例，其中`metrics`变量输入的键值对，字串`'suffix'`和之前定义的字串`'prefix'`\n", "\n", "  将拼接在一起显示到`trainer`的`progress bar`中，故完整的输出形式为`{'prefix#suffix': float}`" ] }, { "cell_type": "code", "execution_count": null, "id": "926a9c50", "metadata": {}, "outputs": [], "source": [ "from fastNLP import Trainer\n", "\n", "trainer = Trainer(\n", " model=model,\n", " driver='torch',\n", " device=0, # 'cuda'\n", " n_epochs=10,\n", " optimizers=optimizers,\n", " train_dataloader=train_dataloader,\n", " evaluate_dataloaders=evaluate_dataloader,\n", " metrics={'suffix': MyMetric()}\n", ")" ] }, { "cell_type": "markdown", "id": "6e723b87", "metadata": {}, "source": [ "## 3. fastNLP 中 trainer 的补充介绍\n", "\n", "### 3.1 trainer 的内部结构\n", "\n", "在`tutorial-0`中，我们已经介绍了`trainer`的基本使用，从`tutorial-1`到`tutorial-4`，我们也已经\n", "\n", "  展示了很多关于`trainer`的使用案例，以下我们先补充介绍训练模块`trainer`的一些内部结构\n", "\n", "\n", "\n", "'accumulation_steps', 'add_callback_fn', 'backward', 'batch_idx_in_epoch', 'batch_step_fn',\n", "'callback_manager', 'check_batch_step_fn', 'cur_epoch_idx', 'data_device', 'dataloader',\n", "'device', 'driver', 'driver_name', 'epoch_evaluate', 'evaluate_batch_step_fn', 'evaluate_dataloaders',\n", "'evaluate_every', 'evaluate_fn', 'evaluator', 'extract_loss_from_outputs', 'fp16',\n", "'get_no_sync_context', 'global_forward_batches', 'has_checked_train_batch_loop',\n", "'input_mapping', 'kwargs', 'larger_better', 'load_checkpoint', 'load_model', 'marker',\n", "'metrics', 'model', 'model_device', 'monitor', 'move_data_to_device', 'n_epochs', 'num_batches_per_epoch',\n", "'on', 'on_after_backward', 'on_after_optimizers_step', 'on_after_trainer_initialized',\n", "'on_after_zero_grad', 'on_before_backward', 'on_before_optimizers_step', 'on_before_zero_grad',\n", "'on_evaluate_begin', 'on_evaluate_end', 'on_exception', 'on_fetch_data_begin', 'on_fetch_data_end',\n", "'on_load_checkpoint', 'on_load_model', 'on_sanity_check_begin', 'on_sanity_check_end',\n", "'on_save_checkpoint', 'on_save_model', 'on_train_batch_begin', 'on_train_batch_end',\n", "'on_train_begin', 'on_train_end', 'on_train_epoch_begin', 'on_train_epoch_end',\n", "'optimizers', 'output_mapping', 'progress_bar', 'run', 'run_evaluate',\n", "'save_checkpoint', 'save_model', 'start_batch_idx_in_epoch', 'state',\n", "'step', 'step_evaluate', 'total_batches', 'train_batch_loop', 'train_dataloader', 'train_fn', 'train_step',\n", "'trainer_state', 'zero_grad'\n", "\n", "  run(num_train_batch_per_epoch: int = -1, num_eval_batch_per_dl: int = -1, num_eval_sanity_batch: int = 2, resume_from: str = None, resume_training: bool = True, catch_KeyboardInterrupt=None)" ] }, { "cell_type": "code", "execution_count": null, "id": "c348864c", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "43be274f", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.13" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } } }, "nbformat": 4, "nbformat_minor": 5 }