{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "213d538c",
   "metadata": {},
   "source": [
    "# T3. dataloader 的内部结构和基本使用\n",
    "\n",
    "  1   \n",
    " \n",
    "    1.1   \n",
    "\n",
    "    1.2   \n",
    "\n",
    "  2   \n",
    "\n",
    "    2.1   \n",
    "\n",
    "    2.2   \n",
    "\n",
    "  3   \n",
    " \n",
    "    3.1   \n",
    "\n",
    "    3.2   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74d0523",
   "metadata": {},
   "source": [
    "## 3. fastNLP 中的 dataloader\n",
    "\n",
    "### 3.1 collator 的概念与使用\n",
    "\n",
    "在`fastNLP 0.8`中，在数据加载模块`DataLoader`之前，还存在其他的一些模块，负责例如对文本数据\n",
    "\n",
    "  进行补零对齐，即 **核对器`collator`模块**，进行分词标注，即 **分词器`tokenizer`模块**\n",
    "\n",
    "  本节将对`fastNLP`中的核对器`collator`等展开介绍，分词器`tokenizer`将在下一节中详细介绍\n",
    "\n",
    "在`fastNLP 0.8`中，**核对器`collator`模块负责文本序列的补零对齐**，通过"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aca72b49",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from fastNLP import Collator\n",
    "\n",
    "collator = Collator()\n",
    "# collator.set_pad(field_name='text', pad_val='<pad>')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51bf0878",
   "metadata": {},
   "source": [
    "&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fd2486f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f9bbd9a7",
   "metadata": {},
   "source": [
    "### 3.2 dataloader 的结构与使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "651baef6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from fastNLP import prepare_torch_dataloader\n",
    "\n",
    "dl_bundle = prepare_torch_dataloader(data_bundle, train_batch_size=2)\n",
    "\n",
    "print(type(dl_bundle), type(dl_bundle['train']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "726ba357",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader = prepare_torch_dataloader(datasets['train'], train_batch_size=2)\n",
    "print(type(dataloader))\n",
    "print(dir(dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0795b3e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader.collate_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0c3c58d",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader.batch_sampler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ed431cc",
   "metadata": {},
   "source": [
    "### 3.3 实例：NG20 的加载预处理\n",
    "\n",
    "在`fastNLP 0.8`中，**`Trainer`模块和`Evaluator`模块分别表示“训练器”和“评测器”**\n",
    "\n",
    "&emsp; 对应于之前的`fastNLP`版本中的`Trainer`模块和`Tester`模块，其定义方法如下所示\n",
    "\n",
    "在`fastNLP 0.8`中，需要注意，在同个`python`脚本中先使用`Trainer`训练，然后使用`Evaluator`评测\n",
    "\n",
    "&emsp; 非常关键的问题在于**如何正确设置二者的`driver`**。这就引入了另一个问题：什么是 `driver`？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a89ef613",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from fastNLP import DataSet\n",
    "from fastNLP import Vocabulary\n",
    "\n",
    "dataset = DataSet.from_pandas(pd.read_csv('./data/ng20_test.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1624b0fa",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "\n",
    "encode = partial(tokenizer.encode_plus, max_length=100, truncation=True,\n",
    "                 return_attention_mask=True)\n",
    "# 会新增 input_ids 、 attention_mask 和 token_type_ids 这三个 field\n",
    "dataset.apply_field_more(encode, field_name='text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0991a8ee",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "target_vocab = Vocabulary(padding=None, unknown=None)\n",
    "\n",
    "target_vocab.from_dataset(*[ds for _, ds in data_bundle.iter_datasets()], field_name='label')\n",
    "target_vocab.index_dataset(*[ds for _, ds in data_bundle.iter_datasets()], field_name='label',\n",
    "                           new_field_name='labels')\n",
    "# 需要将 input_ids 的 pad 值设置为 tokenizer 的 pad 值\n",
    "dataset.set_pad('input_ids', pad_val=tokenizer.pad_token_id)\n",
    "dataset.set_ignore('label', 'text')  # 因为 label 是原始的不需要的 str ，所以我们可以忽略它，让它不要在 batch 的输出中出现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b369137f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}