{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ML2021Spring - HW1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mz0_QVkxCrX3"
      },
      "source": [
        "# **Homework 1: COVID-19 Cases Prediction (Regression)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeZnPAiwDRWG"
      },
      "source": [
        "Author: Heng-Jui Chang\n",
        "\n",
        "Slides: https://github.com/ga642381/ML2021-Spring/blob/main/HW01/HW01.pdf  \n",
        "Video: TBA\n",
        "\n",
        "Objectives:\n",
        "* Solve a regression problem with deep neural networks (DNN).\n",
        "* Understand basic DNN training tips.\n",
        "* Get familiar with PyTorch.\n",
        "\n",
        "If any questions, please contact the TAs via TA hours, NTU COOL, or email.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jx3x1nDkG-Uy"
      },
      "source": [
        "# **Download Data**\n",
        "\n",
        "\n",
        "If the Google drive links are dead, you can download data from [kaggle](https://www.kaggle.com/c/ml2021spring-hw1/data), and upload data manually to the workspace."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tMj55YDKG6ch",
        "outputId": "fc40ecc9-4756-48b1-d5c6-c169a8b453b2"
      },
      "source": [
        "tr_path = 'covid.train.csv'  # path to training data\n",
        "tt_path = 'covid.test.csv'   # path to testing data\n",
        "\n",
        "!gdown --id '19CCyCgJrUxtvgZF53vnctJiOJ23T5mqF' --output covid.train.csv\n",
        "!gdown --id '1CE240jLm2npU-tdz81-oVKEF3T2yfT1O' --output covid.test.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=19CCyCgJrUxtvgZF53vnctJiOJ23T5mqF\n",
            "To: /content/covid.train.csv\n",
            "100% 2.00M/2.00M [00:00<00:00, 31.7MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1CE240jLm2npU-tdz81-oVKEF3T2yfT1O\n",
            "To: /content/covid.test.csv\n",
            "100% 651k/651k [00:00<00:00, 10.2MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wS_4-77xHk44"
      },
      "source": [
        "# **Import Some Packages**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k-onQd4JNA5H"
      },
      "source": [
        "# PyTorch\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "\n",
        "# For data preprocess\n",
        "import numpy as np\n",
        "import csv\n",
        "import os\n",
        "\n",
        "# For plotting\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.pyplot import figure\n",
        "\n",
        "myseed = 42069  # set a random seed for reproducibility\n",
        "torch.backends.cudnn.deterministic = True\n",
        "torch.backends.cudnn.benchmark = False\n",
        "np.random.seed(myseed)\n",
        "torch.manual_seed(myseed)\n",
        "if torch.cuda.is_available():\n",
        "    torch.cuda.manual_seed_all(myseed)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BtE3b6JEH7rw"
      },
      "source": [
        "# **Some Utilities**\n",
        "\n",
        "You do not need to modify this part."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FWMT3uf1NGQp"
      },
      "source": [
        "def get_device():\n",
        "    ''' Get device (if GPU is available, use GPU) '''\n",
        "    return 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "\n",
        "def plot_learning_curve(loss_record, title=''):\n",
        "    ''' Plot learning curve of your DNN (train & dev loss) '''\n",
        "    total_steps = len(loss_record['train'])\n",
        "    x_1 = range(total_steps)\n",
        "    x_2 = x_1[::len(loss_record['train']) // len(loss_record['dev'])]\n",
        "    figure(figsize=(6, 4))\n",
        "    plt.plot(x_1, loss_record['train'], c='tab:red', label='train')\n",
        "    plt.plot(x_2, loss_record['dev'], c='tab:cyan', label='dev')\n",
        "    plt.ylim(0.0, 5.)\n",
        "    plt.xlabel('Training steps')\n",
        "    plt.ylabel('MSE loss')\n",
        "    plt.title('Learning curve of {}'.format(title))\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "def plot_pred(dv_set, model, device, lim=35., preds=None, targets=None):\n",
        "    ''' Plot prediction of your DNN '''\n",
        "    if preds is None or targets is None:\n",
        "        model.eval()\n",
        "        preds, targets = [], []\n",
        "        for x, y in dv_set:\n",
        "            x, y = x.to(device), y.to(device)\n",
        "            with torch.no_grad():\n",
        "                pred = model(x)\n",
        "                preds.append(pred.detach().cpu())\n",
        "                targets.append(y.detach().cpu())\n",
        "        preds = torch.cat(preds, dim=0).numpy()\n",
        "        targets = torch.cat(targets, dim=0).numpy()\n",
        "\n",
        "    figure(figsize=(5, 5))\n",
        "    plt.scatter(targets, preds, c='r', alpha=0.5)\n",
        "    plt.plot([-0.2, lim], [-0.2, lim], c='b')\n",
        "    plt.xlim(-0.2, lim)\n",
        "    plt.ylim(-0.2, lim)\n",
        "    plt.xlabel('ground truth value')\n",
        "    plt.ylabel('predicted value')\n",
        "    plt.title('Ground Truth v.s. Prediction')\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39U_XFX6KOoj"
      },
      "source": [
        "# **Preprocess**\n",
        "\n",
        "We have three kinds of datasets:\n",
        "* `train`: for training\n",
        "* `dev`: for validation\n",
        "* `test`: for testing (w/o target value)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TQ-MdwpLL7Dt"
      },
      "source": [
        "## **Dataset**\n",
        "\n",
        "The `COVID19Dataset` below does:\n",
        "* read `.csv` files\n",
        "* extract features\n",
        "* split `covid.train.csv` into train/dev sets\n",
        "* normalize features\n",
        "\n",
        "Finishing `TODO` below might make you pass medium baseline."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0zlpIp9ANJRU"
      },
      "source": [
        "class COVID19Dataset(Dataset):\n",
        "    ''' Dataset for loading and preprocessing the COVID19 dataset '''\n",
        "    def __init__(self,\n",
        "                 path,\n",
        "                 mode='train',\n",
        "                 target_only=False):\n",
        "        self.mode = mode\n",
        "\n",
        "        # Read data into numpy arrays\n",
        "        with open(path, 'r') as fp:\n",
        "            data = list(csv.reader(fp))\n",
        "            data = np.array(data[1:])[:, 1:].astype(float)\n",
        "        \n",
        "        if not target_only:\n",
        "            feats = list(range(93))\n",
        "        else:\n",
        "            # TODO: Using 40 states & 2 tested_positive features (indices = 57 & 75)\n",
        "            pass\n",
        "\n",
        "        if mode == 'test':\n",
        "            # Testing data\n",
        "            # data: 893 x 93 (40 states + day 1 (18) + day 2 (18) + day 3 (17))\n",
        "            data = data[:, feats]\n",
        "            self.data = torch.FloatTensor(data)\n",
        "        else:\n",
        "            # Training data (train/dev sets)\n",
        "            # data: 2700 x 94 (40 states + day 1 (18) + day 2 (18) + day 3 (18))\n",
        "            target = data[:, -1]\n",
        "            data = data[:, feats]\n",
        "            \n",
        "            # Splitting training data into train & dev sets\n",
        "            if mode == 'train':\n",
        "                indices = [i for i in range(len(data)) if i % 10 != 0]\n",
        "            elif mode == 'dev':\n",
        "                indices = [i for i in range(len(data)) if i % 10 == 0]\n",
        "            \n",
        "            # Convert data into PyTorch tensors\n",
        "            self.data = torch.FloatTensor(data[indices])\n",
        "            self.target = torch.FloatTensor(target[indices])\n",
        "\n",
        "        # Normalize features (you may remove this part to see what will happen)\n",
        "        self.data[:, 40:] = \\\n",
        "            (self.data[:, 40:] - self.data[:, 40:].mean(dim=0, keepdim=True)) \\\n",
        "            / self.data[:, 40:].std(dim=0, keepdim=True)\n",
        "\n",
        "        self.dim = self.data.shape[1]\n",
        "\n",
        "        print('Finished reading the {} set of COVID19 Dataset ({} samples found, each dim = {})'\n",
        "              .format(mode, len(self.data), self.dim))\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        # Returns one sample at a time\n",
        "        if self.mode in ['train', 'dev']:\n",
        "            # For training\n",
        "            return self.data[index], self.target[index]\n",
        "        else:\n",
        "            # For testing (no target)\n",
        "            return self.data[index]\n",
        "\n",
        "    def __len__(self):\n",
        "        # Returns the size of the dataset\n",
        "        return len(self.data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlhTlkE7MDo3"
      },
      "source": [
        "## **DataLoader**\n",
        "\n",
        "A `DataLoader` loads data from a given `Dataset` into batches.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hlhLk5t6MBX3"
      },
      "source": [
        "def prep_dataloader(path, mode, batch_size, n_jobs=0, target_only=False):\n",
        "    ''' Generates a dataset, then is put into a dataloader. '''\n",
        "    dataset = COVID19Dataset(path, mode=mode, target_only=target_only)  # Construct dataset\n",
        "    dataloader = DataLoader(\n",
        "        dataset, batch_size,\n",
        "        shuffle=(mode == 'train'), drop_last=False,\n",
        "        num_workers=n_jobs, pin_memory=True)                            # Construct dataloader\n",
        "    return dataloader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGuycwR0MeQB"
      },
      "source": [
        "# **Deep Neural Network**\n",
        "\n",
        "`NeuralNet` is an `nn.Module` designed for regression.\n",
        "The DNN consists of 2 fully-connected layers with ReLU activation.\n",
        "This module also included a function `cal_loss` for calculating loss.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "49-uXYovOAI0"
      },
      "source": [
        "class NeuralNet(nn.Module):\n",
        "    ''' A simple fully-connected deep neural network '''\n",
        "    def __init__(self, input_dim):\n",
        "        super(NeuralNet, self).__init__()\n",
        "\n",
        "        # Define your neural network here\n",
        "        # TODO: How to modify this model to achieve better performance?\n",
        "        self.net = nn.Sequential(\n",
        "            nn.Linear(input_dim, 64),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(64, 1)\n",
        "        )\n",
        "\n",
        "        # Mean squared error loss\n",
        "        self.criterion = nn.MSELoss(reduction='mean')\n",
        "\n",
        "    def forward(self, x):\n",
        "        ''' Given input of size (batch_size x input_dim), compute output of the network '''\n",
        "        return self.net(x).squeeze(1)\n",
        "\n",
        "    def cal_loss(self, pred, target):\n",
        "        ''' Calculate loss '''\n",
        "        # TODO: you may implement L2 regularization here\n",
        "        return self.criterion(pred, target)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DvFWVjZ5Nvga"
      },
      "source": [
        "# **Train/Dev/Test**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAM8QecJOyqn"
      },
      "source": [
        "## **Training**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOqcmYzMO7jB"
      },
      "source": [
        "def train(tr_set, dv_set, model, config, device):\n",
        "    ''' DNN training '''\n",
        "\n",
        "    n_epochs = config['n_epochs']  # Maximum number of epochs\n",
        "\n",
        "    # Setup optimizer\n",
        "    optimizer = getattr(torch.optim, config['optimizer'])(\n",
        "        model.parameters(), **config['optim_hparas'])\n",
        "\n",
        "    min_mse = 1000.\n",
        "    loss_record = {'train': [], 'dev': []}      # for recording training loss\n",
        "    early_stop_cnt = 0\n",
        "    epoch = 0\n",
        "    while epoch < n_epochs:\n",
        "        model.train()                           # set model to training mode\n",
        "        for x, y in tr_set:                     # iterate through the dataloader\n",
        "            optimizer.zero_grad()               # set gradient to zero\n",
        "            x, y = x.to(device), y.to(device)   # move data to device (cpu/cuda)\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            mse_loss = model.cal_loss(pred, y)  # compute loss\n",
        "            mse_loss.backward()                 # compute gradient (backpropagation)\n",
        "            optimizer.step()                    # update model with optimizer\n",
        "            loss_record['train'].append(mse_loss.detach().cpu().item())\n",
        "\n",
        "        # After each epoch, test your model on the validation (development) set.\n",
        "        dev_mse = dev(dv_set, model, device)\n",
        "        if dev_mse < min_mse:\n",
        "            # Save model if your model improved\n",
        "            min_mse = dev_mse\n",
        "            print('Saving model (epoch = {:4d}, loss = {:.4f})'\n",
        "                .format(epoch + 1, min_mse))\n",
        "            torch.save(model.state_dict(), config['save_path'])  # Save model to specified path\n",
        "            early_stop_cnt = 0\n",
        "        else:\n",
        "            early_stop_cnt += 1\n",
        "\n",
        "        epoch += 1\n",
        "        loss_record['dev'].append(dev_mse)\n",
        "        if early_stop_cnt > config['early_stop']:\n",
        "            # Stop training if your model stops improving for \"config['early_stop']\" epochs.\n",
        "            break\n",
        "\n",
        "    print('Finished training after {} epochs'.format(epoch))\n",
        "    return min_mse, loss_record"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0hSd4Bn3O2PL"
      },
      "source": [
        "## **Validation**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yrxrD3YsN3U2"
      },
      "source": [
        "def dev(dv_set, model, device):\n",
        "    model.eval()                                # set model to evalutation mode\n",
        "    total_loss = 0\n",
        "    for x, y in dv_set:                         # iterate through the dataloader\n",
        "        x, y = x.to(device), y.to(device)       # move data to device (cpu/cuda)\n",
        "        with torch.no_grad():                   # disable gradient calculation\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            mse_loss = model.cal_loss(pred, y)  # compute loss\n",
        "        total_loss += mse_loss.detach().cpu().item() * len(x)  # accumulate loss\n",
        "    total_loss = total_loss / len(dv_set.dataset)              # compute averaged loss\n",
        "\n",
        "    return total_loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0pdrhQAO41L"
      },
      "source": [
        "## **Testing**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aSBMRFlYN5tB"
      },
      "source": [
        "def test(tt_set, model, device):\n",
        "    model.eval()                                # set model to evalutation mode\n",
        "    preds = []\n",
        "    for x in tt_set:                            # iterate through the dataloader\n",
        "        x = x.to(device)                        # move data to device (cpu/cuda)\n",
        "        with torch.no_grad():                   # disable gradient calculation\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            preds.append(pred.detach().cpu())   # collect prediction\n",
        "    preds = torch.cat(preds, dim=0).numpy()     # concatenate all predictions and convert to a numpy array\n",
        "    return preds"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvckkF5dvf0j"
      },
      "source": [
        "# **Setup Hyper-parameters**\n",
        "\n",
        "`config` contains hyper-parameters for training and the path to save your model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NPXpdumwPjE7"
      },
      "source": [
        "device = get_device()                 # get the current available device ('cpu' or 'cuda')\n",
        "os.makedirs('models', exist_ok=True)  # The trained model will be saved to ./models/\n",
        "target_only = False                   # TODO: Using 40 states & 2 tested_positive features\n",
        "\n",
        "# TODO: How to tune these hyper-parameters to improve your model's performance?\n",
        "config = {\n",
        "    'n_epochs': 3000,                # maximum number of epochs\n",
        "    'batch_size': 270,               # mini-batch size for dataloader\n",
        "    'optimizer': 'SGD',              # optimization algorithm (optimizer in torch.optim)\n",
        "    'optim_hparas': {                # hyper-parameters for the optimizer (depends on which optimizer you are using)\n",
        "        'lr': 0.001,                 # learning rate of SGD\n",
        "        'momentum': 0.9              # momentum for SGD\n",
        "    },\n",
        "    'early_stop': 200,               # early stopping epochs (the number epochs since your model's last improvement)\n",
        "    'save_path': 'models/model.pth'  # your model will be saved here\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6j1eOV3TOH-j"
      },
      "source": [
        "# **Load data and model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eNrYBMmePLKm",
        "outputId": "fcd4f175-4f7e-4306-f33c-5f8285f11dce"
      },
      "source": [
        "tr_set = prep_dataloader(tr_path, 'train', config['batch_size'], target_only=target_only)\n",
        "dv_set = prep_dataloader(tr_path, 'dev', config['batch_size'], target_only=target_only)\n",
        "tt_set = prep_dataloader(tt_path, 'test', config['batch_size'], target_only=target_only)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Finished reading the train set of COVID19 Dataset (2430 samples found, each dim = 93)\n",
            "Finished reading the dev set of COVID19 Dataset (270 samples found, each dim = 93)\n",
            "Finished reading the test set of COVID19 Dataset (893 samples found, each dim = 93)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FHylSirLP9oh"
      },
      "source": [
        "model = NeuralNet(tr_set.dataset.dim).to(device)  # Construct model and move to device"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sX2B_zgSOPTJ"
      },
      "source": [
        "# **Start Training!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GrEbUxazQAAZ",
        "outputId": "f4f3bd74-2d97-4275-b69f-6609976b91f9"
      },
      "source": [
        "model_loss, model_loss_record = train(tr_set, dv_set, model, config, device)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving model (epoch =    1, loss = 74.9742)\n",
            "Saving model (epoch =    2, loss = 50.5313)\n",
            "Saving model (epoch =    3, loss = 29.1148)\n",
            "Saving model (epoch =    4, loss = 15.8134)\n",
            "Saving model (epoch =    5, loss = 9.5430)\n",
            "Saving model (epoch =    6, loss = 6.8086)\n",
            "Saving model (epoch =    7, loss = 5.3892)\n",
            "Saving model (epoch =    8, loss = 4.5267)\n",
            "Saving model (epoch =    9, loss = 3.9454)\n",
            "Saving model (epoch =   10, loss = 3.5560)\n",
            "Saving model (epoch =   11, loss = 3.2303)\n",
            "Saving model (epoch =   12, loss = 2.9920)\n",
            "Saving model (epoch =   13, loss = 2.7737)\n",
            "Saving model (epoch =   14, loss = 2.6181)\n",
            "Saving model (epoch =   15, loss = 2.3987)\n",
            "Saving model (epoch =   16, loss = 2.2712)\n",
            "Saving model (epoch =   17, loss = 2.1349)\n",
            "Saving model (epoch =   18, loss = 2.0210)\n",
            "Saving model (epoch =   19, loss = 1.8848)\n",
            "Saving model (epoch =   20, loss = 1.7999)\n",
            "Saving model (epoch =   21, loss = 1.7510)\n",
            "Saving model (epoch =   22, loss = 1.6787)\n",
            "Saving model (epoch =   23, loss = 1.6450)\n",
            "Saving model (epoch =   24, loss = 1.6030)\n",
            "Saving model (epoch =   26, loss = 1.5052)\n",
            "Saving model (epoch =   27, loss = 1.4486)\n",
            "Saving model (epoch =   28, loss = 1.4069)\n",
            "Saving model (epoch =   29, loss = 1.3733)\n",
            "Saving model (epoch =   30, loss = 1.3533)\n",
            "Saving model (epoch =   31, loss = 1.3335)\n",
            "Saving model (epoch =   32, loss = 1.3011)\n",
            "Saving model (epoch =   33, loss = 1.2711)\n",
            "Saving model (epoch =   35, loss = 1.2331)\n",
            "Saving model (epoch =   36, loss = 1.2235)\n",
            "Saving model (epoch =   38, loss = 1.2180)\n",
            "Saving model (epoch =   39, loss = 1.2018)\n",
            "Saving model (epoch =   40, loss = 1.1651)\n",
            "Saving model (epoch =   42, loss = 1.1631)\n",
            "Saving model (epoch =   43, loss = 1.1394)\n",
            "Saving model (epoch =   46, loss = 1.1129)\n",
            "Saving model (epoch =   47, loss = 1.1107)\n",
            "Saving model (epoch =   49, loss = 1.1091)\n",
            "Saving model (epoch =   50, loss = 1.0838)\n",
            "Saving model (epoch =   52, loss = 1.0692)\n",
            "Saving model (epoch =   53, loss = 1.0681)\n",
            "Saving model (epoch =   55, loss = 1.0537)\n",
            "Saving model (epoch =   60, loss = 1.0457)\n",
            "Saving model (epoch =   61, loss = 1.0366)\n",
            "Saving model (epoch =   63, loss = 1.0359)\n",
            "Saving model (epoch =   64, loss = 1.0111)\n",
            "Saving model (epoch =   69, loss = 1.0072)\n",
            "Saving model (epoch =   72, loss = 0.9760)\n",
            "Saving model (epoch =   76, loss = 0.9672)\n",
            "Saving model (epoch =   79, loss = 0.9584)\n",
            "Saving model (epoch =   80, loss = 0.9526)\n",
            "Saving model (epoch =   82, loss = 0.9494)\n",
            "Saving model (epoch =   83, loss = 0.9426)\n",
            "Saving model (epoch =   88, loss = 0.9398)\n",
            "Saving model (epoch =   89, loss = 0.9223)\n",
            "Saving model (epoch =   95, loss = 0.9111)\n",
            "Saving model (epoch =   98, loss = 0.9034)\n",
            "Saving model (epoch =  101, loss = 0.9014)\n",
            "Saving model (epoch =  105, loss = 0.9011)\n",
            "Saving model (epoch =  106, loss = 0.8933)\n",
            "Saving model (epoch =  110, loss = 0.8893)\n",
            "Saving model (epoch =  117, loss = 0.8867)\n",
            "Saving model (epoch =  118, loss = 0.8867)\n",
            "Saving model (epoch =  121, loss = 0.8790)\n",
            "Saving model (epoch =  126, loss = 0.8642)\n",
            "Saving model (epoch =  130, loss = 0.8627)\n",
            "Saving model (epoch =  137, loss = 0.8616)\n",
            "Saving model (epoch =  139, loss = 0.8534)\n",
            "Saving model (epoch =  147, loss = 0.8467)\n",
            "Saving model (epoch =  154, loss = 0.8463)\n",
            "Saving model (epoch =  155, loss = 0.8408)\n",
            "Saving model (epoch =  167, loss = 0.8354)\n",
            "Saving model (epoch =  176, loss = 0.8314)\n",
            "Saving model (epoch =  191, loss = 0.8267)\n",
            "Saving model (epoch =  200, loss = 0.8212)\n",
            "Saving model (epoch =  226, loss = 0.8190)\n",
            "Saving model (epoch =  230, loss = 0.8144)\n",
            "Saving model (epoch =  244, loss = 0.8136)\n",
            "Saving model (epoch =  258, loss = 0.8095)\n",
            "Saving model (epoch =  269, loss = 0.8076)\n",
            "Saving model (epoch =  285, loss = 0.8064)\n",
            "Saving model (epoch =  330, loss = 0.8055)\n",
            "Saving model (epoch =  347, loss = 0.8053)\n",
            "Saving model (epoch =  359, loss = 0.7992)\n",
            "Saving model (epoch =  410, loss = 0.7989)\n",
            "Saving model (epoch =  442, loss = 0.7966)\n",
            "Saving model (epoch =  447, loss = 0.7966)\n",
            "Saving model (epoch =  576, loss = 0.7958)\n",
            "Saving model (epoch =  596, loss = 0.7929)\n",
            "Saving model (epoch =  600, loss = 0.7893)\n",
            "Saving model (epoch =  683, loss = 0.7825)\n",
            "Saving model (epoch =  878, loss = 0.7817)\n",
            "Saving model (epoch =  904, loss = 0.7794)\n",
            "Saving model (epoch =  931, loss = 0.7790)\n",
            "Saving model (epoch =  951, loss = 0.7781)\n",
            "Saving model (epoch =  965, loss = 0.7771)\n",
            "Saving model (epoch = 1018, loss = 0.7717)\n",
            "Saving model (epoch = 1168, loss = 0.7653)\n",
            "Saving model (epoch = 1267, loss = 0.7645)\n",
            "Saving model (epoch = 1428, loss = 0.7644)\n",
            "Saving model (epoch = 1461, loss = 0.7635)\n",
            "Saving model (epoch = 1484, loss = 0.7629)\n",
            "Saving model (epoch = 1493, loss = 0.7590)\n",
            "Finished training after 1694 epochs\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "hsNO9nnXQBvP",
        "outputId": "1626def6-94c7-4a87-9447-d939f827c8eb"
      },
      "source": [
        "plot_learning_curve(model_loss_record, title='deep model')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 20968 characters]\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "id": "3iZTVn5WQFpX",
        "outputId": "a2d5e118-559d-45c6-b644-6792af54663d"
      },
      "source": [
        "del model\n",
        "model = NeuralNet(tr_set.dataset.dim).to(device)\n",
        "ckpt = torch.load(config['save_path'], map_location='cpu')  # Load your best model\n",
        "model.load_state_dict(ckpt)\n",
        "plot_pred(dv_set, model, device)  # Show prediction on the validation set"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 27484 characters]\n",
            "text/plain": [
              "<Figure size 360x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQikz3IPiyPf"
      },
      "source": [
        "# **Testing**\n",
        "The predictions of your model on testing set will be stored at `pred.csv`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O8cTuQjQQOon",
        "outputId": "6bc5de07-4c5a-4e87-9ae3-d09f539c5f2c"
      },
      "source": [
        "def save_pred(preds, file):\n",
        "    ''' Save predictions to specified file '''\n",
        "    print('Saving results to {}'.format(file))\n",
        "    with open(file, 'w') as fp:\n",
        "        writer = csv.writer(fp)\n",
        "        writer.writerow(['id', 'tested_positive'])\n",
        "        for i, p in enumerate(preds):\n",
        "            writer.writerow([i, p])\n",
        "\n",
        "preds = test(tt_set, model, device)  # predict COVID-19 cases with your model\n",
        "save_pred(preds, 'pred.csv')         # save prediction file to pred.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving results to pred.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfrVxqJanGpE"
      },
      "source": [
        "# **Hints**\n",
        "\n",
        "## **Simple Baseline**\n",
        "* Run sample code\n",
        "\n",
        "## **Medium Baseline**\n",
        "* Feature selection: 40 states + 2 `tested_positive` (`TODO` in dataset)\n",
        "\n",
        "## **Strong Baseline**\n",
        "* Feature selection (what other features are useful?)\n",
        "* DNN architecture (layers? dimension? activation function?)\n",
        "* Training (mini-batch? optimizer? learning rate?)\n",
        "* L2 regularization\n",
        "* There are some mistakes in the sample code, can you find them?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tmCwXgpot3t"
      },
      "source": [
        "# **Reference**\n",
        "This code is completely written by Heng-Jui Chang @ NTUEE.  \n",
        "Copying or reusing this code is required to specify the original author. \n",
        "\n",
        "E.g.  \n",
        "Source: Heng-Jui Chang @ NTUEE (https://github.com/ga642381/ML2021-Spring/blob/main/HW01/HW01.ipynb)\n"
      ]
    }
  ]
}