beimingwu
/
learnware
Not watched
1
0
Fork 0
Code
Releases 1 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tree: 257e95da5e
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '257e95da5e'
${ noResults }
learnware/tests/test_text_workflow/get_data.py

16 lines
414 B
Raw Blame History 

  1. import torch

  2. from torchtext.datasets import SST2

  3. 

  4. 

  5. def get_sst2(data_root="./data"):

  6.     train_datapipe = SST2(root="./data", split="train")

  7. 

  8.     X_train = [x[0] for x in train_datapipe]

  9.     y_train = [x[1] for x in train_datapipe]

  10. 

  11.     dev_datapipe = SST2(root="./data", split="dev")

  12. 

  13.     X_test = [x[0] for x in dev_datapipe]

  14.     y_test = [x[1] for x in dev_datapipe]

  15.     return X_train, y_train, X_test, y_test