Huawei_Technology
/
mindspore
Not watched
1
0
Fork 0
Code
Releases 13 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
21109 Commits
11 Branches
1.3 GB
177 Download
Tree: 64689d56a1
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '64689d56a1'
${ noResults }
mindspore/tests/ut/data/dataset/testTokenizerData/bert_tokenizer.txt

bert_tokenizer.txt 264 B
Raw Normal View History

Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp add CaseFold, NormalizeUTF8 add RegexReplace add RegexTokenizer add BasicTokenizer add WordpieceTokenizer add BertTokenizer
5 years ago
​
​
​
​
​
​
​
​
​
​
​
BasicTokenizer not case fold on preserverd words
5 years ago
​
​
 123456789101112131415 
  1. 床前明月光

  2. 疑是地上霜

  3. 举头望明月

  4. 低头思故乡

  5. I am making small mistakes during working hours

  6. 😀嘿嘿😃哈哈😄大笑😁嘻嘻

  7. 繁體字

  8. unused [CLS]

  9. unused [SEP]

  10. unused [UNK]

  11. unused [PAD]

  12. unused [MASK]

  13. [unused1]

  14. [unused10]

  15. 12+/-28=40/-16

MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.