Huawei_Technology
/
mindspore
Not watched
1
0
Fork 0
Code
Releases 13 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
21109 Commits
11 Branches
1.3 GB
177 Download
Tree: 64689d56a1
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '64689d56a1'
${ noResults }
mindspore/tests/ut/data/dataset/testTokenizerData/basic_tokenizer.txt

basic_tokenizer.txt 502 B
Raw Normal View History

Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp add CaseFold, NormalizeUTF8 add RegexReplace add RegexTokenizer add BasicTokenizer add WordpieceTokenizer add BertTokenizer
5 years ago
​
​
​
​
​
​
 1234567 
  1. Welcome to Beijing北京欢迎您

  2. 長風破浪會有時，直掛雲帆濟滄海

  3. 😀嘿嘿😃哈哈😄大笑😁嘻嘻

  4. 明朝（1368—1644年）和清朝（1644—1911年），是中国封建王朝史上最后两个朝代

  5. 明代（1368-1644）と清代（1644-1911）は、中国の封建王朝の歴史における最後の2つの王朝でした

  6. 명나라 (1368-1644)와 청나라 (1644-1911)는 중국 봉건 왕조의 역사에서 마지막 두 왕조였다

  7. Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ

MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.