zzy34407230
/
mindspore2022
Not watched
1
0
Fork 0
Code
Releases 22 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tree: 6ebee5ff2c
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '6ebee5ff2c'
${ noResults }
mindspore2022/docs/api/api_python/mindspore.dataset.text.rst

58 lines
2.1 kB
Raw Blame History 

  1. mindspore.dataset.text

  2. ======================

  3. 

  4. This module is to support text processing for NLP. It includes two parts: transforms and utils. transforms is a high performance NLP text processing module which is developed with ICU4C and cppjieba. utils provides some general methods for NLP text processing.

  5. 

  6. Common imported modules in corresponding API examples are as follows:

  7. 

  8. .. code-block::

  9. 

  10.     import mindspore.dataset as ds

  11.     from mindspore.dataset import text

  12. 

  13. mindspore.dataset.text.transforms

  14. ---------------------------------

  15. 

  16. .. msnoteautosummary::

  17.     :toctree: dataset_text

  18.     :nosignatures:

  19.     :template: classtemplate.rst

  20. 

  21.     mindspore.dataset.text.transforms.BasicTokenizer

  22.     mindspore.dataset.text.transforms.BertTokenizer

  23.     mindspore.dataset.text.transforms.CaseFold

  24.     mindspore.dataset.text.transforms.JiebaTokenizer

  25.     mindspore.dataset.text.transforms.Lookup

  26.     mindspore.dataset.text.transforms.Ngram

  27.     mindspore.dataset.text.transforms.NormalizeUTF8

  28.     mindspore.dataset.text.transforms.PythonTokenizer

  29.     mindspore.dataset.text.transforms.RegexReplace

  30.     mindspore.dataset.text.transforms.RegexTokenizer

  31.     mindspore.dataset.text.transforms.SentencePieceTokenizer

  32.     mindspore.dataset.text.transforms.SlidingWindow

  33.     mindspore.dataset.text.transforms.ToNumber

  34.     mindspore.dataset.text.transforms.TruncateSequencePair

  35.     mindspore.dataset.text.transforms.UnicodeCharTokenizer

  36.     mindspore.dataset.text.transforms.UnicodeScriptTokenizer

  37.     mindspore.dataset.text.transforms.WhitespaceTokenizer

  38.     mindspore.dataset.text.transforms.WordpieceTokenizer

  39. 

  40. 

  41. mindspore.dataset.text.utils

  42. ----------------------------

  43. 

  44. .. msnoteautosummary::

  45.     :toctree: dataset_text

  46.     :nosignatures:

  47.     :template: classtemplate.rst

  48. 

  49.     mindspore.dataset.text.JiebaMode

  50.     mindspore.dataset.text.NormalizeForm

  51.     mindspore.dataset.text.SentencePieceModel

  52.     mindspore.dataset.text.SentencePieceVocab

  53.     mindspore.dataset.text.SPieceTokenizerLoadType

  54.     mindspore.dataset.text.SPieceTokenizerOutType

  55.     mindspore.dataset.text.to_str

  56.     mindspore.dataset.text.to_bytes

  57.     mindspore.dataset.text.Vocab