gy15635282365
/
tensorlayer3
Not watched
1
0
Fork 0
Code
Releases 0 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branch: master
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'master'
${ noResults }
tensorlayer3/docs/modules/nlp.rst

149 lines
2.9 kB
Raw Permalink Blame History 

  1. API - Natural Language Processing

  2. ==================================

  3. 

  4. Natural Language Processing and Word Representation.

  5. 

  6. .. automodule:: tensorlayer.nlp

  7. 

  8. .. autosummary::

  9. 

  10.    generate_skip_gram_batch

  11. 

  12.    sample

  13.    sample_top

  14. 

  15.    SimpleVocabulary

  16.    Vocabulary

  17.    process_sentence

  18.    create_vocab

  19. 

  20.    simple_read_words

  21.    read_words

  22.    read_analogies_file

  23.    build_vocab

  24.    build_reverse_dictionary

  25.    build_words_dataset

  26.    save_vocab

  27. 

  28.    words_to_word_ids

  29.    word_ids_to_words

  30. 

  31.    basic_tokenizer

  32.    create_vocabulary

  33.    initialize_vocabulary

  34.    sentence_to_token_ids

  35.    data_to_token_ids

  36. 

  37.    moses_multi_bleu

  38. 

  39. 

  40. Iteration function for training embedding matrix

  41. -------------------------------------------------

  42. .. autofunction:: generate_skip_gram_batch

  43. 

  44. 

  45. Sampling functions

  46. -------------------

  47. 

  48. Simple sampling

  49. ^^^^^^^^^^^^^^^^^^^

  50. .. autofunction:: sample

  51. 

  52. Sampling from top k

  53. ^^^^^^^^^^^^^^^^^^^^^^

  54. .. autofunction:: sample_top

  55. 

  56. Vector representations of words

  57. -------------------------------

  58. 

  59. Simple vocabulary class

  60. ^^^^^^^^^^^^^^^^^^^^^^^^^

  61. .. autoclass:: SimpleVocabulary

  62. 

  63. Vocabulary class

  64. ^^^^^^^^^^^^^^^^^^^^^^^^^

  65. .. autoclass:: Vocabulary

  66. 

  67. Process sentence

  68. ^^^^^^^^^^^^^^^^^^^^^^^^

  69. .. autofunction:: process_sentence

  70. 

  71. Create vocabulary

  72. ^^^^^^^^^^^^^^^^^^^^^^^^

  73. .. autofunction:: create_vocab

  74. 

  75. Read words from file

  76. ----------------------

  77. 

  78. Simple read file

  79. ^^^^^^^^^^^^^^^^^^

  80. .. autofunction:: simple_read_words

  81. 

  82. Read file

  83. ^^^^^^^^^^^^^^^^^^

  84. .. autofunction:: read_words

  85. 

  86. 

  87. Read analogy question file

  88. -----------------------------

  89. .. autofunction:: read_analogies_file

  90. 

  91. Build vocabulary, word dictionary and word tokenization

  92. --------------------------------------------------------

  93. 

  94. Build dictionary from word to id

  95. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  96. .. autofunction:: build_vocab

  97. 

  98. Build dictionary from id to word

  99. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  100. .. autofunction:: build_reverse_dictionary

  101. 

  102. Build dictionaries for id to word etc

  103. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  104. .. autofunction:: build_words_dataset

  105. 

  106. Save vocabulary

  107. ^^^^^^^^^^^^^^^^^^^^

  108. .. autofunction:: save_vocab

  109. 

  110. 

  111. Convert words to IDs and IDs to words

  112. --------------------------------------------------------

  113. These functions can be done by ``Vocabulary`` class.

  114. 

  115. List of Words to IDs

  116. ^^^^^^^^^^^^^^^^^^^^^^^

  117. .. autofunction:: words_to_word_ids

  118. 

  119. List of IDs to Words

  120. ^^^^^^^^^^^^^^^^^^^^^

  121. .. autofunction:: word_ids_to_words

  122. 

  123. 

  124. 

  125. Functions for translation

  126. ---------------------------

  127. 

  128. Word Tokenization

  129. ^^^^^^^^^^^^^^^^^^^

  130. .. autofunction:: basic_tokenizer

  131. 

  132. Create or read vocabulary

  133. ^^^^^^^^^^^^^^^^^^^^^^^^^^^

  134. .. autofunction:: create_vocabulary

  135. .. autofunction:: initialize_vocabulary

  136. 

  137. Convert words to IDs and IDs to words

  138. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  139. .. autofunction:: sentence_to_token_ids

  140. .. autofunction:: data_to_token_ids

  141. 

  142. 

  143. Metrics

  144. ---------------------------

  145. 

  146. BLEU

  147. ^^^^^^^^^^^^^^^^^^^

  148. .. autofunction:: moses_multi_bleu