zzy34407230
/
mindspore2022
Not watched
1
0
Fork 0
Code
Releases 22 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
34722 Commits
18 Branches
584 MB
67 Download
Tree: a64920c468
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'a64920c468'
${ noResults }
mindspore2022/docs/api/api_python/nn/mindspore.nn.Adam.rst

mindspore.nn.Adam.rst 6.0 kB
Raw Normal View History

modify api annotaion
4 years ago
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
modify api annotation
4 years ago
 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102 
  1. mindspore.nn.Adam

  2. ==================

  3. 

  4. .. py:class:: mindspore.nn.Adam(*args, **kwargs)

  5. 

  6.     ͨAdaptive Moment Estimation (Adam)㷨ݶȡ

  7. 

  8.     `Adam: A Method for Stochastic Optimization <https://arxiv.org/abs/1412.6980>`_

  9. 

  10.     ʽ£

  11. 

  12.     .. math::

  13.         \begin{array}{ll} \\

  14.             m_{t+1} = \beta_1 * m_{t} + (1 - \beta_1) * g \\

  15.             v_{t+1} = \beta_2 * v_{t} + (1 - \beta_2) * g * g \\

  16.             l = \alpha * \frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t} \\

  17.             w_{t+1} = w_{t} - l * \frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon}

  18.         \end{array}

  19. 

  20.     :math:`m` һ `moment1` :math:`v` ڶ `moment2` :math:`g`  `gradients` :math:`l` ӣ:math:`\beta_1,\beta_2`  `beta1`  `beta2` :math:`t` ²裬:math:`beta_1^t` :math:`beta_2^t`  `beta1_power`  `beta2_power` :math:`\alpha`  `learning_rate` :math:`w`  `params` :math:`\epsilon`  `eps` 

  21. 

  22.     .. note::

  23.         ǰʹSparseGatherV2ӣŻִϡ㣬ͨ `target` ΪCPUhostϽϡ㡣

  24.         ϡڳС

  25.   

  26.         ڲδʱŻõ `weight_decay` Ӧƺ"beta""gamma"ͨɵȨ˥ԡʱÿ `weight_decay` δãʹŻõ `weight_decay` 

  27. 

  28. 

  29.     ****

  30. 

  31.     - **params** (Union[list[Parameter], list[dict]]) -  `Parameter` ɵбֵɵббԪֵʱֵļ"params""lr""weight_decay""grad_centralization""order_params"

  32.             

  33.       - **params** - ǰȨأֵ `Parameter` б

  34.       - **lr** - ѡд"lr"ʹöӦֵΪѧϰʡûУʹŻõ `learning_rate` Ϊѧϰʡ

  35.       - **weight_decay** - ѡд"weight_decayʹöӦֵΪȨ˥ֵûУʹŻõ `weight_decay` ΪȨ˥ֵ

  36.       - **grad_centralization** - ѡд"grad_centralization"ʹöӦֵֵΪ͡ûУΪ `grad_centralization` ΪFalseòھ㡣

  37.       - **order_params** - ѡӦֵԤڵIJ˳򡣵ʹò鹦ʱͨʹø `parameters` ˳ܡд"order_params"Ըе"order_params"еIJijһ `params` С

  38.     

  39.     - **learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): Ĭֵ1e-3

  40. 

  41.       - **float** - ̶ѧϰʡڵ㡣

  42.       - **int** - ̶ѧϰʡڵ㡣ͻᱻתΪ

  43.       - **Tensor** - DZһάǹ̶ѧϰʡһάǶ̬ѧϰʣiȡеiֵΪѧϰʡ

  44.       - **Iterable** - ̬ѧϰʡiȡiֵΪѧϰʡ

  45.       - **LearningRateSchedule** - ̬ѧϰʡѵУŻʹòstepΪ룬 `LearningRateSchedule` ʵ㵱ǰѧϰʡ

  46.     

  47.     - **beta1** (float) - `moment1` ָ˥ʡΧ0.0,1.0Ĭֵ0.9

  48.     - **beta2** (float) - `moment2` ָ˥ʡΧ0.0,1.0Ĭֵ0.999

  49.     - **eps** (float) - ӵĸУֵȶԡ0Ĭֵ1e-8

  50.     - **use_locking** (bool) - ǷԲ¼ΪTrue `w` `m`  `v` tensor½ܵıΪFalseԤ⡣ĬֵFalse

  51.     - **use_nesterov** (bool) - ǷʹNesterov Accelerated Gradient (NAG)㷨ݶȡΪTrueʹNAGݶȡΪFalseڲʹNAG¸ݶȡĬֵFalse

  52.     - **weight_decay** (float) - Ȩ˥L2 penaltyڵ0Ĭֵ0.0

  53.     - **loss_scale** (float) - ݶϵ0 `loss_scale` תΪͨʹĬֵѵʱʹ `FixedLossScaleManager`  `FixedLossScaleManager`  `drop_overflow_update` ΪFalseʱֵҪ `FixedLossScaleManager` е `loss_scale` ͬйظϸϢclass`mindspore.FixedLossScaleManager` Ĭֵ1.0

  54. 

  55.     **룺**

  56. 

  57.     **gradients** (tuple[Tensor]) - `params` ݶȣ״shape `params` ͬ

  58. 

  59.     ****

  60. 

  61.     Tensor[bool]ֵΪTrue

  62. 

  63.     **쳣**

  64. 

  65.     - **TypeError** - `learning_rate` intfloatTensorIterableLearningRateSchedule

  66.     - **TypeError** - `parameters` ԪزParameterֵ䡣

  67.     - **TypeError** - `beta1` `beta2`  `eps`  `loss_scale` float

  68.     - **TypeError** - `weight_decay` floatint

  69.     - **TypeError** - `use_locking`  `use_nesterov` bool

  70.     - **ValueError** - `loss_scale`  `eps` Сڻ0

  71.     - **ValueError** - `beta1` `beta2` ڣ0.0,1.0Χڡ

  72.     - **ValueError** - `weight_decay` С0

  73. 

  74.     **֧ƽ̨**

  75. 

  76.     ``Ascend``  ``GPU``  ``CPU``

  77. 

  78.     ****

  79.     

  80.     >>> net = Net()

  81.     >>> #1) вʹͬѧϰʺȨ˥

  82.     >>> optim = nn.Adam(params=net.trainable_params())

  83.     >>>

  84.     >>> #2) ʹò鲢òֵͬ

  85.     >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params()))

  86.     >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params()))

  87.     >>> group_params = [{'params': conv_params, 'weight_decay': 0.01, 'grad_centralization':True},

  88.     ...                 {'params': no_conv_params, 'lr': 0.01},

  89.     ...                 {'order_params': net.trainable_params()}]

  90.     >>> optim = nn.Adam(group_params, learning_rate=0.1, weight_decay=0.0)

  91.     >>> # conv_params齫ʹŻеѧϰ0.1Ȩ˥0.01ݶĻTrue

  92.     >>> # no_conv_params齫ʹøѧϰ0.01ŻеȨ˥0.0ݶĻʹĬֵFalse

  93.     >>> # Ż"order_params"õIJ˳²

  94.     >>>

  95.     >>> loss = nn.SoftmaxCrossEntropyWithLogits()

  96.     >>> model = Model(net, loss_fn=loss, optimizer=optim)

  97.     

  98. 

  99.     .. py:method:: target

  100.         :property:

  101. 

  102.         ָhostϻ豸deviceϸ²Ϊstrֻ'CPU''Ascend''GPU'

阿对对队