zzy34407230
/
mindspore2022
Not watched
1
0
Fork 0
Code
Releases 22 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
34622 Commits
18 Branches
584 MB
265 Download
Tree: 03554c47ca
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '03554c47ca'
${ noResults }
mindspore2022/docs/api/api_python/nn/mindspore.nn.Adam.txt

mindspore.nn.Adam.txt 6.1 kB
Raw Normal View History

optim loss_scale datasethelper chinese api
4 years ago
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116 
  1. Class mindspore.nn.Adam(*args, **kwargs)

  2. 

  3.     ͨAdaptive Moment Estimation (Adam)㷨ݶȡ

  4. 

  5.     `Adam: A Method for Stochastic Optimization <https://arxiv.org/abs/1412.6980>`_

  6. 

  7.     ʽ£

  8. 

  9.     .. math::

  10.         \begin{array}{ll} \\

  11.             m_{t+1} = \beta_1 * m_{t} + (1 - \beta_1) * g \\

  12.             v_{t+1} = \beta_2 * v_{t} + (1 - \beta_2) * g * g \\

  13.             l = \alpha * \frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t} \\

  14.             w_{t+1} = w_{t} - l * \frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon}

  15.         \end{array}

  16. 

  17.     :math:`m`һ`moment1`:math:`v`ڶ`moment2`:math:`g``gradients`:math:`l`ӣ:math:`\beta_1,\beta_2``beta1``beta2`:math:`t`²裬:math:`beta_1^t`:math:`beta_2^t``beta1_power``beta2_power`:math:`\alpha``learning_rate`:math:`w``params`:math:`\epsilon``eps`

  18. 

  19.     ע

  20.         ǰʹSparseGatherV2ӣŻִϡ㣬ͨ`target`ΪCPUhostϽϡ㡣

  21.         ϡڳС

  22. 

  23.         ڲδʱŻõ`weight_decay`Ӧƺ"beta""gamma"ͨɵȨ˥ԡʱÿ`weight_decay`δãʹŻõ`weight_decay`

  24. 

  25. 

  26.     

  27.         params (Union[list[Parameter], list[dict]]): `Parameter`ɵбֵɵббԪֵʱֵļ"params""lr""weight_decay""grad_centralization""order_params"

  28. 

  29.             - params: ǰȨأֵ`Parameter`б

  30. 

  31.             - lr: ѡд"lr"ʹöӦֵΪѧϰʡ

  32.               ûУʹŻõ`learning_rate`Ϊѧϰʡ

  33. 

  34.             - weight_decay: ѡд"weight_decayʹöӦֵΪȨ˥ֵûУʹŻõ`weight_decay`ΪȨ˥ֵ

  35. 

  36.             - grad_centralization: ѡд"grad_centralization"ʹöӦֵֵΪ͡ûУΪ`grad_centralization`ΪFalse

  37.               òھ㡣

  38. 

  39.             - order_params: ѡӦֵԤڵIJ˳򡣵ʹò鹦ʱͨʹø`parameters`˳ܡ

  40.               д"order_params"Ըе"order_params"еIJijһ`params`С

  41. 

  42.         learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): Ĭֵ1e-3

  43. 

  44.             - float: ̶ѧϰʡڵ㡣

  45. 

  46.             - int: ̶ѧϰʡڵ㡣ͻᱻתΪ

  47. 

  48.             - Tensor: DZһάǹ̶ѧϰʡһάǶ̬ѧϰʣiȡеiֵΪѧϰʡ

  49. 

  50.             - Iterable: ̬ѧϰʡiȡiֵΪѧϰʡ

  51. 

  52.             - LearningRateSchedule: ̬ѧϰʡѵУŻʹòstepΪ룬`LearningRateSchedule`ʵ㵱ǰѧϰʡ

  53. 

  54.         beta1 (float): `moment1`ָ˥ʡΧ0.0,1.0

  55.                        Ĭֵ0.9

  56. 

  57.         beta2 (float): `moment2`ָ˥ʡΧ0.0,1.0

  58.                        Ĭֵ0.999

  59. 

  60.         eps (float): ӵĸУֵȶԡ0Ĭֵ1e-8

  61. 

  62.         use_locking (bool): ǷԲ¼

  63.             ΪTrue`w``m``v`tensor½ܵı

  64.             ΪFalseԤ⡣ĬֵFalse

  65. 

  66.         use_nesterov (bool): ǷʹNesterov Accelerated Gradient (NAG)㷨ݶȡ

  67.             ΪTrueʹNAGݶȡ

  68.             ΪFalseڲʹNAG¸ݶȡĬֵFalse

  69. 

  70.         weight_decay (float): Ȩ˥L2 penaltyڵ0Ĭֵ0.0

  71. 

  72.         loss_scale (float):    ݶϵ0`loss_scale`תΪͨʹĬֵѵʱʹ`FixedLossScaleManager``FixedLossScaleManager``drop_overflow_update`ΪFalseʱֵҪ`FixedLossScaleManager`е`loss_scale`ͬйظϸϢclass`mindspore.FixedLossScaleManager`

  73.             Ĭֵ1.0

  74. 

  75.     룺

  76.         - **gradients** (tuple[Tensor]) - `params`ݶȣ״shape`params`ͬ

  77. 

  78.     

  79.         Tensor[bool]ֵΪTrue

  80. 

  81.     쳣

  82.         TypeError: `learning_rate`intfloatTensorIterableLearningRateSchedule

  83.         TypeError: `parameters`ԪزParameterֵ䡣

  84.         TypeError: `beta1``beta2``eps``loss_scale`float

  85.         TypeError: `weight_decay`floatint

  86.         TypeError: `use_locking``use_nesterov`bool

  87.         ValueError: `loss_scale``eps`Сڻ0

  88.         ValueError: `beta1``beta2`ڣ0.0,1.0Χڡ

  89.         ValueError: `weight_decay`С0

  90. 

  91.     ֧ƽ̨

  92.         ``Ascend`` ``GPU``  ``CPU``

  93. 

  94.     ʾ

  95.         >>> net = Net()

  96.         >>> #1) вʹͬѧϰʺȨ˥

  97.         >>> optim = nn.Adam(params=net.trainable_params())

  98.         >>>

  99.         >>> #2) ʹò鲢òֵͬ

  100.         >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params()))

  101.         >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params()))

  102.         >>> group_params = [{'params': conv_params, 'weight_decay': 0.01, 'grad_centralization':True},

  103.         ...                 {'params': no_conv_params, 'lr': 0.01},

  104.         ...                 {'order_params': net.trainable_params()}]

  105.         >>> optim = nn.Adam(group_params, learning_rate=0.1, weight_decay=0.0)

  106.         >>> # conv_params齫ʹŻеѧϰ0.1Ȩ˥0.01ݶĻTrue

  107.         >>> # no_conv_params齫ʹøѧϰ0.01ŻеȨ˥0.0ݶĻʹĬֵFalse

  108.         >>> # Ż"order_params"õIJ˳²

  109.         >>>

  110.         >>> loss = nn.SoftmaxCrossEntropyWithLogits()

  111.         >>> model = Model(net, loss_fn=loss, optimizer=optim)

  112.     

  113. 

  114. target

  115. 

  116.         ָhostϻ豸deviceϸ²Ϊstrֻ'CPU''Ascend''GPU'

阿对对队