Hetu/python/hetu/gpu_ops/README.md

• gpu_ops
• • Executor
  • Operators
  • • Computation
    • Communication

gpu_ops

This directory contains executor and operators for computation and communication. Though the name of directory is "gpu_ops", in each operator we call different API for computation in NumPy(CPU), DNNL(CPU), CUDA(GPU) according to the context specified in executor and the environment.

Executor

• Defined in executor.py, contains all the configurations and controls the training/inference process.

Operators

Computation

Operator	NumPy(CPU)	DNNL(CPU)	CUDA(GPU)	CUDA Backend
AddByConstOp	✔	✔	✔	/
AddOp	✔	✔	✔	/
Avg_Pool2dOp	✔	✔	✔	CuDNN
Avg_Pool2d_GradientOp	✔	✔	✔	CuDNN
BatchMatMulOp	✔	✖	✔	CuBLAS
Batch_NormalizationOp	✔	✔	✔	CuDNN
Batch_Normalization_GradientOp	✔	✔	✔	CuDNN
BinaryCrossEntropyOp	✔	✖	✔	/
BroadcastToOp	✔	✖	✔	/
BroadcastShapeOp	✔	✖	✔	/
ConcatOp	✔	✔	✔	/
Concat_gradientOP	✔	✔	✔	/
Conv2dOp	✔	✔	✔	/
Conv2d_Gradient_of_DataOp	✔	✔	✔	/
Conv2d_Gradient_of_FilterOp	✔	✔	✔	/
Conv2d_BroadcastToOp	✔	✖	✔	/
Conv2d_ReduceSumOp	✔	✖	✔	/
CsrmvOp	✔	✖	✔	/
CsrmmOp	✔	✖	✔	/
DistGCN_15dOp	✖	✖	✔	/
DivOp	✔	✔	✔	/
DivConstOp	✔	✔	✔	/
DropoutOp	✔	✔	✔	CuRAND
Dropout_GradientOp	✔	✔	✔	CuRAND
Dropout2dOp	✖	✖	✔	CuRAND
Dropout2d_GradientOp	✖	✖	✔	CuRAND
EmbeddingLookUp	✔	✖	✔	/
EmbeddingLookUp_Gradient	✔	✖	✔	/
Instance_Normalization2dOp	✖	✖	✔	CuDNN
Instance_Normalization2d_GradientOp	✖	✖	✔	CuDNN
Layer_NormalizationOp	✔	✖	✔	CuDNN
Layer_Normalization_GradientOp	✔	✖	✔	CuDNN
LeakyReluOp	✖	✖	✔	/
LeakyReluGradientOp	✖	✖	✔	/
MatrixDotOp	✔	✖	✔	/
MatMulOp	✔	✔	✔	CuBLAS
Max_Pool2dOp	✔	✔	✔	CuDNN
Max_Pool2d_GradientOp	✔	✔	✔	CuDNN
MulByConstOp	✔	✔	✔	/
MulOp	✔	✔	✔	/
OneHotOp	✔	✖	✔	/
OnesLikeOp	✔	✔	✔	/
OppositeOp	✔	✔	✔	/
PadOp	✔	✔	✔	/
Pad_GradientOp	✔	✔	✔	/
ReduceMeanOp	✔	✖	✔	CuDNN
ReduceSumOp	✔	✖	✔	CuDNN
ReduceSumAxisZeroOp	✔	✔	✔	/
ReluOp	✔	✔	✔	/
ReluGradientOp	✔	✔	✔	/
Array_ReshapeOp	✔	✔	✔	/
SigmoidOp	✔	✔	✔	/
SliceOp	✔	✖	✔	/
SliceGradientOp	✔	✖	✔	/
SoftmaxOp	✔	✔	✔	CuDNN
SoftmaxGradientOp	✔	✖	✔	CuDNN
SoftmaxCrossEntropyOp	✔	✔	✔	CuDNN (Optional)
SoftmaxCrossEntropyGradientOp	✔	✖	✔	CuDNN (Optional)
SplitOp	✔	✖	✔	/
SplitGradientOp	✔	✖	✔	/
SqrtOp	✔	✔	✔	/
ReciprocalSqrtOp	✔	✔	✔	/
TanhOp	✔	✔	✔	/
TransposeOp	✔	✔	✔	/
WhereOp	✔	✖	✔	/
ZerosLikeOp	✔	✔	✔	/
OptimizerOp	✔	✔	✔	/
OptimizerOp for sparse	✔	✖	✔	/
DataloaderOp	✔	✔	/	/

Communication

• DataH2DOp
• DataD2HOp
• DataD2HSparseOp
• AllReduceCommunicateOp
• ParameterServerCommunicateOp
• PipelineSendOp
• PipelineReceiveOp
• Dispatch