xuyongfei
ec4b97ff01
FL, opt allreduce
4 years ago
zhoufeng
f49b195c39
extract common as an independent shared library
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
4 years ago
marui
d47e79b04c
Refactor ccsrc directories and CMakeLists files
4 years ago
xuyongfei
68286c5a45
FL, update model get client info
4 years ago
twc
7c90bdb9c0
1.fix bug in updatemodel with federated learning mode
2.sync code
3.update log level
4 years ago
i-robot
675fa88bce
!29158 端云联邦SignDS算法
Merge pull request !29158 from emmmmtang/master
4 years ago
i-robot
787605d6ca
!29499 fix I4RQOC && sync code
Merge pull request !29499 from tan-wei-cheng-3260/develop-twc-master
4 years ago
emmmmtang
e227c8413f
signds
4 years ago
twc
997ae1133a
fix I4RQOC && sync code
4 years ago
jin-xiulang
eef3dd598f
clean code
4 years ago
TronZhang
deac002bed
refactor kernel mod class and subclass
4 years ago
i-robot
6b78c1a827
!29316 FL, update iterator func_graph_ to weak_ptr
Merge pull request !29316 from 徐永飞/master
4 years ago
xuyongfei
6bad80681b
FL, update iterator func_graph_ to weak_ptr && fix tcp message handler memory bug
4 years ago
twc
7bd718a003
fix fl pclint
4 years ago
i-robot
28c5d74628
!29257 FL, opt kernel launch
Merge pull request !29257 from 徐永飞/master
4 years ago
xuyongfei
98c2cafa69
FL, opt kernel launch
4 years ago
twc
76d9daddf0
fix ISSUE I4QRZP
4 years ago
i-robot
6eae53eb34
!29057 fix ISSUE I4QCJM
Merge pull request !29057 from tan-wei-cheng-3260/develop-fix
4 years ago
twc
5d40613d52
fix ISSUE I4QCJM
4 years ago
xuyongfei
cd4ec2d23a
FL, opt DataPtr in fl, avoid memory copy
4 years ago
ZPaC
9dd90de735
Fix static check.
4 years ago
jin-xiulang
7afab815cd
Add default parameters for federated learning.
4 years ago
i-robot
6d77fc8a14
!27650 fix codedex in fl
Merge pull request !27650 from tan-wei-cheng-3260/develop-twc-pc-lint
4 years ago
i-robot
75ab2ec58a
!27809 Syncronize federated learning's code from enterprise docs
Merge pull request !27809 from jxlang910/master
4 years ago
jin-xiulang
e94ed820d1
Synchronize federated learning's code
4 years ago
ZPaC
0eaa2f3404
Fix scheduler nullptr error and add server group
4 years ago
twc
0f129bd94d
fix codedex in fl
4 years ago
i-robot
9a1dde1f02
!27398 server0 recovery
Merge pull request !27398 from tan-wei-cheng-3260/develop-twc-recovery
4 years ago
w00517672
0064fc7ebc
fix bugs triggered by fl
4 years ago
twc
f62414ce2b
server0 recovery
4 years ago
i-robot
5a26761316
!27120 fix fl on mac
Merge pull request !27120 from xulei/fix_fl_mac
4 years ago
xulei
ed9ad486f2
fix fl on mac
4 years ago
twc
63dac501e0
code sync in mindspore fl
4 years ago
twc
a5fcb08c00
code sync in federated learning
4 years ago
He Wei
41dcac9c49
Replace std::unordered_map/set with robin-hood-hashing
Robin-hood-hashing (https://github.com/martinus/robin-hood-hashing )
is considered faster then std::unordered_map/set,
so we use it to improve mindspore performance.
1. robin_hood head file in `third_party/robin_hood/include`;
2. In `utils/hash_map.h` and `utils/hash_set.h`, we define:
- mindspore::HashMap as an alias of robin_hood::unordered_map;
- mindspore::HashSet as an alias of robin_hood::unordered_set;
3. Replace:
- `#include <unordered_map>` --> `#include "utils/hash_map.h"`;
- `#include <unordered_set>` --> `#include "utils/hash_set.h"`;
- `std::unordered_map` --> `mindspore::HashMap`;
- `std::unordered_set` --> `mindspore::HashSet`;
- `map.insert(std::pair(key, value))` --> `map.emplace(key, value)`;
- `[] (const std::pair<K, V> &p) {..} ` --> `[] (const auto &p) {..} `;
4. Fix issues found by switch to robin_hood:
- AnfNodeConfig hash and equal;
- Fix a bug in `Slice::operator==()`;
- Fix a bug in `CNode::HasPrimalAttr()`;
- Fix map.erase() usage bugs: `map.erase(iter++)` --> `iter = map.erase(iter)`;
- Fix some iterator invalidated problem;
5. Some std::unordered_map/set can not replace by robin_hood:
- As parameter of functions that exposed to python by pybind11;
- Use bad hash that cause robin_hood::map over_flow, such as AbstractBasePtrListHasher;
6. Update cpp unit tests;
7. Add build option '-F' to enable robin_hood, default on.
4 years ago
jin-xiulang
f972de90d0
Add exchangeKeys ops and getKeys ops for STABLE_PW_ENCRYPT
fix review syggestions
4 years ago
zhunaipan
8ce4e62725
optimize the comment and log description
修改: ops/operations/_inner_ops.py
修改: ops/operations/_quant_ops.py
修改: ops/operations/array_ops.py
修改: ops/operations/comm_ops.py
修改: ops/operations/math_ops.py
修改: ops/operations/quantum_ops.py
修改: ops/operations/rl_ops.py
修改: ops/operations/sponge_ops.py
修改: ops/operations/sponge_update_ops.py
修改: train/__init__.py
修改: common/tensor.py
修改: train/serialization.py
修改: ccsrc/pipeline/jit/parse/parse.h
修改: explainer/benchmark/_attribution/metric.py
修改: ops/composite/multitype_ops/_constexpr_utils.py
修改: ops/operations/comm_ops.py
修改: RELEASE.md
修改: mindspore/_extends/parse/standard_method.py
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/concat_offset_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/dynamic_shape_cpu_kernel.cc
修改: mindspore/ccsrc/frontend/parallel/ops_info/reshape_info.cc
修改: mindspore/ccsrc/frontend/parallel/ops_info/tile_info.cc
修改: mindspore/ccsrc/frontend/parallel/ops_info/transpose_info.cc
修改: mindspore/ccsrc/frontend/parallel/strategy.h
修改: mindspore/common/tensor.py
修改: mindspore/core/abstract/prim_arrays.cc
修改: mindspore/core/abstract/prim_nn.cc
修改: mindspore/core/ops/conv2d.cc
修改: mindspore/core/ops/logical_and.h
修改: mindspore/core/ops/logical_not.h
修改: mindspore/core/ops/logical_or.h
修改: mindspore/core/ops/reduce_all.h
修改: mindspore/core/ops/reduce_any.h
修改: mindspore/lite/src/runtime/kernel/arm/fp32_grad/sgd.cc
修改: mindspore/nn/layer/quant.py
修改: mindspore/nn/optim/sgd.py
修改: mindspore/nn/sparse/sparse.py
修改: mindspore/numpy/array_creations.py
修改: mindspore/numpy/array_ops.py
修改: mindspore/numpy/logic_ops.py
修改: mindspore/numpy/math_ops.py
修改: mindspore/ops/operations/_inner_ops.py
修改: mindspore/ops/operations/array_ops.py
修改: mindspore/ops/operations/rl_ops.py
修改: mindspore/train/_utils.py
修改: tests/ut/python/model/test_lenet_core_after_exception.py
修改: mindspore/_extends/parse/standard_method.py
修改: mindspore/ops/operations/rl_ops.py
修改: mindspore/core/abstract/prim_nn.cc
修改: mindspore/core/ops/conv2d.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/ctcloss_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/fl/fused_pull_weight_kernel.h
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/fl/fused_push_weight_kernel.h
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/mkldnn/conv2d_grad_filter_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/mkldnn/conv2d_grad_input_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/ps/sparse_apply_ftrl_ps_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/ps/sparse_apply_lazy_adam_ps_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/rolling_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/scatter_arithmetic_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/split_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/update_cache_cpu_kernel.cc
修改: mindspore/ccsrc/backend/kernel_compiler/gpu/arrays/split_gpu_kernel.h
修改: mindspore/ccsrc/backend/kernel_compiler/gpu/math/broadcast_gpu_kernel.h
修改: mindspore/ccsrc/backend/kernel_compiler/gpu/nn/conv2d_grad_input_gpu_kernel.h
修改: mindspore/ccsrc/fl/server/server.cc
修改: mindspore/ccsrc/frontend/optimizer/ad/kpynative.cc
修改: mindspore/ccsrc/frontend/optimizer/irpass/incorporate_getitem.h
修改: mindspore/ccsrc/frontend/optimizer/irpass/inline.h
修改: mindspore/ccsrc/minddata/dataset/core/device_tensor.cc
修改: mindspore/ccsrc/minddata/dataset/core/tensor.cc
修改: mindspore/ccsrc/minddata/dataset/engine/datasetops/source/emnist_op.cc
修改: mindspore/ccsrc/minddata/dataset/engine/datasetops/source/mnist_op.cc
修改: mindspore/ccsrc/minddata/dataset/engine/datasetops/source/qmnist_op.cc
修改: mindspore/ccsrc/minddata/dataset/engine/ir/datasetops/dataset_node.cc
修改: mindspore/ccsrc/minddata/dataset/engine/opt/pre/epoch_ctrl_pass.cc
修改: mindspore/ccsrc/minddata/dataset/kernels/image/lite_image_utils.cc
修改: mindspore/ccsrc/pipeline/jit/action.cc
修改: mindspore/ccsrc/pipeline/jit/static_analysis/evaluator.cc
修改: mindspore/ccsrc/runtime/device/ascend/executor/tiling/op_tiling_adapter.cc
修改: mindspore/compression/quant/quant_utils.py
修改: mindspore/core/abstract/prim_nn.cc
修改: mindspore/dataset/engine/validators.py
修改: mindspore/lite/micro/coder/opcoders/nnacl/fp32/affine_fp32_coder.cc
修改: mindspore/lite/micro/coder/opcoders/nnacl/int8/affine_int8_coder.cc
修改: mindspore/lite/src/runtime/kernel/ascend310/src/custom_kernel.cc
修改: mindspore/lite/src/runtime/kernel/opencl/kernel/matmul.cc
修改: mindspore/lite/src/runtime/kernel/opencl/kernel/strassen.cc
修改: mindspore/lite/tools/common/graph_util.h
修改: mindspore/lite/tools/optimizer/fisson/fisson_util.cc
修改: mindspore/ops/composite/math_ops.py
修改: mindspore/ops/operations/_inner_ops.py
修改: mindspore/ops/operations/array_ops.py
修改: mindspore/ops/operations/math_ops.py
修改: mindspore/ops/operations/other_ops.py
修改: mindspore/boost/boost_cell_wrapper.py
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/update_cache_cpu_kernel.cc
修改: mindspore/ccsrc/common/trans.cc
修改: mindspore/ccsrc/frontend/parallel/cache_embedding/cache_embedding.cc
修改: mindspore/ccsrc/frontend/parallel/ops_info/gather_info.cc
修改: mindspore/lite/src/common/log_util.h
修改: mindspore/nn/wrap/loss_scale.py
修改: mindspore/parallel/nn/moe.py
修改: tests/mindspore_test_framework/mindspore_test.py
修改: mindspore/ccsrc/backend/kernel_compiler/cpu/split_cpu_kernel.cc
修改: mindspore/lite/tools/common/graph_util.h
修改: mindspore/ccsrc/frontend/parallel/ops_info/gather_info.cc
修改: mindspore/core/ops/conv2d.cc
修改: tests/ut/python/model/test_lenet_core_after_exception.py
4 years ago
ZPaC
01302b9fa9
Fix static
4 years ago
i-robot
3c2b7f63aa
!24816 Add collective allgather broadcast
Merge pull request !24816 from ZPaC/add-collective-allgather-broadcast
4 years ago
ZPaC
3d9620208a
Add collective allgather broadcast
4 years ago
ZPaC
db00c45793
Fix static
4 years ago
ZPaC
f479e144b9
Fix static
4 years ago
i-robot
8da0ca4a7e
!23743 Fix pclint and codeDex Warnings of Federated's secure aggregation
Merge pull request !23743 from jxlang910/master
4 years ago
jin-xiulang
a57becafc6
Fix pclint and codeDex Warnings
4 years ago
ZPaC
e0068e2e2a
Fix file path
4 years ago
i-robot
b23b353319
!23421 Fix code-warnings of federated's secure aggregation.
Merge pull request !23421 from jxlang910/master
4 years ago
jin-xiulang
6a1aa8e4c5
Fix code-warnings of federated's secure aggregation
4 years ago
ZPaC
26d6cf8a3c
Fix real path usage
4 years ago
jin-xiulang
7d9dd343f3
Fix security problems and code-check problems for federated's secure aggregation
fix security check problems for flclient
4 years ago
ZPaC
7fa258f178
Fix cross silo running issue.
4 years ago