mindspore-ci-bot
b86747b9a6
!15813 Convert some pipeline Ops to be inlined
From: @hfarahat
Reviewed-by:
Signed-off-by:
5 years ago
mindspore-ci-bot
b85bf60eb8
!15727 VOCDataset read xml will filter invalid bbox automationally
From: @xiefangqi
Reviewed-by: @heleiwang,@jonyguo
Signed-off-by: @jonyguo
5 years ago
mindspore-ci-bot
46948b3eca
!16004 TFRecordDataset read big tfrecord file has to waiting more than 30s
From: @xiefangqi
Reviewed-by: @heleiwang,@jonyguo
Signed-off-by: @jonyguo
5 years ago
xiefangqi
c8bdf80078
tfrecord add big file warning
5 years ago
Lixia Chen
5967cb2cd7
Unblock mindrecord under cache
5 years ago
hesham
e4c0bd51a5
Modify concat to be an inlined Op
Modify zip to be an inlined Op
Modify take to be an inlined Op
Modify skip to be an inlined Op
Modify rename to be an inlined Op
Modify filter to consume child's rows using one thread
5 years ago
mindspore-ci-bot
1e2e554db3
!15380 Unify minddata include files(except for datasets.h) in scenario of cloud and lite
From: @luoyang42
Reviewed-by: @tiancixiao
Signed-off-by:
5 years ago
xiefangqi
458ff5fed9
md fix voc and manifest error scenario
5 years ago
Peilin Wang
8bc5098d99
fix ci alarms
add 2021 for copyright year for a file
fix ci
5 years ago
Jamie Nisbet
515c936b85
MindRecord becomes a mappable dataset
updates
review touches
ci fixes
ci fix
ci fixin
ci fix
review updates
further cleanup
updates
updates
update
update comment
5 years ago
mindspore-ci-bot
2a37201ea1
!15426 enhance: add info log when first batch push success
From: @jonyguo
Reviewed-by: @heleiwang,@pandoublefeng
Signed-off-by: @pandoublefeng
5 years ago
YangLuo
bf48400c10
unify include file of minddata
5 years ago
hesham
d167095b6d
Fix bug in checking status return in CSVOp
5 years ago
jonyguo
8c8e45ecde
enhance: add first push log info
5 years ago
mindspore-ci-bot
f9c5060cf2
!14930 Handle ignored returned Status
From: @hfarahat
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
5 years ago
hesham
98f0f36a20
Removed unused include
5 years ago
hesham
6e6eaebb77
Fix unused return value in main code
5 years ago
mindspore-ci-bot
1d89109d19
!14929 Cleanup stale comments about DataBuffers
From: @hfarahat
Reviewed-by: @pandoublefeng,@robingrosman
Signed-off-by: @pandoublefeng
5 years ago
mindspore-ci-bot
6c87bd31e1
!14814 fix codedex const keyword in PullBased MD
From: @mhmotallebi
Reviewed-by: @robingrosman
Signed-off-by:
5 years ago
hesham
15354569f2
Cleanup stale comments about DataBuffers
5 years ago
mohammad
cc42b2c221
fix codedex + add docstring
5 years ago
ms_yan
28371e4e41
codex fix
5 years ago
mindspore-ci-bot
74f4c85315
!14750 Address some code issues in MD
From: @hfarahat
Reviewed-by: @robingrosman,@mikef
Signed-off-by: @robingrosman
5 years ago
hesham
97cf4e516e
Address 3 code issues:
1- access out of bound memory in toFlatIndex
2- CreateFromNPArray function is 50>
3- duplicate variable init
5 years ago
xiefangqi
db43f196a7
fix codex&cpplint&pylint
5 years ago
hesham
8535e89e6c
Remove DataBuffer
5 years ago
hesham
0d86cb1423
Remove rows_per_buffer
5 years ago
hesham
a9a80693be
Replace Databuffers with TensorRows
5 years ago
mindspore-ci-bot
0bb4841427
!13291 Add a base class for Non-mappable source ops
From: @hfarahat
Reviewed-by:
Signed-off-by:
5 years ago
mindspore-ci-bot
3d4a1aaff1
!14428 fix codex issue for minddata profiling
From: @robingrosman
Reviewed-by: @pandoublefeng,@nsyca
Signed-off-by: @pandoublefeng
5 years ago
xiefangqi
a81f0fbbfa
tfrecord file name log fix
5 years ago
hesham
c877ac255b
Add NonMappableLeafOp and unify TfReader and TextFile, CSV, Clue and CSV
5 years ago
mindspore-ci-bot
1edbbe56ba
!13290 Add a base class for Mappable source ops
From: @hfarahat
Reviewed-by:
Signed-off-by:
5 years ago
hesham
a220a8d74b
Add MappableLeafOp and unify Album, ImageFolder, CelebA, Cifar, Coco, Manifest, Mnist, VOC and MindRecord
5 years ago
RobinGrosman
f4a735e3b6
fix codex and cpplint errors in cpu_samples.cc and parallel_op.cc
5 years ago
Eric
38944eafc4
Added get dataset size and fix macro
5 years ago
xiefangqi
de856a1bac
md fix gpu core dump problem
5 years ago
RobinGrosman
c616150fd6
Add analysis of profiling to warn if dataset parallel_workers should be increased
This message will currently only be printed if profiling of minddata is enabled.
Profiling of minddata is not enabled by default.
5 years ago
ms_yan
92e86804e1
init add acltdt handle create and destory
add hostpush part modify
optimize previous code
provide aclhandle access method
modify CMakeList format
add device_id parameter into TransferNode
update acltdt api
5 years ago
mindspore-ci-bot
9790a976e7
!13211 Fix CsvDataset OP
From: @lizhenglong1992
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @heleiwang
5 years ago
mindspore-ci-bot
7e8fa13197
!13136 【MD】【BUG】Fix log print for MD
From: @xulei2020
Reviewed-by: @heleiwang
Signed-off-by: @heleiwang
5 years ago
Eric
82f7c6f2c7
Moving data_helper
exposing json
WIP fix for removing templates
Removing templates part 1
replacing core/constants.h
removed template part 2
Changed fake data to filler data
Fixing CI
Fix CI part 2
CI fix part 2
Fix CI 3
Fixed lizard for data_util
Fixed clang
Fixing CI part 4
5 years ago
Zhenglong Li
6b3df8af51
Fix CsvDataset Operation bug
5 years ago
mindspore-ci-bot
f7040ca134
!13198 Fix get_dataset_size error with PKSampler
From: @ziruiwu
Reviewed-by:
Signed-off-by:
5 years ago
Zirui Wu
2692e3cc3d
fix pk sampler's get_dataset error due to num_class unavaiable at pre-runtime
5 years ago
RobinGrosman
a0e75b845a
Increase default level of parallelism in mindddata pipeline to 8.
Adjust memory usage to not increase as parallelism increases. it will
stay at same level it would be with 4 parallelism
5 years ago
xulei2020
4a4f555308
fix log print bug for MD
5 years ago
mindspore-ci-bot
62149265d5
!13066 [MD][usability] Fix batch_op log level
From: @xiefangqi
Reviewed-by: @liucunwei,@pandoublefeng
Signed-off-by: @liucunwei
5 years ago
mindspore-ci-bot
62a6f783b3
!12809 Added pull-based prototype
From: @mahdirahmanihanzaki
Reviewed-by:
Signed-off-by:
5 years ago
xiefangqi
425fdc8beb
fix batch log level
5 years ago