mindspore-ci-bot
8eb8663fbd
!2000 Make GetMutableBuffer protected
Merge pull request !2000 from h.farahat/tensor_mutabble_buffer
5 years ago
mindspore-ci-bot
1b5d9a1194
!1981 Implemented from_dataset, a way to build a vocab from existing dataset
Merge pull request !1981 from ZiruiWu/from_dataset_dev
5 years ago
hesham
0c5e47111b
- Move MutableBuffer to be protected
- Add new CreateTensors
5 years ago
Zirui Wu
880ce5ea26
implemented from_dataset
fix complie error
more tests
address CI complains
fix ci
adress review comments
address review cmts
5 years ago
Alexey Shevlyakov
4e333a2f22
MindData profiler infrastructure
5 years ago
Jesse Lee
a18b04e0f8
Fix coco
5 years ago
mindspore-ci-bot
beefb20c01
!1983 Remove inheritance of Sampler from DatasetOp
Merge pull request !1983 from JesseKLee/sampler
5 years ago
mindspore-ci-bot
b3da41bd7a
!1913 CocoDataset implementation
Merge pull request !1913 from xiefangqi/xfq_support_coco
5 years ago
xiefangqi
097cbcc0cb
CocoDataset implementation
5 years ago
Jesse Lee
255adf7c6e
Remove inheritance of Sampler from DatasetOp
5 years ago
mindspore-ci-bot
5eb95599f6
!1874 Updates to string tensor
Merge pull request !1874 from h.farahat/string_tensor_updates
5 years ago
hesham
f837ddc956
- Bug when empty strings sent to Python
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
5 years ago
xiefangqi
8fdfe34f3c
fix codex problems
5 years ago
mindspore-ci-bot
1973594bd1
!1774 [MD] minddataset support padding samples
Merge pull request !1774 from liyong126/mindrecord_padded_sample
5 years ago
mindspore-ci-bot
58643afc09
!1849 fix tfrecord bug:less data if shuffle is false
Merge pull request !1849 from yanghaitao/yht_fix_tf_noshuffle
5 years ago
liyong
feff8899ac
support padding samples
5 years ago
mindspore-ci-bot
769ae609b4
!1808 consistent design for num_samples
Merge pull request !1808 from Jamie/numsamples
5 years ago
yanghaitao
07877d1e65
fix tfrecord not enough data if shuffle is false
5 years ago
Peilin Wang
9c8ea3c1af
sort columns so schema always has same ColDescriptor ordering
5 years ago
Jamie Nisbet
51bc0c0460
consistent design for num_samples
updates
more update
more work
more fixin
post rebase updates
clang formatting
code review recovery
ci fixes
updates
update
update
update
5 years ago
yanghaitao
dc12598570
minddata-tdt profiling
5 years ago
xiefangqi
4c70d25777
fix codex of voc and graph
5 years ago
Peilin Wang
3ef3d1433d
fixed num_samples bug for RandomSampler
5 years ago
mindspore-ci-bot
4fb59b3787
!1549 fix tfRecordDataset hang
Merge pull request !1549 from yanghaitao/yht_tfreadop_equal_rows_hang
5 years ago
yanghaitao
f55cda2c56
fix tfreadDataset hang when shard_equal_rows is true
5 years ago
xiefangqi
5e4728c50f
add voc support split
5 years ago
mindspore-ci-bot
e7936dedeb
!1480 gpu iiterator weak ref support
Merge pull request !1480 from panfengfeng/iterator_gpu_weak_ref
5 years ago
Peilin Wang
d4c93575e7
fixed bug for split, RandomSampler and some other cleanup
add another test case
typo
merge conflict
another PR changed testing behavior, updated test cases in this commit
added input check for use_sampler
addressed code review comments
fixed pylint, not related to my changes
5 years ago
panfengfeng
636d419af3
gpu iterator weak ref opt
5 years ago
Peilin Wang
0cbcc7200b
made shuffle determinisitc for subsequent epochs
5 years ago
mindspore-ci-bot
6f733ec113
!1308 Stage 2 of adding support for string Tensor
Merge pull request !1308 from h.farahat/string_tensor2
5 years ago
heleiwang
599a449e0b
Support processing GNN data
6 years ago
hesham
df361d1d26
Change mem layout of string tensor
add support for MindRecord and TFRecord
----
optimize tensorshape
optimize tensorshape and FlatIndex
TFRecord and MindRecord support for string tensor
Modify mem layout
Add new constructor
Add method Allocate
Change some GetMutableBuffer usages to AllocateBuffer
6 years ago
jonwe
bb51bb88d7
add compress in mindrecord
6 years ago
mindspore-ci-bot
2e3d55ed87
!1281 Implementation of SplitOp
Merge pull request !1281 from Peilin/splitOp
5 years ago
Peilin Wang
71e8bb1960
general split case done, chaining sampler (basic case) is working
implementation 99% complete
everything and tested except for repeatable shuffling
tested most basic/typical split usecases
cleanup
some more cleanup
fix CI
more ci fix
more ci fixes
more ci fix
more ci fix
more ci fix
added more tests, fixed some bugs
some more clean up and test cases
added shard/shuffle before split warning/error
addressed code review comments and ci
fixed ci
6 years ago
mindspore-ci-bot
661715bad9
!1318 Fix batch error log too simple problem
Merge pull request !1318 from xiefangqi/xfq_fix_batch_log
5 years ago
xiefangqi
25bb3960d4
add detail batch error to log
5 years ago
xiefangqi
394aeb0360
fix too much MS_LOG(INFO) in MD
5 years ago
Junhan Hu
f44d213503
MindData optimizer infrastructure.
6 years ago
ms_yan
e6c25859d4
check wether input column exist in TFRecord files
6 years ago
mindspore-ci-bot
9b8b699eb3
!1235 Auto remove CRLF of VOCDataset txt file
Merge pull request !1235 from xiefangqi/xfq_del_CRLF
6 years ago
xiefangqi
9e48e3daaa
del CR in VOC txt
6 years ago
hesham
e8ca243364
-Add DE_STRING
-replace switch'case by indexing
- Add test case
- Add constructors
- Add getItem string
- Fix bugs
- Add more tests
- Tensor iterator
- asNumpy
- TextFileDataset
- Tensor(Numpy)
- Super > 2D
- Add more test cases for GeneratorDataset
- Change StartAddr to GetBuffer and GetMutableNuffer
- Raise an error if batch is used with strings
Clean-up work
6 years ago
ms_yan
30739001c8
make sure validate column ok first
6 years ago
mindspore-ci-bot
c680cfbf27
!1157 dataset: add concat operation for dataset
Merge pull request !1157 from ms_yan/concat_dataset
6 years ago
mindspore-ci-bot
ab031ee9ea
!1126 VOCDataset support object detection function
Merge pull request !1126 from xiefangqi/voc_support_detection
6 years ago
xiefangqi
c937bad53f
minddata support voc
6 years ago
ms_yan
c0fa7b4b19
init commit of concat dataset
change to use __add__ operation instead ds.concat
6 years ago
mindspore-ci-bot
8f40f36c6c
!924 gpu queue support multi-inputs
Merge pull request !924 from chenweifeng/dataset
6 years ago