Huawei_Technology
/
mindspore
Not watched
1
0
Fork 0
Code
Releases 13 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
1027 Commits
11 Branches
1.3 GB
177 Download
 
 
 
 
 
 
Tree: e64c755ad6
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'e64c755ad6'
${ noResults }
mindspore/example/convert_to_mindrecord
jonwe 353b3b7944 optimize mindrecord writer performance 5 years ago
..
imagenet optimize mindrecord writer performance 5 years ago
template optimize mindrecord writer performance 5 years ago
README.md optimize mindrecord writer performance 5 years ago
run_imagenet.sh optimize mindrecord writer performance 5 years ago
run_template.sh optimize mindrecord writer performance 5 years ago
writer.py optimize mindrecord writer performance 5 years ago

README.md

MindRecord generating guidelines

Create work space

Assume the dataset name is 'xyz'

  • Create work space from template 
    cd ${your_mindspore_home}/example/convert_to_mindrecord
cp -r template xyz

Implement data generator

Edit dictionary data generator

  • Edit file 
    cd ${your_mindspore_home}/example/convert_to_mindrecord
vi xyz/mr_api.py

Two API, 'mindrecord_task_number' and 'mindrecord_dict_data', must be implemented

  • 'mindrecord_task_number()' returns number of tasks. Return 1 if data row is generated serially. Return N if generator can be split into N parallel-run tasks.
  • 'mindrecord_dict_data(task_id)' yields dictionary data row by row. 'task_id' is 0..N-1, if N is return value of mindrecord_task_number()

Tricky for parallel run

  • For imagenet, one directory can be a task.
  • For TFRecord with multiple files, each file can be a task.
  • For TFRecord with 1 file only, it could also be split into N tasks. Task_id=K means: data row is picked only if (count % N == K)

Run data generator

  • run python script 
    cd ${your_mindspore_home}/example/convert_to_mindrecord
python writer.py --mindrecord_script imagenet [...]
Delete task

Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.

cancel
confirm

MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.

C++ Python Text Unity3D Asset C other