Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
|
|
2 years ago | |
|---|---|---|
| .. | ||
| Example_Picture | 2 years ago | |
| README.md | 2 years ago | |
| config.py | 2 years ago | |
| dataset.py | 2 years ago | |
| dataset_distributed.py | 2 years ago | |
| inference.py | 2 years ago | |
| lenet.py | 2 years ago | |
| read_imagenet.py | 2 years ago | |
| train.py | 2 years ago | |
| train_multi_card.py | 2 years ago | |
使用c2net包
#导入包
from c2net.context import prepare, upload_openi
#初始化导入数据集和预训练模型到容器内
c2net_context = prepare()
#获取代码路径,数据集路径,预训练模型路径,输出路径
code_path = c2net_context.code_path
dataset_path = c2net_context.dataset_path
pretrain_model_path = c2net_context.pretrain_model_path
output_path = c2net_context.output_path
#回传结果到openi
upload_output()
主要使用的方法有以下几个:
prepare 准备数据集,模型,输出路径
c2net.context.upload_output 将训练镜像的输出结果拷贝回启智平台
c2net.context.moxing_helper.obs_copy_file 通过mox拷贝文件
c2net.context.moxing_helper.obs_copy_folder 通过mox拷贝文件夹
请在代码中加入 args, unknown = parser.parse_known_args(),可忽略掉 --ckpt_url,--data_url, --multi_date_url等参数报错问题
使用缓存文件和local_rank%8作为判断,若第0卡拷贝完成,则其它卡不拷贝,并在第0卡拷贝数据完成后才执行之后的代码
if local_rank%8==0:
这里省略下载数据的代码...
f = open("/cache/download_input.txt", 'w')
f.close()
try:
if os.path.exists("/cache/download_input.txt"):
print("download_input succeed")
except Exception as e:
print("download_input failed")
while not os.path.exists("/cache/download_input.txt"):
time.sleep(1)