mulin.lyh
4429991646
Merge branch dev/msdataset_event_tracking into master
Title: [to #42322933 ] add event tracking
1. add event tracking for dataset downloading pv/uv
2. change datasets version: <=2.5.2
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10593016
3 years ago
班扬
e45ab2c32d
add event tracking
3 years ago
班扬
63a08e7be6
add event tracking
3 years ago
干劲
2759d538bb
fix ut level for unifold
3 years ago
liugao.lg
40b6770956
[to #42322933 ]fix ocr prepreocess & conflict
修复ocr预处理逻辑不一致问题
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10581697
3 years ago
zhangzhicheng.zzc
06abae4dc6
[to #42322933 ]add token-cls test cases and bug fix
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10585502
3 years ago
zhangzhicheng.zzc
0d3b7b0df2
[to #42322933 ]fix bugs relate to token cls
1.修复token classification preprocessor finetune结果错误问题
2.修复word segmentation output 无用属性
3. 修复nlp preprocessor传use_fast错误
4. 修复torch model exporter bug
5. 修复文档撰写过程中发现trainer相关bug
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10573269
3 years ago
shouzhou.bx
e72988c2ba
add face detection to face_2d_keypoints_pipeline
3 years ago
yichang.zyc
e2d35fbb14
[to #42322933 ]clip支持finetune
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10572842
3 years ago
Yingda Chen
9f7b8b86a3
[to #42322933 ] disble 2dkeypoints training since face_2d_keypoints_dataset is set to be private
3 years ago
Yingda Chen
902019c2e0
[to #42322933 ] disble vgg19_fer
3 years ago
Yingda Chen
29448c0f57
[to #42322933 ] disble vit
3 years ago
mulin.lyh
3791ee7ad2
[to #45821936 ]fix: fix block user specify revision after release_datetime
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10572162
3 years ago
yuze.zyz
4b7e8e89aa
[to #42322933 ] Fix some bugs when downgrade the version of some dependencies
1. Fix bug in model exporting
2. Skip some long trainings in test level 2
3. Refine some comments
4. Fix a bug that mode is not correct when saving checkpoints
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10564716
3 years ago
Yufeng
261c04b8b5
add Mglm ( #5 )
* mglm init
* add mglm requirements
Co-authored-by: Yufeng <zhuyufeng@gmail.com>
Co-authored-by: wenmeng.zwm <wenmeng.zwm@alibaba-inc.com>
3 years ago
Yingda Chen
46cfa177aa
[to #42322933 ]skip timeconsuming test
3 years ago
xianzhe.xxz
88e8d4291a
[to #42322933 ]"fix: set the eps and momentum of BN consistent with training"
To keep consistent between training and evaluation, change the eps and momentum of BN.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10554451
3 years ago
menrui.mr
c7b0787049
修复初始化过程参数未生效问题
此前文生图模型没有加载configuration.json中的参数 影响默认配置
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10558026
3 years ago
hemu.zp
fa415d8720
[to #42322933 ] Fix bug for bloom and gpt_neo
1. 修复 bloom 和 gpt_neo 模型更新 transformers 4.23 后后处理报错的问题
2. 统一使用 ModelOutput 作为模型输出
3. gpt_neo checkpoint 已上线,修改 ut 为 level2
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10553103
3 years ago
xingjun.wxj
78f29cf999
[to #42322933 ] Add delete datasets files and upload mode.
1. Add : MsDataset.delete() , support delete dataset file or dir.
2. Add: upload mode, MsDataset.upload(xx, upload_mode=UploadMode.FORCE_UPLOAD), or MsDataset.upload(xx, upload_mode=UploadMode.APPEND_UPLOAD)
if upload_mode = UploadMode.APPEND_UPLOAD, then skip object in case of this object exists.
3. Add: support reload sts token automatically to avoid expire. (current expiration: 24h)
4. Fix: add cookies in api.py for downloading private datasets.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10524449
3 years ago
Yingda Chen
374fd3090e
[to #42322933 ]skip referring video tests since model is private
3 years ago
yuze.zyz
212cf53318
[to #42322933 ] Fix some bugs
1. Add F1 score to sequence classification metric
2. Fix a bug that the evaluate method in trainer does not support a pure pytorch_model.bin
3. Fix a bug in evaluation of veco trainer
4. Add some tips if lr_scheduler in the trainer needs a higher version torch
5. Add some comments
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10532230
3 years ago
shuying.shu
ddcb57440d
[to #42322933 ]add fine-tune code for referring video object segmentation
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10539423
3 years ago
mulin.lyh
3b75623be4
[to #45773874 ]fix: get_model revision=None bug, and hub case occasionally delete test model failed
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10549680
3 years ago
eniac.xcw
8886c3c1ae
[to #42322933 ]fine tune team on caltech-101
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10525413
3 years ago
yingda.chen
de708dd518
add basic remap column wrapper
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10539917
* add basic remap column wrapper
3 years ago
hemu.zp
69104c0f8a
[to #42322933 ] Refactor text generation model outputs and fix some bugs
1. 将 single_gpu_test 与 multi_gpu_test 中的 model.forward 部分分离为 EpochBasedTrainer 中的 evaluation_step,为部分 evaluation 阶段不调用 forward 的模型提供更好的灵活性
2. 重构代码将文本生成模型 Model 层的输入输出统一为 Tensor,Tensor 到 str 的 decode 过程移动到 pipeline 中完成
3. pipeline 后处理添加对中文和中文标点与英文混杂时空格的处理,使 decode 后中英文混杂输出正确
4. 添加 TextGenerationTrainer 修复了部分模型 evaluation 过程 forward 输出单个 token 计算 metrics 的问题
5. 修复了 rouge 无法接收空字符串的问题
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10473768
3 years ago
liugao.lg
0605376135
[to #42322933 ]add ofa finetune
新增ofa的finetune能力
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10540701
3 years ago
hemu.zp
d0f8547e7e
[to #42322933 ] Fix gpt3 loading checkpoint after finetuning.
1. 修复GPT-3模型无法加载finetune保存的checkpoint的问题
2. 为GPT-3诗词生成模型添加 ut
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10537209
3 years ago
jiaqi.sjq
7b84adc914
[to #42322933 ]Fix remove files in local model not take effect to remote repo after push_model
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10533214
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10533214
3 years ago
hemu.zp
e4a0e046f9
[to #42322933 ] Add ut for mplug and bloom
为新上线的 langboat/bloom-1b4-zh,damo/mplug_visual-question-answering_coco_base_zh,damo/mplug_image-captioning_coco_base_zh 三个模型添加 ut,test_level 设置为 2
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10524221
3 years ago
caorongyu.cry
3b8fb92c13
[to #42322933 ] debug header ids and header names
修复header_ids和header_names命名反了的问题
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10516557
3 years ago
ran.zhou
13f7e9ceca
[to #42322933 ]SEA multilingual NLP (NER & word segmentation)
添加东南亚小语种NLP支持,包括:
1. 针对泰语,越南语NER的预处理
2. 基于XLMR-CRF架构的分词模型和pipeline
3. 针对泰语分词的预处理
添加了相应pipeline的unittest
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492404
3 years ago
mulin.lyh
384377b8f5
* [to #45486649 ]feat: modelscope model version use model repo tag, unsupport branch or commit it, client user-agent header unified
3 years ago
jiaqi.sjq
5190c7de11
[to #41669377 ] tts using default master revision model in UT
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10526747
3 years ago
zhangyanzhao.zyz
781fe49d63
[to #42322933 ]修正finetune text ranking bugs
之前的finetune代码当dataset最后长度不足制定batch size时会出错,现已修正
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10524066
3 years ago
yuanzheng.yuanzhen
bab54bbce8
[to #42322933 ]support uni fold
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10481410
3 years ago
yuze.zyz
c2da44b371
[to #42322933 ] remove dev model inference and fix some bugs
1. Change structbert dev revision to master revision
2. Fix bug: Sample code failed because the updating of model configuration
3. Fix bug: Continue training regression failed
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10519992
3 years ago
zhangzhicheng.zzc
e1ab73b7d8
[to #42322933 ]support type str for for zero-shot labels' input
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10506320
3 years ago
hemu.zp
ffd834fc25
[to #42322933 ] Add bloom model
添加 bloom 模型
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10509187
3 years ago
yichang.zyc
6ddafb3218
[to #42322933 ]caption finetune done, add belu metric
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10318299
3 years ago
yuze.zyz
605cd7f44a
[to #42322933 ] NLP 1030 Refactor
Features:
1. Refactor the directory structure of nlp models. All model files are placed into either the model folder or the task_model folder
2. Refactor all the comments to google style
3. Add detail comments to important tasks and nlp models, to list the description of the model, and its preprocessor&trainer
4. Model Exporting now supports a direct all to TorchModelExporter(no need to derive from it)
5. Refactor model save_pretrained method to support direct running(independent from trainer)
6. Remove the judgement of Model in the pipeline base class, to support outer register models running in our pipelines
7. Nlp trainer now has a NLPTrainingArguments class , user can pass arguments into the dataclass, and use it as a normal cfg_modify_fn, to simplify the operation of modify cfg.
8. Merge the BACKBONES and the MODELS, so user can get a backbone with the Model.from_pretrained call
9. Model.from_pretrained now support a task argument, so user can use a backbone and load it with a specific task class.
10. Support Preprocessor.from_pretrained method
11. Add standard return classes to important nlp tasks, so some of the pipelines and the models are independent now, the return values of the models will always be tensors, and the pipelines will take care of the conversion to numpy and the following stuffs.
12. Split the file of the nlp preprocessors, to make the dir structure more clear.
Bugs Fixing:
1. Fix a bug that lr_scheduler can be called earlier than the optimizer's step
2. Fix a bug that the direct call of Pipelines (not from pipeline(xxx)) throws error
3. Fix a bug that the trainer will not call the correct TaskDataset class
4. Fix a bug that the internal loading of dataset will throws error in the trainer class
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10490585
3 years ago
siyang.ssy
6d51f44dc7
[to #42322933 ]fix input type for video embeding
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10506601
3 years ago
bin.xue
525fa3ea89
[to #42322933 ]test: use 'master' branch in training test
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10509580
3 years ago
huizheng.hz
a1738690c9
[to #42322933 ]test_image_denoise_trainer
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10465138
3 years ago
caorongyu.cry
6178f46910
[to #42322933 ] add ut for multi threads
1. 修复multi thread引起的问题
2. 增加multi thread的unittest
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10502008
3 years ago
yingda.chen
de7b6a06e9
[to #42322933 ] remove revision usage for face detection
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10507910
* [to #42322933 ] remove revision usage for face detection
3 years ago
zhangyanzhao.zyz
c4dbb69d65
[to #42322933 ]增加对text-ranking任务中文模型的单元测试,以方便得到官方模型打标。
增加对text-ranking任务中文模型的单元测试,以方便得到官方模型打标。
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492754
3 years ago
yichang.zyc
35c612a642
[to #42322933 ]去除clip ut中的dev revision
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10507748
* remove clip ut dev revision
3 years ago
ashui.cbh
e223c1b008
[to #42322933 ]merge master after demo service support
demo service 对接,修改输入接口为可调用的方式
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10502169
3 years ago