moyu.wk
|
42896de709
|
[to #42322933]add pipelines for APE\QE\Domain Classifications
Add pipelines for the following new models:
- [Multilingual Quality Estimation](https://modelscope.cn/models/damo/nlp_translation_quality_estimation_multilingual/summary)
- [Automatic Post-Editing (En-De)](https://modelscope.cn/models/damo/nlp_automatic_post_editing_for_translation_en2de/summary)
- [Domain classification (Zh)](https://modelscope.cn/models/damo/nlp_domain_classification_chinese/summary)
- [Style classification (Zh)](https://modelscope.cn/models/damo/nlp_style_classification_chinese/summary)
- [Style classification (En)](https://modelscope.cn/models/damo/nlp_style_classification_english/summary)
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10315370
|
3 years ago |
zhangzhicheng.zzc
|
c35f8cb42b
|
[to #42322933] remove deepspeed and fariseq from requirments
|
3 years ago |
suluyan.sly
|
904374d329
|
[to #42322933] feat: plug inference
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9931748
|
3 years ago |
xiangpeng.wxp
|
054151d92f
|
[to #42322933]nlp_translation_preprocess
* nlp translation preprocess branch
* pull the latest master
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9920445
|
3 years ago |
piaoyu.lxy
|
2dc3286524
|
[to #42322933] add conversational_text_to_sql pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9580066
|
3 years ago |
wenmeng.zwm
|
638cdc632e
|
[to #43115513] update quick_start doc and add pai-easynlp back
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9655518
* update quick_start doc and add pai-easynlp back
* update doc with sndfile
|
3 years ago |
wenmeng.zwm
|
49192f94be
|
[to #43726282] fix bugs and refine docs
1. remove pai-easynlp temporarily due to its hard dependency on scipy==1.5.4
2. fix sentiment classification output
3. update quickstart and trainer doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9646399
|
3 years ago |
zhangzhicheng.zzc
|
9d0b38b4e4
|
[to #42322933] lazy load on trainer
|
3 years ago |
yuze.zyz
|
21fa71baf0
|
[to #42322933] add/refactor nlp models source code and finetune
1. add sbert,veco,palm,space source code
2. support sbert sequence classification, token classification finetune
3. support veco sequence classification finetune
4. support palm nlg finetune
evaluation result: https://sheet.alibaba-inc.com/#/sheet/f7fdcc7f22bd5105 sheet:Maas
5. add ut for finetunes
6. add veco's taskdataset processor
7. add a common trainer for nlp, and a specific trainer for veco
8. merge some duplicate codes of models, preprocessors, pipelines
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9574105
* add basic class of hook&metrics
* pre-commit passed
* change some comments
* pre commit passed
* 1. remove accuracy's groups 2. remove useless hooks 3. simplify priorities
* pre-commit passed
* fix a comment
* Merge branch 'master' into finetune_hooks_metrics
# Conflicts:
# modelscope/metainfo.py
* pre-commit passed
* add basic class of hook&metrics
* pre-commit passed
* change some comments
* pre commit passed
* 1. remove accuracy's groups 2. remove useless hooks 3. simplify priorities
* pre-commit passed
* fix a comment
* Merge branch 'feat/finetune' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into feat/finetune
* mv hooks related to modelscope/trainers/hooks
* mv priority back
* add torch mdoel base and test
* update hooks, trainer, import_util
* add torch epoch based trainer and dis utils
* add hooks
* fix warmup
* format code stype and fix warmup and add warmup unittest
* fix impls
* pre-commit check passed
* update hook and add EpochBasedTrainer
* add trainer unittest
* Merge branch 'feat/add_hooks' into feat/add_task
# Conflicts:
# modelscope/models/base_torch.py
# modelscope/trainers/hooks/hook.py
# modelscope/trainers/trainer.py
* update unittest name
* rewrite taskdataset to trainer
* fix trainer and add unittest
* add unittest
* code: run to forward
* run through... but ugly code
* arrange some cls
* fix some errs
* revert some mistakes
* init check in
* Merge branch 'feat/add_hooks' into feat/add_task
# Conflicts:
# modelscope/trainers/trainer.py
* test with bigger epoch and size
* add the default metrics class
* move build metrics code to a method
* merge add_task
* merge origin add_task
* add device initialization
* remove preprocessor arg for bool
* add task models
* move metric collect logic to metrics class
* pre-commit passed
* fix cr comments
* precommit passed
* add task models
* Merge remote-tracking branch 'origin/feat/add_task' into feat/backbone_head
* add comment
* change comment formats.
* fix comments
* fix ut bug
* fix comments
* add wrapper check
* fix comments
* pre commit passed
* fix cr comments
* solve a loop import problem
* fix ut bug
* fix ut errors
* change dummydataset to msdataset
* precommit passed
* merge add task
* backbone-head is build, model is not correctly loaded
* model load states matched
* result matched
* lint
* add veco/palm_v2 code
* merge master
* merge master success running
* add repr model name level
* Merge branch 'feat/veco_palm' into feat/finetune_sbert_veco
* model test for training
* add token-classification metric add formal ut
* fix running bug
* finetune and pipeline are working with backbone-head
* add nli
* add missing code
* finetune and pipeline are working with backbone-head
* Merge branch 'feat/backbone_head' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib into feat/backbone_head
* add a test repo for pr
* remove merge conflicted file
* remove merge conflicted file 1
* lint check
* import error
* none type bug fix
* forward input unpacking or dict bug
* move head into models, add build_backbone with registry, no base method
* merge master
* feat: 1. add interleave dataset method 2. support multiple dataset in trainer.build_dataset 3. support 3 sub tasks in sequence_classification task
* unfinished
* update the task model structure in NLP field
* merge master
* update by comments
* keep the default model id as current on production
* unfinished
* unfinished
* veco can run
* Merge remote-tracking branch 'origin/master' into feat/backbone_head
* add taskmodel for module management
* remove forward_input_is_dict
* unfinished
* token classification started
* update base model structure
* move space to backbone
* remove 'type' in build_from_cfg method
* test update
* bug fix
* on tesing, mess code
* Merge branch 'feat/backbone_head' into feat/refactor_nlp_730
# Conflicts:
# modelscope/metrics/builder.py
# modelscope/models/__init__.py
# modelscope/models/nlp/__init__.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
# requirements/multi-modal.txt
* add missing merge
* add sofa source code
* refactor
* add veco task dataset
* add veco task dataset
* pre-commit passed
* fix bug of log
* add some features
* merge master
* bug fix
* refine nlp models
* fix the training error
* unfinished
* refactor pipeline
* Merge branch 'feat/backbone_head' into feat/refactor_nlp_730
# Conflicts:
# modelscope/metrics/builder.py
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/backbones/structbert/modeling_sbert.py
# modelscope/models/nlp/palm_v2/palm_for_text_generation.py
# modelscope/preprocessors/base.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
* Merge commit 'ab04ceafc5453ce7daa9aa09e37a55f703072a10' into feat/refactor_nlp_730
# Conflicts:
# modelscope/metainfo.py
# modelscope/metrics/builder.py
# modelscope/models/__init__.py
# modelscope/models/base/base_torch_model.py
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/backbones/space/model/intent_unified_transformer.py
# modelscope/models/nlp/backbones/space/model/model_base.py
# modelscope/models/nlp/palm_v2/palm_for_text_generation.py
# modelscope/models/nlp/sbert_for_sequence_classification.py
# modelscope/models/nlp/sequence_classification.py
# modelscope/models/nlp/space/__init__.py
# modelscope/models/nlp/space_for_dialog_intent_prediction.py
# modelscope/models/nlp/space_for_dialog_modeling.py
# modelscope/models/nlp/space_for_dialog_state_tracking.py
# modelscope/models/nlp/task_model.py
# modelscope/pipelines/nlp/sentiment_classification_pipeline.py
# modelscope/preprocessors/base.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
* revert changes
* unify sentnece classification postprocess
* revert some changes, move some model files
* pipeline first case run through
* ws pipeline passed
* Merge branch 'feat/refactor_nlp_730' into feat/finetune_sbert_veco
* finetune
* revert code
* revert some code
* ws finetune started, only the accuracy is weird
* Merge branch 'feat/veco_taskdataset' into feat/finetune_sbert_veco
# Conflicts:
# modelscope/task_datasets/veco_dataset.py
# tests/taskdataset/test_veco_dataset.py
* veco+nli finetune started
* Merge branch 'master' into feat/finetune_sbert_veco
# Conflicts:
# modelscope/models/nlp/sbert_for_sequence_classification.py
# modelscope/models/nlp/sbert_for_token_classification.py
# modelscope/models/nlp/sbert_for_zero_shot_classification.py
# modelscope/models/nlp/space/space_for_dialog_intent_prediction.py
# modelscope/models/nlp/space/space_for_dialog_modeling.py
# modelscope/trainers/trainer.py
* add trainer for nlp
* trainer: dataset params passed into preprocessor
* test passed by nlptrainer
* fix some bugs
* fix some bugs
* add backbone/head subclass
* fix regression bugs
* fix bug in token-cls finetune
* support cfg modification
* fix bug
* fix bug
* update requirements
* add some comments and fix some t
* add some comments and revert a argument
* split to two test files
* revert code
* fixbug in precessor
(cherry picked from commit 7a648d096e)
* fix ut bug
* support sbert models
* unfinished
* Merge branch 'feat/finetune_sbert_veco' into sly_tmp_veco_finetune
# Conflicts:
# tests/trainers/test_finetune_sequence_classification.py
* fixbug in veco
* fix bug
* fixbug
* correct running params
* remove useless files
* add palm finetuning with cnn_dailymail dataset
* copy space model from sofa
* Merge branch 'feat/finetune_sbert_veco' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into feat/finetune_sbert_veco
* Merge branch 'master' into feat/finetune_sbert_veco
# Conflicts:
# modelscope/metrics/__init__.py
# modelscope/models/__init__.py
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/backbones/__init__.py
# modelscope/models/nlp/backbones/structbert/modeling_sbert.py
# modelscope/models/nlp/heads/__init__.py
# modelscope/models/nlp/masked_language.py
# modelscope/models/nlp/palm_v2/palm_for_text_generation.py
# modelscope/models/nlp/sbert_for_nli.py
# modelscope/models/nlp/sbert_for_sentence_similarity.py
# modelscope/models/nlp/sbert_for_sentiment_classification.py
# modelscope/models/nlp/sbert_for_sequence_classification.py
# modelscope/models/nlp/sbert_for_token_classification.py
# modelscope/models/nlp/sbert_for_zero_shot_classification.py
# modelscope/models/nlp/sequence_classification.py
# modelscope/models/nlp/space/space_for_dialog_intent_prediction.py
# modelscope/models/nlp/space/space_for_dialog_modeling.py
# modelscope/models/nlp/space/space_for_dialog_state_tracking.py
# modelscope/models/nlp/structbert/adv_utils.py
# modelscope/models/nlp/structbert/configuration_sbert.py
# modelscope/models/nlp/task_models/task_model.py
# modelscope/pipelines/__init__.py
# modelscope/pipelines/nlp/__init__.py
# modelscope/pipelines/nlp/fill_mask_pipeline.py
# modelscope/pipelines/nlp/named_entity_recognition_pipeline.py
# modelscope/pipelines/nlp/nli_pipeline.py
# modelscope/pipelines/nlp/sentence_similarity_pipeline.py
# modelscope/pipelines/nlp/sentiment_classification_pipeline.py
# modelscope/pipelines/nlp/text_generation_pipeline.py
# modelscope/pipelines/nlp/word_segmentation_pipeline.py
# modelscope/pipelines/nlp/zero_shot_classification_pipeline.py
# modelscope/preprocessors/nlp.py
# modelscope/task_datasets/__init__.py
# modelscope/trainers/trainer.py
# modelscope/trainers/utils/inference.py
# modelscope/utils/file_utils.py
# requirements/nlp.txt
# tests/pipelines/test_nli.py
# tests/pipelines/test_sentence_similarity.py
# tests/pipelines/test_sentiment_classification.py
* fix imports
* mark backbone in their own modeling
* pre-commit check passed
* pre-commit passed, remove roberta model
* fix a bug in ast import
* skip all finetune uts
* fix bugs
* pre-commit passed
* bug fixed
* bug fixed
* bug fixed
* bug fixed
* fix ut bug
* fix bug
* fix ut bug
* fix bug
* fix bug
* fixbugs
* fixbug
* revert veco
* revert veco because of core dump
* fix palm bug
* revert veco
* revert mistaken code
* add a test print
* pre-commit check
* test exception
* add test code
* for test
* fix bug and test
* remove test code
* remove useless file
* 1. fix some bugs 2. add backbone ut
* Merge branch 'master' into feat/finetune_refactor_730
# Conflicts:
# modelscope/metainfo.py
# modelscope/metrics/sequence_classification_metric.py
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/task_models/task_model.py
# modelscope/preprocessors/__init__.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
# modelscope/trainers/utils/inference.py
# modelscope/utils/file_utils.py
# tests/trainers/test_trainer_with_nlp.py
* pre-commit passed
* revert files
* increase test level
* unregister models
* fix bugs
* fix cr comments
* fix bug in backbone-head
* add sbert backbone
* fix bug
* add test for token-cls-metric
* pre-commit passed
* fix ut comments
* revert normal tokenizer to fast tokenizer
* Merge branch 'master' into feat/finetune_refactor_730
# Conflicts:
# modelscope/models/nlp/__init__.py
# modelscope/models/nlp/backbones/__init__.py
# modelscope/models/nlp/backbones/structbert/__init__.py
# modelscope/models/nlp/masked_language.py
# modelscope/models/nlp/palm_v2/palm_for_text_generation.py
# modelscope/models/nlp/sbert_for_sequence_classification.py
# modelscope/models/nlp/sbert_for_token_classification.py
# modelscope/models/nlp/sbert_for_zero_shot_classification.py
# modelscope/pipelines/nlp/text_generation_pipeline.py
# modelscope/preprocessors/nlp.py
# modelscope/trainers/trainer.py
# modelscope/trainers/utils/inference.py
* fix merge bugs
* pre commit passed
* fix bug
* fix bug
* fix bug
* fix bug from master
* add print
* fix ut bug
* fix bug
* Merge branch 'master' into feat/finetune_refactor_730
* skip task model test
|
3 years ago |
klayzhang.zb
|
ca4b5b2565
|
[to #42322933][NLP] Add text error correction task
NLP新增文本纠错任务
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9540716
|
3 years ago |
mulin.lyh
|
a7ffa7f6ce
|
[to #43588165]fix: nlp case github connection broken
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9528871
* [to #43588165]fix: nlp case github connection broken
|
3 years ago |
hemu.zp
|
ab04ceafc5
|
[to #42322933] set rouge_score version
2022/7/22 更新的 rouge_socre 0.0.7 版本存在 import 报错的问题,暂时将 rouge_score 库版本设置为 <= 0.0.4
|
3 years ago |
yingda.chen
|
dbb450932d
|
[to #42322933] clean requirment
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9458816
|
3 years ago |
hemu.zp
|
0b7b964226
|
[to #42322933] Add palm finetuning
Palm 模型支持 finetuning
|
3 years ago |
mulin.lyh
|
69047b99ae
|
[to #43387011]feat: ci test to new host and running in docker
ci 测试迁移新的机器,并且在容器中运行,减小互相干扰的可能
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9427096
* add docker ci script
|
3 years ago |
ly119399
|
40b1a9566d
|
[to #42322933] remove dependency en_core_web_sm from nlp.txt
将nlp.txt中的en_core_web_sm依赖删除,在代码中下载响应的依赖
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9432253
|
3 years ago |
wenmeng.zwm
|
e62cd756df
|
[to #42322933] relax requirements
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9407594
|
3 years ago |
智丞
|
e5f99bc6d0
|
update requirements/nlp
|
3 years ago |
智丞
|
54122b2dab
|
merge with master
|
3 years ago |
zhangzhicheng.zzc
|
5b98cc1513
|
[to #42322933]新增:nli,sentiment_classification,dialog_intent,dialog_modeling
添加了,nli,sentiment_classification, dialog_intent, dialog_modeling几个pipeline。同时加入了nlp里面sequence classification一些简单的抽象。
去掉了zero_shot_classification
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9159089
|
3 years ago |
wenmeng.zwm
|
8e51a073a6
|
[to #42966122] requirements enchanment and self-host repo support
* add self-hosted repo:
* add extra requirements for different field and reduce necessary requirements
* update docker file with so required by audio
* add requirements checker which will be used later when implement lazy import
* remove repeated requirements and replace opencv-python-headless with opencv-python
example usage:
```shell
pip install model_scope[all] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[cv] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[nlp] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[audio] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[multi-modal] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9211383
|
3 years ago |
ly119399
|
90c5e18183
|
update requirement
|
3 years ago |
hemu.zp
|
a03106172f
|
[to #42322933] Update sofa version 1.0.4.2
更新 sofa 到 1.0.4.2 版本,解决 palm 模型无法并行的 bug
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9217921
|
3 years ago |
ly119399
|
4cfc4b43a0
|
merge feat/nlp
|
3 years ago |
hemu.zp
|
fabea5604e
|
[to #42322933] Add MPLUG model
添加 MPLUG 模型的 visual question answering 任务 pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9182119
|
3 years ago |
ly119399
|
80461600f7
|
merge nlp
|
3 years ago |
suluyan.sly
|
0286dd45cc
|
[to #42322933] Add nlp-structbert/veco-fill-mask-pipeline to maas lib
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9069107
|
3 years ago |
智丞
|
4dfab005ca
|
merge with space
|
3 years ago |
ly119399
|
c6cf0d20c5
|
add dep
|
3 years ago |
ly119399
|
b812cb78c9
|
add dep
|
3 years ago |
hemu.zp
|
99fb503695
|
[to #42322933] Add Palm2.0 model.
接入支持中英文的 Palm2.0 模型,复用 text-generation-pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9066550
|
3 years ago |
suluyan
|
3aa1a70ac8
|
add tests
|
3 years ago |
hemu.zp
|
8a76f40754
|
[to #42322933]Add text-generation-pipeline with Palm model.
将 Palm 中文模型接入 MaaS,添加了文本生成 pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8934393
* add text_generation model and pipeline
* fix bug
* fix bug
* add TextGenerator in pipeline
* fix bug
* update checkpoint and test inputs
* remove magic number..
* fix bug
* adjust code with AutoModel
* clear comments and tidy up the code
* move model.eval() into generator
* update master interface and lint code
* replace 'palm-text-generation' with 'palm'
* add text_generation model and pipeline
* fix bug
* fix bug
* add TextGenerator in pipeline
* fix bug
* fix conflict of pipeline.txt
* remove magic number..
* fix bug
* adjust code with AutoModel
* clear comments and tidy up the code
* move model.eval() into generator
* fix conflict
* replace 'palm-text-generation' with 'palm'
* fix conflict
* add test_run_modelhub
* update sofa version
* modify sofa version
* add test_run_with_model_name
* fix bug
|
3 years ago |