yuze.zyz
605cd7f44a
[to #42322933 ] NLP 1030 Refactor
Features:
1. Refactor the directory structure of nlp models. All model files are placed into either the model folder or the task_model folder
2. Refactor all the comments to google style
3. Add detail comments to important tasks and nlp models, to list the description of the model, and its preprocessor&trainer
4. Model Exporting now supports a direct all to TorchModelExporter(no need to derive from it)
5. Refactor model save_pretrained method to support direct running(independent from trainer)
6. Remove the judgement of Model in the pipeline base class, to support outer register models running in our pipelines
7. Nlp trainer now has a NLPTrainingArguments class , user can pass arguments into the dataclass, and use it as a normal cfg_modify_fn, to simplify the operation of modify cfg.
8. Merge the BACKBONES and the MODELS, so user can get a backbone with the Model.from_pretrained call
9. Model.from_pretrained now support a task argument, so user can use a backbone and load it with a specific task class.
10. Support Preprocessor.from_pretrained method
11. Add standard return classes to important nlp tasks, so some of the pipelines and the models are independent now, the return values of the models will always be tensors, and the pipelines will take care of the conversion to numpy and the following stuffs.
12. Split the file of the nlp preprocessors, to make the dir structure more clear.
Bugs Fixing:
1. Fix a bug that lr_scheduler can be called earlier than the optimizer's step
2. Fix a bug that the direct call of Pipelines (not from pipeline(xxx)) throws error
3. Fix a bug that the trainer will not call the correct TaskDataset class
4. Fix a bug that the internal loading of dataset will throws error in the trainer class
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10490585
3 years ago
yuze.zyz
707cbef013
[to #42322933 ]Fix bug in daily UT
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10491891
3 years ago
zhangzhicheng.zzc
d721fabb34
[to #42322933 ]bert with sequence classification / token classification/ fill mask refactor
1.新增支持原始bert模型(非easynlp的 backbone prefix版本)
2.支持bert的在sequence classification/fill mask /token classification上的backbone head形式
3.统一了sequence classification几个任务的pipeline到一个类
4.fill mask 支持backbone head形式
5.token classification的几个子任务(ner,word seg, part of speech)的preprocessor 统一到了一起TokenClassificationPreprocessor
6. sequence classification的几个子任务(single classification, pair classification)的preprocessor 统一到了一起SequenceClassificationPreprocessor
7. 改动register中 cls的group_key 赋值位置,之前的group_key在多个decorators的情况下,会被覆盖,obj_cls的group_key信息不正确
8. 基于backbone head形式将 原本group_key和 module同名的情况尝试做调整,如下在modelscope/pipelines/nlp/sequence_classification_pipeline.py 中
原本
@PIPELINES.register_module(
Tasks.sentiment_classification, module_name=Pipelines.sentiment_classification)
改成
@PIPELINES.register_module(
Tasks.text_classification, module_name=Pipelines.sentiment_classification)
相应的configuration.json也有改动,这样的改动更符合任务和pipline(子任务)的关系。
8. 其他相应改动为支持上述功能
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10041463
3 years ago
Yingda Chen
e0ef60ca9b
[to #42322933 ] skip demo test by default
3 years ago
lingcai.wl
7a49fa1cc6
[to #44657982 ] add unittest for demo and demotest utils
unittest for demo service
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10006180
3 years ago
yingda.chen
45620dbc7f
[to #42322933 ]clean up test level
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9662182
* clean up test level
3 years ago
wenmeng.zwm
d55525bfb6
[to #43112771 ] requirements check and lazy import support
3 years ago
wenmeng.zwm
4814b198f0
[to #43112534 ] taskdataset refine and auto placement for data and model
* refine taskdataset interface
* add device placement for trainer
* add device placement for pipeline
* add config checker and fix model placement bug
* fix cycling import
* refactor model init for translation_pipeline
* cv pipelines support kwargs
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9463076
3 years ago
feiwu.yfw
2c3875c0e1
[to #43299989 ] Fix msdataset
* fix msdataset
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9436292
* fix msdataset
3 years ago
feiwu.yfw
5da470fd5d
[to #42791465 , #42779255 , #42777959 , #42757844 , #42756050 , #42746916 , #42743595 , #42791863 ] fix: fix msdataset
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9174075
* fix msdataset
3 years ago
yingda.chen
6702b29e21
[to #42794773 ]rename pydataset to msdataset
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9165402
3 years ago
Yingda Chen
b6e3fd80b0
Revert "[to #42794773 ] rename pydataset to msdataset"
This reverts commit c8e2e6de0e .
3 years ago
Yingda Chen
c8e2e6de0e
[to #42794773 ] rename pydataset to msdataset
3 years ago
yingda.chen
e7571a566f
[to #42322933 ] skip dataset test for now
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9155405
3 years ago
yingda.chen
1a0d4af55a
[to #42322933 ] test level check
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9143809
3 years ago
wenmeng.zwm
e288cf076e
[to #42362853 ] refactor pipeline and standardize module_name
* using get_model to validate hub path
* support reading pipeline info from configuration file
* add metainfo const
* update model type and pipeline type and fix UT
* relax requimrent for protobuf
* skip two dataset tests due to temporal failure
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9118154
3 years ago
mulin.lyh
76c6ff6329
[to #42675838 ]merge model hub code
合并model hub 代码
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9096493
3 years ago
feiwu.yfw
c7238a470b
[to #42670107 ]pydataset fetch data from datahub
* pydataset fetch data from datahub
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9060856
3 years ago
wenmeng.zwm
c59833c7ee
[to #42461396 ] feat: test_level support
* add test level support
* update develop doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9021354
3 years ago
yingda.chen
b31c86aa0e
[to #42409340 ] add hub specifier
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9005038
3 years ago
wenmeng.zwm
1f6b376599
[to #42373878 ] refactor maaslib to modelscope
1. refactor maaslib to modelscope
2. fix UT error
3. support pipeline which does not register default model
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8988388
3 years ago
wenmeng.zwm
dd00195814
[to #42362853 ] add default model support and fix circular import
1. add default model support
2. fix circular import
3. temporarily skip ofa and palm test which costs too much time
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8981076
3 years ago
yingda.chen
0d840d519c
[to #42339763 ] move pydataset into maas_lib
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8974892
3 years ago
yingda.chen
e3b8ec3bf1
[to #42339559 ] support multiple models
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8972440
* [to #42339559 ] support multiple models
3 years ago
feiwu.yfw
235880f300
[to #42339763 ] merge pydataset into maas-lib
* merge pydataset to the repo
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8955999
3 years ago
yingda.chen
d6868ddffe
[to #42323743 ] retain local cached model files by default
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8963687
3 years ago
yingda.chen
e075ad2245
[to #42322515 ]support plain pipeline for bert
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8945177
* support plain pipeline for bert
3 years ago
yingda.chen
f8eb699f7f
refine tests and examples
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8898823
3 years ago
wenmeng.zwm
1d01a78c2b
fix: UT error
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8899458
* fix: UT error
3 years ago
yingda.chen
5995cc4607
add PyDataset support
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8868644
3 years ago
wenmeng.zwm
25a2028b54
[to #41401401 ] modelhub and Trainer support
* add trainer interface
* add trainer script
* add model init support for pipelineadd pipeline tutorial and fix bugs
* add text classification evaluation to maas lib
* add quickstart and prepare env doc
* relax requirements for torch and sentencepiece
* merge release/0.1 and fix conflict
* modelhub support for model and pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8868339
3 years ago
wenmeng.zwm
cb416edc2a
[to #41669377 ] add pipeline tutorial and fix bugs
1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8810524
3 years ago
wenmeng.zwm
5e469008fd
[to #41401401 ] add preprocessor, model and pipeline
* add preprocessor module
* add model base and builder
* update task constant
* add load image preprocessor and its dependency
* add pipeline interface and UT covered
* support default pipeline for task
* add image matting pipeline
* refine nlp tokenize interface
* add nlp pipeline
* fix UT failed
* add test for Compose
Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8769235
* add preprocessor module
* add test for Compose
* fix citest error
* fix abs class error
* add model base and builder
* update task constant
* add load image preprocessor and its dependency
* add pipeline interface and UT covered
* support default pipeline for task
* refine models and pipeline interface
* add pipeline folder structure
* add image matting pipeline
* refine nlp tokenize interface
* add nlp pipeline
1.add preprossor model pipeline for nlp text classification
2. add corresponding test
Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8757371
* new nlp pipeline
* format pre-commit code
* update easynlp pipeline
* update model_name for easynlp pipeline; add test for maas_lib/utils/typeassert.py
* update test_typeassert.py
* refactor code
1. rename typeassert to type_assert
2. use lazy import to make easynlp dependency optional
3. refine image matting UT
* fix linter test failed
* update requirements.txt
* fix UT failed
* fix citest script to update requirements
3 years ago