Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10539423

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10549680

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10548014

    * fix image_open func

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10525413

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10539917

    * add basic remap column wrapper

1. 将 single_gpu_test 与 multi_gpu_test 中的 model.forward 部分分离为 EpochBasedTrainer 中的 evaluation_step，为部分 evaluation 阶段不调用 forward 的模型提供更好的灵活性
2. 重构代码将文本生成模型 Model 层的输入输出统一为 Tensor，Tensor 到 str 的 decode 过程移动到 pipeline 中完成
3. pipeline 后处理添加对中文和中文标点与英文混杂时空格的处理，使 decode 后中英文混杂输出正确
4. 添加 TextGenerationTrainer 修复了部分模型 evaluation 过程 forward 输出单个 token 计算 metrics 的问题
5. 修复了 rouge 无法接收空字符串的问题
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10473768

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10536603

新增ofa的finetune能力
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10540701

1. 修复GPT-3模型无法加载finetune保存的checkpoint的问题
2. 为GPT-3诗词生成模型添加 ut
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10537209

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10533214
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10533214

为新上线的 langboat/bloom-1b4-zh，damo/mplug_visual-question-answering_coco_base_zh，damo/mplug_image-captioning_coco_base_zh 三个模型添加 ut，test_level 设置为 2
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10524221

多语言faq，Tokenizer新增类型判别
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10530690

修复header_ids和header_names命名反了的问题
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10516557

添加东南亚小语种NLP支持，包括：
1. 针对泰语，越南语NER的预处理
2. 基于XLMR-CRF架构的分词模型和pipeline
3. 针对泰语分词的预处理

添加了相应pipeline的unittest
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492404

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10526747

之前的finetune代码当dataset最后长度不足制定batch size时会出错，现已修正
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10524066

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10481410

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10525516

修改了I/O的代码，以支持modelscope的demo services
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10518318

1. Change structbert dev revision to master revision
2. Fix bug:  Sample code failed because the updating of model configuration
3. Fix bug: Continue training regression failed
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10519992

在demo service场景，同时调用同一个视频文件，会导致ffmpeg处理同名视频的冲突。通过uuid生成唯一的文件名解决这个冲突。
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10518178

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10521235

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10506320

添加 bloom 模型
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10509187

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10318299

Features:
1. Refactor the directory structure of nlp models. All model files are placed into either the model folder or the task_model folder
2. Refactor all the comments to google style
3. Add detail comments to important tasks and nlp models, to list the description of the model, and its preprocessor&trainer
4. Model Exporting now supports a direct all to TorchModelExporter(no need to derive from it)
5. Refactor model save_pretrained method to support direct running(independent from trainer)
6. Remove the judgement of Model in the pipeline base class, to support outer register models running in our pipelines
7. Nlp trainer now has a NLPTrainingArguments class , user can pass arguments into the dataclass, and use it as a normal cfg_modify_fn, to simplify the operation of modify cfg.
8. Merge the BACKBONES and the MODELS, so user can get a backbone with the Model.from_pretrained call
9. Model.from_pretrained now support a task argument, so user can use a backbone and load it with a specific task class.
10. Support Preprocessor.from_pretrained method
11. Add standard return classes to important nlp tasks, so some of the pipelines and the models are independent now, the return values of the models will always be tensors, and the pipelines will take care of the conversion to numpy and the following stuffs.
12. Split the file of the nlp preprocessors, to make the dir structure more clear.

Bugs Fixing:
1. Fix a bug that lr_scheduler can be called earlier than the optimizer's step
2. Fix a bug that the direct call of Pipelines (not from pipeline(xxx)) throws error
3. Fix a bug that the trainer will not call the correct TaskDataset class
4. Fix a bug that the internal loading of dataset will throws error in the trainer class
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10490585

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10506601

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10509580

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10465138

1. 修复multi thread引起的问题
2. 增加multi thread的unittest
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10502008

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10507910

    * [to #42322933] remove revision usage for face detection

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10501307

增加对text-ranking任务中文模型的单元测试，以方便得到官方模型打标。
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492754

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10507748

    * remove clip ut dev revision

demo service 对接，修改输入接口为可调用的方式
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10502169

服务端文档链接(可能需要登录)：
https://test.modelscope.cn/api/v1/deployer/docs
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10478609

适配demoservice，增加视频时间戳输出，每一个结果对应一个时间戳
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10496879

1. 合并star和star3框架
2. 修改star和star3的model type
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492793

* fix output format
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10491722

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10491994

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10491891

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492024

1. 增加传入table_id
2. 将result和table的结构统一
3. 默认开启is_use_sqlite
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492027

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10489981

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10491024

修复cpu推理
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10468823

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10491951

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10490937