From 4e47d72012a227c8283d75e603d10ed82320d770 Mon Sep 17 00:00:00 2001 From: Gene Date: Mon, 1 Jan 2024 21:19:58 +0800 Subject: [PATCH] [DOC] modify details --- README.md | 74 +++++++++++++++++++------------------- docs/components/market.rst | 2 +- docs/index.rst | 4 ++- docs/start/intro.rst | 8 ++--- docs/workflows/upload.rst | 2 +- 5 files changed, 47 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index bb2b97b..89886ad 100644 --- a/README.md +++ b/README.md @@ -12,57 +12,61 @@ -``Learnware`` is a model sharing platform, which give a basic implementation of the learnware paradigm. A learnware is a well-performed trained machine learning model with a specification that enables it to be adequately identified to reuse according to the requirement of future users who may know nothing about the learnware in advance. The learnware paradigm can solve entangled problems in the current machine learning paradigm, like continual learning and catastrophic forgetting. It also reduces resources for training a well-performed model. +The `learnware` package provides a fundamental implementation of the central concepts and procedures for the learnware paradigm, which is a new paradigm aimed at enabling users to reuse existed well-trained models to solve their AI tasks instead of starting from scratch. +Moreover, the package's well-structured design ensures high scalability and allows for the effortless integration of various new features and techniques in the future. -# Introduction +In addition, the `learnware` package serves as the engine for the [Beimingwu System](https://bmwu.cloud) and can be effectively employed for conducting experiments related to learnware. -## Framework -
- -
+# Introduction +## Learnware Paradigm -At the workflow level, `Learnware` package consists of `Submitting Stage` and `Deploying Stage`. -At the module level, `Learnware` package is a platform that consists of above components. The components are designed as loose-coupled modules and each component could be used stand-alone. +A learnware consists of a high-performance machine learning model and specifications that characterize the model, i.e., "Learnware = Model + Specification". +These specifications, encompassing both semantic and statistical aspects, detail the model's functionality and statistical information, making it easier for future users to identify and reuse these models. +The need for Learnware arises due to challenges in machine learning, such as the need for extensive training data, advanced techniques, continuous learning, catastrophic forgetting, and data privacy issues. Although there are many efforts focusing on one of these issues separately, they are entangled, and solving one problem may exacerbate others. The learnware paradigm aims to address many of these challenges through a unified framework. Its benefits are listed as follows. -## Learnware Paradigm +| Benefit | Description | +| ---- | ---- | +| Lack of training data | Strong models can be built with small data by adapting well-performed learnwares. | +| Lack of training skills | Ordinary users can obtain strong models by leveraging well-performed learnwares instead of building models from scratch. | +| Catastrophic forgetting | Accepted learnwares are always stored in the learnware market, retaining old knowledge. | +| Continual learning | The learnware market continually enriches its knowledge with constant submissions of well-performed learnwares. | +| Data privacy/ proprietary | Developers only submit models, not data, preserving data privacy/proprietary. | +| Unplanned tasks | Open to all legal developers, the learnware market can accommodate helpful learnwares for various tasks. | +| Carbon emission | Assembling small models may offer good-enough performance, reducing interest in training large models and the carbon footprint. | -Machine learning, especially the prevailing big model paradigm, has achieved great success in natural language processing and computer vision applications. However, it still faces challenges such as the requirement of a large amount of labeled training data, difficulty in adapting to changing environments, and catastrophic forgetting when refining trained models incrementally. These big models, while useful in their targeted tasks, often fail to address the above issues and struggle to generalize beyond their specific purposes. +The learnware paradigm consists of two distinct stages: +- `Submitting Stage`: Developers voluntarily submit various learnwares to the learnware market, and the system conducts quality checks and further organization of these learnwares. +- `Deploying Stage`: When users submit task requirements, the learnware market automatically selects whether to recommend a single learnware or a combination of multiple learnwares and provides efficient deployment methods. Whether it’s a single learnware or a combination of multiple learnwares, the system offers convenient learnware reuse interfaces.
-The learnware paradigm introduces the concept of a well-performed, trained machine learning model with a specification that allows future users, who have no prior knowledge of the learnware, to reuse it based on their requirements. +## Learnware Package Design -Developers or owners of trained machine learning models can submit their models to a learnware market. If accepted, the market assigns a specification to the model and accommodates it. The learnware market could host thousands or millions of well-performed models from different developers, for various tasks, using diverse data, and optimizing different objectives. +
+ +
-Instead of building a model from scratch, users can submit their requirements to the learnware market, which then identifies and deploys helpful learnware(s) based on the specifications. Users can apply the learnware directly, adapt it using their data, or exploit it in other ways to improve their model. This process is more efficient and less expensive than building a model from scratch. -## Benefits of the Learnware Paradigm +At the workflow level, the `learnware` package consists of `Submitting Stage` and `Deploying Stage`. +At the module level, the `learnware` package is a platform that consists of above components. The components are designed as loose-coupled modules and each component could be used stand-alone. -| Benefit | Description | -| ---- | ---- | -| Lack of training data | Strong models can be built with small data by adapting well-performed learnwares. | -| Lack of training skills | Ordinary users can obtain strong models by leveraging well-performed learnwares instead of building models from scratch. | -| Catastrophic forgetting | Accepted learnwares are always stored in the learnware market, retaining old knowledge. | -| Continual learning | The learnware market continually enriches its knowledge with constant submissions of well-performed learnwares. | -| Data privacy/ proprietary | Developers only submit models, not data, preserving data privacy/proprietary. | -| Unplanned tasks | Open to all legal developers, the learnware market can accommodate helpful learnwares for various tasks. | -| Carbon emission | Assembling small models may offer good-enough performance, reducing interest in training large models and the carbon footprint. | +# Quick Start ## Installation -Learnware is currently hosted on [PyPI](https://pypi.org/). You can easily install `Learnware` by following these steps: +Learnware is currently hosted on [PyPI](https://pypi.org/project/learnware/). You can easily install `learnware` by following these steps: ```bash pip install learnware ``` -In the `Learnware` package, besides the base classes, many core functionalities such as "learnware specification generation" and "learnware deployment" rely on the `torch` library. Users have the option to manually install `torch`, or they can directly use the following command to install the `learnware` package: +In the `learnware` package, besides the base classes, many core functionalities such as "learnware specification generation" and "learnware deployment" rely on the `torch` library. Users have the option to manually install `torch`, or they can directly use the following command to install the `learnware` package: ```bash pip install learnware[full] @@ -72,7 +76,7 @@ pip install learnware[full] ## Prepare Learnware -In Learnware, each learnware is encapsulated in a `zip` package, which should contain at least the following four files: +In the `learnware` package, each learnware is encapsulated in a `zip` package, which should contain at least the following four files: - `learnware.yaml`: learnware configuration file. - `__init__.py`: methods for using the model. @@ -83,7 +87,7 @@ To facilitate the construction of a learnware, we provide a [Learnware Template] ## Learnware Package Workflow -Users can start a `Learnware` workflow according to the following steps: +Users can start a `learnware` workflow according to the following steps: ### Initialize a Learnware Market @@ -207,7 +211,7 @@ feature_augment_predict_y = reuse_feature_augment.predict(user_data=data_X) ### Auto Workflow Example -The `Learnware` also offers automated workflow examples. This includes preparing learnwares, uploading and deleting learnwares from the market, and searching for learnwares using both semantic and statistical specifications. To experience the basic workflow of the `Learnware` package, the users can run `test/test_workflow/test_workflow.py` to try the basic workflow of `Learnware`. +The `learnware` package also offers automated workflow examples. This includes preparing learnwares, uploading and deleting learnwares from the market, and searching for learnwares using both semantic and statistical specifications. To experience the basic workflow of the `learnware` package, the users can run `test/test_workflow/test_workflow.py` to try the basic workflow of `learnware`. # Experiments and Examples @@ -223,7 +227,7 @@ For all experiments, we used a single Linux server. Details on the specification -## Tabular Data Experiments +## Tabular Scenario Experiments ### Datasets @@ -243,7 +247,7 @@ Based on the specific design of user tasks, our experiments were primarily categ - **heterogeneous experiments** aim to evaluate the performance of identifying and reusing helpful heterogeneous learnwares in situations where no available learnwares match the feature space of the user's task. This helps to highlight the potential of learnwares for applications beyond their original purpose. -### Homogeneous Tabular Dataset +### Homogeneous Tabular Scenario For homogeneous experiments, the 55 stores in the Corporacion dataset act as 55 users, each applying one feature engineering method, and using the test data from their respective store as user data. These users can then search for homogeneous learnwares in the market with the same feature spaces as their tasks. @@ -269,7 +273,7 @@ When users have both test data and limited training data derived from their orig From the figure, it's evident that when users have limited training data, the performance of reusing single/multiple table learnwares is superior to that of the user's own model. This emphasizes the benefit of learnware reuse in significantly reducing the need for extensive training data and achieving enhanced results when available user training data is limited. -### Heterogeneous Tabular Dataset +### Heterogeneous Tabular Scenario In heterogeneous experiments, the learnware market would recommend helpful heterogeneous learnwares with different feature spaces with the user tasks. Based on whether there are learnwares in the market that handle tasks similar to the user's task, the experiments can be further subdivided into the following two types: @@ -301,11 +305,9 @@ Here we have chosen the 10 stores from the M5 dataset to act as users. Although We can observe that heterogeneous learnwares are beneficial when there's a limited amount of the user's labeled training data available, aiding in better alignment with the user's specific task. This underscores the potential of learnwares to be applied to tasks beyond their original purpose. -## Image Data Experiment - -For the CIFAR-10 dataset, we sampled the training set uneven +## Image Scenario Experiment -ly by category and constructed unbalanced training datasets for the 50 learnwares that contained only some of the categories. This makes it unlikely that there exists any learnware in the learnware market that can accurately handle all categories of data; only the learnware whose training data is closest to the data distribution of the target task is likely to perform well on the target task. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with a non-zero probability of sampling on only 4 categories, and the sampling ratio is 0.4: 0.4: 0.1: 0.1. Ultimately, the training set for each learnware contains 12,000 samples covering the data of 4 categories in CIFAR-10. +For the CIFAR-10 dataset, we sampled the training set unevenly by category and constructed unbalanced training datasets for the 50 learnwares that contained only some of the categories. This makes it unlikely that there exists any learnware in the learnware market that can accurately handle all categories of data; only the learnware whose training data is closest to the data distribution of the target task is likely to perform well on the target task. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with a non-zero probability of sampling on only 4 categories, and the sampling ratio is 0.4: 0.4: 0.1: 0.1. Ultimately, the training set for each learnware contains 12,000 samples covering the data of 4 categories in CIFAR-10. We constructed 50 target tasks using data from the test set of CIFAR-10. Similar to constructing the training set for the learnwares, to allow for some variation between tasks, we sampled the test set unevenly. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with non-zero sampling probability on 6 categories, and the sampling ratio is 0.3: 0.3: 0.1: 0.1: 0.1: 0.1. Ultimately, each target task contains 3000 samples covering the data of 6 categories in CIFAR-10. @@ -329,7 +331,7 @@ In some specific settings, the user will have a small number of labelled samples -## Text Data Experiment +## Text Scenario Experiment ### Datasets diff --git a/docs/components/market.rst b/docs/components/market.rst index 4e8747a..3fb46aa 100644 --- a/docs/components/market.rst +++ b/docs/components/market.rst @@ -24,7 +24,7 @@ The ``checker`` is used for checking the learnware in some standards. It should Current Checkers ====================================== -The ``Learnware`` package provide two different implementation of ``market`` where both of them share the same ``checker`` list. So we first introduce the details of ``checker``\ s. +The ``learnware`` package provide two different implementation of ``market`` where both of them share the same ``checker`` list. So we first introduce the details of ``checker``\ s. The ``checker``s check a learnware object in different aspects, including environment configuration (``CondaChecker``), semantic specifications (``EasySemanticChecker``), and statistical specifications (``EasyStatChecker``). The ``__call__`` method of each checker is designed to be invoked as a function to conduct the respective checks on the learnware and return the outcomes. It defines three types of learnwares: ``INVALID_LEARNWARE`` denotes the learnware does not pass the check, ``NONUSABLE_LEARNWARE`` denotes the learnware pass the check but cannot make prediction, ``USABLE_LEARNWARE`` denotes the leanrware pass the check and can make prediction. Currently, we have three ``checker``\ s, which are described below. diff --git a/docs/index.rst b/docs/index.rst index 1137040..946a5c5 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,7 +7,9 @@ ``Learnware`` Documentation ============================================================ -``Learnware`` is a model sharing platform, which give a basic implementation of the learnware paradigm. A learnware is a well-performed trained machine learning model with a specification that enables it to be adequately identified to reuse according to the requirement of future users who may know nothing about the learnware in advance. The learnware paradigm can solve entangled problems in the current machine learning paradigm, like continual learning and catastrophic forgetting. It also reduces resources for training a well-performed model. +The ``learnware`` package provides a fundamental implementation of the central concepts and procedures for the learnware paradigm. +A learnware is a well-performed trained machine learning model with a specification that enables it to be adequately identified to reuse according to the requirement of future users who may know nothing about the learnware in advance. +The learnware paradigm is a new paradigm aimed at enabling users to reuse existed well-trained models to solve their AI tasks instead of starting from scratch. .. _user_guide: diff --git a/docs/start/intro.rst b/docs/start/intro.rst index 6c8279a..29d7d41 100644 --- a/docs/start/intro.rst +++ b/docs/start/intro.rst @@ -12,7 +12,7 @@ In addition, the ``learnware`` package serves as the engine for the `Beimingwu S What is Learnware? ==================== -A learnware consists of high-performance machine learning models and specifications that characterize the models, i.e., "Learnware = Model + Specification." +A learnware consists of a high-performance machine learning model and specifications that characterize the model, i.e., "Learnware = Model + Specification". The learnware specification consists of "semantic specification" and "statistical specification": @@ -29,7 +29,7 @@ The Benefits of Learnware Paradigm Machine learning has achieved great success in many fields but still faces various challenges, such as the need for extensive training data and advanced training techniques, the difficulty of continuous learning, the risk of catastrophic forgetting, and the leakage of data privacy. -Although there are many efforts focusing on one of these issues separately, they are entangled, and solving one problem may exacerbate others. The learnware paradigm aimss to address many of these challenges through a unified framework. +Although there are many efforts focusing on one of these issues separately, they are entangled, and solving one problem may exacerbate others. The learnware paradigm aims to address many of these challenges through a unified framework. +-----------------------+-----------------------------------------------------------------------------------------------+ | Benefit | Description | @@ -79,5 +79,5 @@ Learnware Package Design .. image:: ../_static/img/learnware_framework.svg :align: center -At the workflow level, ``Learnware`` package consists of ``Submitting Stage`` and ``Deploying Stage``. -At the module level, ``Learnware`` package is a platform that consists of above components. The components are designed as loose-coupled modules and each component could be used stand-alone. +At the workflow level, the ``learnware`` package consists of ``Submitting Stage`` and ``Deploying Stage``. +At the module level, the ``learnware`` package is a platform that consists of above components. The components are designed as loose-coupled modules and each component could be used stand-alone. diff --git a/docs/workflows/upload.rst b/docs/workflows/upload.rst index d3797ce..b8e5d8b 100644 --- a/docs/workflows/upload.rst +++ b/docs/workflows/upload.rst @@ -10,7 +10,7 @@ We will first discuss the necessary components of a valid learnware, followed by Prepare Learnware ==================================== -In learnware ``Learnware`` package, each learnware is encapsulated in a ``zip`` package, which should contain at least the following four files: +In the ``learnware`` package, each learnware is encapsulated in a ``zip`` package, which should contain at least the following four files: - ``learnware.yaml``: learnware configuration file. - ``__init__.py``: methods for using the model.