Merge branch 'xiey/docs' of https://github.com/Learnware-LAMDA/Learnware into main

docs/start/intro.rst

@@ -3,61 +3,33 @@
 Introduction
 ================
 
-``Learnware`` is a model sharing platform, which give a basic implementation of the learnware paradigm. A learnware is a well-performed trained machine learning model with a specification that enables it to be adequately identified to reuse according to the requirement of future users who may know nothing about the learnware in advance. The learnware paradigm can solve entangled problems in the current machine learning paradigm, like continual learning and catastrophic forgetting. It also reduces resources for training a well-performed model.
+The ``learnware`` package provides a fundamental implementation of the central concepts and procedures for the learnware paradigm, which is a new paradigm aimed at enabling users to reuse existed well-trained models to solve their AI tasks instead of starting from scratch.
 
+Moreover, the package's well-structured design ensures high scalability and allows for the effortless integration of various new features and techniques in the future.
 
-Motivation
-=================
+In addition, the ``learnware`` package serves as the engine for the `Beimingwu System <https://bmwu.cloud/#/>`_ and can be effectively employed for conducting experiments related to learnware.
 
-.. image:: ../_static/img/learnware_paradigm.jpg
-   :align: center
+What is Learnware?
+====================
 
-Machine learning, especially the prevailing big model paradigm, has achieved great success in natural language processing and computer vision applications. However, it still faces challenges such as the requirement of a large amount of labeled training data, difficulty in adapting to changing environments, and catastrophic forgetting when refining trained models incrementally. These big models, while useful in their targeted tasks, often fail to address the above issues and struggle to generalize beyond their specific purposes.
-
-To better address the entangled issues in machine learning, we should consider the following aspects:
-
-+------------------------------------------------------------------------------------+
-| Aspect                                                                             |
-+====================================================================================+
-| 1. Investigate techniques that address multiple challenges simultaneously,         |
-|    recognizing that these issues are often intertwined in real-world applications. |
-+------------------------------------------------------------------------------------+
-| 2. Explore paradigms like learnware, which offers the possibility of               |
-|    systematically reusing small models for tasks beyond their original purposes,   |
-|    reducing the need for users to build models from scratch.                       |
-+------------------------------------------------------------------------------------+
-| 3. Develop solutions that enable ordinary users to create well-performing models   |
-|    without requiring proficient training skills.                                   |
-+------------------------------------------------------------------------------------+
-| 4. Address data privacy and proprietary concerns to facilitate experience          |
-|    sharing among different users while respecting confidentiality.                 |
-+------------------------------------------------------------------------------------+
-| 5. Adapt to the constraints of big data applications, where it may be              |
-|    unaffordable or infeasible to hold all data for multiple passes of scanning.    |
-+------------------------------------------------------------------------------------+
-| 6. Consider the environmental impact of training large models, as their carbon     |
-|    emissions pose a threat to our environment.                                     |
-+------------------------------------------------------------------------------------+
-
-By considering these factors, we can develop a more comprehensive framework for tackling the complex challenges in machine learning, moving beyond the limitations of the big model paradigm, called Learnware.
-
-
-
-Framework
-=======================
-
-.. image:: ../_static/img/learnware_market.jpg
-   :align: center
+A learnware consists of high-performance machine learning models and specifications that characterize the models, i.e., "Learnware = Model + Specification."
 
-The learnware paradigm introduces the concept of a well-performed, trained machine learning model with a specification that allows future users, who have no prior knowledge of the learnware, to reuse it based on their requirements.
+The learnware specification consists of "semantic specification" and "statistical specification":
 
-Developers or owners of trained machine learning models can submit their models to a learnware market. If accepted, the market assigns a specification to the model and accommodates it. The learnware market could host thousands or millions of well-performed models from different developers, for various tasks, using diverse data, and optimizing different objectives.
+- ``Semantic Specification``: Describe the type and functionality of the model through text.
+- ``Statistical Specification``: Characterize the statistical information contained in the model using various machine learning techniques.
 
-Instead of building a model from scratch, users can submit their requirements to the learnware market, which then identifies and deploys helpful learnware(s) based on the specifications. Users can apply the learnware directly, adapt it using their data, or exploit it in other ways to improve their model. This process is more efficient and less expensive than building a model from scratch.
+Learnware specifications describe the model's capabilities, enabling the model to be identified and reused by future users who may know nothing about the learnware in advance.
 
+Why do we need Learnware?
+============================
 
-Benefits of the Learnware Paradigm
-==============================================
+The Benefits of Learnware Paradigm
+-------------------------------------
+
+Machine learning has achieved great success in many fields but still faces various challenges, such as the need for extensive training data and advanced training techniques, the difficulty of continuous learning, the risk of catastrophic forgetting, and the leakage of data privacy.
+
+Although there are many efforts focusing on one of these issues separately, they are entangled, and solving one problem may exacerbate others. The learnware paradigm aimss to address many of these challenges through a unified framework.
 
 +-----------------------+-----------------------------------------------------------------------------------------------+
 | Benefit               | Description                                                                                   |
@@ -83,11 +55,25 @@ Benefits of the Learnware Paradigm
 |                       | large models and the carbon footprint.                                                        |
 +-----------------------+-----------------------------------------------------------------------------------------------+
 
-Challenges and Future Work
-==============================================
+How to Solve Future Tasks with Learnware Paradigm?
+----------------------------------------------------
+
+.. image:: ../_static/img/learnware_paradigm.jpg
+   :align: center
+
+Instead of building a model from scratch, users can submit their requirements to the learnware market, which then identifies and deploys helpful learnware(s) based on the specifications. Users can apply the learnware directly, adapt it using their data, or exploit it in other ways to improve their models. This process is more efficient and less expensive than building a model from scratch.
 
-Although the learnware proposal shows promise, much work remains to make it a reality. The next sections will present some of the progress made so far.
 
+Procedure of Learnware Paradigm
+==================================
+- ``Submitting Stage``: Developers voluntarily submit various learnwares to the learnware market, and the system conducts quality checks and further organization of these learnwares.
+- ``Deploying Stage``: When users submit task requirements, the learnware market automatically selects whether to recommend a single learnware or a combination of multiple learnwares and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers convenient learnware reuse interfaces.
+
+.. image:: ../_static/img/learnware_market.svg
+   :align: center
 
 
+Learnware Package Design
+==========================
 
+TBD by xiaodong.

examples/dataset_image_workflow/README.md

@@ -2,9 +2,18 @@
 
 ## Introduction
 
-For the CIFAR-10 dataset, we sampled the training set unevenly by category and constructed unbalanced training datasets for the 50 learnwares that contained only some of the categories. This makes it unlikely that there exists any learnware in the learnware market that can accurately handle all categories of data; only the learnware whose training data is closest to the data distribution of the target task is likely to perform well on the target task. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with a non-zero probability of sampling on only 4 categories, and the sampling ratio is 0.4: 0.4: 0.1: 0.1. Ultimately, the training set for each learnware contains 12,000 samples covering the data of 4 categories in CIFAR-10.
+We conducted experiments on the widely used image benchmark dataset: [``CIFAR-10``](https://www.cs.toronto.edu/~kriz/cifar.html).
+The  ``CIFAR-10`` dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.  
 
-We constructed 50 target tasks using data from the test set of CIFAR-10. Similar to constructing the training set for the learnwares, in order to allow for some variation between tasks, we sampled the test set unevenly. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with non-zero sampling probability on 6 categories, and the sampling ratio is 0.3: 0.3: 0.1: 0.1: 0.1: 0.1. Ultimately, each target task contains 3000 samples covering the data of 6 categories in CIFAR-10.
+In the submitting stage, we sampled the training set non-uniformly by category, and constructed unbalanced training datasets for the 50 learnwares that contained only part of the categories randomly. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with positive sampling probability on only 4 categories, and a sampling ratio of 0.4: 0.4: 0.1: 0.1. The training set for each learnware contains 12,500 samples covering data from the 4 categories in CIFAR-10.
+
+In the deploying stage, we constructed 100 user tasks using the CIFAR-10 test set data. Similar to constructing the training set, the probability of each category being sampled obeys a random multinomial distribution, with positive sampling probabilities on only 6 categories, with a sampling ratio of 0.3: 0.3: 0.1: 0.1: 0.1: 0.1. Each user task contains 3,000 samples covering the data of 6 categories in CIFAR-10. 
+
+Our example ``image_example`` shows the performance in two different scenarios:
+
+**Unlabelled Sample Scenario**: This scenario is designed to evaluate performance when users possess only testing data, searching and reusing learnware available in the market.
+
+**Labelled Sample Scenario**: This scenario aims to assess performance when users have both testing and limited training data, searching and reusing learnware directly from the market instead of training a model from scratch. This helps determine the amount of training data saved for the user.
 
 ## Run the code
 
@@ -18,6 +27,8 @@ python workflow.py image_example
 
 With the experimental setup above, we evaluated the performance of RKME Image by calculating the mean accuracy across all users.
 
+### Unlabelled Sample Scenario
+
 | Metric                               | Value               |
 |--------------------------------------|---------------------|
 | Mean in Market (Single)              | 0.346               |
@@ -26,8 +37,12 @@ With the experimental setup above, we evaluated the performance of RKME Image by
 | Job Selector Reuse (Multiple)        | 0.534               |
 | Average Ensemble Reuse (Multiple)    | 0.676               |
 
+### Labelled Sample Scenario
+
 In some specific settings, the user will have a small number of labeled samples. In such settings, learning the weight of selected learnwares on a limited number of labeled samples can result in a better performance than training directly on a limited number of labeled samples.
 
 <div align=center>
   <img src="../../docs/_static/img/image_labeled.svg" alt="Results on Image Experimental Scenario" style="width:50%;" />
-</div>
+</div>
+
+Note that in labelled sample scenario, the labelled samples are repeatedly sampled 3 to 10 times, in order to reduce the estimation error in accuracy due to random sampling.

learnware/client/utils.py

@@ -22,14 +22,15 @@ def system_execute(args, timeout=None, env=None, stdout=subprocess.DEVNULL, stde
             errmsg = err.stderr.decode()
             logger.warning(f"System Execute Error: {errmsg}")
         raise err
-    
+
     return com_process
 
 
 def remove_enviroment(conda_env):
     system_execute(args=["conda", "env", "remove", "-n", f"{conda_env}"])
 
-def install_environment(learnware_dirpath, conda_env):
+
+def install_environment(learnware_dirpath, conda_env, conda_prefix=None):
     """Install environment of a learnware
 
     Parameters
@@ -38,12 +39,21 @@ def install_environment(learnware_dirpath, conda_env):
         Path of the learnware folder
     conda_env : str
         a new conda environment will be created with the given name;
+    conda_prefix: str
+        install env in a specific location, not default env path;
 
     Raises
     ------
     Exception
         Lack of the environment configuration file.
     """
+    if conda_prefix is not None:
+        args_location = ["--prefix", conda_prefix]
+        conda_env = conda_prefix
+    else:
+        args_location = ["--name", conda_env]
+        pass
+
     with tempfile.TemporaryDirectory(prefix="learnware_") as tempdir:
         logger.info(f"learnware_dir namelist: {os.listdir(learnware_dirpath)}")
         if "environment.yaml" in os.listdir(learnware_dirpath):
@@ -53,7 +63,7 @@ def install_environment(learnware_dirpath, conda_env):
             filter_nonexist_conda_packages_file(yaml_file=yaml_path, output_yaml_file=yaml_path_filter)
             # create environment
             logger.info(f"create conda env [{conda_env}] according to .yaml file")
-            system_execute(args=["conda", "env", "create", "--name", f"{conda_env}", "--file", f"{yaml_path_filter}"])
+            system_execute(args=["conda", "env", "create"] + args_location + ["--file", f"{yaml_path_filter}"])
 
         elif "requirements.txt" in os.listdir(learnware_dirpath):
             requirements_path: str = os.path.join(learnware_dirpath, "requirements.txt")
@@ -61,14 +71,15 @@ def install_environment(learnware_dirpath, conda_env):
             logger.info(f"checking the available pip packages for {conda_env}")
             filter_nonexist_pip_packages_file(requirements_file=requirements_path, output_file=requirements_path_filter)
             logger.info(f"create empty conda env [{conda_env}]")
-            system_execute(args=["conda", "create", "-y", "--name", f"{conda_env}", "python=3.8"])
+            system_execute(args=["conda", "create", "-y"] + args_location + ["python=3.8"])
             logger.info(f"install pip requirements for conda env [{conda_env}]")
             system_execute(
                 args=[
                     "conda",
                     "run",
-                    "-n",
-                    f"{conda_env}",
+                ]
+                + args_location
+                + [
                     "--no-capture-output",
                     "python",
                     "-m",
@@ -86,8 +97,9 @@ def install_environment(learnware_dirpath, conda_env):
         args=[
             "conda",
             "run",
-            "-n",
-            f"{conda_env}",
+        ]
+        + args_location
+        + [
             "--no-capture-output",
             "python",
             "-m",