Merge branch 'main' into enh_fix_image_workflow

# Conflicts:
#	examples/dataset_image_workflow/main.py
#	learnware/specification/regular/image/rkme.py

docs/components/market.rst

@@ -3,65 +3,52 @@
 Market
 ================================
 
+The ``learnware market`` receives high-performance machine learning models from developers, incorporates them into the system, and provides services to users by identifying and reusing learnware to help users solve current tasks. Developers voluntarily submit various learnwares to the learnware market, and the market conducts quality checks and further organization of these learnwares. When users submit task requirements, the learnware market automatically selects whether to recommend a single learnware or a combination of multiple learnwares. 
 
-Concepts
-======================================
-
-In the learnware paradigm, there are three key players: *developers*, *users*, and the *market*. 
-
-* Developers: Typically machine learning experts who create and aim to share or sell their high-performing trained machine learning models. 
-* Users: Require machine learning services but often possess limited data and lack the necessary knowledge and expertise in machine learning. 
-* Market: Acquires top-performing trained models from developers, houses them within the marketplace, and offers services to users by identifying and reusing learnwares to help users with their current tasks. 
-
-This process can be broken down into two main stages.
+The ``learnware market`` will receive various kinds of learnwares, and learnwares from different feature/label spaces form numerous islands of specifications. All these islands together constitute the ``specification world`` in the learnware market. The market should discover and establish connections between different islands, and then merge them into a unified specification world. This further organization of learnwares support search learnwares among all learnwares, not just among learnwares which has the same feature space and label space with the user's task requirements.
 
-Submitting Stage
-------------------------------
-
-During the *submitting stage*, developers can voluntarily submit their trained models to the learnware market. The market will then implement a quality assurance mechanism, such as performance validation, to determine if a submitted model is suitable for acceptance. In a learnware market with millions of models, identifying potentially helpful models for a new user is a challenge.
-
-Requiring users to submit their own data to the market for model testing is impractical, time-consuming, and costly, as it could lead to data leakage. Straightforward approaches, such as measuring the similarity between user data and the original training data of models, are also infeasible due to privacy and proprietary concerns. Our design operates under the constraint that the learnware market has no access to the original training data from developers or users. Furthermore, it assumes that users have limited knowledge of the models available in the market.
+Framework
+======================================
 
-The solution's crux lies in the *specification*, which is central to the learnware proposal. Once a submitted model is accepted by the learnware market, it is assigned a specification, which conveys the model's specialty and utility without revealing its original training data. For simplicity, consider models as functions that map input domain :math:`\mathcal{X}` to output domain :math:`\mathcal{Y}` with respect to objective 'obj.' These models exist in a functional space :math:`\mathcal{F}: \mathcal{X} \mapsto \mathcal{Y}` with respect to 'obj.' Each model has a specification, and all specifications form a specification space, where those for models that serve similar tasks are situated closely.
+The ``learnware market`` is combined with a ``organizer``, a ``searcher``, and a list of ``checker``s. 
 
-In a learnware market, heterogeneous models may have different :math:`\mathcal{X}`, :math:`\mathcal{Y}`, or objectives. If we refer to the specification space that covers all possible models in all possible functional spaces as the 'specification world' analogously, then each specification space corresponding to one possible functional space can be called a 'specification island.' Designing an elegant specification format that encompasses the entire specification world and allows all possible models to be efficiently and adequately identified is a significant challenge. Currently, we adopt a practical design, where each learnware's specification consists of two parts.
+The ``organizer`` can store and organize learnwares in the market. It supports ``add``, ``delete``, and ``update`` operations for learnwares. It also provides the interface for ``searcher`` to search learnwares based on user requirement.
 
+The ``searcher`` can search learnwares based on user requirement. The implementation of ``searcher`` is dependent on the concrete implementation and interface for ``organizer``, where usually an ``organizer`` can be compatible with multiple different ``searcher``s.
 
-Reusing Stage
-------------------------------
+The ``checker`` is used for checking the learnware in some standards. It should check the utility of a learnware and is supposed to return the status and a message related to the learnware's check result. Only the learnwares who passed the ``checker`` could be able to be stored and added into the ``learnware market``. 
 
-Creating Learnware Specifications
-++++++++++++++++++++++++++++++++++++
 
-The first part of the learnware specification can be realized by a string consisting of a set of descriptions/tags given by the learnware market. These tags address aspects such as the task, input, output, and objective. Based on the user's provided descriptions/tags, the corresponding specification island can be efficiently and accurately located. The designer of the learnware market can create an initial set of descriptions/tags, which can grow as new models are accepted, and new functional spaces and specification islands are created.
 
-Merging Specification Islands
-+++++++++++++++++++++++++++++++++
+Current Checkers
+======================================
 
-Specification islands can merge into larger ones. For example, when a new model about :math:`F: \mathcal{X}_1 \cup \mathcal{X}_2 \mapsto \mathcal{Y}` with respect to 'obj' is accepted by the learnware market, two islands can be merged. This is possible because the market can have synthetic data by randomly generating inputs, feeding them to models, and concatenating each input with its corresponding output to construct a dataset reflecting the function of a model. In principle, specification islands can be merged if there are common ingredients in :math:`\mathcal{X}`, :math:`\mathcal{Y}`, and 'obj.'
+The ``learnware`` package provide two different implementation of ``market`` where both of them share the same ``checker`` list. So we first introduce the details of ``checker``s.
 
-Deploying Learnware Models
-++++++++++++++++++++++++++++++
+The ``checker``s check a learnware object in different aspects, including environment configuration (``CondaChecker``), semantic specifications (``EasySemanticChecker``), and statistical specifications (``EasyStatChecker``). The ``__call__`` method of each checker is designed to be invoked as a function to conduct the respective checks on the learnware and return the outcomes. It defines three types of learnwares: ``INVALID_LEARNWARE`` denotes the learnware does not pass the check, ``NONUSABLE_LEARNWARE`` denotes the learnware pass the check but cannot make prediction, ``USABLE_LEARWARE`` denotes the leanrware pass the check and can make prediction. Currently, we have three ``checker``s, which are described below.
 
-In the deploying stage, the user submits their requirement to the learnware market, which then identifies and returns helpful learnwares to the user. There are two issues to address: how to identify learnwares matching the user requirement and how to reuse the returned learnwares.
 
-The learnware market can house thousands or millions of models. Efficiently identifying helpful learnwares is challenging, especially given that the market has no access to the original training data of learnwares or the current user's data. With the specification design mentioned earlier, the market can request users to describe their intentions using a set of descriptions/tags, through a user interface or a learnware description language. Based on this information, the task becomes identifying helpful learnwares in a specification island.
+``CondaChecker``
+------------------
+This ``checker`` checks a the environment of the learnware object. It creates a ``LearnwaresContainer`` instance to handle the Learnware and uses ``inner_checker`` to check the Learnware. If an exception occurs, it logs the error and returns ``NONUSABLE_LEARNWARE`` status and error message.
 
-Reusing Learnwares
-++++++++++++++++++++++
 
-Once helpful learnwares are identified and delivered to the user, they can be reused in various ways. Users can apply the received learnware directly to their data, use multiple learnwares to create an ensemble, or adapt and polish the received learnware(s) using their own data. Learnwares can also be used as feature augmentors, with their outputs used as augmented features for building the final model.
+``EasySemanticChecker``
+-------------------------
+This ``checker`` checks the semantic specification of a learnware object. It checks if the given semantic specification conforms to predefined standards. It verifies each key in predefined dictionary. If the check fails, it logs the error and returns ``NONUSABLE_LEARNWARE`` status and error message.
 
-Helpful learnwares may be trained for tasks that are not exactly the same as the user's current task. In such cases, users can tackle their tasks in a divide-and-conquer way or reuse the learnwares collectively through measuring the utility of each model on each testing instance. If users find it difficult to express their requirements accurately, they can adapt and polish the received learnwares directly using their own data.
 
+``EasyStatChecker``
+---------------------
 
-Framework
-======================================
+This ``checker`` checks the statistical specification and functionality of a learnware object. It performs multiple checks to validate the learnware. It checks for model instantiation, verifies input shape and statistical specifications, and test output shape using random generated data. In case of any exceptions, it logs the error and returns ``NONUSABLE_LEARNWARE`` status and error message.
 
 
 Current Markets
 ======================================
 
+The ``learnware`` package provide two different implementation of ``market``, i.e. ``Easy Market`` and ``Hetero Market``. They have different implementation of ``organizer`` and ``searcher``.
+
 Easy Market
 -------------
 
@@ -77,6 +64,3 @@ One important case is that models have different feature spaces. In order to ena
 - First, design a method for the market to connect different feature spaces to a common subspace and implement the function ``HeterogeneousFeatureMarket.learn_mapping_functions``. This function uses specifications of all submitted models to learn mapping functions that can map the data in the original feature space to the common subspace and vice verse.
 - Second, use learned mapping functions to implement the functions ``HeterogeneousFeatureMarket.transform_original_to_subspace`` and ``HeterogeneousFeatureMarket.transform_subspace_to_original``.
 - Third, use the functions ``HeterogeneousFeatureMarket.transform_original_to_subspace`` and ``HeterogeneousFeatureMarket.transform_subspace_to_original`` to overwrite the mehtod ``EvolvedMarket.generate_new_stat_specification`` and  ``EvolvedMarket.EvolvedMarket.evolve_learnware_list`` of the base class ``EvolvedMarket``.
-
-Current Checkers
-======================================

docs/components/spec.rst

@@ -80,6 +80,51 @@ Table Specification
 Image Specification
 --------------------------
 
+Image data lives in a higher dimensional space than other data types. Unlike lower dimensional spaces, metrics defined based on Euclidean distances (or similar distances) will fail in higher dimensional spaces. This means that measuring the similarity between image samples becomes difficult. 
+
+To address these issues, we use the Neural Tangent Kernel (NTK) based on Convolutional Neural Networks (CNN) to measure the similarity of image samples.  As we all know, CNN has greatly advanced the field of computer vision and is still a mainstream deep learning technique. 
+
+Usage & Example
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this part, we show that how to generate Image Specification for the training set of the CIFAR-10 dataset. 
+Note that the Image Specification is generated on a subset of the CIFAR-10 dataset with ``generate_rkme_image_spec``. 
+Then, it is saved to file "cifar10.json" using ``spec.save``. 
+
+In many cases, it is difficult to construct Image Specification on the full dataset. 
+By randomly sampling a subset of the dataset, we can construct Image Specification based on it efficiently, with a strong enough statistical description of the full dataset.
+
+.. tip::
+   Typically, sampling 3,000 to 10,000 images is sufficient to generate the Image Specification.
+
+.. code-block:: python
+
+   import torchvision
+   from torch.utils.data import DataLoader
+   from learnware.specification import generate_rkme_image_spec
+
+   SAMPLED_SIZE = 5000
+
+   full_set = torchvision.datasets.CIFAR10(
+      root='./data', train=True, download=True, transform=torchvision.transforms.ToTensor())
+   loader =  DataLoader(full_set, batch_size=SAMPLED_SIZE, shuffle=True)
+   sampled_X, _ = next(iter(loader))
+
+   spec = generate_rkme_image_spec(sampled_X)
+   spec.save("cifar10.json")
+
+Privacy Protection
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the third row of the figure, we show the eight pseudo-data with the largest weights :math:`\beta` in the Image Specification generated on the CIFAR-10 dataset.
+Notice that the Image Specification generated based on Neural Tangent Kernel (NTK) protects the user's privacy very well.
+
+In contrast, we show the performance of the RBF kernel on image dat in the first row of the figure below. 
+The RBF not only exposes the real data (plotted in the corresponding position in the second row), but also fails to fully utilise the weights :math:`\beta`.
+
+.. image:: ../_static/img/image_spec.png
+   :align: center
+
 Text Specification
 --------------------------
 

docs/index.rst

@@ -39,7 +39,7 @@ Document Structure
 
 .. toctree::
    :maxdepth: 3
-   :caption: CONCEPTS:
+   :caption: COMPONENTS:
 
    Market <components/market.rst>
    Learnware & Reuser <components/learnware.rst>

docs/workflows/client.rst

@@ -6,13 +6,13 @@ Learnware Client
 Introduction
 ====================
 
-``Learnware Client`` is a python api that provides a convenient interface for interacting with the official market. You can easily use the client to upload, download and search learnwares.
+``Learnware Client`` is a ``Python API`` that provides a convenient interface for interacting with the ``BeimingWu`` system. You can easily use the client to upload, download, delete, update, and search learnwares.
 
 
 Prepare access token
 ====================
 
-Before using the ``Learnware Client``, you'll need to obtain a token from the `official website <https://www.lamda.nju.edu.cn/learnware/>`_. Just login to the website and click "client token" tab in the user center.
+Before using the ``Learnware Client``, you'll need to obtain a token from the `official website <https://bmwu.cloud/>`_. Just login to the website and click ``Client Token`` tab in the ``Personal Information``.
 
 
 How to Use Client
@@ -25,142 +25,209 @@ Initialize a Learware Client
 
 .. code-block:: python
     
-    import learnware
-    from learnware.client import LearnwareClient
+    from learnware.client import LearnwareClient, SemanticSpecificationKey
 
+    # Login to Beiming system
     client = LearnwareClient()
-
-    # login to official market
     client.login(email="your email", token="your token")
 
+Where email is the registered mailbox of the system and token is the token obtained in the previous section.
 
 Upload Leanware
 -------------------------------
 
-Before uploading a learnware, you'll need to prepare the semantic specification of your learnware. You can create a semantic specification by a helper function ``create_semantic_specification``.
+Before uploading a learnware, you'll need to prepare the semantic specification of your learnware. Let's take the classification task for tabular data as an example. You can create a semantic specification by a helper function ``create_semantic_specification``.
 
 .. code-block:: python
 
+    # Prepare input description when data_type="Table"
     input_description = {
-        "Dimension": 16,
-        {
-            "Description": {
-                "0": "gender",
-                "1": "age",
-                "2": "f2",
-                "5": "f5"
-            }            
-        }
+        "Dimension": 5,
+        "Description": {
+            "0": "age",
+            "1": "weight",
+            "2": "body length",
+            "3": "animal type",
+            "4": "claw length"
+        },
     }
+
+    # Prepare output description when task_type in ["Classification", "Regression"]
     output_description = {
         "Dimension": 3,
         "Description": {
-            "0": "the probability of being a cat",
-            "1": "the probability of being a dog",
-            "2": "the probability of being a bird"
-        }
+            "0": "cat",
+            "1": "dog",
+            "2": "bird",
+        },
     }
+
+    # Create semantic specification
     semantic_spec = client.create_semantic_specification(
-        name="mylearnware1", 
-        description="this is my learnware", 
-        data_type="Table", 
-        task_type="Classification", 
-        library_type="Scikit-learn", 
-        senarioes=["Business", "Financial"],
-        input_description, output_description)
-    # data_type, task_type, library_type, senarioes are enums, you can find possible values in `learnware.C`
+        name="learnware_example",
+        description="Just a example for uploading a learnware",
+        data_type="Table",
+        task_type="Classification",
+        library_type="Scikit-learn",
+        scenarios=["Business", "Financial"],
+        license=["Apache-2.0"],
+        input_description=input_description,
+        output_description=output_description,
+    )
     
+Ensure that the input parameters for the semantic specification fall within the specified ranges provided by ``client.list_semantic_specification_values(key)``:
+
+* "data_type" must be within the range of ``key=SemanticSpecificationKey.DATA_TYPE``.
+* "task_type" must be within the range of ``key=SemanticSpecificationKey.TASK_TYPE``.
+* "library_type" must be within the range of ``key=SemanticSpecificationKey.LIBRARY_TYPE``.
+* "scenarios" must be a subset of ``key=SemanticSpecificationKey.SENARIOS``.
+* "license" must be a subset of ``key=SemanticSpecificationKey.LICENSE``.
+* When "data_type" is set to "Table", it is necessary to provide "input_description".
+* When "task_type" is either "Classification" or "Regression", it is necessary to provide "output_description".
+
+Finally, the semantic specification and the zip package path of the learnware were filled in to upload the learnware.
+
+Remember to verify the learnware before uploading it, as shown in the following code example:
 
-After defining the semantic specification, 
-you can upload your learnware using ``upload_learnware`` function:
-    
 .. code-block:: python
-    
-    learnware_id = client.upload_learnware(
-        semantic_spec=semantic_spec, 
-        zip_path="path to your learnware zipfile")
 
-Here, ``zip_path`` is the local path of your learnware zipfile.
+    # Prepare your learnware zip file
+    zip_path = "your learnware zip"
 
+    # Check your learnware before upload
+    client.check_learnware(
+        learnware_zip_path=zip_path, semantic_specification=semantic_spec
+    )
 
-Semantic Specification Search
+    # Upload your learnware
+    learnware_id = client.upload_learnware(
+        learnware_zip_path=zip_path, semantic_specification=semantic_spec
+    )
+
+After uploading the learnware successfully, you can see it in ``My Learnware``, the background will check it. Click on the learnware, which can be viewed in the ``Verify Status``. After the check passes, the Unverified tag of the learnware will disappear, and the uploaded learnware will appear in the system.
+
+Update Learnware
 -------------------------------
 
-You can search learnwares in official market using semantic specification. All the learnwares that match the semantic specification will be returned by the api. For example, the code below searches learnwares with `Table` data type:
+The ``update_learnware`` method is used to update the metadata and content of an existing learnware on the server. You can upload a new semantic specification, or directly upload a new learnware.
 
 .. code-block:: python
 
+    # Replace with the actual learnware ID
+    learnware_id = "123456789"
+
+    # Create new semantic specification
     semantic_spec = client.create_semantic_specification(
-        name="", 
-        description="", 
-        data_type="Table", 
-        task_type="", 
-        library_type="", 
-        senarioes=[],
-        input_description={}, output_description={})
-    
-    specification = learnware.specification.Specification()
-    specification.update_semantic_spec(specification)
-    learnware_list = client.search_learnware(specification)
-    
+        name="new learnware name",
+        description="new description",
+        data_type="Table",
+        task_type="Classification",
+        library_type="Scikit-learn",
+        scenarios=["Computer", "Internet"],
+        license=["CC-BY-4.0"],
+        input_description=new_input_description,
+        output_description=new_output_description,
+    )
+
+    # Update metadata without changing the content
+    client.update_learnware(learnware_id, semantic_spec)
+
+    # Update metadata and content with a new ZIP file
+    updated_zip_path = "/path/to/updated_learnware.zip"
+    client.update_learnware(learnware_id, semantic_spec, learnware_zip_path=updated_zip_path)
+
+Delete Learnware
+-------------------------------
 
-Statistical Specification Search
----------------------------------
+The ``delete_learnware`` method is used to delete a learnware from the server.
+
+.. code-block:: python
+
+    # Replace with the actual learnware ID to delete
+    learnware_id = "123456789"
+
+    # Delete the specified learnware
+    client.delete_learnware(learnware_id)
 
-You can search learnware by providing a statistical specification. The statistical specification is a json file that contains the statistical information of your training data. For example, the code below searches learnwares with `RKMETableSpecification`:
+
+Semantic Specification Search
+-------------------------------
+
+You can search the learnware in the system through the semantic specification, and all the learnware conforming to the semantic specification will be returned through the API. For example, the following code will give you all the learnware in the system whose task type is classified:
 
 .. code-block:: python
 
-    import learnware.specification as specification
+    from learnware.market import BaseUserInfo
 
-    user_spec = specification.RKMETableSpecification()
-    user_spec.load(os.path.join(unzip_path, "rkme.json"))
+    user_semantic = client.create_semantic_specification(
+        task_type="Classification"
+    )
+    user_info = BaseUserInfo(semantic_spec=user_semantic)
+    learnware_list = client.search_learnware(user_info, page_size=None)
     
-    specification = learnware.specification.Specification()
-    specification.update_stat_spec(user_spec)
 
-    learnware_list = client.search_learnware(specification)
+Statistical Specification Search
+---------------------------------
 
-    # you can view the scores of the searched learnwares
-    for learnware in learnware_list:
-        print(f'learnware_id: {learnware["learnware_id"]}, score: {learnware["matching"]}')
+You can also search the learnware in the system through the statistical specification, and all the learnware with similar distribution will be returned through the API. Using the ``generate_stat_spec`` function mentioned above, you can easily get the ``stat_spec`` for your current task, and then get the learnware that meets the statistical specification for the same type of data in the system by using the following code:
+
+.. code-block:: python
+
+    user_info = BaseUserInfo(stat_info={stat_spec.type: stat_spec})
+    learnware_list = client.search_learnware(user_info, page_size=None)
 
 
 Combine Semantic and Statistical Search
 ----------------------------------------
-You can provide both semantic and statistical specification to search learnwares. The engine will first filter learnwares by semantic specification and then search by statistical specification. For example, the code below searches learnwares with `Table` data type and `RKMETableSpecification`:
+By combining statistical and semantic specifications, you can perform more detailed searches, such as the following code that searches tabular data for pieces of learnware that satisfy your semantic specifications:
 
 .. code-block:: python
 
-    semantic_spec = client.create_semantic_specification(
-        name="", 
-        description="", 
-        data_type="Table", 
-        task_type="", 
-        library_type="", 
-        senarioes=[],
-        input_description={}, output_description={})
+    user_semantic = client.create_semantic_specification(
+        task_type="Classification",
+        scenarios=["Business"],
+    )
+    rkme_table = generate_stat_spec(type="table", X=train_x)
+    user_info = BaseUserInfo(
+        semantic_spec=user_semantic, stat_info={rkme_table.type: rkme_table}
+    )
+    learnware_list = client.search_learnware(user_info, page_size=None)
+
+Heterogeneous Table Search
+----------------------------------------
+When you provide a statistical specification for tabular data, the task type is "Classification" or "Regression", and your semantic specification includes descriptions for each dimension, the system will automatically enable heterogeneous table search. It won't only search in the tabular learnwares with same dimensions. The following code will perform heterogeneous table search through the API:
 
-    stat_spec = specification.RKMETableSpecification()
-    stat_spec.load(os.path.join(unzip_path, "rkme.json"))
-    specification = learnware.specification.Specification()
-    specification.update_semantic_spec(semantic_spec)
-    specification.update_stat_spec(stat_spec)
+.. code-block:: python
 
-    learnware_list = client.search_learnware(specification)
+    input_description = {
+        "Dimension": 2,
+        "Description": {
+            "0": "leaf width",
+            "1": "leaf length",
+        },
+    }
+    user_semantic = client.create_semantic_specification(
+        task_type="Classification",
+        scenarios=["Business"],
+        input_description=input_description,
+    )
+    rkme_table = generate_stat_spec(type="table", X=train_x)
+    user_info = BaseUserInfo(
+        semantic_spec=user_semantic, stat_info={rkme_table.type: rkme_table}
+    )
+    learnware_list = client.search_learnware(user_info)
 
 
 Download and Use Learnware
 -------------------------------
-When you get a learnware id, you can download and initiate the learnware with the following code:
+When the search is complete, you can download the learnware and configure the environment through the following code:
 
 .. code-block:: python
 
-    client.download_learnware(learnware_id, zip_path)
-    client.install_environment(zip_path)
-    learnware = client.load_learnware(zip_path)
-    # you can use the learnware to make prediction now
-
-
-
+    for temp_learnware in learnware_list:
+        learnware_id = temp_learnware["learnware_id"]
 
+        # you can use the learnware to make prediction now
+        learnware = client.load_learnware(
+            learnware_id=learnware_id, runnable_option="conda"
+        )

examples/dataset_m5_workflow/main.py

@@ -36,6 +36,7 @@ semantic_specs = [
         "Name": {"Values": "learnware_1", "Type": "String"},
         "Input": input_description,
         "Output": output_description,
+        "License": {"Values": ["MIT"], "Type": "Class"},
     }
 ]
 
@@ -48,6 +49,7 @@ user_semantic = {
     "Name": {"Values": "", "Type": "String"},
     "Input": input_description,
     "Output": output_description,
+    "License": {"Values": ["MIT"], "Type": "Class"},
 }
 
 
@@ -158,7 +160,7 @@ class M5DatasetWorkflow:
             search_result = easy_market.search_learnware(user_info)
             single_result = search_result.get_single_results()
             multiple_result = search_result.get_multiple_results()
-            
+
             print(f"search result of user{idx}:")
             print(
                 f"single model num: {len(single_result)}, max_score: {single_result[0].score}, min_score: {single_result[-1].score}"

examples/dataset_m5_workflow/upload.py

@@ -66,6 +66,7 @@ def main():
             "Scenario": {"Values": ["Business"], "Type": "Tag"},
             "Description": {"Values": "A sales-forecasting model from Walmart store", "Type": "String"},
             "Name": {"Values": name, "Type": "String"},
+            "License": {"Values": ["MIT"], "Type": "Class"},
         }
         res = session.post(
             submit_url,

examples/dataset_pfs_workflow/main.py

@@ -35,6 +35,7 @@ semantic_specs = [
         "Name": {"Values": "learnware_1", "Type": "String"},
         "Input": input_description,
         "Output": output_description,
+        "License": {"Values": ["MIT"], "Type": "Class"},
     }
 ]
 
@@ -47,6 +48,7 @@ user_semantic = {
     "Name": {"Values": "", "Type": "String"},
     "Input": input_description,
     "Output": output_description,
+    "License": {"Values": ["MIT"], "Type": "Class"},
 }
 
 
@@ -155,7 +157,7 @@ class PFSDatasetWorkflow:
             search_result = easy_market.search_learnware(user_info)
             single_result = search_result.get_single_results()
             multiple_result = search_result.get_multiple_results()
-            
+
             print(f"search result of user{idx}:")
             print(
                 f"single model num: {len(single_result)}, max_score: {single_result[0].score}, min_score: {single_result[-1].score}"

examples/dataset_pfs_workflow/upload.py

@@ -69,6 +69,7 @@ def main():
                 "Type": "String",
             },
             "Name": {"Values": name, "Type": "String"},
+            "License": {"Values": ["MIT"], "Type": "Class"},
         }
         res = session.post(
             submit_url,

examples/dataset_text_workflow/main.py

@@ -48,6 +48,7 @@ semantic_specs = [
         "Description": {"Values": "", "Type": "String"},
         "Name": {"Values": "learnware_1", "Type": "String"},
         "Output": output_description,
+        "License": {"Values": ["MIT"], "Type": "Class"},
     }
 ]
 
@@ -59,6 +60,7 @@ user_semantic = {
     "Description": {"Values": "", "Type": "String"},
     "Name": {"Values": "", "Type": "String"},
     "Output": output_description,
+    "License": {"Values": ["MIT"], "Type": "Class"},
 }
 
 
@@ -73,7 +75,6 @@ class TextDatasetWorkflow:
         generate_uploader(X_train, y_train, n_uploaders=n_uploaders, data_save_root=uploader_save_root)
         generate_user(X_test, y_test, n_users=n_users, data_save_root=user_save_root)
 
-
     def _prepare_model(self):
         dataloader = TextDataLoader(data_save_root, train=True)
         for i in range(n_uploaders):
@@ -92,9 +93,8 @@ class TextDatasetWorkflow:
 
             logger.info("Model saved to '%s' and '%s'" % (modelv_save_path, modell_save_path))
 
-
-    def _prepare_learnware(self, 
-        data_path, modelv_path, modell_path, init_file_path, yaml_path, env_file_path, save_root, zip_name
+    def _prepare_learnware(
+        self, data_path, modelv_path, modell_path, init_file_path, yaml_path, env_file_path, save_root, zip_name
     ):
         os.makedirs(save_root, exist_ok=True)
         tmp_spec_path = os.path.join(save_root, "rkme.json")
@@ -139,7 +139,6 @@ class TextDatasetWorkflow:
         logger.info("New Learnware Saved to %s" % (zip_file_name))
         return zip_file_name
 
-
     def prepare_market(self, regenerate_flag=False):
         if regenerate_flag:
             self._init_text_dataset()
@@ -175,7 +174,6 @@ class TextDatasetWorkflow:
 
         logger.info("Total Item: %d" % (len(text_market)))
 
-
     def test(self, regenerate_flag=False):
         self.prepare_market(regenerate_flag)
         text_market = instantiate_learnware_market(market_id="ae")
@@ -199,11 +197,11 @@ class TextDatasetWorkflow:
             user_stat_spec.generate_stat_spec_from_data(X=user_data)
             user_info = BaseUserInfo(semantic_spec=user_semantic, stat_info={"RKMETextSpecification": user_stat_spec})
             logger.info("Searching Market for user: %d" % (i))
-            
+
             search_result = text_market.search_learnware(user_info)
             single_result = search_result.get_single_results()
             multiple_result = search_result.get_multiple_results()
-            
+
             print(f"search result of user{i}:")
             print(
                 f"single model num: {len(single_result)}, max_score: {single_result[0].score}, min_score: {single_result[-1].score}"
@@ -220,7 +218,7 @@ class TextDatasetWorkflow:
             print(
                 f"Top1-score: {single_result[0].score}, learnware_id: {single_result[0].learnware.id}, acc: {acc_list[0]}"
             )
-            
+
             if len(multiple_result) > 0:
                 mixture_id = " ".join([learnware.id for learnware in multiple_result[0].learnwares])
                 print(f"mixture_score: {multiple_result[0].score}, mixture_learnware: {mixture_id}")

learnware/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.2.0.4"
+__version__ = "0.2.0.7"
 
 import os
 import json

learnware/client/container.py

@@ -43,19 +43,22 @@ class ModelContainer(BaseModel):
         """We must set `input_shape` and `output_shape`"""
         if self.build:
             self.cleanup_flag = True
-            self._init_env()
             atexit.register(self.remove_env)
-
+            self._init_env()
         self._setup_env_and_metadata()
 
     def remove_env(self):
         if self.cleanup_flag is True:
-            self.cleanup_flag = False
             try:
+                self.cleanup_flag = False
                 self._remove_env()
+            except KeyboardInterrupt:
+                self.cleanup_flag = True
+                logger.warning("The KeyboardInterrupt is ignored when removing the container env!")
+                self.remove_env()
             except Exception as err:
                 self.cleanup_flag = True
-                raise err
+                logger.error(f"Failed to remove container env due to {err}!")
 
     def _setup_env_and_metadata(self):
         raise NotImplementedError("_setup_env_and_metadata method is not implemented!")
@@ -83,9 +86,11 @@ class ModelCondaContainer(ModelContainer):
 
     def _init_env(self):
         install_environment(self.learnware_dirpath, self.conda_env)
+        logger.info(f"Conda env {self.conda_env} is generated.")
 
     def _remove_env(self):
         remove_enviroment(self.conda_env)
+        logger.info(f"Conda env {self.conda_env} is removed.")
 
     def _setup_env_and_metadata(self):
         with tempfile.TemporaryDirectory(prefix="learnware_") as tempdir:
@@ -106,7 +111,7 @@ class ModelCondaContainer(ModelContainer):
                     "-n",
                     f"{self.conda_env}",
                     "--no-capture-output",
-                    "python3",
+                    "python",
                     f"{self.model_script}",
                     "--model-path",
                     f"{model_path}",
@@ -149,7 +154,7 @@ class ModelCondaContainer(ModelContainer):
                     "-n",
                     f"{self.conda_env}",
                     "--no-capture-output",
-                    "python3",
+                    "python",
                     f"{self.model_script}",
                     "--model-path",
                     f"{model_path}",
@@ -198,6 +203,7 @@ class ModelDockerContainer(ModelContainer):
         self.conda_env = f"learnware_{shortuuid.uuid()}"
         self.docker_model_config = None
         self.docker_model_script_path = None
+        self.env_script = os.path.join(C.package_path, "client", "scripts", "install_env.py")
         # call init method of parent of parent class
         super(ModelDockerContainer, self).__init__(model_config, learnware_dirpath, build)
 
@@ -214,34 +220,70 @@ class ModelDockerContainer(ModelContainer):
             "tty": True,
             "command": "bash",
             "environment": {"http_proxy": http_proxy, "https_proxy": https_proxy},
+            "pids_limit": -1,
         }
         container = client.containers.run(**container_config)
-        environment_cmd = [
-            "pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple",
-            "conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
-            "conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
-            "conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
-            "conda config --set show_channel_urls yes",
-        ]
-        for _cmd in environment_cmd:
-            container.exec_run(_cmd)
+        logger.info(f"Docker container {container.id[:12]} is generated.")
+        
+        try:
+            environment_cmd = [
+                "pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple",
+                "conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
+                "conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
+                "conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
+                "conda config --set show_channel_urls yes",
+            ]
+            for _cmd in environment_cmd:
+                container.exec_run(_cmd)
+
+            logger.info("Install learnware package in docker.")
+            result = container.exec_run(
+                " ".join(
+                    [
+                        "conda",
+                        "run",
+                        "-n",
+                        "base",
+                        "--no-capture-output",
+                        "python",
+                        "-m",
+                        "pip",
+                        "install",
+                        "learnware",
+                    ]
+                )
+            )
+            if result.exit_code != 0:
+                logger.error(f"Install learnware package in docker failed!\n{result.output.decode('utf-8')}")
+        except KeyboardInterrupt:
+            logger.error("The container docker generation and setup procedure is KeyboardInterrupted!")
+            return container
         return container
 
     @staticmethod
-    def _destroy_docker_container(docker_container):
+    def _destroy_docker_container(docker_container) -> None:
         if isinstance(docker_container, docker.models.containers.Container):
             client = docker.from_env()
             container_ids = [container.id for container in client.containers.list()]
-
+            
             if docker_container.id in container_ids:
                 docker_container.stop()
                 docker_container.remove()
-                logger.info("Docker container is stopped and removed.")
+                logger.info(f"Docker container {docker_container.id[:12]} is stopped and removed.")
             else:
-                logger.info("Docker container has already been removed.")
+                logger.info(f"Docker container {docker_container.id[:12]} has already been removed.")
         else:
             logger.error("Type of docker_container is not docker.models.containers.Container.")
 
+    @staticmethod
+    def _change_path_to_container(path):
+        file_name = os.path.basename(path)
+        if "." in file_name:
+            file_dir = os.path.basename(os.path.dirname(path))
+            return f"/tmp/{file_dir}/{file_name}"
+        else:
+            return f"/tmp/{file_name}"
+
     def _copy_file_to_container(self, local_path, container_path):
         directory_path = os.path.dirname(container_path)
         container_name = os.path.basename(container_path)
@@ -270,13 +312,11 @@ class ModelDockerContainer(ModelContainer):
         except docker.errors.NotFound as err:
             logger.error(f"Copy file from container failed due to {err}")
 
-    def _install_environment(self, learnware_dirpath, conda_env):
+    def _install_environment(self, conda_env):
         """Install environment of a learnware in docker container
 
         Parameters
         ----------
-        learnware_dirpath : str
-            Path of the learnware dir
         conda_env : str
             a new conda environment will be created with the given name
 
@@ -286,117 +326,63 @@ class ModelDockerContainer(ModelContainer):
             Lack of the environment configuration file.
         """
         run_cmd_times = 10
+        self.learnware_dirpath_container = self._change_path_to_container(self.learnware_dirpath)
+        self._copy_file_to_container(self.learnware_dirpath, self.learnware_dirpath_container)
+
         with tempfile.TemporaryDirectory(prefix="learnware_") as tempdir:
+            env_script_container_path = self._change_path_to_container(os.path.join(tempdir, "install_env.py"))
+            self._copy_file_to_container(self.env_script, env_script_container_path)
+
             success_flag = False
-            logger.info(f"learnware_dir namelist: {os.listdir(learnware_dirpath)}")
-
-            if "environment.yaml" in os.listdir(learnware_dirpath):
-                yaml_path: str = os.path.join(learnware_dirpath, "environment.yaml")
-                yaml_path_filter: str = os.path.join(tempdir, "environment_filter.yaml")
-                logger.info(f"checking the avaliabe conda packages for {conda_env}")
-                filter_nonexist_conda_packages_file(yaml_file=yaml_path, output_yaml_file=yaml_path_filter)
-                self._copy_file_to_container(yaml_path_filter, yaml_path_filter)
-
-                # create environment
-                logger.info(f"Create and update conda env [{conda_env}] according to .yaml file")
-                for i in range(run_cmd_times):
-                    result = self.docker_container.exec_run(
-                        " ".join(["conda", "env", "create", "--name", f"{conda_env}", "--file", f"{yaml_path_filter}"])
+            logger.info("Install environment dependencies in docker.")
+            for i in range(run_cmd_times):
+                result = self.docker_container.exec_run(
+                    " ".join(
+                        [
+                            "conda",
+                            "run",
+                            "-n",
+                            "base",
+                            "--no-capture-output",
+                            "python",
+                            f"{env_script_container_path}",
+                            "--learnware-dirpath",
+                            f"{self.learnware_dirpath_container}",
+                            "--conda-env",
+                            f"{conda_env}",
+                        ]
                     )
-                    if result.exit_code == 0:
-                        success_flag = True
-                        break
-
-            elif "requirements.txt" in os.listdir(learnware_dirpath):
-                requirements_path: str = os.path.join(learnware_dirpath, "requirements.txt")
-                requirements_path_filter: str = os.path.join(tempdir, "requirements_filter.txt")
-                logger.info(f"checking the avaliabe pip packages for {conda_env}.")
-                filter_nonexist_pip_packages_file(
-                    requirements_file=requirements_path, output_file=requirements_path_filter
                 )
-                logger.info(f"Create empty conda env [{conda_env}] in docker.")
-                for i in range(run_cmd_times):
-                    result = self.docker_container.exec_run(
-                        " ".join(["conda", "create", "-y", "--name", f"{conda_env}", "python=3.8"])
-                    )
-                    if result.exit_code == 0:
-                        break
-                logger.info(f"install pip requirements for conda env [{conda_env}] in docker.")
-
-                self._copy_file_to_container(requirements_path_filter, requirements_path_filter)
-                for i in range(run_cmd_times):
-                    result = self.docker_container.exec_run(
-                        " ".join(
-                            [
-                                "conda",
-                                "run",
-                                "-n",
-                                f"{conda_env}",
-                                "--no-capture-output",
-                                "python3",
-                                "-m",
-                                "pip",
-                                "install",
-                                "-r",
-                                f"{requirements_path_filter}",
-                            ]
-                        )
-                    )
-                    if result.exit_code == 0:
-                        success_flag = True
-                        break
-            else:
-                raise Exception("Environment.yaml or requirements.txt not found in the learnware zip file.")
-
+                if result.exit_code == 0:
+                    success_flag = True
+                    break
+                else:
+                    self.docker_container.exec_run("conda clean --all")
             if not success_flag:
-                logger.error(f"Install conda env [{conda_env}] in docker failed!")
-
-        success_flag = False
-        logger.info(f"Install learnware package for conda env [{conda_env}] in docker.")
-        for i in range(run_cmd_times):
-            result = self.docker_container.exec_run(
-                " ".join(
-                    [
-                        "conda",
-                        "run",
-                        "-n",
-                        f"{conda_env}",
-                        "--no-capture-output",
-                        "python3",
-                        "-m",
-                        "pip",
-                        "install",
-                        "learnware",
-                    ]
-                )
-            )
-            if result.exit_code == 0:
-                success_flag = True
-                break
-
-        if not success_flag:
-            logger.error(f"Install learnware package for conda env [{conda_env}] in docker failed!")
+                logger.error(f"Install environment dependencies in docker failed!\n{result.output.decode('utf-8')}")
 
     def _setup_env_and_metadata(self):
         """setup env and set the input and output shape by communicating with docker"""
-        self._install_environment(self.learnware_dirpath, self.conda_env)
+        self._install_environment(self.conda_env)
         with tempfile.TemporaryDirectory(prefix="learnware_") as tempdir:
             output_path = os.path.join(tempdir, "output.pkl")
             model_path = os.path.join(tempdir, "model.pkl")
-            self.docker_model_script_path = os.path.join(tempdir, "run_model.py")
+            self.docker_model_script_path = self._change_path_to_container(os.path.join(tempdir, "run_model.py"))
+
+            output_path_container = self._change_path_to_container(output_path)
+            model_path_container = self._change_path_to_container(model_path)
 
             docker_model_config = self.model_config.copy()
-            docker_model_config["module_path"] = Learnware.get_model_module_abspath(
-                self.learnware_dirpath, docker_model_config["module_path"]
+            docker_model_config["module_path"] = (
+                self.learnware_dirpath_container + "/" + docker_model_config["module_path"]
             )
-            self._copy_file_to_container(self.learnware_dirpath, self.learnware_dirpath)
 
             with open(model_path, "wb") as model_fp:
                 pickle.dump(docker_model_config, model_fp)
-
-            self._copy_file_to_container(model_path, model_path)
+            self._copy_file_to_container(model_path, model_path_container)
             self._copy_file_to_container(self.model_script, self.docker_model_script_path)
-            self.docker_container.exec_run(
+
+            result = self.docker_container.exec_run(
                 " ".join(
                     [
                         "conda",
@@ -404,16 +390,18 @@ class ModelDockerContainer(ModelContainer):
                         "-n",
                         f"{self.conda_env}",
                         "--no-capture-output",
-                        "python3",
+                        "python",
                         f"{self.docker_model_script_path}",
                         "--model-path",
-                        f"{model_path}",
+                        f"{model_path_container}",
                         "--output-path",
-                        f"{output_path}",
+                        f"{output_path_container}",
                     ]
                 )
             )
-            self._copy_file_from_container(output_path, output_path)
+            if result.exit_code != 0:
+                logger.error(f"Instantiate learnware in docker failed!\n{result.output.decode('utf-8')}")
+            self._copy_file_from_container(output_path_container, output_path)
 
             with open(output_path, "rb") as output_fp:
                 output_results = pickle.load(output_fp)
@@ -437,9 +425,13 @@ class ModelDockerContainer(ModelContainer):
             output_path = os.path.join(tempdir, "output.pkl")
             model_path = os.path.join(tempdir, "model.pkl")
 
+            input_path_container = self._change_path_to_container(input_path)
+            output_path_container = self._change_path_to_container(output_path)
+            model_path_container = self._change_path_to_container(model_path)
+
             docker_model_config = self.model_config.copy()
-            docker_model_config["module_path"] = Learnware.get_model_module_abspath(
-                self.learnware_dirpath, docker_model_config["module_path"]
+            docker_model_config["module_path"] = (
+                self.learnware_dirpath_container + "/" + docker_model_config["module_path"]
             )
             with open(model_path, "wb") as model_fp:
                 pickle.dump(docker_model_config, model_fp)
@@ -447,10 +439,10 @@ class ModelDockerContainer(ModelContainer):
             with open(input_path, "wb") as input_fp:
                 pickle.dump({"method": method, "kargs": kargs}, input_fp)
 
-            self._copy_file_to_container(model_path, model_path)
-            self._copy_file_to_container(input_path, input_path)
+            self._copy_file_to_container(model_path, model_path_container)
+            self._copy_file_to_container(input_path, input_path_container)
 
-            self.docker_container.exec_run(
+            result = self.docker_container.exec_run(
                 " ".join(
                     [
                         "conda",
@@ -458,18 +450,20 @@ class ModelDockerContainer(ModelContainer):
                         "-n",
                         f"{self.conda_env}",
                         "--no-capture-output",
-                        "python3",
+                        "python",
                         f"{self.docker_model_script_path}",
                         "--model-path",
-                        f"{model_path}",
+                        f"{model_path_container}",
                         "--input-path",
-                        f"{input_path}",
+                        f"{input_path_container}",
                         "--output-path",
-                        f"{output_path}",
+                        f"{output_path_container}",
                     ]
                 )
             )
-            self._copy_file_from_container(output_path, output_path)
+            if result.exit_code != 0:
+                logger.error(f"Run learnware in docker failed!\n{result.output.decode('utf-8')}")
+            self._copy_file_from_container(output_path_container, output_path)
 
             with open(output_path, "rb") as output_fp:
                 output_results = pickle.load(output_fp)
@@ -520,6 +514,14 @@ class LearnwaresContainer:
         self.cleanup = cleanup
         self.ignore_error = ignore_error
 
+    def _destroy_docker_container(self):
+        try:
+            if self._docker_container is not None:
+                ModelDockerContainer._destroy_docker_container(self._docker_container)
+        except KeyboardInterrupt:
+            logger.warning("The KeyboardInterrupt is ignored when removing the container env!")
+            self._destroy_docker_container()
+            
     def __enter__(self):
         if self.mode == "conda":
             self.learnware_containers = [
@@ -532,6 +534,8 @@ class LearnwaresContainer:
                 for _learnware in self.learnware_list
             ]
         else:
+            self._docker_container = None
+            atexit.register(self._destroy_docker_container)
             self._docker_container = ModelDockerContainer._generate_docker_container()
             self.learnware_containers = [
                 Learnware(
@@ -544,23 +548,28 @@ class LearnwaresContainer:
                 )
                 for _learnware in self.learnware_list
             ]
-            atexit.register(ModelDockerContainer._destroy_docker_container, self._docker_container)
 
         model_list = [_learnware.get_model() for _learnware in self.learnware_containers]
-        with ThreadPoolExecutor(max_workers=max(os.cpu_count() // 2, 1)) as executor:
-            results = executor.map(self._initialize_model_container, model_list, [self.ignore_error] * len(model_list))
-        self.results = list(results)
+        if self.mode == "conda":
+            with ThreadPoolExecutor(max_workers=max(os.cpu_count() // 2, 1)) as executor:
+                results = executor.map(
+                    self._initialize_model_container, model_list, [self.ignore_error] * len(model_list)
+                )
+            self.results = list(results)
+        else:
+            self.results = []
+            for model_item in model_list:
+                self.results.append(self._initialize_model_container(model_item, self.ignore_error))
 
         if sum(self.results) < len(self.learnware_list):
             logger.warning(
-                f"{len(self.learnware_list) - sum(results)} of {len(self.learnware_list)} learnwares init failed! This learnware will be ignored"
+                f"{len(self.learnware_list) - sum(results)} of {len(self.learnware_list)} learnwares init failed! These learnwares will be ignored"
             )
 
         return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         if not self.cleanup:
-            logger.warning(f"Notice, the learnware container env is not cleaned up!")
             self.learnware_containers = None
             self.results = None
             return

learnware/client/learnware_client.py

@@ -12,11 +12,9 @@ from tqdm import tqdm
 from typing import Union, List, Optional
 
 from ..config import C
-from .. import learnware
 from .container import LearnwaresContainer
-from ..market import BaseChecker, EasySemanticChecker, EasyStatChecker
+from ..market import BaseChecker
 from ..logger import get_module_logger
-from ..specification import Specification
 from ..learnware import get_learnware_from_dirpath
 from ..market import BaseUserInfo
 from ..tests import get_semantic_specification
@@ -54,6 +52,7 @@ class SemanticSpecificationKey(Enum):
     DATA_TYPE = "Data"
     TASK_TYPE = "Task"
     LIBRARY_TYPE = "Library"
+    LICENSE = "License"
     SENARIOES = "Scenario"
 
 
@@ -217,55 +216,59 @@ class LearnwareClient:
         else:
             stat_spec = None
 
-        returns = []
-        with tempfile.NamedTemporaryFile(prefix="learnware_stat_", suffix=".json") as ftemp:
+        returns = {
+            "single": {
+                "learnware_ids": [],
+                "semantic_specifications": [],
+                "matching": [],
+            },
+            "multiple": {
+                "learnware_ids": [],
+                "semantic_specifications": [],
+                "matching": None,
+            },
+        }
+        with tempfile.NamedTemporaryFile(prefix="learnware_stat_", suffix=".json", delete=False) as ftemp:
+            temp_file_name = ftemp.name
             if stat_spec is not None:
-                stat_spec.save(ftemp.name)
-
-            with open(ftemp.name, "r") as fin:
-                semantic_specification = user_info.get_semantic_spec()
-                if stat_spec is None:
-                    files = None
-                else:
-                    files = {"statistical_specification": fin}
-
-                response = requests.post(
-                    url,
-                    files=files,
-                    data={
-                        "semantic_specification": json.dumps(semantic_specification),
-                        "limit": page_size,
-                        "page": page_index,
-                    },
-                    headers=self.headers,
-                )
-
-                result = response.json()
-
-                if result["code"] != 0:
-                    raise Exception("search failed: " + json.dumps(result))
-
-                for learnware in result["data"]["learnware_list_single"]:
-                    returns.append(
-                        {   
-                            "type": "single",
-                            "learnware_id": learnware["learnware_id"],
-                            "semantic_specification": learnware["semantic_specification"],
-                            "matching": learnware["matching"],
-                        }
-                    )
-                if len(result["data"]["learnware_list_multi"]) > 0:
-                    multiple_learnware = {
-                        "type": "multiple",
-                        "learnware_ids": [],
-                        "semantic_specifications": [],
-                        "matching": result["data"]["learnware_list_multi"][0]["matching"]
-                    }
-                    for learnware in result["data"]["learnware_list_multi"]:
-                        multiple_learnware["learnware_ids"].append(learnware["learnware_id"])
-                        multiple_learnware["semantic_specifications"].append(learnware["semantic_specification"])
-                        
-                    returns.append(multiple_learnware)
+                stat_spec.save(temp_file_name)
+
+        with open(temp_file_name, "r") as fin:
+            semantic_specification = user_info.get_semantic_spec()
+            if stat_spec is None:
+                files = None
+            else:
+                files = {"statistical_specification": fin}
+
+            response = requests.post(
+                url,
+                files=files,
+                data={
+                    "semantic_specification": json.dumps(semantic_specification),
+                    "limit": page_size,
+                    "page": page_index,
+                },
+                headers=self.headers,
+            )
+            result = response.json()
+
+            if result["code"] != 0:
+                raise Exception("search failed: " + json.dumps(result))
+
+            for learnware in result["data"]["learnware_list_single"]:
+                returns["single"]["learnware_ids"].append(learnware["learnware_id"])
+                returns["single"]["semantic_specifications"].append(learnware["semantic_specification"])
+                returns["single"]["matching"].append(learnware["matching"])
+
+            if len(result["data"]["learnware_list_multi"]) > 0:
+                multi_learnware = result["data"]["learnware_list_multi"][0]
+                returns["multiple"]["learnware_ids"].append(multi_learnware["learnware_id"])
+                returns["multiple"]["semantic_specifications"].append(multi_learnware["semantic_specification"])
+                returns["multiple"]["matching"] = learnware["matching"]
+        
+        # Delete temp json file
+        os.remove(temp_file_name)
+
         return returns
 
     @require_login
@@ -280,14 +283,15 @@ class LearnwareClient:
 
     def create_semantic_specification(
         self,
-        name=None,
-        description=None,
-        data_type=None,
-        task_type=None,
-        library_type=None,
-        scenarios=None,
-        input_description=None,
-        output_description=None,
+        name: Optional[str] = None,
+        description: Optional[str] = None,
+        data_type: Optional[str] = None,
+        task_type: Optional[str] = None,
+        library_type: Optional[str] = None,
+        scenarios: Optional[Union[str, List[str]]] = None,
+        license: Optional[Union[str, List[str]]] = None,
+        input_description: Optional[dict] = None,
+        output_description: Optional[dict] = None,
     ):
         semantic_specification = dict()
         semantic_specification["Data"] = {"Type": "Class", "Values": [data_type] if data_type is not None else []}
@@ -296,7 +300,12 @@ class LearnwareClient:
             "Type": "Class",
             "Values": [library_type] if library_type is not None else [],
         }
+
+        license = [license] if isinstance(license, str) else license
+        semantic_specification["License"] = {"Type": "Class", "Values": license if license is not None else []}
+        scenarios = [scenarios] if isinstance(scenarios, str) else scenarios
         semantic_specification["Scenario"] = {"Type": "Tag", "Values": scenarios if scenarios is not None else []}
+
         semantic_specification["Name"] = {"Type": "String", "Values": name if name is not None else ""}
         semantic_specification["Description"] = {
             "Type": "String",
@@ -377,7 +386,7 @@ class LearnwareClient:
                 with open(os.path.join(tempdir, semantic_file), "r") as fin:
                     semantic_specification = json.load(fin)
 
-            return learnware.get_learnware_from_dirpath(learnware_id, semantic_specification, tempdir)
+            return get_learnware_from_dirpath(learnware_id, semantic_specification, tempdir)
 
         learnware_list = []
         if learnware_path is not None:
@@ -410,12 +419,14 @@ class LearnwareClient:
 
     @staticmethod
     def _check_semantic_specification(semantic_spec):
+        from ..market import EasySemanticChecker
+
         check_status, message = EasySemanticChecker.check_semantic_spec(semantic_spec)
         return check_status != BaseChecker.INVALID_LEARNWARE, message
 
     @staticmethod
     def _check_stat_specification(learnware):
-        from ..market import CondaChecker
+        from ..market import EasyStatChecker, CondaChecker
 
         stat_checker = CondaChecker(inner_checker=EasyStatChecker())
         check_status, message = stat_checker(learnware)

learnware/client/package_utils.py

@@ -6,6 +6,7 @@ import tempfile
 import subprocess
 from typing import List, Tuple
 from . import utils
+from concurrent.futures import ThreadPoolExecutor
 
 
 from ..logger import get_module_logger
@@ -13,38 +14,38 @@ from ..logger import get_module_logger
 logger = get_module_logger("package_utils")
 
 
-def try_to_run(args, timeout=5, retry=5):
-    sucess = False
+def try_to_run(args, timeout=10, retry=3):
     for i in range(retry):
         try:
-            utils.system_execute(args=args, timeout=timeout, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-            sucess = True
-            break
+            result = utils.system_execute(args=args, timeout=timeout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+            return result.stdout.decode()
         except subprocess.TimeoutExpired as e:
             pass
 
-    if not sucess:
-        raise subprocess.TimeoutExpired(args, timeout)
+    raise subprocess.TimeoutExpired(args, timeout)
 
 
 def parse_pip_requirement(line: str):
-    """Parse pip requirement line to package name"""
+    """Parse pip requirement line to package name and version"""
 
     line = line.strip()
+    if len(line) == 0 or line[0] in ("#", "-"):
+        return None, None
 
-    if len(line) == 0:
-        return None
-
-    if line[0] in ("#", "-"):
-        return None
+    package_name, package_version = line, line
+    for split_ch in ("=", ">", "<", "!", "~", " ", "="):
+        split_ch_index = package_name.find(split_ch)
+        if split_ch_index != -1:
+            package_name = package_name[:split_ch_index]
 
-    package_str = line
-    for split_ch in ("=", ">", "<", "!", "~", " "):
-        split_ch_index = package_str.find(split_ch)
+        split_ch_index = package_version.find(split_ch)
         if split_ch_index != -1:
-            package_str = package_str[:split_ch_index]
+            package_version = package_version[split_ch_index + 1 :]
 
-    return package_str
+    if package_version == package_name:
+        package_version = ""
+
+    return package_name, package_version
 
 
 def read_pip_packages_from_requirements(requirements_file: str) -> Tuple[List[str], List[str]]:
@@ -54,7 +55,7 @@ def read_pip_packages_from_requirements(requirements_file: str) -> Tuple[List[st
     lines = []
     with open(requirements_file, "r") as fin:
         for line in fin:
-            package_str = parse_pip_requirement(line)
+            package_str, package_version = parse_pip_requirement(line)
             packages.append(package_str)
             lines.append(line)
 
@@ -71,21 +72,38 @@ def filter_nonexist_pip_packages(packages: list) -> Tuple[List[str], List[str]]:
         nonexist_packages: list of non-exist packages
     """
 
-    exist_packages = []
-    nonexist_packages = []
-    for package in packages:
-        if package is None:
-            continue
+    def _filter_nonexist_pip_package_worker(package):
+        # Return filtered package
         try:
-            package_name = parse_pip_requirement(package)
+            package_name, package_version = parse_pip_requirement(package)
             if package_name is not None and package_name != "learnware":
-                try_to_run(args=["pip", "index", "versions", package_name], timeout=5)
-                exist_packages.append(package)
-                continue
-        except Exception as e:
-            logger.error(e)
-        nonexist_packages.append(package)
+                result = try_to_run(args=["pip", "index", "versions", package_name], timeout=10)
+                if len(package_version) and package_version not in result:
+                    return package_name
+                else:
+                    return package
+        except subprocess.CalledProcessError:
+            pass
+        except Exception as err:
+            logger.error(err)
+            
+        return None
+
+    exist_packages = []
+    nonexist_packages = []
+    packages = [package for package in packages if package is not None]
+
+    with ThreadPoolExecutor(max_workers=max(os.cpu_count() // 5, 1)) as executor:
+        results = executor.map(_filter_nonexist_pip_package_worker, packages)
 
+    for result, package in zip(list(results), packages):
+        if result is not None:
+            exist_packages.append(result)
+        else:
+            nonexist_packages.append(package)
+    
+    if len(nonexist_packages) > 0:
+        logger.info(f"Filtered out {len(nonexist_packages)} non-exist pip packages.")
     return exist_packages, nonexist_packages
 
 
@@ -116,7 +134,14 @@ def filter_nonexist_conda_packages(packages: list) -> Tuple[List[str], List[str]
             last_bracket = stdout.rfind("\n{")
             if last_bracket != -1:
                 stdout = stdout[last_bracket:]
-            return json.loads(stdout).get("bad_deps", [])
+
+            stdout_json = json.loads(stdout)
+            if "error" in stdout_json:
+                if "bad_deps" in stdout_json:
+                    return stdout_json["bad_deps"]
+                elif "packages" in stdout_json:
+                    return stdout_json["packages"]
+            return []
 
     org_yaml = {
         "channels": ["defaults"],
@@ -169,7 +194,6 @@ def read_conda_packages_from_dict(env_desc: dict) -> Tuple[List[str], List[str]]
         for package in conda_packages:
             if isinstance(package, dict) and "pip" in package:
                 pip_packages = package["pip"]
-                # pip_packages = [parse_pip_requirement(line) for line in pip_packages]
             elif isinstance(package, str):
                 conda_packages_.append(package)
 

learnware/client/scripts/install_env.py

@@ -0,0 +1,15 @@
+import argparse
+from learnware.client.utils import install_environment
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--learnware-dirpath", type=str, required=True, help="path of learnware dir")
+    parser.add_argument("--conda-env", type=str, required=False, help="name of conda env")
+
+    args = parser.parse_args()
+
+    learnware_dirpath = args.learnware_dirpath
+    conda_env = args.conda_env
+
+    install_environment(learnware_dirpath, conda_env)

learnware/client/utils.py

@@ -10,24 +10,20 @@ logger = get_module_logger(module_name="client_utils")
 
 
 def system_execute(args, timeout=None, env=None, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE):
-    if env is None:
-        env = os.environ.copy()
-        pass
-
-    if isinstance(args, str):
-        pass
-    else:
-        args = " ".join(args)
-        pass
+    env = os.environ.copy() if env is None else env
+    args = args if isinstance(args, str) else " ".join(args)
 
     com_process = subprocess.run(args, stdout=stdout, stderr=stderr, timeout=timeout, env=env, shell=True)
 
     try:
         com_process.check_returncode()
     except subprocess.CalledProcessError as err:
-        errmsg = com_process.stderr.decode()
-        logger.warning(f"System Execute Error: {errmsg}")
-        raise Exception(errmsg)
+        if err.stderr is not None:
+            errmsg = err.stderr.decode()
+            logger.warning(f"System Execute Error: {errmsg}")
+        raise err
+    
+    return com_process
 
 
 def remove_enviroment(conda_env):
@@ -76,7 +72,7 @@ def install_environment(learnware_dirpath, conda_env):
                     "-n",
                     f"{conda_env}",
                     "--no-capture-output",
-                    "python3",
+                    "python",
                     "-m",
                     "pip",
                     "install",
@@ -95,7 +91,7 @@ def install_environment(learnware_dirpath, conda_env):
             "-n",
             f"{conda_env}",
             "--no-capture-output",
-            "python3",
+            "python",
             "-m",
             "pip",
             "install",

learnware/config.py

@@ -128,6 +128,25 @@ semantic_config = {
         "Values": None,
         "Type": "String",
     },
+    "License": {
+        "Values": [
+            "MIT",
+            "Apache-2.0",
+            "BSD-2-Clause",
+            "BSD-3-Clause",
+            "GPL-2.0",
+            "GPL-3.0",
+            "LGPL-2.1",
+            "LGPL-3.0",
+            "AGPL-3.0",
+            "ECL-2.0",
+            "AFL-3.0",
+            "CC-BY-4.0",
+            "CC-BY-SA-4.0",
+            "Others",
+        ],
+        "Type": "Class",
+    },
 }
 
 _DEFAULT_CONFIG = {

learnware/learnware/utils.py

@@ -44,7 +44,6 @@ def get_stat_spec_from_config(stat_spec: dict) -> BaseStatSpecification:
         raise TypeError(
             f"Statistic specification must be type of BaseStatSpecification, not {BaseStatSpecification.__class__.__name__}"
         )
-    if stat_spec_inst.load(stat_spec["file_name"]) is False:
-        raise ValueError("Load statistic specification failed!")
-
+    stat_spec_inst.load(stat_spec["file_name"])
+    
     return stat_spec_inst

learnware/market/anchor/organizer.py

@@ -1,9 +1,8 @@
-from typing import List, Dict, Tuple, Any
+from typing import Dict
 
 from ..easy.organizer import EasyOrganizer
 from ...logger import get_module_logger
 from ...learnware import Learnware
-from ...specification import BaseStatSpecification
 
 logger = get_module_logger("anchor_organizer")
 

learnware/market/anchor/searcher.py

@@ -1,7 +1,6 @@
-from typing import List, Dict, Tuple, Any, Union
+from typing import List, Tuple, Any
 
 from .user_info import AnchoredUserInfo
-from ..base import BaseUserInfo
 from ..easy.searcher import EasySearcher
 from ...logger import get_module_logger
 from ...learnware import Learnware

learnware/market/base.py

@@ -3,7 +3,7 @@ from __future__ import annotations
 import traceback
 import zipfile
 import tempfile
-from typing import Tuple, Any, List, Union, Dict, Optional
+from typing import Tuple, Any, List, Union, Optional
 from dataclasses import dataclass
 from ..learnware import Learnware, get_learnware_from_dirpath
 from ..logger import get_module_logger
@@ -45,7 +45,7 @@ class BaseUserInfo:
 
     def update_semantic_spec(self, semantic_spec: dict):
         self.semantic_spec = semantic_spec
-        
+
     def update_stat_info(self, name: str, item: Any):
         """Update stat_info by market
 
@@ -64,28 +64,35 @@ class SingleSearchItem:
     learnware: Learnware
     score: Optional[float] = None
 
+
 @dataclass
 class MultipleSearchItem:
     learnwares: List[Learnware]
     score: float
-    
+
+
 class SearchResults:
-    def __init__(self, single_results: Optional[List[SingleSearchItem]] = None, multiple_results: Optional[List[MultipleSearchItem]] = None):
+    def __init__(
+        self,
+        single_results: Optional[List[SingleSearchItem]] = None,
+        multiple_results: Optional[List[MultipleSearchItem]] = None,
+    ):
         self.update_single_results([] if single_results is None else single_results)
         self.update_multiple_results([] if multiple_results is None else multiple_results)
-        
+
     def get_single_results(self) -> List[SingleSearchItem]:
         return self.single_results
-    
+
     def get_multiple_results(self) -> List[MultipleSearchItem]:
         return self.multiple_results
-    
+
     def update_single_results(self, single_results: List[SingleSearchItem]):
         self.single_results = single_results
-    
+
     def update_multiple_results(self, multiple_results: List[MultipleSearchItem]):
         self.multiple_results = multiple_results
 
+
 class LearnwareMarket:
     """Base interface for market, it provide the interface of search/add/detele/update learnwares"""
 
@@ -179,9 +186,7 @@ class LearnwareMarket:
             zip_path=zip_path, semantic_spec=semantic_spec, check_status=check_status, **kwargs
         )
 
-    def search_learnware(
-        self, user_info: BaseUserInfo, check_status: int = None, **kwargs
-    ) -> SearchResults:
+    def search_learnware(self, user_info: BaseUserInfo, check_status: int = None, **kwargs) -> SearchResults:
         """Search learnwares based on user_info from learnwares with check_status
 
         Parameters

learnware/market/easy/checker.py

@@ -57,7 +57,7 @@ class EasySemanticChecker(BaseChecker):
 
             return EasySemanticChecker.NONUSABLE_LEARNWARE, "EasySemanticChecker Success"
 
-        except AssertionError as err:
+        except Exception as err:
             logger.warning(f"semantic_specification is not valid due to {err}!")
             return EasySemanticChecker.INVALID_LEARNWARE, traceback.format_exc()
 
@@ -95,8 +95,8 @@ class EasyStatChecker(BaseChecker):
             logger.warning(f"The learnware [{learnware.id}] is instantiated failed! Due to {e}.")
             return self.INVALID_LEARNWARE, traceback.format_exc()
         try:
-            learnware_model = learnware.get_model()
             # Check input shape
+            learnware_model = learnware.get_model()
             input_shape = learnware_model.input_shape
 
             if semantic_spec["Data"]["Values"][0] == "Table" and input_shape != (
@@ -106,14 +106,22 @@ class EasyStatChecker(BaseChecker):
                 logger.warning(message)
                 return self.INVALID_LEARNWARE, message
 
+            # Check statistical specification
             spec_type = parse_specification_type(learnware.get_specification().stat_spec)
             if spec_type is None:
                 message = f"No valid specification is found in stat spec {spec_type}"
                 logger.warning(message)
                 return self.INVALID_LEARNWARE, message
 
+            # Check if statistical specification is computable in dist()
+            stat_spec = learnware.get_specification().get_stat_spec_by_name(spec_type)
+            distance = float(stat_spec.dist(stat_spec))
+            if not np.isfinite(distance):
+                message = f"The distance between statistical specifications is not finite, where distance={distance}"
+                logger.warning(message)
+                return self.INVALID_LEARNWARE, message
+
             if spec_type == "RKMETableSpecification":
-                stat_spec = learnware.get_specification().get_stat_spec_by_name(spec_type)
                 if not isinstance(input_shape, tuple) or not all(isinstance(item, int) for item in input_shape):
                     raise ValueError(
                         f"For RKMETableSpecification, input_shape should be tuple of int, but got {input_shape}"
@@ -124,14 +132,17 @@ class EasyStatChecker(BaseChecker):
                     logger.warning(message)
                     return self.INVALID_LEARNWARE, message
                 inputs = np.random.randn(10, *input_shape)
+
             elif spec_type == "RKMETextSpecification":
                 inputs = EasyStatChecker._generate_random_text_list(10)
+
             elif spec_type == "RKMEImageSpecification":
                 if not isinstance(input_shape, tuple) or not all(isinstance(item, int) for item in input_shape):
                     raise ValueError(
                         f"For RKMEImageSpecification, input_shape should be tuple of int, but got {input_shape}"
                     )
                 inputs = np.random.randint(0, 255, size=(10, *input_shape))
+
             else:
                 raise ValueError(f"not supported spec type for spec_type = {spec_type}")
 

learnware/market/easy/database_ops.py

@@ -1,6 +1,6 @@
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy import create_engine, text
-from sqlalchemy import Column, Integer, Text, DateTime, String
+from sqlalchemy import Column, Text, String
 import os
 import json
 import traceback
@@ -177,6 +177,11 @@ class DatabaseOperations(object):
                 id = id.strip()
                 try:
                     semantic_spec_dict = json.loads(semantic_spec)
+                    if "License" not in semantic_spec_dict:
+                        semantic_spec_dict["License"] = {
+                            "Values": ["Apache-2.0"],
+                            "Type": "Class",
+                        }
                     new_learnware = get_learnware_from_dirpath(
                         id=id, semantic_spec=semantic_spec_dict, learnware_dirpath=folder_path, ignore_error=False
                     )

learnware/market/easy/searcher.py

@@ -45,11 +45,9 @@ class EasyExactSemanticSearcher(BaseSearcher):
                     return False
 
                 if semantic_spec1[key]["Type"] == "Class":
-                    if isinstance(v1, list):
-                        v1 = v1[0]
                     if isinstance(v2, list):
                         v2 = v2[0]
-                    if v1 != v2:
+                    if v2 not in v1:
                         return False
                 elif semantic_spec1[key]["Type"] == "Tag":
                     if not (set(v1) & set(v2)):
@@ -67,6 +65,7 @@ class EasyExactSemanticSearcher(BaseSearcher):
         logger.info("semantic_spec search: choose %d from %d learnwares" % (len(match_learnwares), len(learnware_list)))
         return SearchResults(single_results=[SingleSearchItem(learnware=_learnware) for _learnware in match_learnwares])
 
+
 class EasyFuzzSemanticSearcher(BaseSearcher):
     def _match_semantic_spec_tag(self, semantic_spec1, semantic_spec2) -> bool:
         """Judge if tags of two semantic specs are consistent
@@ -85,23 +84,19 @@ class EasyFuzzSemanticSearcher(BaseSearcher):
         """
         for key in semantic_spec1.keys():
             v1 = semantic_spec1[key].get("Values", "")
-            v2 = semantic_spec2[key].get("Values", "")
-
-            if len(v1) == 0:
-                # user input is empty, no need to search
+            if key not in semantic_spec2 or len(v1) == 0:
                 continue
 
-            if key not in "Name":
+            v2 = semantic_spec2[key].get("Values", "")
+            if key not in ("Name", "Description"):
                 if len(v2) == 0:
                     # user input contains some key that is not in database
                     return False
 
                 if semantic_spec1[key]["Type"] == "Class":
-                    if isinstance(v1, list):
-                        v1 = v1[0]
                     if isinstance(v2, list):
                         v2 = v2[0]
-                    if v1 != v2:
+                    if v2 not in v1:
                         return False
                 elif semantic_spec1[key]["Type"] == "Tag":
                     if not (set(v1) & set(v2)):
@@ -543,14 +538,20 @@ class EasyStatSearcher(BaseSearcher):
             both lists are sorted by mmd dist
         """
         rkme_list = [learnware.specification.get_stat_spec_by_name(self.stat_spec_type) for learnware in learnware_list]
-        mmd_dist_list = []
-        for rkme in rkme_list:
-            mmd_dist = rkme.dist(user_rkme)
-            mmd_dist_list.append(mmd_dist)
+        filtered_idx_list, mmd_dist_list = [], []
+        for idx in range(len(rkme_list)):
+            mmd_dist = float(rkme_list[idx].dist(user_rkme))
+            if np.isfinite(mmd_dist):
+                mmd_dist_list.append(mmd_dist)
+                filtered_idx_list.append(idx)
+            else:
+                logger.warning(
+                    f"The distance between user_spec and learnware_spec (id: {learnware_list[idx].id}) is not finite, where distance is {mmd_dist}"
+                )
 
-        sorted_idx_list = sorted(range(len(learnware_list)), key=lambda k: mmd_dist_list[k])
+        sorted_idx_list = sorted(range(len(mmd_dist_list)), key=lambda k: mmd_dist_list[k])
         sorted_dist_list = [mmd_dist_list[idx] for idx in sorted_idx_list]
-        sorted_learnware_list = [learnware_list[idx] for idx in sorted_idx_list]
+        sorted_learnware_list = [learnware_list[filtered_idx_list[idx]] for idx in sorted_idx_list]
 
         return sorted_dist_list, sorted_learnware_list
 
@@ -566,6 +567,9 @@ class EasyStatSearcher(BaseSearcher):
             raise KeyError("No supported stat specification is given in the user info")
 
         user_rkme = user_info.stat_info[self.stat_spec_type]
+        if not np.isfinite(float(user_rkme.dist(user_rkme))):
+            raise ValueError("The distance between uploaded statistical specifications is not finite!")
+
         learnware_list = self._filter_by_rkme_spec_metadata(learnware_list, user_rkme)
         logger.info(f"After filter by rkme dimension, learnware_list length is {len(learnware_list)}")
 
@@ -622,13 +626,16 @@ class EasyStatSearcher(BaseSearcher):
         logger.info(f"After filter by rkme spec, learnware_list length is {len(learnware_list)}")
 
         search_results = SearchResults()
-        
+
         search_results.update_single_results(
-            [SingleSearchItem(learnware=_learnware, score=_score) for _score, _learnware in zip(sorted_score_list, single_learnware_list)]
+            [
+                SingleSearchItem(learnware=_learnware, score=_score)
+                for _score, _learnware in zip(sorted_score_list, single_learnware_list)
+            ]
         )
         if mixture_score is not None and len(mixture_learnware_list) > 0:
             search_results.update_multiple_results(
-                [MultipleSearchItem(learnwares=mixture_learnware_list, score=mixture_score)]           
+                [MultipleSearchItem(learnwares=mixture_learnware_list, score=mixture_score)]
             )
         return search_results
 

learnware/market/heterogeneous/__init__.py

@@ -1,2 +1,12 @@
-from .organizer import HeteroMapTableOrganizer
-from .searcher import HeteroSearcher
+from ...utils import is_torch_available
+from ...logger import get_module_logger
+
+logger = get_module_logger("market_hetero")
+
+if not is_torch_available(verbose=False):
+    HeteroMapTableOrganizer = None
+    HeteroSearcher = None
+    logger.error("HeteroMapTableOrganizer and HeteroSearcher are not available because 'torch' is not installed!")
+else:
+    from .organizer import HeteroMapTableOrganizer
+    from .searcher import HeteroSearcher

learnware/market/heterogeneous/organizer/__init__.py

@@ -96,7 +96,7 @@ class HeteroMapTableOrganizer(EasyOrganizer):
         )
 
         if learnwere_status == BaseChecker.USABLE_LEARWARE and len(self._get_hetero_learnware_ids(learnware_id)):
-            self._update_learware_hetero_sepc(learnware_id)
+            self._update_learware_hetero_spec(learnware_id)
 
             if self.auto_update:
                 self.count_down -= 1
@@ -113,7 +113,7 @@ class HeteroMapTableOrganizer(EasyOrganizer):
                         f"Market mapping train completed. Now update HeteroMapTableSpecification for {training_learnware_ids}"
                     )
                     self.market_mapping = updated_market_mapping
-                    self._update_learware_hetero_sepc(training_learnware_ids)
+                    self._update_learware_hetero_spec(training_learnware_ids)
 
                     self.count_down = self.auto_update_limit
 
@@ -167,7 +167,7 @@ class HeteroMapTableOrganizer(EasyOrganizer):
         """
         final_status = super(HeteroMapTableOrganizer, self).update_learnware(id, zip_path, semantic_spec, check_status)
         if final_status == BaseChecker.USABLE_LEARWARE and len(self._get_hetero_learnware_ids(id)):
-            self._update_learware_hetero_sepc(id)
+            self._update_learware_hetero_spec(id)
         return final_status
 
     def _reload_learnware_hetero_spec(self, learnware_id):
@@ -178,7 +178,7 @@ class HeteroMapTableOrganizer(EasyOrganizer):
                 hetero_spec.load(hetero_spec_path)
                 self.learnware_list[learnware_id].update_stat_spec(hetero_spec.type, hetero_spec)
             else:
-                self._update_learware_hetero_sepc(learnware_id)
+                self._update_learware_hetero_spec(learnware_id)
             logger.info(f"Reload HeteroMapTableSpecification for hetero spec {learnware_id} succeed!")
         except Exception as err:
             logger.error(f"Reload HeteroMapTableSpecification for hetero spec {learnware_id} failed! due to {err}.")
@@ -196,7 +196,7 @@ class HeteroMapTableOrganizer(EasyOrganizer):
         if len(self._get_hetero_learnware_ids(learnware_id)):
             self._reload_learnware_hetero_spec(learnware_id)
 
-    def _update_learware_hetero_sepc(self, ids: Union[str, List[str]]):
+    def _update_learware_hetero_spec(self, ids: Union[str, List[str]]):
         """Update learnware by ids, attempting to generate HeteroMapTableSpecification for them.
 
         Parameters

learnware/market/heterogeneous/organizer/hetero_map/__init__.py

@@ -1,15 +1,15 @@
-from typing import Callable, List, Optional, Union
+from typing import Callable, Union
 
 import numpy as np
 import pandas as pd
 import torch
 import torch.nn.functional as F
-from torch import Tensor, nn
+from torch import nn
 
 from .....utils import allocate_cuda_idx, choose_device
 from .....specification import HeteroMapTableSpecification, RKMETableSpecification
 from .feature_extractor import CLSToken, FeatureProcessor, FeatureTokenizer
-from .trainer import Trainer, TransTabCollatorForCL
+from .trainer import TransTabCollatorForCL, Trainer
 
 
 class HeteroMap(nn.Module):
@@ -127,6 +127,7 @@ class HeteroMap(nn.Module):
         self.base_temperature = base_temperature
         self.num_partition = num_partition
         self.overlap_ratio = overlap_ratio
+        self.max_process_size = 20480
         self.to(device)
 
     def to(self, device: Union[str, torch.device]):
@@ -306,6 +307,10 @@ class HeteroMap(nn.Module):
         """
         self.eval()
         output_feas_list = []
+
+        if eval_batch_size * x_test.shape[1] > self.max_process_size:
+            eval_batch_size = max(1, self.max_process_size // x_test.shape[1])
+
         for i in range(0, len(x_test), eval_batch_size):
             bs_x_test = x_test.iloc[i : i + eval_batch_size]
             with torch.no_grad():

learnware/market/heterogeneous/organizer/hetero_map/feature_extractor.py

@@ -1,6 +1,6 @@
 import math
 import os
-from typing import Callable, Dict, List, Union
+from typing import Dict, List, Union
 
 import numpy as np
 import pandas as pd

learnware/market/heterogeneous/searcher.py

@@ -1,11 +1,9 @@
-import traceback
-from typing import Tuple, List
+from typing import Optional
 
 from .utils import is_hetero
 from ..base import BaseUserInfo, SearchResults
 from ..easy import EasySearcher
 from ..utils import parse_specification_type
-from ...learnware import Learnware
 from ...logger import get_module_logger
 
 
@@ -14,7 +12,7 @@ logger = get_module_logger("hetero_searcher")
 
 class HeteroSearcher(EasySearcher):
     def __call__(
-        self, user_info: BaseUserInfo, check_status: int = None, max_search_num: int = 5, search_method: str = "greedy"
+        self, user_info: BaseUserInfo, check_status: Optional[int] = None, max_search_num: int = 5, search_method: str = "greedy"
     ) -> SearchResults:
         """Search learnwares based on user_info from learnwares with check_status.
            Employs heterogeneous learnware search if specific requirements are met, otherwise resorts to homogeneous search methods.

learnware/specification/base.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import copy
 import numpy as np
 from typing import Dict
@@ -22,6 +24,9 @@ class BaseStatSpecification:
     def get_states(self):
         return {k: v for k, v in self.__dict__.items() if not k.startswith("_")}
 
+    def dist(self, stat_spec: BaseStatSpecification):
+        raise NotImplementedError("dist is not implemented")
+        
     def save(self, filepath: str):
         """Save the statistical specification into file in filepath
 

learnware/specification/regular/image/rkme.py

@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import codecs
-import copy
 import functools
 import json
 import os
@@ -20,8 +19,11 @@ from . import cnn_gp
 from ..base import RegularStatSpecification
 from ..table.rkme import rkme_solve_qp
 from .... import setup_seed
+from ....logger import get_module_logger
 from ....utils import choose_device, allocate_cuda_idx
 
+logger = get_module_logger("image_rkme")
+
 
 class RKMEImageSpecification(RegularStatSpecification):
     # INNER_PRODUCT_COUNT = 0
@@ -135,8 +137,10 @@ class RKMEImageSpecification(RegularStatSpecification):
         try:
             from torchvision.transforms import Resize
         except ModuleNotFoundError:
-            raise ModuleNotFoundError(f"RKMEImageSpecification is not available because 'torchvision' is not installed! Please install it manually." )
-            
+            raise ModuleNotFoundError(
+                f"RKMEImageSpecification is not available because 'torchvision' is not installed! Please install it manually."
+            )
+
         if X.shape[2] != RKMEImageSpecification.IMAGE_WIDTH or X.shape[3] != RKMEImageSpecification.IMAGE_WIDTH:
             X = Resize((RKMEImageSpecification.IMAGE_WIDTH, RKMEImageSpecification.IMAGE_WIDTH), antialias=True)(X)
 
@@ -417,15 +421,13 @@ class RKMEImageSpecification(RegularStatSpecification):
 
             for d in self.get_states():
                 if d in rkme_load.keys():
+                    if d == "type" and rkme_load[d] != self.type:
+                        raise TypeError(f"The type of loaded RKME ({rkme_load[d]}) is different from the expected type ({self.type})!")
                     setattr(self, d, rkme_load[d])
 
             self.beta = self.beta.to(self._device)
             self.z = self.z.to(self._device)
 
-            return True
-        else:
-            return False
-
 
 def _get_zca_matrix(X, reg_coef=0.1):
     X_flat = X.reshape(X.shape[0], -1)

learnware/specification/regular/table/rkme.py

@@ -6,7 +6,7 @@ import json
 import codecs
 import scipy
 import numpy as np
-from qpsolvers import solve_qp, Problem, solve_problem
+from qpsolvers import Problem, solve_problem
 from collections import Counter
 from typing import Any, Union
 
@@ -140,15 +140,17 @@ class RKMETableSpecification(RegularStatSpecification):
         if isinstance(X, np.ndarray):
             X = X.astype("float32")
             X = torch.from_numpy(X)
-            
+
         X = X.to(self._device)
-        
+
         try:
             from fast_pytorch_kmeans import KMeans
         except ModuleNotFoundError:
-            raise ModuleNotFoundError(f"RKMETableSpecification is not available because 'fast_pytorch_kmeans' is not installed! Please install it manually." )
+            raise ModuleNotFoundError(
+                f"RKMETableSpecification is not available because 'fast_pytorch_kmeans' is not installed! Please install it manually."
+            )
 
-        kmeans = KMeans(n_clusters=K, mode='euclidean', max_iter=100, verbose=0)
+        kmeans = KMeans(n_clusters=K, mode="euclidean", max_iter=100, verbose=0)
         kmeans.fit(X)
         self.z = kmeans.centroids.double()
 
@@ -454,10 +456,9 @@ class RKMETableSpecification(RegularStatSpecification):
 
             for d in self.get_states():
                 if d in rkme_load.keys():
+                    if d == "type" and rkme_load[d] != self.type:
+                        raise TypeError(f"The type of loaded RKME ({rkme_load[d]}) is different from the expected type ({self.type})!")
                     setattr(self, d, rkme_load[d])
-            return True
-        else:
-            return False
 
 
 class RKMEStatSpecification(RKMETableSpecification):

learnware/specification/system/hetero_table.py

@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import os
-import copy
 import json
 import torch
 import codecs
@@ -10,8 +9,11 @@ import numpy as np
 from .base import SystemStatSpecification
 from ..regular import RKMETableSpecification
 from ..regular.table.rkme import torch_rbf_kernel
+from ...logger import get_module_logger
 from ...utils import choose_device, allocate_cuda_idx
 
+logger = get_module_logger("hetero_map_table_spec")
+
 
 class HeteroMapTableSpecification(SystemStatSpecification):
     """Heterogeneous Map-Table Specification"""
@@ -133,12 +135,10 @@ class HeteroMapTableSpecification(SystemStatSpecification):
 
             for d in self.get_states():
                 if d in embedding_load.keys():
+                    if d == "type" and embedding_load[d] != self.type:
+                        raise TypeError(f"The type of loaded RKME ({embedding_load[d]}) is different from the expected type ({self.type})!")
                     setattr(self, d, embedding_load[d])
 
-            return True
-        else:
-            return False
-
     def save(self, filepath: str) -> bool:
         """Save the computed HeteroMapTableSpecification to a specified path in JSON format.
 

learnware/tests/module.py

@@ -4,6 +4,7 @@ def get_semantic_specification():
     semantic_specification["Task"] = {"Type": "Class", "Values": ["Segmentation"]}
     semantic_specification["Library"] = {"Type": "Class", "Values": ["Scikit-learn"]}
     semantic_specification["Scenario"] = {"Type": "Tag", "Values": ["Financial"]}
+    semantic_specification["License"] = {"Type": "Class", "Values": ["Apache-2.0"]}
     semantic_specification["Name"] = {"Type": "String", "Values": "test"}
     semantic_specification["Description"] = {"Type": "String", "Values": "test"}
     return semantic_specification

tests/test_hetero_market/test_hetero.py

@@ -5,10 +5,10 @@ import copy
 import joblib
 import zipfile
 import numpy as np
+import multiprocessing
 from sklearn.linear_model import Ridge
 from sklearn.datasets import make_regression
 from shutil import copyfile, rmtree
-from multiprocessing import Pool
 from learnware.client import LearnwareClient
 from sklearn.metrics import mean_squared_error
 
@@ -35,6 +35,7 @@ user_semantic = {
     "Scenario": {"Values": ["Education"], "Type": "Tag"},
     "Description": {"Values": "", "Type": "String"},
     "Name": {"Values": "", "Type": "String"},
+    "License": {"Values": ["MIT"], "Type": "Class"},
 }
 
 
@@ -120,7 +121,8 @@ class TestMarket(unittest.TestCase):
         dir_path = os.path.join(curr_root, "learnware_pool")
 
         # Execute multi-process checking using Pool
-        with Pool() as pool:
+        mp_context = multiprocessing.get_context("spawn")
+        with mp_context.Pool() as pool:
             results = pool.starmap(check_learnware, [(name, dir_path) for name in os.listdir(dir_path)])
 
         # Use an assert statement to ensure that all checks return True
@@ -259,15 +261,15 @@ class TestMarket(unittest.TestCase):
                 str(key): semantic_spec["Input"]["Description"][str(key)] for key in range(user_dim)
             }
             user_info = BaseUserInfo(semantic_spec=semantic_spec, stat_info={"RKMETableSpecification": user_spec})
-            
+
             search_result = hetero_market.search_learnware(user_info)
             single_result = search_result.get_single_results()
             multiple_result = search_result.get_multiple_results()
-            
+
             print(f"search result of user{idx}:")
             for single_item in single_result:
                 print(f"score: {single_item.score}, learnware_id: {single_item.learnware.id}")
-                
+
             for multiple_item in multiple_result:
                 print(
                     f"mixture_score: {multiple_item.score}, mixture_learnware_ids: {[item.id for item in multiple_item.learnwares]}"
@@ -335,7 +337,7 @@ class TestMarket(unittest.TestCase):
             print(f"search result of user{idx}:")
             for single_item in single_result:
                 print(f"score: {single_item.score}, learnware_id: {single_item.learnware.id}")
-            
+
             for multiple_item in multiple_result:
                 print(f"mixture_score: {multiple_item.score}\n")
                 mixture_id = " ".join([learnware.id for learnware in multiple_item.learnwares])
@@ -363,9 +365,11 @@ class TestMarket(unittest.TestCase):
         # print search results
         for single_item in single_result:
             print(f"score: {single_item.score}, learnware_id: {single_item.learnware.id}")
-        
+
         for multiple_item in multiple_result:
-            print(f"mixture_score: {multiple_item.score}, mixture_learnware_ids: {[item.id for item in multiple_item.learnwares]}")
+            print(
+                f"mixture_score: {multiple_item.score}, mixture_learnware_ids: {[item.id for item in multiple_item.learnwares]}"
+            )
 
         # single model reuse
         hetero_learnware = HeteroMapAlignLearnware(single_result[0].learnware, mode="regression")

tests/test_learnware_client/test_all_learnware.py

@@ -30,15 +30,17 @@ class TestAllLearnware(unittest.TestCase):
         self.client.login(email, token)
 
     def test_all_learnware(self):
-        max_learnware_num = 1000
+        max_learnware_num = 2000
         semantic_spec = self.client.create_semantic_specification()
         user_info = BaseUserInfo(semantic_spec=semantic_spec, stat_info={})
         result = self.client.search_learnware(user_info, page_size=max_learnware_num)
-        print(f"result size: {len(result)}")
-        print(f"key in result: {[key for key in result[0]]}")
+        
+        learnware_ids = result["single"]["learnware_ids"]
+        keys = [key for key in result["single"]["semantic_specifications"][0]]
+        print(f"result size: {len(learnware_ids)}")
+        print(f"key in result: {keys}")
 
         failed_ids = []
-        learnware_ids = [res["learnware_id"] for res in result]
         with tempfile.TemporaryDirectory(prefix="learnware_") as tempdir:
             for idx in learnware_ids:
                 zip_path = os.path.join(tempdir, f"test_{idx}.zip")

tests/test_learnware_client/test_load_conda.py

@@ -16,7 +16,7 @@ class TestLearnwareLoad(unittest.TestCase):
         self.client = LearnwareClient()
 
         root = os.path.dirname(__file__)
-        self.learnware_ids = ["00000084", "00000154", "00000155"]
+        self.learnware_ids = ["00000910", "00000899", "00000900"]
         self.zip_paths = [os.path.join(root, x) for x in ["1.zip", "2.zip", "3.zip"]]
 
     def test_load_single_learnware_by_zippath(self):
@@ -26,8 +26,8 @@ class TestLearnwareLoad(unittest.TestCase):
         learnware_list = [
             self.client.load_learnware(learnware_path=zippath, runnable_option="conda") for zippath in self.zip_paths
         ]
-        reuser = AveragingReuser(learnware_list, mode="vote_by_label")
-        input_array = np.random.random(size=(20, 13))
+        reuser = AveragingReuser(learnware_list, mode="mean")
+        input_array = np.random.random(size=(20, 40))
         print(reuser.predict(input_array))
 
         for learnware in learnware_list:
@@ -38,8 +38,8 @@ class TestLearnwareLoad(unittest.TestCase):
             self.client.download_learnware(learnware_id, zip_path)
 
         learnware_list = self.client.load_learnware(learnware_path=self.zip_paths, runnable_option="conda")
-        reuser = AveragingReuser(learnware_list, mode="vote_by_label")
-        input_array = np.random.random(size=(20, 13))
+        reuser = AveragingReuser(learnware_list, mode="mean")
+        input_array = np.random.random(size=(20, 40))
         print(reuser.predict(input_array))
 
         for learnware in learnware_list:
@@ -49,8 +49,8 @@ class TestLearnwareLoad(unittest.TestCase):
         learnware_list = [
             self.client.load_learnware(learnware_id=idx, runnable_option="conda") for idx in self.learnware_ids
         ]
-        reuser = AveragingReuser(learnware_list, mode="vote_by_label")
-        input_array = np.random.random(size=(20, 13))
+        reuser = AveragingReuser(learnware_list, mode="mean")
+        input_array = np.random.random(size=(20, 40))
         print(reuser.predict(input_array))
 
         for learnware in learnware_list:
@@ -58,8 +58,8 @@ class TestLearnwareLoad(unittest.TestCase):
 
     def test_load_multi_learnware_by_id(self):
         learnware_list = self.client.load_learnware(learnware_id=self.learnware_ids, runnable_option="conda")
-        reuser = AveragingReuser(learnware_list, mode="vote_by_label")
-        input_array = np.random.random(size=(20, 13))
+        reuser = AveragingReuser(learnware_list, mode="mean")
+        input_array = np.random.random(size=(20, 40))
         print(reuser.predict(input_array))
 
         for learnware in learnware_list:

tests/test_learnware_client/test_load_docker.py

@@ -16,7 +16,7 @@ class TestLearnwareLoad(unittest.TestCase):
         self.client = LearnwareClient()
 
         root = os.path.dirname(__file__)
-        self.learnware_ids = ["00000084", "00000154", "00000155"]
+        self.learnware_ids = ["00000910", "00000899", "00000900"]
         self.zip_paths = [os.path.join(root, x) for x in ["1.zip", "2.zip", "3.zip"]]
 
     def test_load_multi_learnware_by_zippath(self):
@@ -24,8 +24,8 @@ class TestLearnwareLoad(unittest.TestCase):
             self.client.download_learnware(learnware_id, zip_path)
 
         learnware_list = self.client.load_learnware(learnware_path=self.zip_paths, runnable_option="docker")
-        reuser = AveragingReuser(learnware_list, mode="vote_by_label")
-        input_array = np.random.random(size=(20, 13))
+        reuser = AveragingReuser(learnware_list, mode="mean")
+        input_array = np.random.random(size=(20, 40))
         print(reuser.predict(input_array))
 
         for learnware in learnware_list:
@@ -33,8 +33,8 @@ class TestLearnwareLoad(unittest.TestCase):
 
     def test_load_multi_learnware_by_id(self):
         learnware_list = self.client.load_learnware(learnware_id=self.learnware_ids, runnable_option="docker")
-        reuser = AveragingReuser(learnware_list, mode="vote_by_label")
-        input_array = np.random.random(size=(20, 13))
+        reuser = AveragingReuser(learnware_list, mode="mean")
+        input_array = np.random.random(size=(20, 40))
         print(reuser.predict(input_array))
 
         for learnware in learnware_list:

tests/test_workflow/test_workflow.py

@@ -27,6 +27,7 @@ user_semantic = {
     "Scenario": {"Values": ["Education"], "Type": "Tag"},
     "Description": {"Values": "", "Type": "String"},
     "Name": {"Values": "", "Type": "String"},
+    "License": {"Values": ["MIT"], "Type": "Class"},
 }
 
 
@@ -145,11 +146,15 @@ class TestWorkflow(unittest.TestCase):
         user_info = BaseUserInfo(semantic_spec=semantic_spec)
         search_result = easy_market.search_learnware(user_info)
         single_result = search_result.get_single_results()
-        
+
         print("User info:", user_info.get_semantic_spec())
         print(f"Search result:")
         for search_item in single_result:
-            print("Choose learnware:", search_item.learnware.id, search_item.learnware.get_specification().get_semantic_spec())
+            print(
+                "Choose learnware:",
+                search_item.learnware.id,
+                search_item.learnware.get_specification().get_semantic_spec(),
+            )
 
         rmtree(test_folder)  # rm -r test_folder
 
@@ -176,12 +181,12 @@ class TestWorkflow(unittest.TestCase):
 
             single_result = search_results.get_single_results()
             multiple_result = search_results.get_multiple_results()
-            
+
             assert len(single_result) >= 1, f"Statistical search failed!"
             print(f"search result of user{idx}:")
             for search_item in single_result:
                 print(f"score: {search_item.score}, learnware_id: {search_item.learnware.id}")
-            
+
             for mixture_item in multiple_result:
                 print(f"mixture_score: {mixture_item.score}\n")
                 mixture_id = " ".join([learnware.id for learnware in mixture_item.learnwares])
@@ -229,8 +234,8 @@ class TestWorkflow(unittest.TestCase):
 
 def suite():
     _suite = unittest.TestSuite()
-    #_suite.addTest(TestWorkflow("test_prepare_learnware_randomly"))
-    #_suite.addTest(TestWorkflow("test_upload_delete_learnware"))
+    # _suite.addTest(TestWorkflow("test_prepare_learnware_randomly"))
+    # _suite.addTest(TestWorkflow("test_upload_delete_learnware"))
     _suite.addTest(TestWorkflow("test_search_semantics"))
     _suite.addTest(TestWorkflow("test_stat_search"))
     _suite.addTest(TestWorkflow("test_learnware_reuse"))