[DOC] solve merge conflicts

docs/advanced/anchor.rst

@@ -4,15 +4,15 @@ Anchor learnware
 
 Anchor learnwares are a small fraction of representative learnwares that helps locate user's requirements through user feedback. The learnware market can choose or generate several learnwares as anchor learnwares corresponding to the specification island. If the user does not have sufficient training data for constructing an RKME requirement, the learnware market can send several anchor learnwares to the user. By feeding her own data to these anchor learnwares, some information such as (precision, recall) or other performance indicators, can be generated and returned to the market. These information could help the market identify potentially helpful models, e.g., by identifying models that are far from anchors exhibiting poor performance whereas close to anchors exhibiting relatively better performance in the specification island.
 
-To fulfill the anchor learnware method, you need to implement the following functions in ``anchor.py``: 
+To fulfill the anchor learnware method, you need to implement the following functions in the folder ``Learnware/market/anchor``: 
 
-- First, you should design how the market chooses or generates anchor learnwares. This can be realized by selecting prototype models through functional space clustering, and more interesting designs can be explored. The function ``AnchoredMarket.update_anchor_learnware_list`` is reserved for it. The functions ``AnchoredMarket._update_anchor_learnware`` and ``AnchoredMarket._delete_anchor_learnware`` have been completed as auxiliary.
+- First, you should design how the market chooses or generates anchor learnwares. This can be realized by selecting prototype models through functional space clustering, and more interesting designs can be explored. The class ``AnchoredOrganizer`` in ``organizer.py`` is designed for it. The function ``AnchoredOrganizer.update_anchor_learnware_list`` is reserved for choosing or generating anchor learnwares. The functions ``AnchoredOrganizer._update_anchor_learnware`` and ``AnchoredOrganizer._delete_anchor_learnware`` have been completed as auxiliary.
 
-- Second, when a user comes with no RKME(or other statistical) specifications, the market should choose several anchor learnwares  and send them to the user. This process is done by ``AnchoredMarket.search_anchor_learnware``, and the chosen anchors are stored in ``AnchoredUserInfo`` by ``AnchoredUserInfo.add_anchor_learnware``.
+- Second, when a user comes with no RKME(or other statistical) specifications, the market should choose several anchor learnwares and send them to the user. This process is done by ``AnchoredSearcher.search_anchor_learnware`` in ``searcher.py``. Besides the list of anchor learnwares, it also returns an item specifying which performance indicator should the user return. 
   
-- Third, the market should specify which performance indicator should the user return. By feeding the user's data to these anchor learnwares, the returned information is calculated and stored in ``AnchoredUserInfo`` by ``AnchoredUserInfo.update_stat_info``.
+- Third, by feeding the user's data to these anchor learnwares, the returned information is calculated and stored in ``AnchoredUserInfo`` in ``user_info.py``. The user should add the anchor learnwares into it using ``AnchoredUserInfo.add_anchor_learnware_ids``, and then fill in performance indicator using ``AnchoredUserInfo.update_stat_info``.
   
-- Fourth, according to the returned information from the user, the market should identify the helpful learnwares for the user. This process is done in ``AnchoredMarket.search_learnware``.
+- Fourth, according to the returned information from the user, the market should identify the helpful learnwares for the user. This process is done in ``AnchoredSearcher.search_learnware`` in ``searcher.py``, which returns the recommended comination of helpful learnwares and the list of helpful learnwares.
 
 
 

docs/components/learnware.rst

@@ -1,4 +1,5 @@
 .. _learnware:
+
 ==========================================
 Learnware & Reuser
 ==========================================
@@ -15,7 +16,16 @@ In our implementation, the class ``Learnware`` has 3 important member variables:
 - ``model``: The model in the learnware, can be a ``BaseModel`` or a dict including model name and path. When it is a dict, the function ``Learnware.instantiate_model`` is used to transform it to a ``BaseModel``. The function ``Learnware.predict`` use the model to predict for an input ``X``. See more in `COMPONENTS: Model <./model.html>`_.
 - ``specification``: The specification including the semantic specification and the statistic specification.
 
+In the learnware paradigm, a learnware is a well-performed trained machine learning model with a specification which enables it to be adequately identified to reuse according to the requirement of future users who know nothing about the learnware in advance. The introduction of specifications are shown in `COMPONENTS: Specification <./spec.html>`_.
+
+In our implementation, the class ``Learnware`` has 3 important member variables:
+
+- ``id``: The learnware id is generated by ``market``.
+- ``model``: The model in the learnware, can be a ``BaseModel`` or a dict including model name and path. When it is a dict, the function ``Learnware.instantiate_model`` is used to transform it to a ``BaseModel``. The function ``Learnware.predict`` use the model to predict for an input ``X``. See more in `COMPONENTS: Model <./model.html>`_.
+- ``specification``: The specification including the semantic specification and the statistic specification.
+
 
+Learnware for Hetero Reuse (Feature Align + Hetero Map Learnware)
 Learnware for Hetero Reuse (Feature Align + Hetero Map Learnware)
 =======================================================================
 

docs/components/market.rst

@@ -1,6 +1,7 @@
 .. _market:
+
 ================================
-Market
+Learnware Market
 ================================
 
 The ``learnware market`` receives high-performance machine learning models from developers, incorporates them into the system, and provides services to users by identifying and reusing learnware to help users solve current tasks. Developers voluntarily submit various learnwares to the learnware market, and the market conducts quality checks and further organization of these learnwares. When users submit task requirements, the learnware market automatically selects whether to recommend a single learnware or a combination of multiple learnwares. 

docs/components/spec.rst

@@ -99,19 +99,17 @@ By randomly sampling a subset of the dataset, we can construct Image Specificati
 
 .. code-block:: python
 
-   import torchvision
-   from torch.utils.data import DataLoader
-   from learnware.specification import generate_rkme_image_spec
+    import torchvision
+    from torch.utils.data import DataLoader
+    from learnware.specification import generate_rkme_image_spec
 
-   SAMPLED_SIZE = 5000
 
-   full_set = torchvision.datasets.CIFAR10(
-      root='./data', train=True, download=True, transform=torchvision.transforms.ToTensor())
-   loader =  DataLoader(full_set, batch_size=SAMPLED_SIZE, shuffle=True)
-   sampled_X, _ = next(iter(loader))
+    cifar10 = torchvision.datasets.CIFAR10(
+       root='./data', train=True, download=True, transform=torchvision.transforms.ToTensor())
+    X, _ = next(iter(DataLoader(cifar10, batch_size=len(cifar10))))
 
-   spec = generate_rkme_image_spec(sampled_X)
-   spec.save("cifar10.json")
+    spec = generate_rkme_image_spec(X, sample_size=5000)
+    spec.save("cifar10.json")
 
 Privacy Protection
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -127,7 +125,8 @@ The RBF not only exposes the real data (plotted in the corresponding position in
 
 Text Specification
 --------------------------
-
+Different from tabular data, each text input is a string of different length, so we should first transform them to equal-length arrays. Sentence embedding is used here to complete this transformation. We choose the model ``paraphrase-multilingual-MiniLM-L12-v2``, a lightweight multilingual embedding model. Then, we calculate the RKME specification on the embedding, 
+just like we do with tabular data. Besides, we use the package ``langdetect`` to detect and store the language of the text inputs for further search. We hope to search for the learnware which supports the language of the user task.
 
 System Specification
 ======================================

docs/start/exp.rst

@@ -53,11 +53,51 @@ M5                      2.066 +/- 0.424    2.116 +/- 0.472                    2.
 CIFAR10                 0.619 +/- 0.138    0.585 +/- 0.056                    0.715 +/- 0.075
 ====================  ==================== ================================= =================================
 
-
-
 Text Experiment
 ====================
 
+Datasets
+------------------
+We conducted experiments on the widely used text benchmark dataset: `20-newsgroup <http://qwone.com/~jason/20Newsgroups/>`_.
+20-newsgroup is a renowned text classification benchmark with a hierarchical structure, featuring 5 superclasses {comp, rec, sci, talk, misc}.
+
+In the submitting stage, we enumerated all combinations of three superclasses from the five available, randomly sampling 50% of each combination from the training set to create datasets for 50 uploaders.
+
+In the deploying stage, we considered all combinations of two superclasses out of the five, selecting all data for each combination from the testing set as a test dataset for one user. This resulted in 10 users.
+The user's own training data was generated using the same sampling procedure as the user test data, despite originating from the training dataset.
+
+Model training comprised two parts: the first part involved training a tfidf feature extractor, and the second part used the extracted text feature vectors to train a naive Bayes classifier.
+
+Our experiments comprises two components:
+
+* ``test_unlabeled`` is designed to evaluate performance when users possess only testing data, searching and reusing learnware available in the market.
+* ``test_labeled`` aims to assess performance when users have both testing and limited training data, searching and reusing learnware directly from the market instead of training a model from scratch. This helps determine the amount of training data saved for the user.
+
+Results
+----------------
+
+* ``test_unlabeled``:
+
+The accuracy of search and reuse is presented in the table below:
+
+==================== ================================= =================================
+ Top-1 Performance         Job Selector Reuse                Average Ensemble Reuse
+==================== ================================= =================================
+  0.859 +/- 0.051          0.844 +/- 0.053                    0.858 +/- 0.051
+==================== ================================= =================================
+
+* ``test_labeled``:
+
+We present the change curves in classification error rates for both the user's self-trained model and the multiple learnware reuse(EnsemblePrune), showcasing their performance on the user's test data as the user's training data increases. The average results across 10 users are depicted below:
+
+.. image:: ../_static/img/text_example_labeled_curves.png
+   :width: 300
+   :height: 200
+   :alt: Text Limited Labeled Data
+
+
+From the figure above, it is evident that when the user's own training data is limited, the performance of multiple learnware reuse surpasses that of the user's own model. As the user's training data grows, it is expected that the user's model will eventually outperform the learnware reuse. This underscores the value of reusing learnware to significantly conserve training data and achieve superior performance when user training data is limited.
+
 
 Image Experiment
 ====================
@@ -67,4 +107,13 @@ Get Start Examples
 =========================
 Examples for `PFS, M5` and `CIFAR10` are available at [xxx]. You can run { main.py } directly to reproduce related experiments.
 The test code is mainly composed of three parts, namely data preparation (optional), specification generation and market construction, and search test.
-You can load data prepared by as and skip the data preparation step.
+You can load data prepared by as and skip the data preparation step.
+
+
+Examples for the `20-newsgroup` dataset are available at [examples/dataset_text_workflow].
+We utilize the `fire` module to construct our experiments. You can execute the experiment with the following commands:
+
+* `python main.py prepare_market`: Prepares the market.
+* `python main.py test_unlabeled`: Executes the test_unlabeled experiment; the results will be printed in the terminal.
+* `python main.py test_labeled`: Executes the test_labeled experiment; result curves will be automatically saved in the `figs` directory.
+* Additionally, you can use `python main.py test_unlabeled True` to combine steps 1 and 2. The same approach applies to running test_labeled directly.

docs/workflows/search.rst

@@ -6,7 +6,6 @@ Learnwares Search
 
 All the searchers are implemented as a subclass of ``BaseSearcher``. When initializing, you should assign a ``organizer`` to it. The introduction of ``organizer`` is shown in `COMPONENTS: Market - Framework <../components/market.html>`_. Then these searchers can be called with ``UserInfo`` and return ``SearchResults``.
 
-
 Homo Search
 ======================
 
@@ -14,6 +13,7 @@ The homogeneous search of helpful learnwares can be divided into two stages: sem
 
 .. code-block:: python
 
+    # generate BaseUserInfo(semantic_spec + stat_info)
     # generate BaseUserInfo(semantic_spec + stat_info)
     user_semantic = {
         "Data": {"Values": ["Table"], "Type": "Class"},
@@ -47,7 +47,6 @@ The homogeneous search of helpful learnwares can be divided into two stages: sem
     mixture_id = " ".join([learnware.id for learnware in multiple_result[0].learnwares])
     print(f"mixture_score: {multiple_result[0].score}, mixture_learnwares: {mixture_id}")
 
-
 Hetero Search
 ======================