Browse Source

[DOC] unify results format

tags/v0.3.2
Gene 2 years ago
parent
commit
85731a5b5c
1 changed files with 44 additions and 32 deletions
  1. +44
    -32
      docs/start/exp.rst

+ 44
- 32
docs/start/exp.rst View File

@@ -51,11 +51,17 @@ and using the test data from their respective store as user data. These users ca

The Mean Squared Error (MSE) of search and reuse is presented in the table below:

=================== ====================== ================= ==================
Top-1 Reuse Average Ensemble Reuse Best in Market Average in Market
=================== ====================== ================= ==================
0.280 +/- 0.090 0.267 +/- 0.051 0.151 +/- 0.046 0.331 +/- 0.040
=================== ====================== ================= ==================
+-----------------------------------+---------------------+
| Mean in Market (Single) | 0.331 ± 0.040 |
+-----------------------------------+---------------------+
| Best in Market (Single) | 0.151 ± 0.046 |
+-----------------------------------+---------------------+
| Top-1 Reuse (Single) | 0.280 ± 0.090 |
+-----------------------------------+---------------------+
| Job Selector Reuse (Multiple) | 0.274 ± 0.064 |
+-----------------------------------+---------------------+
| Average Ensemble Reuse (Multiple) | 0.267 ± 0.051 |
+-----------------------------------+---------------------+

When users have both test data and limited training data derived from their original data, reusing single or multiple searched learnwares from the market can often yield
better results than training models from scratch on limited training data. We present the change curves in MSE for the user's self-trained model, as well as for the Feature Augmentation single learnware reuse method and the Ensemble Pruning multiple learnware reuse method.
@@ -84,17 +90,17 @@ Consequently, while the market's learnwares from the PFS dataset undertake tasks
we tested various heterogeneous learnware reuse methods (without using user's labeled data) and compared them to the user's self-trained model based on a small amount of training data.
The average MSE performance across 41 users are as follows:

+---------------------------------------+---------------------+
| **Mean in Market (Single)** | 1.459 +/- 1.066 |
+---------------------------------------+---------------------+
| **Best in Market (Single)** | 1.226 +/- 1.032 |
+---------------------------------------+---------------------+
| **Top-1 Reuse (Single)** | 1.407 +/- 1.061 |
+---------------------------------------+---------------------+
| **Average Ensemble Reuse (Multiple)** | 1.312 +/- 1.099 |
+---------------------------------------+---------------------+
| **User model with 50 labeled data** | 1.267 +/- 1.055 |
+---------------------------------------+---------------------+
+-----------------------------------+---------------------+
| Mean in Market (Single) | 1.459 ± 1.066 |
+-----------------------------------+---------------------+
| Best in Market (Single) | 1.226 ± 1.032 |
+-----------------------------------+---------------------+
| Top-1 Reuse (Single) | 1.407 ± 1.061 |
+-----------------------------------+---------------------+
| Average Ensemble Reuse (Multiple) | 1.312 ± 1.099 |
+-----------------------------------+---------------------+
| User model with 50 labeled data | 1.267 ± 1.055 |
+-----------------------------------+---------------------+

From the results, it is noticeable that the learnware market still perform quite well even when users lack labeled data,
provided it includes learnwares addressing tasks that are similar but not identical to the user's.
@@ -143,17 +149,17 @@ Results

The accuracy of search and reuse is presented in the table below:

+--------------------------------------+---------------------+
| **Mean in Market (Single)** | 0.507 +/- 0.030 |
+--------------------------------------+---------------------+
| **Best in Market (Single)** | 0.859 +/- 0.051 |
+--------------------------------------+---------------------+
| **Top-1 Reuse (Single)** | 0.846 +/- 0.054 |
+--------------------------------------+---------------------+
| **Job Selector Reuse (Multiple)** | 0.845 +/- 0.053 |
+--------------------------------------+---------------------+
|**Average Ensemble Reuse (Multiple)** | 0.862 +/- 0.051 |
+--------------------------------------+---------------------+
+-----------------------------------+---------------------+
| Mean in Market (Single) | 0.507 ± 0.030 |
+-----------------------------------+---------------------+
| Best in Market (Single) | 0.859 ± 0.051 |
+-----------------------------------+---------------------+
| Top-1 Reuse (Single) | 0.846 ± 0.054 |
+-----------------------------------+---------------------+
| Job Selector Reuse (Multiple) | 0.845 ± 0.053 |
+-----------------------------------+---------------------+
| Average Ensemble Reuse (Multiple) | 0.862 ± 0.051 |
+-----------------------------------+---------------------+

* ``test_labeled``:

@@ -176,11 +182,17 @@ We constructed 50 target tasks using data from the test set of CIFAR-10. Similar

With this experimental setup, we evaluated the performance of RKME Image using 1 - Accuracy as the loss.

==================== ==================== ==================== ====================
Top-1 Reuse Job Selector Reuse Voting Reuse Best in Market
==================== ==================== ==================== ====================
0.406 +/- 0.128 0.406 +/- 0.128 0.310 +/- 0.112 0.304 ± 0.046
==================== ==================== ==================== ====================
+-----------------------------------+---------------------+
| Mean in Market (Single) | 0.655 ± 0.021 |
+-----------------------------------+---------------------+
| Best in Market (Single) | 0.304 ± 0.046 |
+-----------------------------------+---------------------+
| Top-1 Reuse (Single) | 0.406 ± 0.128 |
+-----------------------------------+---------------------+
| Job Selector Reuse (Multiple) | 0.406 ± 0.128 |
+-----------------------------------+---------------------+
| Average Ensemble Reuse (Multiple) | 0.310 ± 0.112 |
+-----------------------------------+---------------------+

In some specific settings, the user will have a small number of labelled samples. In such settings, learning the weight of selected learnwares on a limited number of labelled samples can result in a better performance than training directly on a limited number of labelled samples.



Loading…
Cancel
Save