diff --git a/README.md b/README.md index f05bb41..da0f1e1 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,8 @@ These specifications, encompassing both semantic and statistical aspects, detail The need for Learnware arises due to challenges in machine learning, such as the need for extensive training data, advanced techniques, continuous learning, catastrophic forgetting, and data privacy issues. Although there are many efforts focusing on one of these issues separately, they are entangled, and solving one problem may exacerbate others. The learnware paradigm aims to address many of these challenges through a unified framework. Its benefits are listed as follows. +
+ | Benefit | Description | | ---- | ---- | | Lack of training data | Strong models can be built with small data by adapting well-performed learnwares. | @@ -59,6 +61,7 @@ The need for Learnware arises due to challenges in machine learning, such as the | Data privacy/ proprietary | Developers only submit models, not data, preserving data privacy/proprietary. | | Unplanned tasks | Open to all legal developers, the learnware market can accommodate helpful learnwares for various tasks. | | Carbon emission | Assembling small models may offer good-enough performance, reducing interest in training large models and the carbon footprint. | +
The learnware paradigm consists of two distinct stages: - `Submitting Stage`: Developers voluntarily submit various learnwares to the learnware market, and the system conducts quality checks and further organization of these learnwares. @@ -68,15 +71,41 @@ The learnware paradigm consists of two distinct stages: -## Learnware Package Design +## Framework and Infrastructure Design
+The architecture is designed based on the guidelines including _decoupling_, _autonomy_, _reusability_, and _scalability_. The above architecture diagram illustrates the architecture and framework from the perspectives of both modules and workflows. + +- At the workflow level, the `learnware` package consists of `Submitting Stage` and `Deploying Stage`. + +
+ +| Module | Workflow | +| ---- | ---- | +| `Submitting Stage` | The learnware developers submit learnwares to the learnware market, which conducts usability checks and further organization of these learnwares. | +| `Deploying Stage` | The `learnware` package identifies learnwares according to users’ task requirements and provides efficient reuse and deployment methods. | + +
+ +- At the module level, the `learnware` package is a platform that consists of `Learnware`, `Market`, `Specification`, `Model`, `Reuse`, and `Interface` modules. + +
+ +| Module | Description | +| ---- | ---- | +| `Learnware` | The specific learnware, consisting of specification module, and user model module. | +| `Market` | Designed for learnware organization, identification, and usability testing. | +| `Specification` | Generating and storing statistical and semantic information of learnware, which can be used for learnware search and reuse. | +| `Model` | Including the base model and the model container, which can provide unified interfaces and automatically create isolated runtime environments. | +| `Reuse` | Including the data-free reuser, data-dependent reuser, and aligner, which can deploy and reuse learnware for user tasks. | +| `Interface` | The interface for network communication with the `Beimingwu` backend.| + +
+ -At the workflow level, the `learnware` package consists of `Submitting Stage` and `Deploying Stage`. -At the module level, the `learnware` package is a platform that consists of above components. The components are designed as loose-coupled modules and each component could be used stand-alone. # Quick Start diff --git a/docs/_static/img/learnware_framework.svg b/docs/_static/img/learnware_framework.svg index 92e0d37..bf4326b 100644 --- a/docs/_static/img/learnware_framework.svg +++ b/docs/_static/img/learnware_framework.svg @@ -1,4 +1,4 @@ -
Market
Market
Specification
Specification
Interface
Interface
Workflow
Workflow
Learnware Standard
Learnware Standard
Zip Package





Zip Package...
Semantic
Specification

Semantic...
Yaml Config
Yaml Config
Model File
Model File
Env Dependencies
Env Dependencies
Learnware
Learnware
Model



Model...
Classfication
Classficat...
Feature Extraction
Feature Extract...
Regression
Regression
Specification



Specification...
Semantic
Semantic
Others
Others
Statistical
Statistical
create
cre...
Market
Market
Checker







Checker...
Searcher



Searcher...
Semantic
Semantic
Statistical
Statisti...
Conda
Conda
Semantic
Semantic
Statictical
Statictica...
Database
Database
Organizer


Organizer...
Easy
Easy
Hetero
Hetero
submit
sub...
Submitting Workflow
Submitting Workflow
Deploying Workflow
Deploying Workflow
UserInfo






UserInfo...
Semantic
Spec
Semantic...
Statistical
Info
Statisti...
SearchResult






SearchResult...
Scores
Scores
Learnwares



Learnwares...
search
sea...
return
ret...
Executable
Learnwares





Executable...
ModelContainer


ModelContaine...
Docker
Docker
Conda
Conda
Reuser
Reuser
Model
Model
Specification
Specificat...
ContainerManager
ContainerManager

Executable
Learnwares

Executable...
Data Free Reuser


Data Free Reuser...
Averaging
Averaging
JobSelector
JobSelecto...
Data-Dependent Reuser



Data-Dependent Reuser...
Ensemble
Purning
Ensemble...
Feature
Argument
Feature...
Aligner







Aligner...
Feature
Aligner
Featur...
Label
Aligner
Label...
Specification
Specificati...
Learnware Client





Learnware Client...
login
login
BeimingWu Server



BeimingWu Server...
search
search
upload
upload
delete
delete
update
update
load
load
Network
Network
Statistical Specification
Statistical Specification
Semantic Specification










Semantic Specificati...
Data
Data
Feature
Featur...
Task
Task
Label
Label
Library
Librar...
Description
Descripti...
Name
Name
Scenario
Scenario
License
Licens...
Others
Others
System Specification
System Specification
Regular Specification
Regular Specification
RKMEStatSpecification










RKMEStatSpecification...
Data Set


Data Set...
Table
Table
Image
Image
Text
Text
RKME
RKME
Reduced Set


Reduced Set...
Weights
Weight...
Reduced Points
Reduced Point...
HeteroMapSpecification









HeteroMapSpecification...
RKME Reduced Set
RKME Reduced Set
HeteroMapping
HeteroMapping
Hetero Reduced Set



Hetero Reduced Set...
Weights
Weight...
Homo Embeddings
Homo Embeddings
Organizer
Organizer
Searcher
Searcher
EasyOrganizer
EasyOrganizer
HeteroOrganizer
HeteroOrganizer
Database



Database...
SqlLite
SqlLite
PostgresSql
PostgresSq...
HeteroMapping
HeteroMapping
HeteroMap
Specification
HeteroMap...
update
upd...
generate
gen...
save
save
load
load
save
save
load
load
Homo
StatSearcher
Homo...
Semantic Searcher



Semantic Searcher...
Checker











Checker...
Conda Checker








Conda Checker...
Semantic Checker
Semantic Checke...
Executable
Learnwares



Executable...
ModelContainer
ModelContaine...
Statistical Checker
Statistical Che...
check
che...
HeteroMap
Specification
HeteroMap...
RKMEStat
Specification
RKMEStat...
Heterom
StatSearcher
Heterom...
Fuzzy
Semantic Searcher
Fuzzy...
Exact
Semantic Searcher
Exact...
match
mat...
match
mat...
Text is not SVG - cannot display
\ No newline at end of file +
Market
Market
Specification
Specification
Interface
Interface
Workflow
Workflow
Learnware Standard
Learnware Standard
Zip Package





Zip Package...
Semantic
Specification

Semantic...
Yaml Config
Yaml Config
Model File
Model File
Env Dependencies
Env Dependencies
Learnware
Learnware
Model



Model...
Classfication
Classficat...
Feature Extraction
Feature Extract...
Regression
Regression
Specification



Specification...
Semantic
Semantic
Others
Others
Statistical
Statistical
create
cre...
Market
Market
Checker







Checker...
Searcher



Searcher...
Semantic
Semantic
Statistical
Statisti...
Conda
Conda
Semantic
Semantic
Statictical
Statictica...
Database
Database
Organizer


Organizer...
Easy
Easy
Hetero
Hetero
submit
sub...
Submitting Workflow
Submitting Workflow
Deploying Workflow
Deploying Workflow
UserInfo






UserInfo...
Semantic
Spec
Semantic...
Statistical
Info
Statisti...
SearchResult






SearchResult...
Scores
Scores
Learnwares



Learnwares...
search
sea...
return
ret...
Executable
Learnwares





Executable...
ModelContainer


ModelContaine...
Docker
Docker
Conda
Conda
Reuser
Reuser
Model
Model
Specification
Specificat...
ContainerManager
ContainerManager

Executable
Learnwares

Executable...
Data Free Reuser


Data Free Reuser...
Averaging
Averaging
JobSelector
JobSelecto...
Data-Dependent Reuser



Data-Dependent Reuser...
Ensemble
Purning
Ensemble...
Feature
Argument
Feature...
Aligner







Aligner...
Feature
Aligner
Featur...
Label
Aligner
Label...
Specification
Specificati...
Learnware Client





Learnware Client...
login
login
BeimingWu Server



BeimingWu Server...
search
search
upload
upload
delete
delete
update
update
load
load
Network
Network
Statistical Specification
Statistical Specification
Semantic Specification










Semantic Specificati...
Data
Data
Feature
Feature
Task
Task
Label
Label
Library
Library
Description
Descripti...
Name
Name
Scenario
Scenario
License
License
Others
Others
System Specification
System Specification
Regular Specification
Regular Specification
RKMEStatSpecification










RKMEStatSpecification...
Data Set


Data Set...
Table
Table
Image
Image
Text
Text
RKME
RKME
Reduced Set


Reduced Set...
Weights
Weights
Reduced Points
Reduced Point...
HeteroMapSpecification









HeteroMapSpecification...
RKME Reduced Set
RKME Reduced Set
HeteroMapping
HeteroMapping
Hetero Reduced Set



Hetero Reduced Set...
Weights
Weights
Homo Embeddings
Homo Embeddings
Organizer
Organizer
Searcher
Searcher
EasyOrganizer
EasyOrganizer
HeteroOrganizer
HeteroOrganizer
Database



Database...
SqlLite
SqlLite
PostgresSql
PostgresSq...
HeteroMapping
HeteroMapping
HeteroMap
Specification
HeteroMap...
update
upd...
generate
gen...
save
save
load
load
save
save
load
load
Homo
StatSearcher
Homo...
Semantic Searcher



Semantic Searcher...
Checker











Checker...
Conda Checker








Conda Checker...
Semantic Checker
Semantic Checke...
Executable
Learnwares



Executable...
ModelContainer
ModelContaine...
Statistical Checker
Statistical Che...
check
che...
HeteroMap
Specification
HeteroMap...
RKMEStat
Specification
RKMEStat...
Heterom
StatSearcher
Heterom...
Fuzzy
Semantic Searcher
Fuzzy...
Exact
Semantic Searcher
Exact...
match
mat...
match
mat...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/start/intro.rst b/docs/start/intro.rst index 9d0e753..947d081 100644 --- a/docs/start/intro.rst +++ b/docs/start/intro.rst @@ -76,11 +76,44 @@ Procedure of Learnware Paradigm :align: center -Learnware Package Design -========================== +Framework and Architecture Design +================================== .. image:: ../_static/img/learnware_framework.svg :align: center -At the workflow level, the ``learnware`` package consists of ``Submitting Stage`` and ``Deploying Stage``. -At the module level, the ``learnware`` package is a platform that consists of above components. The components are designed as loose-coupled modules and each component could be used stand-alone. +The architecture is designed based on the guidelines including *decoupling*, *autonomy*, *reusability*, and *scalability*. The above architecture diagram illustrates the architecture and framework from the perspectives of both modules and workflows. + +- At the workflow level, the ``learnware`` package consists of ``Submitting Stage`` and ``Deploying Stage``. + ++---------------------+-------------------------------------------------------------------------------------------------------------------+ +| Module | Workflow | ++=====================+===================================================================================================================+ +| ``Submitting Stage``| The learnware developers submit learnwares to the learnware market, which conducts usability checks and further | +| | organization of these learnwares. | ++---------------------+-------------------------------------------------------------------------------------------------------------------+ +| ``Deploying Stage`` | The `learnware` package identifies learnwares according to users’ task requirements and provides efficient | +| | reuse and deployment methods. | ++---------------------+-------------------------------------------------------------------------------------------------------------------+ + +- At the module level, the ``learnware`` package is a platform that consists of ``Learnware``, ``Market``, ``Specification``, ``Model``, ``Reuse``, and ``Interface`` modules. + ++------------------+------------------------------------------------------------------------------------------------------------+ +| Module | Description | ++==================+============================================================================================================+ +| ``Learnware`` | The specific learnware, consisting of specification module, and user model module. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``Market`` | Designed for learnware organization, identification, and usability testing. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``Specification``| Generating and storing statistical and semantic information of learnware, which can be used for learnware | +| | search and reuse. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``Model`` | Including the base model and the model container, which can provide unified interfaces and automatically | +| | create isolated runtime environments. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``Reuse`` | Including the data-free reuser, data-dependent reuser, and aligner, which can deploy and reuse learnware | +| | for user tasks. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``Interface`` | The interface for network communication with the `Beimingwu` backend. | ++------------------+------------------------------------------------------------------------------------------------------------+ +