[DOC] modify Quick Start and other module intros

2 years ago · 3b45e4484d
--- a/docs/Intro/Basics.rst
+++ b/docs/Intro/Basics.rst
@@ -1,3 +1,11 @@
 **Learn the Basics** ||
 `Quick Start <QuickStart.html>`_ ||
 `Dataset & Data Structure <Datasets.html>`_ ||
 `Machine Learning Part <Learning.html>`_ ||
 `Reasoning Part <Reasoning.html>`_ ||
 `Evaluation Metrics <Evaluation.html>`_ ||
 `Bridge <Bridge.html>`_ 

 Learn the Basics
 ================

--- a/docs/Intro/Bridge.rst
+++ b/docs/Intro/Bridge.rst
@@ -1,46 +1,67 @@
 .. _
 `Learn the Basics <Basics.html>`_ ||
 `Quick Start <QuickStart.html>`_ ||
 `Dataset & Data Structure <Datasets.html>`_ ||
 `Machine Learning Part <Learning.html>`_ ||
 `Reasoning Part <Reasoning.html>`_ ||
 `Evaluation Metrics <Evaluation.html>`_ ||
 **Bridge**

 Bridge the machine learning and reasoning parts
 ===============================================

 We next need to bridge the machine learning and reasoning parts. In ABL-Package, the ``BaseBridge`` class gives necessary abstract interface definitions to bridge the two parts and ``SimpleBridge`` provides a basic implementation. 
 We build a bridge with previously defined ``model``, ``reasoner``, and ``metric_list`` as follows:
 Bridge
 ======

 .. code:: python
 Bridging machine learning and reasoning to train the model is the fundamental idea of Abductive Learning, ABL-Package implements a set of `bridge class <../API/abl.bridge.html>`_ to achieve this.

    bridge = SimpleBridge(model, reasoner, metric_list)
 ``BaseBridge`` is an abstract class with the following initialization parameters:

 ``BaseBridge.train`` and ``BaseBridge.test`` trigger the training and testing processes, respectively.
 - ``model``: an object of type ``ABLModel``. Machine Learning part are wrapped in this object.
 - ``reasoner``: a object of type ``ReasonerBase``. Reasoning part are wrapped in this object.

 The two methods take the previous prepared ``train_data`` and ``test_data`` as input.
 ``BaseBridge`` has the following important methods that need to be overridden in subclasses:

 .. code:: python
 +-----------------------------------+--------------------------------------------------------------------------------------+
 | Method Signature                  | Description                                                                          |
 +===================================+======================================================================================+
 | predict(data_samples)             | Predicts class probabilities and indices for the given data samples.                 |
 +-----------------------------------+--------------------------------------------------------------------------------------+
 | abduce_pseudo_label(data_samples) | Abduces pseudo labels for the given data samples.                                    |
 +-----------------------------------+--------------------------------------------------------------------------------------+
 | idx_to_pseudo_label(data_samples) | Converts indices to pseudo labels using the provided or default mapping.             |
 +-----------------------------------+--------------------------------------------------------------------------------------+
 | pseudo_label_to_idx(data_samples) | Converts pseudo labels to indices using the provided or default remapping.           |
 +-----------------------------------+--------------------------------------------------------------------------------------+
 | train(train_data)                 | Train the model.                                                                     |
 +-----------------------------------+--------------------------------------------------------------------------------------+
 | test(test_data)                   | Test the model.                                                                      |
 +-----------------------------------+--------------------------------------------------------------------------------------+

    bridge.train(train_data)
    bridge.test(test_data)

 Aside from data, ``BaseBridge.train`` can also take some other training configs shown as follows:
 ``SimpleBridge`` inherits from ``BaseBridge`` and provides a basic implementation. Besides the ``model`` and ``reasoner``, ``SimpleBridge`` has an extra initialization arguments, ``metric_list``, which will be used to evaluate model performance. Its training process involves several Abductive Learning loops and each loop consists of the following five steps:

 .. code:: python
  1. Predict class probabilities and indices for the given data samples.
  2. Transform indices into pseudo labels.
  3. Revise pseudo labels based on abdutive reasoning.
  4. Transform the revised pseudo labels to indices.
  5. Train the model.

    bridge.train(
        # training data
        train_data,
        # number of Abductive Learning loops
        loops=5,
        # data will be divided into segments and each segment will be used to train the model iteratively
        segment_size=10000,
        # evaluate the model every eval_interval loops
        eval_interval=1,
        # save the model every save_interval loops
        save_interval=1,
        # directory to save the model
        save_dir='./save_dir',
    )
 The fundamental part of the ``train`` method is as follows:

 In the MNIST Add example, the code to train and test looks like
 .. code-block:: python

 .. code:: python
    def train(self, train_data, loops=50, segment_size=10000):
        if isinstance(train_data, ListData):
            data_samples = train_data
        else:
            data_samples = self.data_preprocess(*train_data)

        for loop in range(loops):
            for seg_idx in range((len(data_samples) - 1) // segment_size + 1):
                sub_data_samples = data_samples[
                    seg_idx * segment_size : (seg_idx + 1) * segment_size
                ]
                self.predict(sub_data_samples)                  # 1
                self.idx_to_pseudo_label(sub_data_samples)      # 2
                self.abduce_pseudo_label(sub_data_samples)      # 3
                self.pseudo_label_to_idx(sub_data_samples)      # 4
                loss = self.model.train(sub_data_samples)       # 5, self.model is an ABLModel object

    bridge.train(train_data, loops=5, segment_size=10000, save_interval=1, save_dir=weights_dir)
    bridge.test(test_data)
--- a/docs/Intro/Datasets.rst
+++ b/docs/Intro/Datasets.rst
@@ -1,13 +1,59 @@
 Prepare datasets
 ================
 `Learn the Basics <Basics.html>`_ ||
 `Quick Start <QuickStart.html>`_ ||
 **Dataset & Data Structure** ||
 `Machine Learning Part <Learning.html>`_ ||
 `Reasoning Part <Reasoning.html>`_ ||
 `Evaluation Metrics <Evaluation.html>`_ ||
 `Bridge <Bridge.html>`_

 Next, we need to build datasets. ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` where ``X`` is the input of the machine learning model, ``Y`` is the ground truth of the reasoning result and ``gt_pseudo_label`` is the ground truth label of each element in ``X``. ``X`` should be of type ``List[List[Any]]``, ``Y`` should be of type ``List[Any]`` and ``gt_pseudo_label`` can be ``None`` or of the type ``List[List[Any]]``. 

 In the MNIST Add example, the data loading looks like
 Dataset & Data Structure
 ========================

 .. code:: python
 Dataset
 -------

    # train_data and test_data are all tuples consist of X, gt_pseudo_label and Y.
    train_data = get_mnist_add(train=True, get_pseudo_label=True)
    test_data = get_mnist_add(train=False, get_pseudo_label=True)
 ABL-Package offers several `dataset classes <../API/abl.dataset.html>`_ for different usage, such as ``ClassificationDataset``, ``RegressionDataset`` and ``PredictionDataset``, while users are only required to organize the dataset into a tuple consists of the following three components

 - ``X``: List[List[Any]]
    A list of instances representing the input data. We refer to each List in ``X`` as an instance and one instance may contain several elements.
 - ``gt_pseudo_label``: List[List[Any]], optional
    A list of objects representing the ground truth label of each element in ``X``. It should have the same shape as ``X``. This component is only used to evaluate the performance of the machine learning part but not to train the model. If elements are unlabeled, this component can be ``None``.
 - ``Y``: List[Any]
    A list of objects representing the ground truth label of the reasoning result of each instance in ``X``.

 In the MNIST Add example, the data used for training looks like:

 .. image:: ../img/Datasets_1.png
   :width: 350px
   :align: center

 Data Structure
 --------------

 In Abductive Learning, there are various types of data in the training and testing process, such as raw data, pseudo label, index of the pseudo label, abduced pseudo label, etc. To make the interface stable and possessing good versatility, ABL-Package uses `abstract data interfaces <../API/abl.structures.html>`_ to encapsulate various data during the implementation of the model.

 One of the most commonly used abstract data interface is ``ListData``. Besides orginizing data into tuple, we can also prepare data to be in the form of this data interface.

 .. code-block:: python

    import torch
    from abl.structures import ListData

    # prepare data
    X = [list(torch.randn(3, 28, 28)), list(torch.randn(3, 28, 28))]
    gt_pseudo_label = [[1, 2, 3], [4, 5, 6]]
    Y = [1, 2]

    # convert data into ListData
    data = ListData(X=X, Y=Y, gt_pseudo_label=gt_pseudo_label)

    # get data
    X = data.X
    Y = data.Y
    gt_pseudo_label = data.gt_pseudo_label

    # set data
    data.X = X
    data.Y = Y
    data.gt_pseudo_label = gt_pseudo_label
--- a/docs/Intro/Evaluation.rst
+++ b/docs/Intro/Evaluation.rst
@@ -1,12 +1,40 @@
 Define Evaluation Metrics
 =========================
 `Learn the Basics <Basics.html>`_ ||
 `Quick Start <QuickStart.html>`_ ||
 `Dataset & Data Structure <Datasets.html>`_ ||
 `Machine Learning Part <Learning.html>`_ ||
 `Reasoning Part <Reasoning.html>`_ ||
 **Evaluation Metrics** ||
 `Bridge <Bridge.html>`_

 To validate and test the model, we need to inherit from ``BaseMetric`` to define metrics and implement the ``process`` and ``compute_metrics`` methods where the process method accepts a batch of outputs. After processing this batch of data, we save the information to ``self.results`` property. The input results of ``compute_metrics`` is all the information saved in ``process``. Use these information to calculate and return a dict that holds the results of the evaluation metrics. 

 We provide two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively.
 Evaluation Metrics
 ==================

 In the case of MNIST Add example, the metric definition looks like
 ABL-Package seperates the evaluation process as na independent class from the ``BaseBridge`` which accounts for training and testing. To customize our own metrics, we need to inherit from ``BaseMetric`` and implement the ``process`` and ``compute_metrics`` methods. The ``process`` method accepts a batch of model prediction. After processing this batch, we save the information to ``self.results`` property. The input results of ``compute_metrics`` is all the information saved in ``process`` and it uses these information to calculate and return a dict that holds the evaluation results. 

 We provide two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. Using ``SymbolMetric`` as an example, the following code shows how to implement a custom metrics.

 .. code:: python

    metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")]
    class SymbolMetric(BaseMetric):
        def __init__(self, prefix: Optional[str] = None) -> None:
            # prefix is used to distinguish different metrics
            super().__init__(prefix)

        def process(self, data_samples: Sequence[dict]) -> None:
            # pred_pseudo_label and gt_pseudo_label are both of type List[List[Any]] 
            # and have the same length
            pred_pseudo_label = data_samples.pred_pseudo_label
            gt_pseudo_label = data_samples.gt_pseudo_label
            
            for pred_z, z in zip(pred_pseudo_label, gt_pseudo_label):
                correct_num = 0
                for pred_symbol, symbol in zip(pred_z, z):
                    if pred_symbol == symbol:
                        correct_num += 1
                self.results.append(correct_num / len(z))
        
        def compute_metrics(self, results: list) -> dict:
            metrics = dict()
            metrics["character_accuracy"] = sum(results) / len(results)
            return metrics
--- a/docs/Intro/Learning.rst
+++ b/docs/Intro/Learning.rst
@@ -1,46 +1,64 @@
 Build the machine learning part
 ===============================
 `Learn the Basics <Basics.html>`_ ||
 `Quick Start <QuickStart.html>`_ ||
 `Dataset & Data Structure <Datasets.html>`_ ||
 **Machine Learning Part** ||
 `Reasoning Part <Reasoning.html>`_ ||
 `Evaluation Metrics <Evaluation.html>`_ ||
 `Bridge <Bridge.html>`_

 First, we build the machine learning part, which needs to be wrapped in the ``ABLModel`` class. We can use machine learning models from scikit-learn or based on PyTorch to create an instance of ``ABLModel``. 

 - for a scikit-learn model, we can directly use the model to create an instance of ``ABLModel``. For example, we can customize our machine learning model by
 Machine Learning Part
 =====================

  .. code:: python
 ``ABLModel`` class serves as a unified interface to all machine learning models. Its constructor, the ``__init__`` method, takes a singular argument, ``base_model``. This argument denotes the fundamental machine learning model, which must implement the ``fit`` and ``predict`` methods.

      # Load a scikit-learn model
      base_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=3)
 .. code:: python

      model = ABLModel(base_model)
    class ABLModel:
        def __init__(self, base_model: Any) -> None:
            if not (hasattr(base_model, "fit") and hasattr(base_model, "predict")):
                raise NotImplementedError("The base_model should implement fit and predict methods.")

 - for a PyTorch-based neural network, we first need to encapsulate it within a ``BasicNN`` object and then use this object to instantiate an instance of ``ABLModel``.  For example, we can customize our machine learning model by
            self.base_model = base_model

  .. code:: python
 All scikit-learn models satisify this requiremnts, so we can directly use the model to create an instance of ``ABLModel``. For example, we can customize our machine learning model by

      # Load a PyTorch-based neural network
      cls = torchvision.models.resnet18(pretrained=True)
 .. code:: python

      # criterion and optimizer are used for training
      criterion = torch.nn.CrossEntropyLoss() 
      optimizer = torch.optim.Adam(cls.parameters())
    import sklearn
    from abl.learning import ABLModel

      base_model = BasicNN(cls, criterion, optimizer)
      model = ABLModel(base_model)
    base_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=3)
    model = ABLModel(base_model)

 In the MNIST Add example, the machine learning model looks like
 For a PyTorch-based neural network, we first need to encapsulate it within a ``BasicNN`` object and then use this object to instantiate an instance of ``ABLModel``.  For example, we can customize our machine learning model by

 .. code:: python

    cls = LeNet5(num_classes=10)
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99))

    base_model = BasicNN(
        cls,
        criterion,
        optimizer,
        device=device,
        batch_size=32,
        num_epochs=1,
    )
    model = ABLModel(base_model)
    # Load a PyTorch-based neural network
    cls = torchvision.models.resnet18(pretrained=True)

    # criterion and optimizer are used for training
    criterion = torch.nn.CrossEntropyLoss() 
    optimizer = torch.optim.Adam(cls.parameters())

    base_model = BasicNN(cls, criterion, optimizer)
    model = ABLModel(base_model)

 Besides ``fit`` and ``predict``, ``BasicNN`` also implements the following methods:

 +---------------------------+----------------------------------------+
 | Method                    | Function                               |
 +===========================+========================================+
 | train_epoch(data_loader)  | Train the neural network for one epoch.|
 +---------------------------+----------------------------------------+
 | predict_proba(X)          | Predict the class probabilities of X.  |
 +---------------------------+----------------------------------------+
 | score(X, y)               | Calculate the accuracy of the model on |
 |                           | test data.                             |
 +---------------------------+----------------------------------------+
 | save(epoch_id, save_path) | Save the model.                        |
 +---------------------------+----------------------------------------+
 | load(load_path)           | Load the model.                        |
 +---------------------------+----------------------------------------+

--- a/docs/Intro/Quick
+++ b/docs/Intro/Quick
@@ -1,113 +0,0 @@
 Quick Start
 ===========

 We use the MNIST Add benchmark as a quick start example. In this task, the inputs are 
 pairs of MNIST handwritten images, and the outputs are their sums. 
 To complete this task, we first process the images through a machine learning model 
 to get their corresponding pseudo labels (the number each image represents). 
 Then, the recognized labels undergo logical reasoning which calculates their sum. 

 Load Data
 ---------

 ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` 
 where ``X`` is the input of the machine learning model, 
 ``Y`` is the ground truth of the reasoning result and 
 ``gt_pseudo_label`` is the ground truth label of each element in ``X``. 

 .. code:: python

    from examples.mnist_add.datasets.get_mnist_add import get_mnist_add

    train_data = get_mnist_add(train=True, get_pseudo_label=True)
    test_data = get_mnist_add(train=False, get_pseudo_label=True)

 In the ``get_mnist_add`` above, the return values are tuples of ``(X, gt_pseudo_label, Y)``.

 Read more about `prepare datasets <Datasets.html>`_.

 Build Machine Learning Models
 -----------------------------

 We use a simple LeNet5 model to recognize the pseudo labels (numbers) in the images. 
 We first build the model and define its corresponding criterion and optimizer for training.

 .. code:: python

    import torch
    import torch.nn as nn
    from examples.models.nn import LeNet5

    cls = LeNet5(num_classes=10)
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99))
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

 Afterward, we wrap it in ``ABLModel``.

 .. code:: python

    from abl.learning import ABLModel, BasicNN

    base_model = BasicNN(cls, criterion, optimizer, device)
    model = ABLModel(base_model)

 Read more about `build machine learning models <Learning.html>`_.

 Build the Reasoning Part
 ------------------------

 First, we build a knowledge base that defines how to deduce 
 logical results (i.e., calculate summation) from the pseudo labels 
 obtained by machine learning.

 .. code:: python

    from abl.reasoning import KBBase, ReasonerBase

    class AddKB(KBBase):
        def __init__(self, pseudo_label_list=list(range(10))):
            super().__init__(pseudo_label_list)

        def logic_forward(self, nums):
            return sum(nums)


    kb = AddKB(pseudo_label_list=list(range(10)))

 Then, we define a reasoner, which defines 
 how to minimize the inconsistency between the knowledge base and machine learning.

 .. code:: python

    reasoner = ReasonerBase(kb, dist_func="confidence")  

 Read more about `build the reasoning part <Reasoning.html>`_.  

 Bridge Machine Learning and Reasoning
 -------------------------------------

 Before bridging, we first define the metrics to measure accuracy during validation and testing.

 .. code:: python

    from abl.evaluation import SemanticsMetric, SymbolMetric

    metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")]


 Now, we may use ``SimpleBridge`` to combine machine learning and reasoning together,
 setting the stage for subsequent integrated training, validation, and testing.

 .. code:: python

    from abl.bridge import SimpleBridge

 Finally, we proceed with testing and training.

 .. code:: python

    bridge.train(train_data, loops=5, segment_size=10000)
    bridge.test(test_data)

 Read more about `defining evaluation metrics <Evaluation.html>`_ and `bridge machine learning and reasoning <Bridge.html>`_.
--- a/docs/Intro/QuickStart.rst
+++ b/docs/Intro/QuickStart.rst
@@ -0,0 +1,223 @@
 `Learn the Basics <Basics.html>`_ ||
 **Quick Start** ||
 `Dataset & Data Structure <Datasets.html>`_ ||
 `Machine Learning Part <Learning.html>`_ ||
 `Reasoning Part <Reasoning.html>`_ ||
 `Evaluation Metrics <Evaluation.html>`_ ||
 `Bridge <Bridge.html>`_ 

 Quick Start
 ===========


 This section runs through the API for the neural-symbolic task, MNITST Add. Refer to the links in each section to dive deeper.

 Working with Data
 -----------------

 ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)``  where ``X`` is the input of the machine learning model, 
 ``gt_pseudo_label`` is the ground truth label of each element in ``X`` and ``Y`` is the ground truth reasoning result of each instance in ``X``. 

 In the MNIST Add task, the data loading looks like

 .. code:: python

   from examples.mnist_add.datasets.get_mnist_add import get_mnist_add
   
   # train_data and test_data are all tuples consist of X, gt_pseudo_label and Y.
   # If get_pseudo_label is False, gt_pseudo_label will be None
   train_data = get_mnist_add(train=True, get_pseudo_label=True)
   test_data = get_mnist_add(train=False, get_pseudo_label=True)

 ABL-Package assumes ``X`` to be of type ``List[List[Any]]``, ``gt_pseudo_label`` can be ``None`` or of the type ``List[List[Any]]`` and ``Y`` should be of type ``List[Any]``. 

 .. code:: python

   def describe_structure(lst):
      if not isinstance(lst, list):
         return type(lst).__name__ 
      return [describe_structure(item) for item in lst]
    
   X, gt_pseudo_label, Y = train_data

   print(f"Length of X List[List[Any]]: {len(X)}")
   print(f"Length of gt_pseudo_label List[List[Any]]: {len(gt_pseudo_label)}")
   print(f"Length of Y List[Any]: {len(Y)}\n")

   structure_X = describe_structure(X[:3])
   print(f"Structure of X: {structure_X}")
   structure_gt_pseudo_label = describe_structure(gt_pseudo_label[:3])
   print(f"Structure of gt_pseudo_label: {structure_gt_pseudo_label}")
   structure_Y = describe_structure(Y[:3])
   print(f"Structure of Y: {structure_Y}\n")

   print(f"Shape of X [C, H, W]: {X[0][0].shape}")

 Out:

 .. code-block:: none

   Length of X List[List[Any]]: 30000
   Length of gt_pseudo_label List[List[Any]]: 30000
   Length of gt_pseudo_label List[Any]: 30000

   Structure of X: [['Tensor', 'Tensor'], ['Tensor', 'Tensor'], ['Tensor', 'Tensor']]                   
   Structure of gt_pseudo_label: [['int', 'int'], ['int', 'int'], ['int', 'int']]
   Structure of Y: ['int', 'int', 'int']

   Shape of X [C, H, W]: torch.Size([1, 28, 28])


 ABL-Package offers several `dataset classes <../API/abl.dataset.html>`_ for different usage, such as ``ClassificationDataset``, ``RegressionDataset`` and ``PredictionDataset``, while users are only required to organize the dataset into the aforementioned format. 

 Read more about `preparing datasets <Datasets.html>`_.

 Building the Machine Learning Part
 ----------------------------------

 To build the machine learning part, we need to wrap our machine learning model into the ``ABLModel`` class. The machine learning model can either be a scikit-learn model or a PyTorch neural network. We use a simple LeNet5 in the MNIST Add example.

 .. code:: python

   from examples.models.nn import LeNet5

   # The number of pseudo labels is 10
   cls = LeNet5(num_classes=10)

 Aside from the network, we need to define a criterion, an optimizer, and a device so as to create a ``BasicNN`` object. This class implements ``fit``, ``predict``, ``predict_proba`` and several other methods to enable the PyTorch-based neural network to work as a scikit-learn model.

 .. code:: python

   import torch
   from abl.learning import BasicNN

   criterion = torch.nn.CrossEntropyLoss()
   optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99))
   device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
   base_model = BasicNN(cls, criterion, optimizer, device)

 .. code:: python

   pred_idx = base_model.predict(X=[torch.randn(1, 28, 28).to(device) for _ in range(32)])
   print(f"Shape of pred_idx : {pred_idx.shape}")
   pred_prob = base_model.predict_proba(X=[torch.randn(1, 28, 28).to(device) for _ in range(32)])
   print(f"Shape of pred_prob : {pred_prob.shape}")

 Out:

 .. code-block:: none

   Shape of pred_idx : (32,)
   Shape of pred_prob : (32, 10)

 Afterward, we wrap the ``base_model`` into ``ABLModel``.

 .. code:: python

    from abl.learning import ABLModel

    model = ABLModel(base_model)

 Read more about `building the machine learning part <Learning.html>`_.

 Building the Reasoning Part
 ---------------------------

 To build the reasoning part, we first create a class that inherits from ``KBBase`` to define the knowledge base.
 We pass the list of pseudo labels to the ``__init__`` function and specify how to deduce logical results in the ``logic_forward`` function.

 .. code:: python

   from abl.reasoning import KBBase

   class AddKB(KBBase):
      def __init__(self, pseudo_label_list=list(range(10))):
         super().__init__(pseudo_label_list)

      def logic_forward(self, nums):
         return sum(nums)

   kb = AddKB(pseudo_label_list=list(range(10)))
   print(kb)

 Out:

 .. code-block:: none

   AddKB is a KB with pseudo_label_list=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], max_err=1e-10, use_cache=True.

 Then, we create a reasoner. Aside from the knowledge base, the instantiation of ``ReasonerBase`` also needs to set an extra argument called ``dist_func``, which measures the consistency between the knowledge base and machine learning.

 .. code:: python

   from abl.reasoning import ReasonerBase
   
   reasoner = ReasonerBase(kb, dist_func="confidence")

 Read more about `building the reasoning part <Reasoning.html>`_. 


 Building Evaluation Metrics
 ---------------------------

 ABL-Package provides two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively.

 In the case of MNIST Add example, the metric definition looks like

 .. code:: python

   from abl.evaluation import SemanticsMetric, SymbolMetric

   metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")]

 Read more about `building evaluation metrics <Evaluation.html>`_

 Bridging Machine Learning and Reasoning
 ---------------------------------------

 Now, we use ``SimpleBridge`` to combine machine learning and reasoning together.

 .. code:: python

   from abl.bridge import SimpleBridge

   bridge = SimpleBridge(model, reasoner, metric_list)

 Finally, we proceed with testing and training.

 .. code:: python

   bridge.train(train_data, loops=5, segment_size=10000)
   bridge.test(test_data)

 Training log would be similar to this:

 .. code-block:: none

   2023/12/02 21:26:57 - abl - INFO - Abductive Learning on the MNIST Add example.
   2023/12/02 21:32:20 - abl - INFO - Abductive Learning on the MNIST Add example.
   2023/12/02 21:32:51 - abl - INFO - loop(train) [1/5] segment(train) [1/3] model loss is 1.85589
   2023/12/02 21:32:56 - abl - INFO - loop(train) [1/5] segment(train) [2/3] model loss is 1.50332
   2023/12/02 21:33:02 - abl - INFO - loop(train) [1/5] segment(train) [3/3] model loss is 1.17501
   2023/12/02 21:33:02 - abl - INFO - Evaluation start: loop(val) [1]
   2023/12/02 21:33:07 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.350 mnist_add/semantics_accuracy: 0.254 
   2023/12/02 21:33:07 - abl - INFO - Saving model: loop(save) [1]
   2023/12/02 21:33:07 - abl - INFO - Checkpoints will be saved to results/20231202_21_26_57/weights/model_checkpoint_loop_1.pth
   2023/12/02 21:33:13 - abl - INFO - loop(train) [2/5] segment(train) [1/3] model loss is 0.97188
   2023/12/02 21:33:18 - abl - INFO - loop(train) [2/5] segment(train) [2/3] model loss is 0.85622
   2023/12/02 21:33:24 - abl - INFO - loop(train) [2/5] segment(train) [3/3] model loss is 0.81511
   2023/12/02 21:33:24 - abl - INFO - Evaluation start: loop(val) [2]
   2023/12/02 21:33:29 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.546 mnist_add/semantics_accuracy: 0.399 
   2023/12/02 21:33:29 - abl - INFO - Saving model: loop(save) [2]
   ...
   2023/12/02 21:34:17 - abl - INFO - loop(train) [5/5] segment(train) [1/3] model loss is 0.03935
   2023/12/02 21:34:23 - abl - INFO - loop(train) [5/5] segment(train) [2/3] model loss is 0.03716
   2023/12/02 21:34:28 - abl - INFO - loop(train) [5/5] segment(train) [3/3] model loss is 0.03346
   2023/12/02 21:34:28 - abl - INFO - Evaluation start: loop(val) [5]
   2023/12/02 21:34:33 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.993 mnist_add/semantics_accuracy: 0.986 
   2023/12/02 21:34:33 - abl - INFO - Saving model: loop(save) [5]
   2023/12/02 21:34:33 - abl - INFO - Checkpoints will be saved to results/20231202_21_26_57/weights/model_checkpoint_loop_5.pth
   2023/12/02 21:34:34 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.989 mnist_add/semantics_accuracy: 0.978 


 Read more about `bridging machine learning and reasoning <Bridge.html>`_.
--- a/docs/Intro/Reasoning.rst
+++ b/docs/Intro/Reasoning.rst
@@ -1,3 +1,12 @@
 Build the reasoning part
 ========================
 `Learn the Basics <Basics.html>`_ ||
 `Quick Start <QuickStart.html>`_ ||
 `Dataset & Data Structure <Datasets.html>`_ ||
 `Machine Learning Part <Learning.html>`_ ||
 **Reasoning Part** ||
 `Evaluation Metrics <Evaluation.html>`_ ||
 `Bridge <Bridge.html>`_


 Reasoning part
 ===============

--- a/docs/img/Datasets_1.png
+++ b/docs/img/Datasets_1.png
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -12,7 +12,7 @@
   :caption: Introduction to ABL-Package

   Intro/Basics
   Intro/Quick Start
   Intro/QuickStart
   Intro/Datasets
   Intro/Learning
   Intro/Reasoning