diff --git a/docs/Intro/Basics.rst b/docs/Intro/Basics.rst index a6bae42..46a2c1b 100644 --- a/docs/Intro/Basics.rst +++ b/docs/Intro/Basics.rst @@ -1,3 +1,11 @@ +**Learn the Basics** || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ + Learn the Basics ================ diff --git a/docs/Intro/Bridge.rst b/docs/Intro/Bridge.rst index a94ae1e..2529ff4 100644 --- a/docs/Intro/Bridge.rst +++ b/docs/Intro/Bridge.rst @@ -1,46 +1,67 @@ -.. _ +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +**Bridge** -Bridge the machine learning and reasoning parts -=============================================== -We next need to bridge the machine learning and reasoning parts. In ABL-Package, the ``BaseBridge`` class gives necessary abstract interface definitions to bridge the two parts and ``SimpleBridge`` provides a basic implementation. -We build a bridge with previously defined ``model``, ``reasoner``, and ``metric_list`` as follows: +Bridge +====== -.. code:: python +Bridging machine learning and reasoning to train the model is the fundamental idea of Abductive Learning, ABL-Package implements a set of `bridge class <../API/abl.bridge.html>`_ to achieve this. - bridge = SimpleBridge(model, reasoner, metric_list) +``BaseBridge`` is an abstract class with the following initialization parameters: -``BaseBridge.train`` and ``BaseBridge.test`` trigger the training and testing processes, respectively. +- ``model``: an object of type ``ABLModel``. Machine Learning part are wrapped in this object. +- ``reasoner``: a object of type ``ReasonerBase``. Reasoning part are wrapped in this object. -The two methods take the previous prepared ``train_data`` and ``test_data`` as input. +``BaseBridge`` has the following important methods that need to be overridden in subclasses: -.. code:: python ++-----------------------------------+--------------------------------------------------------------------------------------+ +| Method Signature | Description | ++===================================+======================================================================================+ +| predict(data_samples) | Predicts class probabilities and indices for the given data samples. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| abduce_pseudo_label(data_samples) | Abduces pseudo labels for the given data samples. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| idx_to_pseudo_label(data_samples) | Converts indices to pseudo labels using the provided or default mapping. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| pseudo_label_to_idx(data_samples) | Converts pseudo labels to indices using the provided or default remapping. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| train(train_data) | Train the model. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| test(test_data) | Test the model. | ++-----------------------------------+--------------------------------------------------------------------------------------+ - bridge.train(train_data) - bridge.test(test_data) -Aside from data, ``BaseBridge.train`` can also take some other training configs shown as follows: +``SimpleBridge`` inherits from ``BaseBridge`` and provides a basic implementation. Besides the ``model`` and ``reasoner``, ``SimpleBridge`` has an extra initialization arguments, ``metric_list``, which will be used to evaluate model performance. Its training process involves several Abductive Learning loops and each loop consists of the following five steps: -.. code:: python + 1. Predict class probabilities and indices for the given data samples. + 2. Transform indices into pseudo labels. + 3. Revise pseudo labels based on abdutive reasoning. + 4. Transform the revised pseudo labels to indices. + 5. Train the model. - bridge.train( - # training data - train_data, - # number of Abductive Learning loops - loops=5, - # data will be divided into segments and each segment will be used to train the model iteratively - segment_size=10000, - # evaluate the model every eval_interval loops - eval_interval=1, - # save the model every save_interval loops - save_interval=1, - # directory to save the model - save_dir='./save_dir', - ) +The fundamental part of the ``train`` method is as follows: -In the MNIST Add example, the code to train and test looks like +.. code-block:: python -.. code:: python + def train(self, train_data, loops=50, segment_size=10000): + if isinstance(train_data, ListData): + data_samples = train_data + else: + data_samples = self.data_preprocess(*train_data) + + for loop in range(loops): + for seg_idx in range((len(data_samples) - 1) // segment_size + 1): + sub_data_samples = data_samples[ + seg_idx * segment_size : (seg_idx + 1) * segment_size + ] + self.predict(sub_data_samples) # 1 + self.idx_to_pseudo_label(sub_data_samples) # 2 + self.abduce_pseudo_label(sub_data_samples) # 3 + self.pseudo_label_to_idx(sub_data_samples) # 4 + loss = self.model.train(sub_data_samples) # 5, self.model is an ABLModel object - bridge.train(train_data, loops=5, segment_size=10000, save_interval=1, save_dir=weights_dir) - bridge.test(test_data) diff --git a/docs/Intro/Datasets.rst b/docs/Intro/Datasets.rst index d5eaa7b..0ea1be6 100644 --- a/docs/Intro/Datasets.rst +++ b/docs/Intro/Datasets.rst @@ -1,13 +1,59 @@ -Prepare datasets -================ +`Learn the Basics `_ || +`Quick Start `_ || +**Dataset & Data Structure** || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ -Next, we need to build datasets. ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` where ``X`` is the input of the machine learning model, ``Y`` is the ground truth of the reasoning result and ``gt_pseudo_label`` is the ground truth label of each element in ``X``. ``X`` should be of type ``List[List[Any]]``, ``Y`` should be of type ``List[Any]`` and ``gt_pseudo_label`` can be ``None`` or of the type ``List[List[Any]]``. -In the MNIST Add example, the data loading looks like +Dataset & Data Structure +======================== -.. code:: python +Dataset +------- - # train_data and test_data are all tuples consist of X, gt_pseudo_label and Y. - train_data = get_mnist_add(train=True, get_pseudo_label=True) - test_data = get_mnist_add(train=False, get_pseudo_label=True) +ABL-Package offers several `dataset classes <../API/abl.dataset.html>`_ for different usage, such as ``ClassificationDataset``, ``RegressionDataset`` and ``PredictionDataset``, while users are only required to organize the dataset into a tuple consists of the following three components +- ``X``: List[List[Any]] + A list of instances representing the input data. We refer to each List in ``X`` as an instance and one instance may contain several elements. +- ``gt_pseudo_label``: List[List[Any]], optional + A list of objects representing the ground truth label of each element in ``X``. It should have the same shape as ``X``. This component is only used to evaluate the performance of the machine learning part but not to train the model. If elements are unlabeled, this component can be ``None``. +- ``Y``: List[Any] + A list of objects representing the ground truth label of the reasoning result of each instance in ``X``. + +In the MNIST Add example, the data used for training looks like: + +.. image:: ../img/Datasets_1.png + :width: 350px + :align: center + +Data Structure +-------------- + +In Abductive Learning, there are various types of data in the training and testing process, such as raw data, pseudo label, index of the pseudo label, abduced pseudo label, etc. To make the interface stable and possessing good versatility, ABL-Package uses `abstract data interfaces <../API/abl.structures.html>`_ to encapsulate various data during the implementation of the model. + +One of the most commonly used abstract data interface is ``ListData``. Besides orginizing data into tuple, we can also prepare data to be in the form of this data interface. + +.. code-block:: python + + import torch + from abl.structures import ListData + + # prepare data + X = [list(torch.randn(3, 28, 28)), list(torch.randn(3, 28, 28))] + gt_pseudo_label = [[1, 2, 3], [4, 5, 6]] + Y = [1, 2] + + # convert data into ListData + data = ListData(X=X, Y=Y, gt_pseudo_label=gt_pseudo_label) + + # get data + X = data.X + Y = data.Y + gt_pseudo_label = data.gt_pseudo_label + + # set data + data.X = X + data.Y = Y + data.gt_pseudo_label = gt_pseudo_label \ No newline at end of file diff --git a/docs/Intro/Evaluation.rst b/docs/Intro/Evaluation.rst index 3c1ad6b..23e500d 100644 --- a/docs/Intro/Evaluation.rst +++ b/docs/Intro/Evaluation.rst @@ -1,12 +1,40 @@ -Define Evaluation Metrics -========================= +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +**Evaluation Metrics** || +`Bridge `_ -To validate and test the model, we need to inherit from ``BaseMetric`` to define metrics and implement the ``process`` and ``compute_metrics`` methods where the process method accepts a batch of outputs. After processing this batch of data, we save the information to ``self.results`` property. The input results of ``compute_metrics`` is all the information saved in ``process``. Use these information to calculate and return a dict that holds the results of the evaluation metrics. -We provide two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. +Evaluation Metrics +================== -In the case of MNIST Add example, the metric definition looks like +ABL-Package seperates the evaluation process as na independent class from the ``BaseBridge`` which accounts for training and testing. To customize our own metrics, we need to inherit from ``BaseMetric`` and implement the ``process`` and ``compute_metrics`` methods. The ``process`` method accepts a batch of model prediction. After processing this batch, we save the information to ``self.results`` property. The input results of ``compute_metrics`` is all the information saved in ``process`` and it uses these information to calculate and return a dict that holds the evaluation results. + +We provide two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. Using ``SymbolMetric`` as an example, the following code shows how to implement a custom metrics. .. code:: python - metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")] \ No newline at end of file + class SymbolMetric(BaseMetric): + def __init__(self, prefix: Optional[str] = None) -> None: + # prefix is used to distinguish different metrics + super().__init__(prefix) + + def process(self, data_samples: Sequence[dict]) -> None: + # pred_pseudo_label and gt_pseudo_label are both of type List[List[Any]] + # and have the same length + pred_pseudo_label = data_samples.pred_pseudo_label + gt_pseudo_label = data_samples.gt_pseudo_label + + for pred_z, z in zip(pred_pseudo_label, gt_pseudo_label): + correct_num = 0 + for pred_symbol, symbol in zip(pred_z, z): + if pred_symbol == symbol: + correct_num += 1 + self.results.append(correct_num / len(z)) + + def compute_metrics(self, results: list) -> dict: + metrics = dict() + metrics["character_accuracy"] = sum(results) / len(results) + return metrics \ No newline at end of file diff --git a/docs/Intro/Learning.rst b/docs/Intro/Learning.rst index 21b82ae..ed6d1d6 100644 --- a/docs/Intro/Learning.rst +++ b/docs/Intro/Learning.rst @@ -1,46 +1,64 @@ -Build the machine learning part -=============================== +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +**Machine Learning Part** || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ -First, we build the machine learning part, which needs to be wrapped in the ``ABLModel`` class. We can use machine learning models from scikit-learn or based on PyTorch to create an instance of ``ABLModel``. -- for a scikit-learn model, we can directly use the model to create an instance of ``ABLModel``. For example, we can customize our machine learning model by +Machine Learning Part +===================== - .. code:: python +``ABLModel`` class serves as a unified interface to all machine learning models. Its constructor, the ``__init__`` method, takes a singular argument, ``base_model``. This argument denotes the fundamental machine learning model, which must implement the ``fit`` and ``predict`` methods. - # Load a scikit-learn model - base_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=3) +.. code:: python - model = ABLModel(base_model) + class ABLModel: + def __init__(self, base_model: Any) -> None: + if not (hasattr(base_model, "fit") and hasattr(base_model, "predict")): + raise NotImplementedError("The base_model should implement fit and predict methods.") -- for a PyTorch-based neural network, we first need to encapsulate it within a ``BasicNN`` object and then use this object to instantiate an instance of ``ABLModel``. For example, we can customize our machine learning model by + self.base_model = base_model - .. code:: python +All scikit-learn models satisify this requiremnts, so we can directly use the model to create an instance of ``ABLModel``. For example, we can customize our machine learning model by - # Load a PyTorch-based neural network - cls = torchvision.models.resnet18(pretrained=True) +.. code:: python - # criterion and optimizer are used for training - criterion = torch.nn.CrossEntropyLoss() - optimizer = torch.optim.Adam(cls.parameters()) + import sklearn + from abl.learning import ABLModel - base_model = BasicNN(cls, criterion, optimizer) - model = ABLModel(base_model) + base_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=3) + model = ABLModel(base_model) -In the MNIST Add example, the machine learning model looks like +For a PyTorch-based neural network, we first need to encapsulate it within a ``BasicNN`` object and then use this object to instantiate an instance of ``ABLModel``. For example, we can customize our machine learning model by .. code:: python - cls = LeNet5(num_classes=10) - device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") - criterion = torch.nn.CrossEntropyLoss() - optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99)) - - base_model = BasicNN( - cls, - criterion, - optimizer, - device=device, - batch_size=32, - num_epochs=1, - ) - model = ABLModel(base_model) \ No newline at end of file + # Load a PyTorch-based neural network + cls = torchvision.models.resnet18(pretrained=True) + + # criterion and optimizer are used for training + criterion = torch.nn.CrossEntropyLoss() + optimizer = torch.optim.Adam(cls.parameters()) + + base_model = BasicNN(cls, criterion, optimizer) + model = ABLModel(base_model) + +Besides ``fit`` and ``predict``, ``BasicNN`` also implements the following methods: + ++---------------------------+----------------------------------------+ +| Method | Function | ++===========================+========================================+ +| train_epoch(data_loader) | Train the neural network for one epoch.| ++---------------------------+----------------------------------------+ +| predict_proba(X) | Predict the class probabilities of X. | ++---------------------------+----------------------------------------+ +| score(X, y) | Calculate the accuracy of the model on | +| | test data. | ++---------------------------+----------------------------------------+ +| save(epoch_id, save_path) | Save the model. | ++---------------------------+----------------------------------------+ +| load(load_path) | Load the model. | ++---------------------------+----------------------------------------+ + diff --git a/docs/Intro/Quick Start.rst b/docs/Intro/Quick Start.rst deleted file mode 100644 index 122ecbb..0000000 --- a/docs/Intro/Quick Start.rst +++ /dev/null @@ -1,113 +0,0 @@ -Quick Start -=========== - -We use the MNIST Add benchmark as a quick start example. In this task, the inputs are -pairs of MNIST handwritten images, and the outputs are their sums. -To complete this task, we first process the images through a machine learning model -to get their corresponding pseudo labels (the number each image represents). -Then, the recognized labels undergo logical reasoning which calculates their sum. - -Load Data ---------- - -ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` -where ``X`` is the input of the machine learning model, -``Y`` is the ground truth of the reasoning result and -``gt_pseudo_label`` is the ground truth label of each element in ``X``. - -.. code:: python - - from examples.mnist_add.datasets.get_mnist_add import get_mnist_add - - train_data = get_mnist_add(train=True, get_pseudo_label=True) - test_data = get_mnist_add(train=False, get_pseudo_label=True) - -In the ``get_mnist_add`` above, the return values are tuples of ``(X, gt_pseudo_label, Y)``. - -Read more about `prepare datasets `_. - -Build Machine Learning Models ------------------------------ - -We use a simple LeNet5 model to recognize the pseudo labels (numbers) in the images. -We first build the model and define its corresponding criterion and optimizer for training. - -.. code:: python - - import torch - import torch.nn as nn - from examples.models.nn import LeNet5 - - cls = LeNet5(num_classes=10) - criterion = nn.CrossEntropyLoss() - optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99)) - device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") - -Afterward, we wrap it in ``ABLModel``. - -.. code:: python - - from abl.learning import ABLModel, BasicNN - - base_model = BasicNN(cls, criterion, optimizer, device) - model = ABLModel(base_model) - -Read more about `build machine learning models `_. - -Build the Reasoning Part ------------------------- - -First, we build a knowledge base that defines how to deduce -logical results (i.e., calculate summation) from the pseudo labels -obtained by machine learning. - -.. code:: python - - from abl.reasoning import KBBase, ReasonerBase - - class AddKB(KBBase): - def __init__(self, pseudo_label_list=list(range(10))): - super().__init__(pseudo_label_list) - - def logic_forward(self, nums): - return sum(nums) - - - kb = AddKB(pseudo_label_list=list(range(10))) - -Then, we define a reasoner, which defines -how to minimize the inconsistency between the knowledge base and machine learning. - -.. code:: python - - reasoner = ReasonerBase(kb, dist_func="confidence") - -Read more about `build the reasoning part `_. - -Bridge Machine Learning and Reasoning -------------------------------------- - -Before bridging, we first define the metrics to measure accuracy during validation and testing. - -.. code:: python - - from abl.evaluation import SemanticsMetric, SymbolMetric - - metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")] - - -Now, we may use ``SimpleBridge`` to combine machine learning and reasoning together, -setting the stage for subsequent integrated training, validation, and testing. - -.. code:: python - - from abl.bridge import SimpleBridge - -Finally, we proceed with testing and training. - -.. code:: python - - bridge.train(train_data, loops=5, segment_size=10000) - bridge.test(test_data) - -Read more about `defining evaluation metrics `_ and `bridge machine learning and reasoning `_. diff --git a/docs/Intro/QuickStart.rst b/docs/Intro/QuickStart.rst new file mode 100644 index 0000000..5b1c68b --- /dev/null +++ b/docs/Intro/QuickStart.rst @@ -0,0 +1,223 @@ +`Learn the Basics `_ || +**Quick Start** || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ + +Quick Start +=========== + + +This section runs through the API for the neural-symbolic task, MNITST Add. Refer to the links in each section to dive deeper. + +Working with Data +----------------- + +ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` where ``X`` is the input of the machine learning model, +``gt_pseudo_label`` is the ground truth label of each element in ``X`` and ``Y`` is the ground truth reasoning result of each instance in ``X``. + +In the MNIST Add task, the data loading looks like + +.. code:: python + + from examples.mnist_add.datasets.get_mnist_add import get_mnist_add + + # train_data and test_data are all tuples consist of X, gt_pseudo_label and Y. + # If get_pseudo_label is False, gt_pseudo_label will be None + train_data = get_mnist_add(train=True, get_pseudo_label=True) + test_data = get_mnist_add(train=False, get_pseudo_label=True) + +ABL-Package assumes ``X`` to be of type ``List[List[Any]]``, ``gt_pseudo_label`` can be ``None`` or of the type ``List[List[Any]]`` and ``Y`` should be of type ``List[Any]``. + +.. code:: python + + def describe_structure(lst): + if not isinstance(lst, list): + return type(lst).__name__ + return [describe_structure(item) for item in lst] + + X, gt_pseudo_label, Y = train_data + + print(f"Length of X List[List[Any]]: {len(X)}") + print(f"Length of gt_pseudo_label List[List[Any]]: {len(gt_pseudo_label)}") + print(f"Length of Y List[Any]: {len(Y)}\n") + + structure_X = describe_structure(X[:3]) + print(f"Structure of X: {structure_X}") + structure_gt_pseudo_label = describe_structure(gt_pseudo_label[:3]) + print(f"Structure of gt_pseudo_label: {structure_gt_pseudo_label}") + structure_Y = describe_structure(Y[:3]) + print(f"Structure of Y: {structure_Y}\n") + + print(f"Shape of X [C, H, W]: {X[0][0].shape}") + +Out: + +.. code-block:: none + + Length of X List[List[Any]]: 30000 + Length of gt_pseudo_label List[List[Any]]: 30000 + Length of gt_pseudo_label List[Any]: 30000 + + Structure of X: [['Tensor', 'Tensor'], ['Tensor', 'Tensor'], ['Tensor', 'Tensor']] + Structure of gt_pseudo_label: [['int', 'int'], ['int', 'int'], ['int', 'int']] + Structure of Y: ['int', 'int', 'int'] + + Shape of X [C, H, W]: torch.Size([1, 28, 28]) + + +ABL-Package offers several `dataset classes <../API/abl.dataset.html>`_ for different usage, such as ``ClassificationDataset``, ``RegressionDataset`` and ``PredictionDataset``, while users are only required to organize the dataset into the aforementioned format. + +Read more about `preparing datasets `_. + +Building the Machine Learning Part +---------------------------------- + +To build the machine learning part, we need to wrap our machine learning model into the ``ABLModel`` class. The machine learning model can either be a scikit-learn model or a PyTorch neural network. We use a simple LeNet5 in the MNIST Add example. + +.. code:: python + + from examples.models.nn import LeNet5 + + # The number of pseudo labels is 10 + cls = LeNet5(num_classes=10) + +Aside from the network, we need to define a criterion, an optimizer, and a device so as to create a ``BasicNN`` object. This class implements ``fit``, ``predict``, ``predict_proba`` and several other methods to enable the PyTorch-based neural network to work as a scikit-learn model. + +.. code:: python + + import torch + from abl.learning import BasicNN + + criterion = torch.nn.CrossEntropyLoss() + optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99)) + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + base_model = BasicNN(cls, criterion, optimizer, device) + +.. code:: python + + pred_idx = base_model.predict(X=[torch.randn(1, 28, 28).to(device) for _ in range(32)]) + print(f"Shape of pred_idx : {pred_idx.shape}") + pred_prob = base_model.predict_proba(X=[torch.randn(1, 28, 28).to(device) for _ in range(32)]) + print(f"Shape of pred_prob : {pred_prob.shape}") + +Out: + +.. code-block:: none + + Shape of pred_idx : (32,) + Shape of pred_prob : (32, 10) + +Afterward, we wrap the ``base_model`` into ``ABLModel``. + +.. code:: python + + from abl.learning import ABLModel + + model = ABLModel(base_model) + +Read more about `building the machine learning part `_. + +Building the Reasoning Part +--------------------------- + +To build the reasoning part, we first create a class that inherits from ``KBBase`` to define the knowledge base. +We pass the list of pseudo labels to the ``__init__`` function and specify how to deduce logical results in the ``logic_forward`` function. + +.. code:: python + + from abl.reasoning import KBBase + + class AddKB(KBBase): + def __init__(self, pseudo_label_list=list(range(10))): + super().__init__(pseudo_label_list) + + def logic_forward(self, nums): + return sum(nums) + + kb = AddKB(pseudo_label_list=list(range(10))) + print(kb) + +Out: + +.. code-block:: none + + AddKB is a KB with pseudo_label_list=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], max_err=1e-10, use_cache=True. + +Then, we create a reasoner. Aside from the knowledge base, the instantiation of ``ReasonerBase`` also needs to set an extra argument called ``dist_func``, which measures the consistency between the knowledge base and machine learning. + +.. code:: python + + from abl.reasoning import ReasonerBase + + reasoner = ReasonerBase(kb, dist_func="confidence") + +Read more about `building the reasoning part `_. + + +Building Evaluation Metrics +--------------------------- + +ABL-Package provides two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. + +In the case of MNIST Add example, the metric definition looks like + +.. code:: python + + from abl.evaluation import SemanticsMetric, SymbolMetric + + metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")] + +Read more about `building evaluation metrics `_ + +Bridging Machine Learning and Reasoning +--------------------------------------- + +Now, we use ``SimpleBridge`` to combine machine learning and reasoning together. + +.. code:: python + + from abl.bridge import SimpleBridge + + bridge = SimpleBridge(model, reasoner, metric_list) + +Finally, we proceed with testing and training. + +.. code:: python + + bridge.train(train_data, loops=5, segment_size=10000) + bridge.test(test_data) + +Training log would be similar to this: + +.. code-block:: none + + 2023/12/02 21:26:57 - abl - INFO - Abductive Learning on the MNIST Add example. + 2023/12/02 21:32:20 - abl - INFO - Abductive Learning on the MNIST Add example. + 2023/12/02 21:32:51 - abl - INFO - loop(train) [1/5] segment(train) [1/3] model loss is 1.85589 + 2023/12/02 21:32:56 - abl - INFO - loop(train) [1/5] segment(train) [2/3] model loss is 1.50332 + 2023/12/02 21:33:02 - abl - INFO - loop(train) [1/5] segment(train) [3/3] model loss is 1.17501 + 2023/12/02 21:33:02 - abl - INFO - Evaluation start: loop(val) [1] + 2023/12/02 21:33:07 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.350 mnist_add/semantics_accuracy: 0.254 + 2023/12/02 21:33:07 - abl - INFO - Saving model: loop(save) [1] + 2023/12/02 21:33:07 - abl - INFO - Checkpoints will be saved to results/20231202_21_26_57/weights/model_checkpoint_loop_1.pth + 2023/12/02 21:33:13 - abl - INFO - loop(train) [2/5] segment(train) [1/3] model loss is 0.97188 + 2023/12/02 21:33:18 - abl - INFO - loop(train) [2/5] segment(train) [2/3] model loss is 0.85622 + 2023/12/02 21:33:24 - abl - INFO - loop(train) [2/5] segment(train) [3/3] model loss is 0.81511 + 2023/12/02 21:33:24 - abl - INFO - Evaluation start: loop(val) [2] + 2023/12/02 21:33:29 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.546 mnist_add/semantics_accuracy: 0.399 + 2023/12/02 21:33:29 - abl - INFO - Saving model: loop(save) [2] + ... + 2023/12/02 21:34:17 - abl - INFO - loop(train) [5/5] segment(train) [1/3] model loss is 0.03935 + 2023/12/02 21:34:23 - abl - INFO - loop(train) [5/5] segment(train) [2/3] model loss is 0.03716 + 2023/12/02 21:34:28 - abl - INFO - loop(train) [5/5] segment(train) [3/3] model loss is 0.03346 + 2023/12/02 21:34:28 - abl - INFO - Evaluation start: loop(val) [5] + 2023/12/02 21:34:33 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.993 mnist_add/semantics_accuracy: 0.986 + 2023/12/02 21:34:33 - abl - INFO - Saving model: loop(save) [5] + 2023/12/02 21:34:33 - abl - INFO - Checkpoints will be saved to results/20231202_21_26_57/weights/model_checkpoint_loop_5.pth + 2023/12/02 21:34:34 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.989 mnist_add/semantics_accuracy: 0.978 + + +Read more about `bridging machine learning and reasoning `_. diff --git a/docs/Intro/Reasoning.rst b/docs/Intro/Reasoning.rst index 79e78bb..a845c4b 100644 --- a/docs/Intro/Reasoning.rst +++ b/docs/Intro/Reasoning.rst @@ -1,3 +1,12 @@ -Build the reasoning part -======================== +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +**Reasoning Part** || +`Evaluation Metrics `_ || +`Bridge `_ + + +Reasoning part +=============== diff --git a/docs/img/Datasets_1.png b/docs/img/Datasets_1.png new file mode 100644 index 0000000..900b2e5 Binary files /dev/null and b/docs/img/Datasets_1.png differ diff --git a/docs/index.rst b/docs/index.rst index f81e833..eb0c15b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,7 +12,7 @@ :caption: Introduction to ABL-Package Intro/Basics - Intro/Quick Start + Intro/QuickStart Intro/Datasets Intro/Learning Intro/Reasoning