From 3b45e4484d2b2c08472ca1ab1c70ee3038324755 Mon Sep 17 00:00:00 2001 From: Gao Enhao Date: Sun, 3 Dec 2023 01:12:12 +0800 Subject: [PATCH] [DOC] modify Quick Start and other module intros --- docs/Intro/Basics.rst | 8 ++ docs/Intro/Bridge.rst | 85 ++++++++------ docs/Intro/Datasets.rst | 62 +++++++++-- docs/Intro/Evaluation.rst | 40 ++++++- docs/Intro/Learning.rst | 82 ++++++++------ docs/Intro/Quick Start.rst | 113 ------------------- docs/Intro/QuickStart.rst | 223 +++++++++++++++++++++++++++++++++++++ docs/Intro/Reasoning.rst | 13 ++- docs/img/Datasets_1.png | Bin 0 -> 33974 bytes docs/index.rst | 2 +- 10 files changed, 434 insertions(+), 194 deletions(-) delete mode 100644 docs/Intro/Quick Start.rst create mode 100644 docs/Intro/QuickStart.rst create mode 100644 docs/img/Datasets_1.png diff --git a/docs/Intro/Basics.rst b/docs/Intro/Basics.rst index a6bae42..46a2c1b 100644 --- a/docs/Intro/Basics.rst +++ b/docs/Intro/Basics.rst @@ -1,3 +1,11 @@ +**Learn the Basics** || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ + Learn the Basics ================ diff --git a/docs/Intro/Bridge.rst b/docs/Intro/Bridge.rst index a94ae1e..2529ff4 100644 --- a/docs/Intro/Bridge.rst +++ b/docs/Intro/Bridge.rst @@ -1,46 +1,67 @@ -.. _ +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +**Bridge** -Bridge the machine learning and reasoning parts -=============================================== -We next need to bridge the machine learning and reasoning parts. In ABL-Package, the ``BaseBridge`` class gives necessary abstract interface definitions to bridge the two parts and ``SimpleBridge`` provides a basic implementation. -We build a bridge with previously defined ``model``, ``reasoner``, and ``metric_list`` as follows: +Bridge +====== -.. code:: python +Bridging machine learning and reasoning to train the model is the fundamental idea of Abductive Learning, ABL-Package implements a set of `bridge class <../API/abl.bridge.html>`_ to achieve this. - bridge = SimpleBridge(model, reasoner, metric_list) +``BaseBridge`` is an abstract class with the following initialization parameters: -``BaseBridge.train`` and ``BaseBridge.test`` trigger the training and testing processes, respectively. +- ``model``: an object of type ``ABLModel``. Machine Learning part are wrapped in this object. +- ``reasoner``: a object of type ``ReasonerBase``. Reasoning part are wrapped in this object. -The two methods take the previous prepared ``train_data`` and ``test_data`` as input. +``BaseBridge`` has the following important methods that need to be overridden in subclasses: -.. code:: python ++-----------------------------------+--------------------------------------------------------------------------------------+ +| Method Signature | Description | ++===================================+======================================================================================+ +| predict(data_samples) | Predicts class probabilities and indices for the given data samples. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| abduce_pseudo_label(data_samples) | Abduces pseudo labels for the given data samples. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| idx_to_pseudo_label(data_samples) | Converts indices to pseudo labels using the provided or default mapping. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| pseudo_label_to_idx(data_samples) | Converts pseudo labels to indices using the provided or default remapping. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| train(train_data) | Train the model. | ++-----------------------------------+--------------------------------------------------------------------------------------+ +| test(test_data) | Test the model. | ++-----------------------------------+--------------------------------------------------------------------------------------+ - bridge.train(train_data) - bridge.test(test_data) -Aside from data, ``BaseBridge.train`` can also take some other training configs shown as follows: +``SimpleBridge`` inherits from ``BaseBridge`` and provides a basic implementation. Besides the ``model`` and ``reasoner``, ``SimpleBridge`` has an extra initialization arguments, ``metric_list``, which will be used to evaluate model performance. Its training process involves several Abductive Learning loops and each loop consists of the following five steps: -.. code:: python + 1. Predict class probabilities and indices for the given data samples. + 2. Transform indices into pseudo labels. + 3. Revise pseudo labels based on abdutive reasoning. + 4. Transform the revised pseudo labels to indices. + 5. Train the model. - bridge.train( - # training data - train_data, - # number of Abductive Learning loops - loops=5, - # data will be divided into segments and each segment will be used to train the model iteratively - segment_size=10000, - # evaluate the model every eval_interval loops - eval_interval=1, - # save the model every save_interval loops - save_interval=1, - # directory to save the model - save_dir='./save_dir', - ) +The fundamental part of the ``train`` method is as follows: -In the MNIST Add example, the code to train and test looks like +.. code-block:: python -.. code:: python + def train(self, train_data, loops=50, segment_size=10000): + if isinstance(train_data, ListData): + data_samples = train_data + else: + data_samples = self.data_preprocess(*train_data) + + for loop in range(loops): + for seg_idx in range((len(data_samples) - 1) // segment_size + 1): + sub_data_samples = data_samples[ + seg_idx * segment_size : (seg_idx + 1) * segment_size + ] + self.predict(sub_data_samples) # 1 + self.idx_to_pseudo_label(sub_data_samples) # 2 + self.abduce_pseudo_label(sub_data_samples) # 3 + self.pseudo_label_to_idx(sub_data_samples) # 4 + loss = self.model.train(sub_data_samples) # 5, self.model is an ABLModel object - bridge.train(train_data, loops=5, segment_size=10000, save_interval=1, save_dir=weights_dir) - bridge.test(test_data) diff --git a/docs/Intro/Datasets.rst b/docs/Intro/Datasets.rst index d5eaa7b..0ea1be6 100644 --- a/docs/Intro/Datasets.rst +++ b/docs/Intro/Datasets.rst @@ -1,13 +1,59 @@ -Prepare datasets -================ +`Learn the Basics `_ || +`Quick Start `_ || +**Dataset & Data Structure** || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ -Next, we need to build datasets. ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` where ``X`` is the input of the machine learning model, ``Y`` is the ground truth of the reasoning result and ``gt_pseudo_label`` is the ground truth label of each element in ``X``. ``X`` should be of type ``List[List[Any]]``, ``Y`` should be of type ``List[Any]`` and ``gt_pseudo_label`` can be ``None`` or of the type ``List[List[Any]]``. -In the MNIST Add example, the data loading looks like +Dataset & Data Structure +======================== -.. code:: python +Dataset +------- - # train_data and test_data are all tuples consist of X, gt_pseudo_label and Y. - train_data = get_mnist_add(train=True, get_pseudo_label=True) - test_data = get_mnist_add(train=False, get_pseudo_label=True) +ABL-Package offers several `dataset classes <../API/abl.dataset.html>`_ for different usage, such as ``ClassificationDataset``, ``RegressionDataset`` and ``PredictionDataset``, while users are only required to organize the dataset into a tuple consists of the following three components +- ``X``: List[List[Any]] + A list of instances representing the input data. We refer to each List in ``X`` as an instance and one instance may contain several elements. +- ``gt_pseudo_label``: List[List[Any]], optional + A list of objects representing the ground truth label of each element in ``X``. It should have the same shape as ``X``. This component is only used to evaluate the performance of the machine learning part but not to train the model. If elements are unlabeled, this component can be ``None``. +- ``Y``: List[Any] + A list of objects representing the ground truth label of the reasoning result of each instance in ``X``. + +In the MNIST Add example, the data used for training looks like: + +.. image:: ../img/Datasets_1.png + :width: 350px + :align: center + +Data Structure +-------------- + +In Abductive Learning, there are various types of data in the training and testing process, such as raw data, pseudo label, index of the pseudo label, abduced pseudo label, etc. To make the interface stable and possessing good versatility, ABL-Package uses `abstract data interfaces <../API/abl.structures.html>`_ to encapsulate various data during the implementation of the model. + +One of the most commonly used abstract data interface is ``ListData``. Besides orginizing data into tuple, we can also prepare data to be in the form of this data interface. + +.. code-block:: python + + import torch + from abl.structures import ListData + + # prepare data + X = [list(torch.randn(3, 28, 28)), list(torch.randn(3, 28, 28))] + gt_pseudo_label = [[1, 2, 3], [4, 5, 6]] + Y = [1, 2] + + # convert data into ListData + data = ListData(X=X, Y=Y, gt_pseudo_label=gt_pseudo_label) + + # get data + X = data.X + Y = data.Y + gt_pseudo_label = data.gt_pseudo_label + + # set data + data.X = X + data.Y = Y + data.gt_pseudo_label = gt_pseudo_label \ No newline at end of file diff --git a/docs/Intro/Evaluation.rst b/docs/Intro/Evaluation.rst index 3c1ad6b..23e500d 100644 --- a/docs/Intro/Evaluation.rst +++ b/docs/Intro/Evaluation.rst @@ -1,12 +1,40 @@ -Define Evaluation Metrics -========================= +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +**Evaluation Metrics** || +`Bridge `_ -To validate and test the model, we need to inherit from ``BaseMetric`` to define metrics and implement the ``process`` and ``compute_metrics`` methods where the process method accepts a batch of outputs. After processing this batch of data, we save the information to ``self.results`` property. The input results of ``compute_metrics`` is all the information saved in ``process``. Use these information to calculate and return a dict that holds the results of the evaluation metrics. -We provide two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. +Evaluation Metrics +================== -In the case of MNIST Add example, the metric definition looks like +ABL-Package seperates the evaluation process as na independent class from the ``BaseBridge`` which accounts for training and testing. To customize our own metrics, we need to inherit from ``BaseMetric`` and implement the ``process`` and ``compute_metrics`` methods. The ``process`` method accepts a batch of model prediction. After processing this batch, we save the information to ``self.results`` property. The input results of ``compute_metrics`` is all the information saved in ``process`` and it uses these information to calculate and return a dict that holds the evaluation results. + +We provide two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. Using ``SymbolMetric`` as an example, the following code shows how to implement a custom metrics. .. code:: python - metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")] \ No newline at end of file + class SymbolMetric(BaseMetric): + def __init__(self, prefix: Optional[str] = None) -> None: + # prefix is used to distinguish different metrics + super().__init__(prefix) + + def process(self, data_samples: Sequence[dict]) -> None: + # pred_pseudo_label and gt_pseudo_label are both of type List[List[Any]] + # and have the same length + pred_pseudo_label = data_samples.pred_pseudo_label + gt_pseudo_label = data_samples.gt_pseudo_label + + for pred_z, z in zip(pred_pseudo_label, gt_pseudo_label): + correct_num = 0 + for pred_symbol, symbol in zip(pred_z, z): + if pred_symbol == symbol: + correct_num += 1 + self.results.append(correct_num / len(z)) + + def compute_metrics(self, results: list) -> dict: + metrics = dict() + metrics["character_accuracy"] = sum(results) / len(results) + return metrics \ No newline at end of file diff --git a/docs/Intro/Learning.rst b/docs/Intro/Learning.rst index 21b82ae..ed6d1d6 100644 --- a/docs/Intro/Learning.rst +++ b/docs/Intro/Learning.rst @@ -1,46 +1,64 @@ -Build the machine learning part -=============================== +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +**Machine Learning Part** || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ -First, we build the machine learning part, which needs to be wrapped in the ``ABLModel`` class. We can use machine learning models from scikit-learn or based on PyTorch to create an instance of ``ABLModel``. -- for a scikit-learn model, we can directly use the model to create an instance of ``ABLModel``. For example, we can customize our machine learning model by +Machine Learning Part +===================== - .. code:: python +``ABLModel`` class serves as a unified interface to all machine learning models. Its constructor, the ``__init__`` method, takes a singular argument, ``base_model``. This argument denotes the fundamental machine learning model, which must implement the ``fit`` and ``predict`` methods. - # Load a scikit-learn model - base_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=3) +.. code:: python - model = ABLModel(base_model) + class ABLModel: + def __init__(self, base_model: Any) -> None: + if not (hasattr(base_model, "fit") and hasattr(base_model, "predict")): + raise NotImplementedError("The base_model should implement fit and predict methods.") -- for a PyTorch-based neural network, we first need to encapsulate it within a ``BasicNN`` object and then use this object to instantiate an instance of ``ABLModel``. For example, we can customize our machine learning model by + self.base_model = base_model - .. code:: python +All scikit-learn models satisify this requiremnts, so we can directly use the model to create an instance of ``ABLModel``. For example, we can customize our machine learning model by - # Load a PyTorch-based neural network - cls = torchvision.models.resnet18(pretrained=True) +.. code:: python - # criterion and optimizer are used for training - criterion = torch.nn.CrossEntropyLoss() - optimizer = torch.optim.Adam(cls.parameters()) + import sklearn + from abl.learning import ABLModel - base_model = BasicNN(cls, criterion, optimizer) - model = ABLModel(base_model) + base_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=3) + model = ABLModel(base_model) -In the MNIST Add example, the machine learning model looks like +For a PyTorch-based neural network, we first need to encapsulate it within a ``BasicNN`` object and then use this object to instantiate an instance of ``ABLModel``. For example, we can customize our machine learning model by .. code:: python - cls = LeNet5(num_classes=10) - device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") - criterion = torch.nn.CrossEntropyLoss() - optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99)) - - base_model = BasicNN( - cls, - criterion, - optimizer, - device=device, - batch_size=32, - num_epochs=1, - ) - model = ABLModel(base_model) \ No newline at end of file + # Load a PyTorch-based neural network + cls = torchvision.models.resnet18(pretrained=True) + + # criterion and optimizer are used for training + criterion = torch.nn.CrossEntropyLoss() + optimizer = torch.optim.Adam(cls.parameters()) + + base_model = BasicNN(cls, criterion, optimizer) + model = ABLModel(base_model) + +Besides ``fit`` and ``predict``, ``BasicNN`` also implements the following methods: + ++---------------------------+----------------------------------------+ +| Method | Function | ++===========================+========================================+ +| train_epoch(data_loader) | Train the neural network for one epoch.| ++---------------------------+----------------------------------------+ +| predict_proba(X) | Predict the class probabilities of X. | ++---------------------------+----------------------------------------+ +| score(X, y) | Calculate the accuracy of the model on | +| | test data. | ++---------------------------+----------------------------------------+ +| save(epoch_id, save_path) | Save the model. | ++---------------------------+----------------------------------------+ +| load(load_path) | Load the model. | ++---------------------------+----------------------------------------+ + diff --git a/docs/Intro/Quick Start.rst b/docs/Intro/Quick Start.rst deleted file mode 100644 index 122ecbb..0000000 --- a/docs/Intro/Quick Start.rst +++ /dev/null @@ -1,113 +0,0 @@ -Quick Start -=========== - -We use the MNIST Add benchmark as a quick start example. In this task, the inputs are -pairs of MNIST handwritten images, and the outputs are their sums. -To complete this task, we first process the images through a machine learning model -to get their corresponding pseudo labels (the number each image represents). -Then, the recognized labels undergo logical reasoning which calculates their sum. - -Load Data ---------- - -ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` -where ``X`` is the input of the machine learning model, -``Y`` is the ground truth of the reasoning result and -``gt_pseudo_label`` is the ground truth label of each element in ``X``. - -.. code:: python - - from examples.mnist_add.datasets.get_mnist_add import get_mnist_add - - train_data = get_mnist_add(train=True, get_pseudo_label=True) - test_data = get_mnist_add(train=False, get_pseudo_label=True) - -In the ``get_mnist_add`` above, the return values are tuples of ``(X, gt_pseudo_label, Y)``. - -Read more about `prepare datasets `_. - -Build Machine Learning Models ------------------------------ - -We use a simple LeNet5 model to recognize the pseudo labels (numbers) in the images. -We first build the model and define its corresponding criterion and optimizer for training. - -.. code:: python - - import torch - import torch.nn as nn - from examples.models.nn import LeNet5 - - cls = LeNet5(num_classes=10) - criterion = nn.CrossEntropyLoss() - optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99)) - device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") - -Afterward, we wrap it in ``ABLModel``. - -.. code:: python - - from abl.learning import ABLModel, BasicNN - - base_model = BasicNN(cls, criterion, optimizer, device) - model = ABLModel(base_model) - -Read more about `build machine learning models `_. - -Build the Reasoning Part ------------------------- - -First, we build a knowledge base that defines how to deduce -logical results (i.e., calculate summation) from the pseudo labels -obtained by machine learning. - -.. code:: python - - from abl.reasoning import KBBase, ReasonerBase - - class AddKB(KBBase): - def __init__(self, pseudo_label_list=list(range(10))): - super().__init__(pseudo_label_list) - - def logic_forward(self, nums): - return sum(nums) - - - kb = AddKB(pseudo_label_list=list(range(10))) - -Then, we define a reasoner, which defines -how to minimize the inconsistency between the knowledge base and machine learning. - -.. code:: python - - reasoner = ReasonerBase(kb, dist_func="confidence") - -Read more about `build the reasoning part `_. - -Bridge Machine Learning and Reasoning -------------------------------------- - -Before bridging, we first define the metrics to measure accuracy during validation and testing. - -.. code:: python - - from abl.evaluation import SemanticsMetric, SymbolMetric - - metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")] - - -Now, we may use ``SimpleBridge`` to combine machine learning and reasoning together, -setting the stage for subsequent integrated training, validation, and testing. - -.. code:: python - - from abl.bridge import SimpleBridge - -Finally, we proceed with testing and training. - -.. code:: python - - bridge.train(train_data, loops=5, segment_size=10000) - bridge.test(test_data) - -Read more about `defining evaluation metrics `_ and `bridge machine learning and reasoning `_. diff --git a/docs/Intro/QuickStart.rst b/docs/Intro/QuickStart.rst new file mode 100644 index 0000000..5b1c68b --- /dev/null +++ b/docs/Intro/QuickStart.rst @@ -0,0 +1,223 @@ +`Learn the Basics `_ || +**Quick Start** || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +`Reasoning Part `_ || +`Evaluation Metrics `_ || +`Bridge `_ + +Quick Start +=========== + + +This section runs through the API for the neural-symbolic task, MNITST Add. Refer to the links in each section to dive deeper. + +Working with Data +----------------- + +ABL-Package assumes data to be in the form of ``(X, gt_pseudo_label, Y)`` where ``X`` is the input of the machine learning model, +``gt_pseudo_label`` is the ground truth label of each element in ``X`` and ``Y`` is the ground truth reasoning result of each instance in ``X``. + +In the MNIST Add task, the data loading looks like + +.. code:: python + + from examples.mnist_add.datasets.get_mnist_add import get_mnist_add + + # train_data and test_data are all tuples consist of X, gt_pseudo_label and Y. + # If get_pseudo_label is False, gt_pseudo_label will be None + train_data = get_mnist_add(train=True, get_pseudo_label=True) + test_data = get_mnist_add(train=False, get_pseudo_label=True) + +ABL-Package assumes ``X`` to be of type ``List[List[Any]]``, ``gt_pseudo_label`` can be ``None`` or of the type ``List[List[Any]]`` and ``Y`` should be of type ``List[Any]``. + +.. code:: python + + def describe_structure(lst): + if not isinstance(lst, list): + return type(lst).__name__ + return [describe_structure(item) for item in lst] + + X, gt_pseudo_label, Y = train_data + + print(f"Length of X List[List[Any]]: {len(X)}") + print(f"Length of gt_pseudo_label List[List[Any]]: {len(gt_pseudo_label)}") + print(f"Length of Y List[Any]: {len(Y)}\n") + + structure_X = describe_structure(X[:3]) + print(f"Structure of X: {structure_X}") + structure_gt_pseudo_label = describe_structure(gt_pseudo_label[:3]) + print(f"Structure of gt_pseudo_label: {structure_gt_pseudo_label}") + structure_Y = describe_structure(Y[:3]) + print(f"Structure of Y: {structure_Y}\n") + + print(f"Shape of X [C, H, W]: {X[0][0].shape}") + +Out: + +.. code-block:: none + + Length of X List[List[Any]]: 30000 + Length of gt_pseudo_label List[List[Any]]: 30000 + Length of gt_pseudo_label List[Any]: 30000 + + Structure of X: [['Tensor', 'Tensor'], ['Tensor', 'Tensor'], ['Tensor', 'Tensor']] + Structure of gt_pseudo_label: [['int', 'int'], ['int', 'int'], ['int', 'int']] + Structure of Y: ['int', 'int', 'int'] + + Shape of X [C, H, W]: torch.Size([1, 28, 28]) + + +ABL-Package offers several `dataset classes <../API/abl.dataset.html>`_ for different usage, such as ``ClassificationDataset``, ``RegressionDataset`` and ``PredictionDataset``, while users are only required to organize the dataset into the aforementioned format. + +Read more about `preparing datasets `_. + +Building the Machine Learning Part +---------------------------------- + +To build the machine learning part, we need to wrap our machine learning model into the ``ABLModel`` class. The machine learning model can either be a scikit-learn model or a PyTorch neural network. We use a simple LeNet5 in the MNIST Add example. + +.. code:: python + + from examples.models.nn import LeNet5 + + # The number of pseudo labels is 10 + cls = LeNet5(num_classes=10) + +Aside from the network, we need to define a criterion, an optimizer, and a device so as to create a ``BasicNN`` object. This class implements ``fit``, ``predict``, ``predict_proba`` and several other methods to enable the PyTorch-based neural network to work as a scikit-learn model. + +.. code:: python + + import torch + from abl.learning import BasicNN + + criterion = torch.nn.CrossEntropyLoss() + optimizer = torch.optim.Adam(cls.parameters(), lr=0.001, betas=(0.9, 0.99)) + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + base_model = BasicNN(cls, criterion, optimizer, device) + +.. code:: python + + pred_idx = base_model.predict(X=[torch.randn(1, 28, 28).to(device) for _ in range(32)]) + print(f"Shape of pred_idx : {pred_idx.shape}") + pred_prob = base_model.predict_proba(X=[torch.randn(1, 28, 28).to(device) for _ in range(32)]) + print(f"Shape of pred_prob : {pred_prob.shape}") + +Out: + +.. code-block:: none + + Shape of pred_idx : (32,) + Shape of pred_prob : (32, 10) + +Afterward, we wrap the ``base_model`` into ``ABLModel``. + +.. code:: python + + from abl.learning import ABLModel + + model = ABLModel(base_model) + +Read more about `building the machine learning part `_. + +Building the Reasoning Part +--------------------------- + +To build the reasoning part, we first create a class that inherits from ``KBBase`` to define the knowledge base. +We pass the list of pseudo labels to the ``__init__`` function and specify how to deduce logical results in the ``logic_forward`` function. + +.. code:: python + + from abl.reasoning import KBBase + + class AddKB(KBBase): + def __init__(self, pseudo_label_list=list(range(10))): + super().__init__(pseudo_label_list) + + def logic_forward(self, nums): + return sum(nums) + + kb = AddKB(pseudo_label_list=list(range(10))) + print(kb) + +Out: + +.. code-block:: none + + AddKB is a KB with pseudo_label_list=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], max_err=1e-10, use_cache=True. + +Then, we create a reasoner. Aside from the knowledge base, the instantiation of ``ReasonerBase`` also needs to set an extra argument called ``dist_func``, which measures the consistency between the knowledge base and machine learning. + +.. code:: python + + from abl.reasoning import ReasonerBase + + reasoner = ReasonerBase(kb, dist_func="confidence") + +Read more about `building the reasoning part `_. + + +Building Evaluation Metrics +--------------------------- + +ABL-Package provides two basic metrics, namely ``SymbolMetric`` and ``SemanticsMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. + +In the case of MNIST Add example, the metric definition looks like + +.. code:: python + + from abl.evaluation import SemanticsMetric, SymbolMetric + + metric_list = [SymbolMetric(prefix="mnist_add"), SemanticsMetric(kb=kb, prefix="mnist_add")] + +Read more about `building evaluation metrics `_ + +Bridging Machine Learning and Reasoning +--------------------------------------- + +Now, we use ``SimpleBridge`` to combine machine learning and reasoning together. + +.. code:: python + + from abl.bridge import SimpleBridge + + bridge = SimpleBridge(model, reasoner, metric_list) + +Finally, we proceed with testing and training. + +.. code:: python + + bridge.train(train_data, loops=5, segment_size=10000) + bridge.test(test_data) + +Training log would be similar to this: + +.. code-block:: none + + 2023/12/02 21:26:57 - abl - INFO - Abductive Learning on the MNIST Add example. + 2023/12/02 21:32:20 - abl - INFO - Abductive Learning on the MNIST Add example. + 2023/12/02 21:32:51 - abl - INFO - loop(train) [1/5] segment(train) [1/3] model loss is 1.85589 + 2023/12/02 21:32:56 - abl - INFO - loop(train) [1/5] segment(train) [2/3] model loss is 1.50332 + 2023/12/02 21:33:02 - abl - INFO - loop(train) [1/5] segment(train) [3/3] model loss is 1.17501 + 2023/12/02 21:33:02 - abl - INFO - Evaluation start: loop(val) [1] + 2023/12/02 21:33:07 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.350 mnist_add/semantics_accuracy: 0.254 + 2023/12/02 21:33:07 - abl - INFO - Saving model: loop(save) [1] + 2023/12/02 21:33:07 - abl - INFO - Checkpoints will be saved to results/20231202_21_26_57/weights/model_checkpoint_loop_1.pth + 2023/12/02 21:33:13 - abl - INFO - loop(train) [2/5] segment(train) [1/3] model loss is 0.97188 + 2023/12/02 21:33:18 - abl - INFO - loop(train) [2/5] segment(train) [2/3] model loss is 0.85622 + 2023/12/02 21:33:24 - abl - INFO - loop(train) [2/5] segment(train) [3/3] model loss is 0.81511 + 2023/12/02 21:33:24 - abl - INFO - Evaluation start: loop(val) [2] + 2023/12/02 21:33:29 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.546 mnist_add/semantics_accuracy: 0.399 + 2023/12/02 21:33:29 - abl - INFO - Saving model: loop(save) [2] + ... + 2023/12/02 21:34:17 - abl - INFO - loop(train) [5/5] segment(train) [1/3] model loss is 0.03935 + 2023/12/02 21:34:23 - abl - INFO - loop(train) [5/5] segment(train) [2/3] model loss is 0.03716 + 2023/12/02 21:34:28 - abl - INFO - loop(train) [5/5] segment(train) [3/3] model loss is 0.03346 + 2023/12/02 21:34:28 - abl - INFO - Evaluation start: loop(val) [5] + 2023/12/02 21:34:33 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.993 mnist_add/semantics_accuracy: 0.986 + 2023/12/02 21:34:33 - abl - INFO - Saving model: loop(save) [5] + 2023/12/02 21:34:33 - abl - INFO - Checkpoints will be saved to results/20231202_21_26_57/weights/model_checkpoint_loop_5.pth + 2023/12/02 21:34:34 - abl - INFO - Evaluation ended, mnist_add/character_accuracy: 0.989 mnist_add/semantics_accuracy: 0.978 + + +Read more about `bridging machine learning and reasoning `_. diff --git a/docs/Intro/Reasoning.rst b/docs/Intro/Reasoning.rst index 79e78bb..a845c4b 100644 --- a/docs/Intro/Reasoning.rst +++ b/docs/Intro/Reasoning.rst @@ -1,3 +1,12 @@ -Build the reasoning part -======================== +`Learn the Basics `_ || +`Quick Start `_ || +`Dataset & Data Structure `_ || +`Machine Learning Part `_ || +**Reasoning Part** || +`Evaluation Metrics `_ || +`Bridge `_ + + +Reasoning part +=============== diff --git a/docs/img/Datasets_1.png b/docs/img/Datasets_1.png new file mode 100644 index 0000000000000000000000000000000000000000..900b2e5bfb50cf63ea7064945375345654feedfc GIT binary patch literal 33974 zcmce-cTiLB*9VBApj4IK?TAQ6Y5)-d=~4s)0@9=gqy&=C^oxkpNbgMuy_bNY3PKc= z5}Fcv5ePjHdf6L)W#`>LcHi0A*~}nw@!mY=InSw|^SNQqbk*r;IB6&-DCjkwJ~5!6 zxB#c1IHz=p3iuagAd(FH$2l(p^~V(Dz1+*d#RUgt9c2oNDkSZ(^+n+N^6RHByeKHx zu9APwb>H1qqM&%ErSU}B2xhsKLA`%L9iF8k(H#CtwE0pRuNmDL?x5eKcQ4wRmb}qk@=KRC!`M<7aJdw1$@Qd}%CDG?n;OpAzl?Q>j2f|)nH31u^ ze!d$=pw>mVg0)rPQ$N+aekuq^K|%56Zt*_tgnrtZf8zm_&qROyS1b2)BUvymekry> z_~*cu?>$RhOSB+L$TGg&C%j|U-Gr5ADN>@hF!N;9!DPdKQ9g~c0z)cu2ZuF`U(_?7 zSP=Ft>S1edJzfwI!QJmquB8m~o7~PkkV{u>*iW!JNC1()DO!>1v*h{9XBBOcR|5BE zLZwjx`rzsFY=!GEti03$?aFBzwc^Ku6#olyN2@h#XF3AIbG&;Ego%wnUlPHj6?1sJ zsFa0`F!-hG(YlX^s+j0n-#qQRJmVg)V$SBo^6Tx#-JL-__ihsS>OM&alDt&f}b zs<(}lPL}o9JT`!3h%kK`JO?gn@Qf{MuRcbV8nX_qXdHTT!@KTBwq4%S?@1Yi#QLU% zo9^*)mc4VQv|5B|XdOG;)%xl(N;OxbrP-gQ)wR}m3v}{(%OuZ!|Ey&}PCBX;LMOy`{SkF|WPrshKK-1yXeS*HGG*<5mDI^J&hwag-g0)_ceV;A?%Y_tEB#r6giC6X z)RS;p^qBS3z0|q}aZf7{Kuu`W=A>R5dnet6`?fvLuk;x)P07c8s*ZYBi40h^b!sZu z0t69WjZ`|!RU(4mhgW=ZXu*Yu1wpG6z@z5F7|-Ab0y&wZt*0b5ClyE8wO$;$iq{GR z0TT?8h-U&fR_}mw9(0?uT$2kuRTqlk zl|jV7jzx@d>zt`>%Une-ymYwMts{E9e)m|S5hFF3&rc?%PBLZ})eOF!Z0hGq2jBHDyxe{f(rRg6R^6-F zP%w>s_sq#4t?q)f#M_6_ASWIJLbqO#)?Dk$JbKEB{7v`zE<1^G(`c^V+w+cF2X)!~MANV?rAFb&VLJ9;0PDB!77In54 zVCmA8iE;i~8-6~ila|h{f+Q#Wi|)9JTUM+nNt8E4`!^gj4b@(e%bPsc^GmLFQSe7l zMs4YB9hqxQF+`B;{abpuO^^cvP+2$4xmyBdjV`9)kkHA{=AFdBB6(y&t9W%DjAsyo z9`<8~3J&RK@gA_E3t_SR-kElv(VujjF!R2#wNV)b<)!|ThM2-mft+}0dcn@5Z;X++ zpd+x;=r!Y!n63KVM?%n%d~tc}i$k1+xwKQFmR3g@-t$#cojvXNu{>`%p#k4{jTKQA z2;Lw<9Bt2Qv43vBKJ6ukOT_`Wn396x5tp2=*XcU7q4-V7^QHZDhC6E@Ltk_)NQYA4 zTW`9a*MzyIo=!YxzMd_Q4Pm=$=*Ex^&jhn~=F_D2OL)^2Hw&Lh?d0lh$Dyl;?(8VB zTto8!-2P6hU!0a6q%GCdgwl7}PUW-DaR{Ya29~e!t(&8F?$7@?-hY0-)EgKV(UJE= zEeSQzXz^e1xg(-w=&dVVRC;4UC!|8D!iuNmg!+X;^Sb%U=xtolWS!zW+1ikb*4i`7 zx?%E-F*pL-%#Qy;jL!bTukf0A>bXulZdBloRe5fFMM>&CR7G4|{Y<{b%1t7|m@!~U zrDU)5>}0rkBVYqm9tMJB7!d9CX@Z45)#e&v)~7*Wk4@S6k1EB27Be(6X9s;jr-X5@ zzB0GK?X=K^Cpr&MCxDl7v~WRxS%*RoVzt;b4_M{qA|=}0dqR~oAK*@s&WKjUHNA2r zh|OR=ZnRQVgy$=TeaAC+8rxWIm@1}md_i~^Ke>zUXP8BdHd^jSZ3&>nwX7@eGz`ZD z&N$qIpQ`mNeqw8OL{-fCub5922ObV){h?H6a>5XRG-zek8y_n&s^87#O(-fbo;o|- z>-chGN32P5a_H;Lj7`R0CtdExYN=BTKM) zKL(~PIMWFXhdVWapWNQ~;xHeYGIgA9=EZb+&2df1^d;TD130nhFBQO8g1^cSDEq`3 z@T8W8864{59>RH=$}w89BrrdHEF`#3xn9+fhoRc?KXyt{DNJOnK2RUC5QEL_l&MVZ zth6a@36M@~=Hh?k2&EEUe(vb(w=mk2XgG`42tQv^*UU_7<8 zrYv^7iyN1JTuFY7)8L;_{9}H2vvJYO_4Dr0kZed(24gzNoiBbgzBfPs2jv0doEp1E zWKZdb66|Mcc>TFaRXL>90an2o@QU;YbU5h+T%bb0drY#IYalQp7)%WEXe&4vJZn!x z?urU-$3H&Cc``Yft!?g`%xH}~cjC!UfUxKs96q~OAGg3X4F6ct(d}4!J9XB;tXt;( z>6(S*<%QwP*M&zC?Ptuz_9Gz?`_ldWz;d=D_{&yrPk7Baugp#v6Kl<+4x|gui~Sfi zsYd^Ux-tmm@n=m$2GVW~Z(;d@${Kz>O_e^HvfsQ+t;?H)V;)tr&fu!@H^~2mP81*N zEk`MeGAA&dm1h$dLHni?h!^X3h|r<38swPk_@?0oC8>(GqO6%V}!D%eBq?<2&qV@v6UGe ztg5xVw5L71?$liNGCZ2z4(kK|H3njB~YH>duhmu%U?!e zB8$jX*1tYKxYC0votC`J-%rk))DL#8YRjE>kv$EEEIH~x z$KM}TFhPO_&6z?gHz1f{$3+c9(uV2C@YVd{sk`^;-daA#=-?}bM+THeh8JI|XxC8| ztY5-_w5`6j(DRfPMI+9Ev}uF;5(jH3Sw^AM6}h*TsJeJVvgho<5E#t6q@eF0Yqcwdds0>m;`{&E z&i~zS$?T+usPm?5PD+DZoYq?XM{fhLsEWR!5_N?pk#4VF@E|srDQ}C0{|SQ9>QHk) zF5X9%#9J+r|CqY&Fr1L%Ysbe7h2+>%WeRmrr=;gpeA@|rh#Jzz6a(=>PL_}zK*h^1 zQlathbTfGyp0+@8HDE&RKs5z~BsN`dIHch+QZ{}m-V>qJQy~BOdy=%qT$@+Rc818Y z;v7d`q2f*sqflo`>bF7R?)&#pwLjWwP>zcWl+VJk^*P>Ulzkz>-Z_LAq`w^r$V4qb zmk?Jh(JPaCExc#wtHD^0)hTiK7Vp*v!cX@=uc8MI#J7w1sZ$@`k*@}3bsgk3h9KJq z*M$~oft>}&X~R9X>+NF8UbDO7hF$={aCpV)?2w>ziaks2l8(uz#q4*ASOx&;raW-B zChNI7VlWNY5RVh5>|>4F=w}Ac`5xGXbvyFFyk9kr4VUL{X7ejZ$KrT8r!4mA;t4 zjAE}og?bEDVLc}C25P=1Eze^*Mi};m_WhJeGvWELpygJ$WO%3 z#F_Y=Lt*pVCr>++ebNO3rU-9Nm?AL-X(WOeh_I0nj20Cv>Gr?r0EXRJw9l|_M2o@seiacT1&pL)-mYx)6wy$KRAl>0dJ=bU!$ZOT4q zxmO0E#Oj1#V0tr0Pdw>QQ)ylhslW z@)y2}`!ID_HbrwL;!_pzywCrSgi5k-heDLVUk6a#*Zs+Og)xcyBx|)iaR)dYG-d_7 zVHR-EW5vKD1eM|^YDA;H+|=XR*+K?2F|X%Q>jSZ0X4A2M;AKjvde}~oW~%TKP~vDp zH~gmj?u^IB3XgWWRB8#JnBWb%_sq)it(|YN2c1^Osv;G)V@~xh&pb9BL9+ybglqNCf&8N7lndUfJA77GIvg18;5*%{w79i6wk3^-JDv(bZd3x42o@D4nRo zl)3oJf|ncEEh12|_kS`4OZRdUvV+G~UVo&9fz7$Sn_ope{80oCrk}}+)5QcNy+`lg z3SGalz-2a?>}x+A7R1P17dl+9-NwYmm>yiJH7bQ?hMPOWpDImp5y4@Q z0ZjS6^p(-(gSS{?i&ZvUB^XA2ib7dtA{nawi9%c}IdEmsnZ z8@7haL(Y<+k*KntAYCZ8VV|VCp)5Q-aa3;e7maR@iL>C)B@QVWWJ!l0$Myq-YIl!< zp@iJm5lRn*5vQtS{x9}XF-5;(gjUmAqVhe7AatpZDd(F(@52hryAKyS*@$IB!--sd z-=c{EK^AoL_QL}Y!IvTCNj{Ny`R{FW3v^}m3+BJAwl-)~M#V1bKwfz`bhf^|3h_uy zsH;pqtPM$QTs>0G6O0hVQFxS!gnnRv-C5g{85}T^3JJasc|vR6biKQM@XM>^>gy*W z)eU3O_GfkL&w^sQ>T+tGR2&wBPmp+o_K=OastSh|bp~1q?Dlv$UP+h=F<<5e|lNvZdPi8)wpz$kjGV^*CJWUPyb&0`!s1=MxRD7#~X_OSdpuR;f}yBwauA= z`?xWa>6qxk*XG#5x7c=htDAK*!N}5re9x{@+XrC$4;;OZ@FKpze(;Mq$apg)_B6?u z`wR_yedd&WFp73W&+q`AE`s0aqRPa0ZhN=Sk3+Lq_Fhr=kW?Kbtn_I=hoPTTRsmcmH^p@iv~C9UA3@nSKvR;il$j z{c8IA`kBrgjv+Qg?Tr+t0xiqeY9vhA5OBOxvF-mWR4XpwmohY7nNge@$CPs zluQ5DEeg$%l?|Oz<`c#Yppb_Pz7Kx3(>Q5X>~2EugTZFB8=(C}+`|NLJ8t9<^~B7$ zXq6r9O_?39J3q6l)WD_PDVGN^zB%Q7a9JpEVvN-(HGc;%_vhFK8KL_ilSs3#3-r%! z<||;i3ZCh%U7rtEehUJf`WDp7dne_|TunmOAY7`gY_VEJ0_rr>5!lZQuM8BTdwrt| z5^(66vjYmwbhJlRjH%_~TkNR4Gq*Q>wmBVr&HyrPFVHN+Cc^l@ZK&;*zXJ50B`GW- zDsQF#%s&HdA?^5J0(#BNVN$m{D``0ol9&tnY%_EXsF@RWNsdk5KPI6T%GZ8z3Hr}? zU3G4_A!zP$=SFRX(eSj6YZr8eSM{ly>;%3)X`m)fljJf{*p zc$bKA{_JGMpeC;$QqWy-t8w;V?xwaj2I=Tv%I~CP-qQvEDF5SAIpMgjd;bkIX1AvP z9>p#j^q-F2BihQV?65a)sihbw`x0|_?k32{pYDbL*YrR|^(n*W8&HG}(W`+?v2Afa z#j_>K4qqL#QM-dupl->0NIeH+Z+UAkkS6&`!uPfs`*~_j#Y-erdmV+%JQ5S3&FgN( zM%`RR|2c8eTWQ}RtO0SCBcj`O`uF47xno>$xuN&k15IgCTO30)c*WzRhJ^L-%0>Ve3Cn!3r!bLO?Zdu;PAyD4z~w-fcu3(hW%FHCPCdd7Vbs zw;7aj?ZB8~?W@P)-a2Rv^rp?edB(A_v-OSop{l~DhWW`K%QL80h${H;8XzxF$?AAK zDvX56ClwSHBs-^hjdgP$cMg5%PFjfnRM|99muUcIPGmT*nB4Qpf(?bx4(Wb1{o9~+ zJ5^47G&S+Z`a9Hizw-J@>TI-e(WF`L3BQ?1rtZB53eQfi zq2qmMnpTYrKM~pgrbH+mXk~VOzY(Gxuo(`G|9qJp_HuFN(^4bd+P%QtRH- zY2-{DvcXl`c~&sQV8E35iVsiwGB=MJHS{iP(~>(;DtNRcV!%-PC5*$++kQ->qn z2jh#1Ce23^8|9WjQ5@1>(DFn@AmMq*cGlVP@I~R*ZwZgtFwZ`oCgD;ve@re71YU6u zI>MKbYu8T6{G9l1vzReck_-U#umShj-G4hE(Y_>08ZuN-dQFD1-cRiT9mue~h~IA; z-a%joEYKl+_2)`Qzxy78TTV$8CS)+XL<<10QMJ(8PFlU-0m@2iMOz}}=4zoL%+iUb zsZ=8}lZ?;}lS?wuHz(87N`-E~v`A$3Hiz8Q$+~aH^*vx4z5?ahWPz5{q%ry=VATrp z1WHArQ@Zuwr|ACVvaT<|?|gE`z&3!7=lliiPD`3QoeDD0vBG50E`sN8vK0YU^x|_X zpm`|fJ6Td64&%>cC)`{Q+-qdpzD?d<|I&cR!BBx>9Rh%vOciFYA`8pCQGy|x@q%Gc3W*4#yFMn6;be(sD-MN*O8bMmaN80b-VJ3(Q8 zHxh2rWJCJG7$`b~VxIbRtwszH4I+Lc@GBi?Y`%0lh&R-|jEC2xO}O5HCWPlj>8b1S8yhFl0inBzp~#IMcDa+wp04(jviAHPd0+iY%qm64-&kv@{?jaS{&kl z{VoPC>etIFoVK2ho|!4EIq8`{=(%nvH)Eau-yK{DP4@0zmhNO!(_qS!j;io0%gi(y z1-icCKAkgr;`1*CG@gKf9+2ewA}f`_MGqFoBC{~ z4AS5b(0%;Y$57Di(iJlNJV{Kfe@{v{^ydX7x_z_5R_{j_)N4Li-mC7TyJ#n9vA>V{{+TOm^5DJ|WBB zUWxzb-I2A@)cI!IU@Yk|VcM3|Rn`Y>T z$#^=u!9_}vPkrNr0f#3>={K=r>GWHfkV`!L=GgiZ;>Lt)K0Dc{2s~X3 zQq(h7H*eaMWLKbYdGbD*q4p>Q^~(Q|?*i`)ejSO6mLq0SnGxVsn_<4w89rkmGS#_p`I6JIh8@zjM^1YKOm(!uK0vR>#LT0+%@8rL)?<-D3>;>F+k00CNn_%zDBsGKoBZ`lR)nR~Vh8Q?0`l6?(&(vL4)Zh!1(bq}0GOS|_&6FjDDe;+b< zx_ih7XEbrhRqxV@=DpvaUc8fNpdMym$`!HWw)1T;*W5T26oB*_&&4b%Y`kbb9Ihw_ zFo%tIFdd5#;`4IP1aEK=!7+j@lPZ_`W#879m*TmLIgA00>|Z^1jJn9k~qYJZ6gTR zs(!yTc)2`O%=l9oA5Ew4Rw=I}6mP2Tkn3QBDira3$ob#2S|mg8*8*xw=Oz(VvSYj8o2P)&S(All6vU+Eqot&ud> zNzQ$%vWo299xqg+?Lcj>p0!R;sW_}tEtCV9N-<2lkVUB%|%>8 zLO$)v3*Y5Xec)TAXV7`uFSu?kW_fo7Ry)-U@^DIlR{;u#{K7EI+bymP9oqKOuu2o` z;!15b+$wo5pie_)GP~)I%C!T;t3B_oQ5!44weAX5h?rFj&kQvb9$s*dT25vk;^+w; zkjTy|Dzlwr;TaNh#4~~NJUj+OW^F|L&1Y?{>eWaGV7Y3`th2dBqH7vCNr&3Qf3;*j z+RLH9`An!U{0KbLGRa=6@7lbm{qR?b&Mf*z?^2bx-7g-aRw9YDl}0g7H<35Dz(J|T zzhH!eSOr@C#a?^WzuCj%ZOCVj5f`qNZmlgz^WVO;OmkEZsGN?hFwXM)YSbW*aI_lF zcYs_4IC@3JHhZ;Jud7)DO9vp#(lJZWb98v zljFum(T;YN#pfrr+}jWTa6}}n9oBpeikej5*d-P6pI~n1y5~-Co$Z{CwFSx zpb{EjmZlOhDV9wUnzenF&HN|6o#hqYG`;qZeM;DyzvCxH4n1hhB;Gs-JHcuO^9QjYZu&=I8=q`CDG%E>cLYHQkB@gT-_;q z?LVD5KGjv?{Vc+Ue}-pMm>epwl1vl;{Tw0jRKTJ<^!?5=>3G;~dZuv>4kqhH zDASnJq4n$d&*|ktYXx3izHRR-g}^a+(V2aj4X3OLT_8Qx*d_$g@kS;6u64uWJ|`87 z9L>)N;jiJ6aVMB>qxs2qUc%L^W*EPW7Ll;oEDzjpaZw3#?~#(UbLx23mBE!faY-v4 zM3)rW%Gl5;M%r^0dmf0iL%VLQ96$fW*l+po9eU4A7UsDm1y0T6L zo?5mdZ#*;hS(eu=^#^3+qDXy_SQ_rhEn5#_)P02Oq2&Vq0#phSk7mxa@rY7G7Qvv4 zx0mmFl^{}3Qo8Ovo!M`MM%_w6ea9TU=g^`dDJ)51c zoo_-HW@L@m4e(}%&5{UA`hM-kgEhN2q?Y4RU39}Qj=C^(N3C~cr(DTT1Xu@)&>mwv zjCWXWZy2ke1TW31y0sHoj&edmqzu!r**C}Tq9p3p7i-JGhZ)^QWBEQcu!10h_gG_h>k6=Js`#9xP&yl%_Wq5(Ud{_3`^&;2?C2srhy2}|d z#pVjh6UAQp1Z+WPHO}gB{IUjOAt&l8+k7V*_W1+S5W|l4yw2~TMO{uzWkpt}Ixi+Y z6@w{rCe`O82uk~i>B1~5(&?u+-?|Q82`(omZ$26YV#~H*2YLMaknj!S8GnM?*eo}9b)<-&_k@dW@RCdMBEZL z+{#PIuM+2^XVj`$54XMfy`e|d~s%a96mi@Cbd?s{)Q!F#ow){HM+`z-} zT;ABbv-{bR@YwKaXw^;O-Ny%GLdCI6w;cMA*d!&DN0IV zQGwHaKa^TuUXCDCMk`8E)oHl4&VK*a-ZAWA;o6DQb=#r)f-V{SRjpB~Sau?co+u@N zACn@3!7Q>U(uZD0$V+oHsZ3y znmK>k)9z%WD%jGai7(^&xf??Nn_5CLdrwnCL~2>$RuKgSI;una!1T}DxVbh(C3gR1 z_S^FmFWG=5+>}YK@R-6Y1A;*pMXP9Q#{H3Puf6Oa0Mx>jCHpd=K*M|2V6LF&t&dV1 z&W*ETPKsUOuq&2;sMZj6k!H4+Y=2$5(acvH!?k#AtyO9M$a3s`RmVy=@SK{PvnF+@ zocDEto#z$U#2>}iMbYBkn)h_yU!lv57|2D7a7btBsmZSjbLMJdt(-6Paa7z=V{D_^ zq%4;dy;r(!mpNyz?Q|WUf`7U5R2Bm%+yt7HeqHbp`i8C7Q^j9AO&dRc&n-f~0$9S8 zWtJE&_5f-c1#1)tNuxFRanz|72ENnE(?Cy}q-p{mWuJZsr5CARx~vUTsqudZ|*-J846_I@kaU6}uJMNfJZ@7~AZVGrH4 z&t;KLDpN}MH;2zg&AU_iN1)}&-Q%q8%ckfp)9uqJGOBPq%_aVNbvvE+Cr6=+L`a|0 z&6@fmmvzcAJ*p(a=Dt5e-?Bcn>gl^YBZr|pr?Fj)RAcAI_H`)I~Qn-&~5m?^nu-A}FrE8IVxmG+jKsv{61?|L4#8#0(29_5b~%szrOG!bFnjDMSCT$;!AFvTXDc` z_h8|*(NGN9($^~a7Mx3aVg&yL-15qY>ufT%o`ur!gO7ni+s3JPgtNw&WDM>-*KJXjd-n@ycVY9tOfQ!M_a5PlGOPeJ=S1Hu*SPD~d~6 zj^Up=P7t~8D^3Eq+rt2u^1b_@r1h|bu;w1*N5TdjV~5Gq0-KgX0K_OxpgN?rm_9#z zQ3Uqo8Q;k+_c8uo!jN!UR)@I$T}D_NW%w2u@VMWNCjo5jAhuhj(dd0}z7t-d+?b33 zm5}MlxxxJpxEH2n@d*K7FjomTP60-26Q-S|ynm7XDM8=(2;XiYOHVf$&ngi;=92~hE;vO4JscZtF`k&FlD@Z4 zk}K3AqO$gAX$n`A4Z~b#lAfVHv;ne6W_@op3OC&wRd zMPK6O`x9Sf7e5Gyx%kVjwiJ{UO<15o+ zPClVO`(OBVJY1abxny+A#bW2HlbT&gQSSb^hrjy*S;tpQnr2xdoS6Q8FCXBn9U3O} zBi@au_MWXcj=JX=lIgO?P@Sbr$d!3x3eZMnu;) zLr4h~{&({H`h$7n6ZOfGca8VWQ%@=Ly>pR<1#U|RNu6PYqM z-#wEi1Cmo_4GILV^?jus9*$ zHl4P-+sTqm7pNH9w6|nIhdNR8KYz+z8gbj)A5PCb+9or7lH*M^vmD6=f&p-SHp13X z>)BD~KGlE2v#R*@UYWVbO30WR2g_{6hQEVYz+Y$<0XQ}nU8alWcB=IYXdG7k{YN** zpJfT)$$u+07Icq{gQ}eE?-VG3wE=`4b)6@jgTEL&kWDqbJ!err4*Ef# zNuD)Ch#Ha&JLMClE-x>Z@v&2!yQz8zHKS2R zcC)9*R}p8Me5Wd6Ez19~1@oB6vf1q$fjhZ;A&{ZVYCWmhC-DU!RI-ROu-$*W&ktfA z>S^k&IVmC&?Jx0oe!Br+<^ZNRWuJAM#E6B(-i=5CnhSly6$?LMl{=s&Tr{w9Yk zd6-gC0KoghF2Inafxp+*(5vxXB#(~Ss9|~$f5Gi2M)8q{9xs?m;u@~8gD50AvVPo; z33PE8tVc_^Z9e6Ea}ve_zpDuh8~}MfQkm1YWC7u*`r6rQKL1ENIsR!Ycg86g2lp&CU8?^xw9FLW}yk9%0Thanxv+qwoZnvK8 zot2XrP0ZVB^!7#Ei)0!Veh~h}bgydWDiZd@jr(sFy}I7%)Xu2p6!%v+$F{+q#q=JF zKjNK#0e`dDK9QItjV|QmBSEBIPlCgLQob|vWW`DeG;@;?Iq417H0%f5s%&XAaBcLT zdJW_a-{jU?|24de%d82RZn(8MZjx{FC8PRC+yN9}WZ|yR+s)Z+AltiQxrZ4u(O(MF znCQl>+<&s`omleMNj5dQ5xJgB%gT74XFeK(2~nINgq)U_oOhkb@#RcT*;6S3TPVth z=xKaASF0YrQ~j_xSM9h%>A0hx?FRba0W@LK`sRAj*%8~E%Y1{!FnU6+V&A=)0;b@Fp5`iCMyOPuW#4J{yB<&N+18C#p$yy z2C@@$+IMz2IJG}}d>}XWeNhV}Z`m9r%iy^d!HRs!gMtN-+CG}c>#S@yd^%PQ2Eb~z zj<7r+oa%BKk5U`rcIt3p&n+bt!b|m#@i@EUT;a*7N0UC*Icx#oFRmzeQ$j42=LJPTM!*TqLpG0#r zoRB-bhX4di-(sYv(wWOsrUJ1 z68Shu1FKx@cEZyh@WIoitmE%l4?6cg8nRhp4G=ei)$3#Kz+n?Tg16L}QbFdHAqDdblA6J#t&y*92J=7E1P4SK_8Hz3 z@=}$nx~#)0+$ml71Ps%()GhPwne7w-AdvEGKq{vegQmERB(g}`;wY!QmaK#=|2sI( z4y+mZ&4f^stNS6U5jXS;078d=)`Fwu@3_pJyiM@KfIJhC>q8*92}_rQH?PadaiKq~ zjva)c(c1VtVEDM)d;O96!oEP z&jX&7o}J;y;m#Psusflu!*PQz6zHmes%c^~Szi3Z=%1OCXR^S1c5qq4BEWMgylu?s zc)?$MKi_8~kMwzTw!qB>0ze5mKBO75mlI*465pNr7|0AgAC?X!r<6ZImTWIXkXwhM zh5fG!<=fFj&wK+Q^H3~(Wv1%GEEA2sDdUEuXf~V+WvPFUcuBLSj=eH(l6Rgvj#>fB z6b9Z1{#0_#{|eBhJmNp(1byC(F_3>Mpm=GoIh+mtSvcXdJJ5!8$6OuQV;xY=mLrs{ zsF0g8_aTLW`wz)Y)v73%o_8slY97N6eTkSCT0Z))r(8*2Kjy>HCHqE!{~^p%fPvyA z{$7-W@c$P*<_Catc$rUe>nl@9EHhgi(B#siwtle0QM9$haDv1xQM_DynNIo33&RP@ zylWCUDX6)2y>n?GzB^iuJbj^oTw%`L3kNnShH$ThCqF2AS#|GkPLRjdYx!Md>cp z-d#6rSM+yTF(16p#=ztv^Z{YyS)`n3Despww^oN;g>~*8`h{`h`#WMw=x``eU+XcCRbEJFnPZ@{B3pEK};WO`KkJz>>p~& z+iuPWr`p;duq<{Vv9VXB-Z${ntoR0=*Vx@sLk{P!RlWx5YCX3&G)YR#8!x0+Rbk%x z_n>Y$YWK@=(al(rs&*RPH%&W{Tkj*}*O%Dq-i3&8-b$kSC)t&m)6^)bVX0>_MK|J$ zDz3H|dGfwQ`=sl(plSMgCn7Tef?Rk2afzgv_t50a4L9Wk$Aq5i#!)5ZEv_7nl}^XIMTt&SgLi(e7M9e82~GxD^$l9Cu}O z!M_0!gC66x^nOB4yDA!gCLE-PP0$F*7T8A4Rs3ulEy3U4%>l1;4no6$WOd7{_?;m4? z#0sA#Nd_cWmn!~3K()D(HdHi@gzm0{EFvRvIm|N`=y|6bWVC|SYde$vy@wLe3SUN4 zMAfguG2wPLwl1+k7OloKD&%t?j|A@E#7&tDQ=9(tNC_S1%}XNkJa*i)T2aBBYeGkS zXwr}+8W0?NZmiwb+$=98ocj8BS#D(ahBthg8fPZEk zaWthtN|z1Q2xM&ngaV0p3>1$T%Vd4=n_&hRdxqTJxT)iLUMSO|_9vVd+o2P;Hz1cv zh{N5{VM>%<%cV`A`x)G}V!Jkgzp~u~WCGi6!bQ1RHaf9FCKci; zjiI~Wg9?x6<)MA4LcOl+I}aFwu~prNO&@Jg3cZqo!3FDiv(^TKk4nxx$RAv*e(i_S zO}raW@#R@?0ej}GlXisi5KiO|b7%HVjma+WfMv7%u|WdC3t`br&nsx#S@Xww*Dt@Q z0cj!Db6%vU){9CnJfX(!`0F&KgJVGcWb{?iy}(eR-&n4PuqPH3c~#~|0b*fR)fkev zsDhQnxM2Rck03vHu!o23FS(8uk*J{dSu+u|*MqfQPEeG|MP7%f7nqAzXSKPcVGqh- zR^noI1_|Zw+ZoDMZbM;;0{>DduUj#sTQ(&^=Uu(6 zacs?XgxClDOUm6~=MMB%SG%9zDG*>dLG6MNGdY;y95CjNnU}v5T<~-A>0Q5rq>bJSF zg#rprhk#)4l%A_qlMTW(x0U~!0~Fm^pqDCy8|&ThE_rI2+Id8~dUh|kKsf6uE|03u zCcp_6?dvQ+rSlk^A_ub>?|$OkXPf$Uy6P-S?}p^ef%Xwl6572irFEgzDcm@@Fg^s4 z$OCCrn4fd*WBKiKu|wP~tDP#S$C<{6O@ODhaTY!|Ybn-=Hb4EIxyh|L=mSe94fpD4 zn;lL(guQ$CUianl+tP)u;9t_l2R8k7u-_gY7fZ7GR!qk$&p49Sq%%_v5G)V!&}wF^ z5)MA<>?hIm2&%kT;Zu-0Q2Y>~^%c4^rl!@}NQGozA56wyjN|(TpB%vkd7=qr<8A@20 zK?P6Dkxd2zV?^Fh;F`R~%f@ya?o6Fv0Jp6~lnzfygRB?3sJd>Krp}4zrh;nla_4b8Q6a-4VGzR$Kd)tLVKWCa*M~gZ#7hrmEncUj*RhcORm)62vRN zNp{~V+&z-R5Cxb&+`RT}en!W;&Y`O$_g37C zmawBmc#h|@l;L*cgRYbofwV^dFwZcxgl(yuFhz~KJE*k3i!IhAyc>IC(3Tl1=V4=r zui8Y0_F2X3CHsu;N1vBkYkV3TYx9@lj%MuKN>>24tF{NX46=)n`EH}N(6bs~T9^=; z^Fc?$=wLA5eQPxnFdn{1U&PyYDlYV~vO-X$m5LGR6|zVs&TyEYln~zVr#$DnH`4`Q zt3%JxS>rGL>{at|<>hgT_mv}f1cp3~eEqnG$G#abi8ESm+|3DVjQ|gRQwUlMmUzr-c6)GYURtmhj-1 zLD$|aU+P~8F{|z%t&d}nWvMD7cS`jzCof!|MZFXp;foBLiv#VkcO2fVew$MOvUKN9 z<7z8GK3-~fb832bV{7HMd3K2W%`BNKyUblfxrvQekwcQ!FA>c!qtK-(tsoVnU?=Un zNd~1n{N0er*$#m#KbbB$w1n+z8s#x(aG!rK%5MQtD|VQ*&eOPsa2 zI8}@<`**^cbGb2P*!}^!8ng0+q#sd%2$rU?Ddhw3c8O(}s(gQ8S*L8m74X>0Cwk3W@jVPf=I+v1sH~4iZJf zD9v*(KK{Yv8r~Phjz(O%aCzcJ5C*_WJd}}4#wyPt*QfNSdCDnJw<1@Ncy$E!@HVc5 z&!A6lqyxB5XyKY&=3#Ah7OytaTWzkB{g5Swpo)d4^xI=`Mr8zh)z`m63~XJu$9+{4 zu7825!)$(9U*7b)E~HK~Eqr0dSa(cf;P1qdlK4LgvmE+UbA#}#XefEf_}14yN6V^# zn?uqi{Cx{+#OBS$xAPKBI;JBZ=`MISl0?1T+E<%C10eozgT{1ac3BYx0kNSC7m<>b zN9@11uf8VFiLI>q{BV$QMSP-tYnt6Tg;l#@`GXoT*kXLeZT#WsG7fZ6osw@$+;>O` zC4TXOUaw_58of_?s@H5T&{q=zI~3)Pu+2wBW*2!hwe3uo9pLYAlps5>;u5a>$H161CT@7 z`|W@H>wR58Crhz@edy%7<)7JWk`JwK{^v^%=NJq#g1U#g?6KE?3{N?WaPM!$R}{Y1 zbG}gb>h=kg?2urS4NcVVtJESUHc1J(*7tZ`*7DpIOY>teaEuj`C- zPP00>r9p35?xxtLFmaAlkn#b_KB%9ItWluBCDolRjvu20`kKWH%Xo3MeEm2GZ+qy4 z&Pk^;)sVR@K2@mOD0?%-7RbXW)Wv@cwcaFWK+GwBb=t2S@lcfh)n4ay1I>9b@->}Hyd|%t9sL)?hVpu4>D)VN=pI$E5o&YZh9X7BL>FXt^EhP2 z|CLq~lW?jCEKy$WJT*vAUgD4~wW43;Hwh;CKH+41{u@KMo?IO|Dx7fEX1C4r6Wk^x z9Hhu~Zd~}LY@2;i*1WKHh!~m$u+87%^SaaR@AZQ&V)fOH5xdJBdvLnd3qUhie15I8 z9Pc7I4n4OxivWF7YzYGgnk#~G43)%1#9>pZ@uWScnK7O2U@GXolC9L`G+m zD4A2h>7s3L>y^8>u_qM#k(5<|>qmbsNuPGzy_(#@GkY6c>N{nQK;Dtt%RWtOisw2c z5B6MVb9OY1V5|G}+gs_GWuhn0-lx`9md0kmZSgJrpqD1glkhB+pdu_3$hQW%XfrIO z|L)LR1Mz=Un~gH*^oFhBDFV)T?lpjo4+iR^NaB zn2X^WnhwDXc>Q*=_+v8Syw>*U4n4!!1&*)js9Ul`)$I9qm*{Y+^6_fGF7|^uZZFO| zUZX=t4|*@3`QJW){$Faqepw*b@pUWGj zL-v~NmRRs2Y^jSNohnO5C+rl`E7dvj4*EGq7N*+_%NZ_Js>yp{RdvBLV=%xqhBA

VVSXLtkg%wcBx;`d>5XtQ@b9O+hTTHLw=aG9Rw4D87%r(O<+-I5vW+M!=;id!KPZQ3`HI`0pS`d3fDq2DU zJ>-l-OFteUT)89C*3H0X zvpH*~RSxQi#7fF*l$`>bSbEMO{W?wki6$`hY`+#GS9mHrKr5}{VXl6rH06hC%6G85 z&e3+LLb2O14lrRZ)#yRlX`cgwH+=wcj1ZCCH!S{E00e-PDI7lyR1^zcfC`EMtuPnn z!#@1)C`6-{#d3eW13RFhT_U;lwY*Z_77+tBR+0~Uf+ugs*%B6`f8z`7rPXZkLdWw$ zg)9NvK_fK&o_%3@XnYHx71^FsD6^94UI>6r0t=dt2(E@S-8(ZHFRO@#_mQ0(;4jGv zlF%w(MyYnJ#{tZ^+1u`Idq82)nhfYAG|{j@Ism?5tB^*!z|;6*v|4J5k+R}HGflIi z$}$}gl{27GxLA*lPDM?E*e9*ZQf2V~=H_RM+0l~a@yqc5X^$M-)TU=^t6$Pjv)sh8 z(Envv*KX9*K0$PgVDtgild(3Vr}HT@zii0>ORn@3E6^KC;~*B&m`x)yzj|i$y&4xl z)Kpfegt z=ZG43vQ>0K$~e{|cStsiNHu+@p;-FSj_gN;B>;D;?Aa-m4+-eIltAUk!>{a&^o}%M zvJQvRcINEYG+BHftteIvOi92mA zqO?op0f|2Y8u5~Kp^c`#GCN6zFI+G4enO_f&X6F>g8yYA)GEDbgqQvky+x7_fVMG{ z`!&Z>Rtj*MLfBs4(&Ge1A=E~B)qfkD$jxjTNNsL8#O3DE^wyCDqKh|A zpN4|D!7LQPcJD%t&$3ts8;8;*F@K;h20A(`x}2-_g(8--j#(r&PltT&myT{h9sq=J zesQ3^&^DwNHWV;{vf}NiV<^Q{zNKMv*p9Fj{E^C@pIwwvzHyF-NbO-T``ad=Q^1>-G*muq+FUpe>iQyO4R4v3i5VVD~om>2ew35AL=^ zyrj&#RLKG{l@=Ng@llE_4KlX!g9c@qn>@@BLKXn{APar-r3Z$*rG5V%A!pZi4@PDB z8(w?-nSo2F2M+t>IFw!f;-vTSZU)!pX2X>tm0EG`RhpPvV%!&&t;JK4DvdN4mZT~A zA{e`57_ekms9`q$M=RYZQ~nh~^uNdE8yTM}SCR{FPw^iVGGwP^hrLDThz52#&+2xt z+H~veG}jz$O2pMM1R9cP`{3Q~NLj!`sW(fhq_7E^0@0mGp>YrEJ%F*wQ^f}5G~%}D zs1%YRJ`s6?t2A#eejAMF8Ifb0K<`f@Esh2S$=0*P9qtguN)&Cq}r16UDx4^J>Lcy zJ2YOB`T-~XNhhIxIH%C7g*x>~&I4Y^Sd!4R@iMHrn9bg|ahHaBQC(PNZk&Gh*;ps} zLTmy~2l=(LG9#R=Z_wXq2DS`DpHJYg=_Snoqk<10q@HMo#nP zvCH~;^w_P|UR4qEJK1wRkqFPbbyZvM}TrQ=p$J>ajmD;CH*!yPYp&pYLeY^~iv{ zrX48r&S?f8N&dROo&C>xfpSVWyb~N37t^+E+ffl3nIX?_lSBloqd0NH?Uoi6+Q2OK z1&G?pYV2w~kiPE`q$H-xNgc;@a4($f{!o#3i$-I@duj1R%;Mq*B$!k!jI>g;PyJY_ z5($!SU%ERwj^841q1e9>re z%)#VN?w~X*+pw&_7Q#LNM_JStJFJM}UnF*_4i(cQud90nBltFy8 zQKwA^bnxdr@@f~r;p8;UECg`3nCnB({C^G}b_8*CvqDfpoV|xD>$0!(Udl8n_Z7yt z1yBZKectV}HSYrm$r>N+aGk)tt2DNTYl(4XX8bKrXbmebLDLfeM(EtSv71pLfyQ?a zEt~Uh>}NgL^qZm5GZ+_ibN+WUn4+ItLd*m3lyf%c)-;kHdamxMO#QmQM|VvyXIdny!6$hn(<@;W)#c{4%J`TDK+ZDur2Fla0Oo9a zlP%CVpHgP=pft(RcP5v)^MKG&fAoz<8?;E$C2hnR{3E#!0%Xo7byK^_8 zVQB*Y{wI$Z1cA6d$ z)BvGV3_OVFNrEOl@RD~p`eP**i7EFGB zI$n3B?=IyJ+hvl+my|e(AMt=6b@48!<(jDvD@*cCF8R2^mH*!tj?asW!`!tpc5-WK z+7~04PM&{W_~}r4?DT8QFlSP&c;_UG>&RY8*&|CyOrA`ZMhf5o*#$G!l|I^z$X{F> z*ie?XQ|{Kk$oywd2+{bd@0$h=g71V|ggfTnYs$I7zAUG`p8mdANu(`uW5G08C;O^& z##15Ga*Dp2{wnf!0J6_mDAp(IZhg>YKh)sE2>@DIM9$D+)~4))Ui?yHvP&tXnWpyF zMr9=VAhedeNN3hj+Vax=Y1u2>&IJ_V<cg z56WpCtF`^@r6Jey{G!fDTy6l#=tyY^id=X^Cds(|6KTYd)?COdfK9J6k z#myhc71jxM+?@4k?$vFHANuo%)U+wM{`LuTw{H4{*umlHn2x!M*e(!1@|kAk&}3&; zg7Es2VN+og7&$CuI23vo|1m4uBQmx_MjJ1J9!(1i{}}|FGg47^gRw-RpPqnwMt6zKt5ZY0^VY9@U1@& zAiliPN%-v5vl_Wflgc@Tfxh}Zu8TACnz?>9>vu|-1SzaoUO50{v1gLQdn6hXEDw?6 z25V1+*Xpl}+G0*eaL^fqyd-!O6E>apsm!*s)woOSja!vaBUcgfVGw42>>44h)URss zygVTF(zbBj_e!PhIt{0NKNplQ?*U>#$g2*;4UY{(mqA9xj|?6RJ*3|Z#nnB3XxMkD9PobQ*2V0q$gR^sVnY}Yl#cY&C- zS47%J+9t{iIL*f&sfUpH(`4(7hJFT-MMro%;QIi3g5T^e^(VKAzg#xd{1s;UTBRnD zto@+p??DeQq>rZ8Un3zjA;MJ7vB7gLxNu&F4~*|q&%%kS^;gUGxz}7O2!^$#kb{{a5K}4DAice4f z$j_LEC^by8%jT->#}4hO?;pifnJb;l7H%=x_0n@9VP|Q#e>zR|wMSO*(DbB&2wtIg zPnwU_?kyO3iIS^t^n4fu=aCOa&#xG9b(mdyKm6A;8@TT`^V;e;)&sX%Eyd>Jfk91h z)9q)KN9lmljXv#07q&(Z1T7tH*ZOAmYhOPbM1B!GO?bhcvg=K+UOuF((6V4a4(_#D z=Ox_>VC@Eczxj))i1=K8{<8n1q2BeqP3+qF=zbfTepVN><_FEKx4L>rQ7PyZk@(bB z479HVMry)8BweOP_FYTvI<$EHPm*tBuGcDp4aq6y%x!s+`mbXW<^?=5=pD)~C0cRa zi=Rv!;Eli{t-7B?L%cmiDg<-Izt&uU)oigJ&A(X1jPLY=Wc6g9S8ruU`9aBVLSz8@ zaUZaTxHU3|ZJx-VzfubWp5nPm5mA<} zwtuLZV)si2L(hf8Bm|wN23U;qPRYN*;a zCg3^B0~U+eRmaluuaoF zA;%Hh|D+jXrIk2^Fmnk-TtVNWS*`0vG#W41;|uz%hWY-VZ}cGaZG%R5YgS$nfuC%! z-vdXj<=?zS`@QL3E`4*fbvUk+>p?VXj%!ebeK58T1qS&Bh_$l!Oi|QkW;}l4E=LXo z+B0sF zS?5Nj*$~}sdeK=4jMH5W=U{&kpL!-~58HV)?c%2hJa+9igM;aQv_Ysluvtc3FFLbF z?h%BSdO@tv^c(zKtPlHB9&vr|JZl>DKU$UY$y7iGMLiVx`jx-DRawZx{MKvD?nSk7 zi5R_?1rf{Qs!oADbpd)-xNNw)1G|=#k4d?|+3;Vk7m=62EY5PH+BaGDKh{Or2`dyw z%`UzP81KF0m8ZodEX+8}YLlTEi~@7JX4PF9bT6ARa!xUT*7Bsv3K%t?ZOX4e;8VSz zCZV^|a@jmYaMRW}OSk6|4kgHzeg-&^^c^3&HPj?s=4e{Na>%6FBXh};A-pAQ=Pw6M zl@4bo6~s0ILw2vIX0TVtqfS*->}Ve0hQ-|j?r3;*$)__^_h9$3LBkd@218?N_LJk| ztdC*lz|%s|HdC$KbBk|`t&kNSJK_J+zE?!1uSC#~ClUCR(xCuEYmu~c4yCCv1q9nmx!bA(&lugL&&iTYT9RP9zNU{pu^r^ z>eWAv4k6U+nKUfqvINWjjmkS~z!I2{D7fsV(g)2j9Fn&=fk-|KILlk1WvEZNC5OOTGSevSw7;y!SdrE-#WLP=8K6=3! zfsLkpY1H8O`LX*T(!jp0L}mqX4=cRpxrb$SOptB)_MDV$R9DHxV(q&upDu0CHwm45 z2G*_@%En@<+=$2mi7*}1^OwL_8eC-1B|K?tPY`^&t}iQceXz&z^?UFdG~)&Oq@*|P zZLdRvca^vUM3QqGYxzJ-ZKm_KDq73OY1hb1K6rfOpu4s`fk>{qY~HmFn+|T&a;}5V zRstZ_`v^yEjf4apzm-2_elwsLn98j@`ZQfa<#E}K5gUk4=}2z1{MfZy5P*P9^DH?) zEI=A9QZ52B1&G62dx2n;upH3(qA>@1(x@AN@O45p!HM<_F%&{O6DJ7VDN&=2`H>5f z50;nWOKHc8F@mGEG^plvH~FgOHae1d`#oRcCj3kp561>)^j+#X30XJKLAoa~Vk%_y zbqDgMEVqm4kD7`t&v>F&wIS7NX?}46z%mrSakpt9zjTeaeQAe-;{D;tD-V0zUc;Y5 z1V$#%hJEoNE8QtfP?%}Dq|VNF*O!9G&ocBipDE&T>qfr@gYB7I?(r%-BQ#gm##9XX zSg!cMj?(DQh4fWYGEnCYC757C$wWqNb}omRWDoJZYaA z0k=lL?JyQnUfA+I`{XMR1bVI+ga0!$Nx#Xz^WGf3`8ma=oiX+idoP7UGByVEDS#u( zly@Q}x0)mwmr|-iy|3KLEVjSXUEwRR;Wz|MXJw>*9vg?PX`p8pg8B-YmZ0 zG^xn?^mFXkaDmNrA}~5?o$JnNUv@Q0-N1hp>ayKPx3u^41)?mD;al1_5_&J=$-i=Q z${OedOSR{WWeRqidv`tA&R1D}@o%DsXjo!9uzUP$9!Mcs^_P!(_Oj#0s(&_DcpN4U zV9Nxw%8uq1eJJUVW95aE3L(n^NWAcx?1;hL!yAr8WjSmJXV}$;)aU)@J};%TKhct1 zjw)EUi=pVxRJ=&2kSIujO{{yfKeATn!5dXlI1*8q(It%jg13czn<8dg!U|{N2KB^? z#k<(pG<0c=Wg)L+S%oH1y?MUa*t_Mmo2jBjexMt6wVqr1NQhb(CH?DfM;}l3>(WGK z;su-Q)4$U%K~#w{`-Ir{XL-iD&|~JL_1T&k9oFND@VGB3g|#RA{=>7V6_}HIn5FNp zoGtl3;>+|`Ol-tm`|hk?i?1Qs51QnfLlrIhKHD zN?GtjIlQD%#lfF9S}GXd1SwNX%S%iNi*Vw9%AoIHE@0bT0unO?78DN@YTp?Ke{?PD?_`HvXzu` z3{gj=Z4K61M&oG2GzuELI zxXf%a%3-LnIKO%(hx3s|p9oIitO2#FYv;I>_asl*&dwp#0hC3QRl?gH8cZt~JW#PV zuCKOivteVuvoBhzJ#y3$}sZSTJ6e znhwLn|GNPgCVn7wk|tzc5xdg{!>>4PlnNc2`I0{OLDhGk5Q0${VhP7x^R5bydiROd z?dYm4O)a4>J(m_NhR#r9%7h%vq3MHvjZwe-Ajaa&ID`fKU#6Mw*@3!J66}(fDx$|K zw>+I1zKBNhO;zv$kQ!|(pY%J_C+lmvQ0tbGDX9uXxC|QoK6&A}U7GA8T*9c{2c63m z$OM_TQT*?Gq*h~)lR-uVnU$}Blz(jBLpGYcJ4%1pJzFg!1Ou6haDJ=mhRf=K^b-*K zhjnwfvE@B4tNPHixN4y_4}PtJ2$t?n8c9_^qhh}N6E}!^5m*Ocq1OxatiEr( zTp^*hzhp8q@pLt)Qf9PRo^L@hJUKOa0baH2&+qcK(H!PO$OuD|oqN2`59vsSusF+B zB0?*E;;RgML12ra9O@&y^Nh3Opqsggvvv=Q3x?(^{EG2YAArW3CrQ zK5Xy*YxVl`ZT)eRT3j9X>8w!GD~tS&fyx*cP0(|N`|`Rk=CTAI)1K62hc9aMF&Vkz zR?yr2TaE3jczW^*=^)10htT`>Q#Lsq0h8tA@9D~d&xu4g zx6+3#8Ja$9zBVq=flz7-uGzBe0L`7gJt3L0zeUeWQ-kredFvX zv&J5LY(W@po$LzJwlg|Bo#%QNA*RO+iOf~w<4y{7i;!{g|4eWV{Il|kYQ~$s!7{VQ zKrsH-4FmN&!Wu&7mbRPq>qtdDqhDQ_R;Sw-2D7lA$&?3sYgiV#mt_5z|3GlSBZxKi zD}D1d0m{_dop@}$Yn7Zar8Fv-GaCgib0>`3!!?-e^QSxNmv#s+52(qnd>f~f{$@Vs z_1gL_r|cC3sqkX zJA!UL1UQjWlrwd+x8c0b+(;WH<3OD>Nebs5rj{!^PZ1hJxr3hHU!=j=MTPhr^=HZ{ z#f>Mry55?i=lQZcrsI4koZw7FeiO&oG=GERH_|>;FgQDdv=@IyWRl`mVO1mIX#OjK zu1F=ozdG_*cIgM z-fl4}0cG>|vgBHTBNH*tAXmT6tF!hh-nze0(^>GTjPu-CT!5+2u)tK8TBAeCH9H4V zci_(A%qXWtjoFRq&hr&jIYKq!>2w0ko8M;+y>(4}1;^(OugI`F z!qe8|b*3amkN4a0NkF-Y7CS`9_*`7qYbks<&3~*qBSe9%dzrsqHy`Fs7-dJf#6FU% zBD5omUd=F5r&fmpOfA=+L!15WvM}sqAPhDO5~Lq{-+R_z zi{yVYR}i_|JWhc`gx+W-K3aWC!4_$GuI4Xf>*w@z`(X&`#oB5S(+1UHFm2yQt`8>#%make zM&mJE<~JdZKhY$gCi?s#5Lm--g)SOwiRk4zXExdX_fJ0Erh7*$e4XXnxe(~5jKgy^ zSjC?=IO#R10=FW%^29{~D=g!QCPoCXUT}^MH9N>R&r?oS9js9e=$<}zz*F83@!p2N z1}ZK!$@%+t?%uaJQbuNakMZU5E;)H*ekThJCUgN>gu{DH5)OXYNK?#US+NQtYM^A% z=W0*yfDJB=-6v?>;V_

Wb8HQar+p>fytl7*mAh+c!{GFw`eAktahHD}VJ?qo+$T z8d8=Xa`L$yqiOrCaeE(N8!Ht&S^?n&-OeE3YIVgRGIFgyYC*<>b7%MPjfJccsLJw% zW%kO$qZ|h#^|L^4SfQBT5GLc8+SF&IH)5HAwZ0 zdNQ$Mk+6l9xGDUv%}+X!H>>@=5ay)BbC54@>_t(MM#P%r8OJ7Xpl_;dwDdk?Pbslq zPo{+qS;#-KtQ?=`89Q!A>oPclR82>5_~CJJAJ7Od!BB6O^=vb>0=HgXZ<*eFz`Y=Ef;}VthP_{v0zMDf@C^k;e4sG z?iln8reFe+z!LL_YXhP4Vr^)*%R_CcY8GfZtX=!>Tw!jN7M=Rcq1Tmv5@H|ou~ywc>j{0}cjij_ zgU+hQzPJZtc@>%-J&tz2GZ_tZx4w9x@^*1*b)#e}5VBpZfU$TjkBy8rysI8s-RUO^ zxX8SUmgRK9ywY<@TOf|_I4AT=UBSI(2VF9v0%$^sV(Yf4NnZBxuHCB}P=feYxS-5E zy04T0qm-x?AO&-_0=B2}!&f1)w(J_kY9;JXjdkYq>Ymt2;{J4J7JIDYLhX@RIpQnU zVc6T=XR=j{a-|3f-kX_yRX$TaeRnJ%lG;=41?9xqwirxN%lr_n*o)vk^yR#~bFLNF zg21g^yQq#03vWq!S3XwSBOYC2qGfeioK49VAKTCUt_)MSh%6_6R&{uet%eK%>vwH& zz?0fMAQTxAVAOEw7K;FkGA_YfedT)|_(tTcR(MZ9_g$_s+p$Sc(ui>L;& zTd+_4U&Fj&n4TP#@NgHs(= zE+Y`9bYj9i`js)t&9eQ?^OBIaT?*$Nrr^FA={ORb`r%&DROE6beKGsGQ-s&z=sk)A zhxm?crRXtQ?eHHGMmbOSrU`u3;|dzfsfWLu^3+W3EB{U3PmnjYI7V7-^#Aw=(q*bS zLdSdhF2Lf1EUkN{#aiQzfLVU3ot^H97ib2eLnqVQXb7G1(9?PX-FU!%rLf!lJOy2+-(y0ondB89bDk2HNI zO!oDc`YxP8JLquQ|D=jvmjI-WUw4<2yDqr@?EqTpy^<=szJ%rJInagVLDi286X#>; ztpryO0xQ&xJT8i-J5KgulKT>?I6K&1Pd!ci`o&*2umXd?=3)Jcf zUBjEmg?f=3H7WlNR1GifDaK_68Q1icX5vsXQ68ZVTwgoI1lEFKTni5nCeK&D@qBa# zr#;CP>7!;kov;6@7Z{)15mRrRB;`woW(DmA@UL?iC0{79k!@;?R$rj6Gz~@7fKS_1 z$1|OieoeWp^nDjUZ(~Vo*8#Ezm&iRb68k|X(W9m1a1uxP941M&j2k)Ubv|dE1E0kKQF2^MsouFZJhKbu%gEpe&)!AN=x`-z3yFuS~J({G}-iEUVk~E-syg$<~ zrpgY*)Vv&n#H2UanH(Lan(oT(fc<4B&&E;R^E6JZlxGh5^|U znZPiW3$&WY#_u`%>Z*sqXhEJy-m1v`t>GpQ4;SWZfGv8y8&7ltHSFATw(ibdxt@5| zAuw(s&nk|23226476}b`Kh8qZ`En3eH9=Xu8<%7sntss=$ zQNG-+gl9pNgdFsIxf=0U`kdC0LP8`1=fs(Du0a)CMP~;pGVhPmaZnqCbQB~i!a1dU zMnl=l)~M-={RQsw=%xk+tEok#TiT+SuP$7Y%W5A^wE;zSZ}A-(eK_qhW-#p4Jt^QV zt3#;iq+-9Z$W(a*Yi*SR$2A|z$!j?_ztMSK4Ri&G?dO8IRNz1*V>c00Vx4!cwo{|vx^S&;69H&KFO_}VxTocTz2!;b+!HW z`W6bhBwuSNN!vH!tk=7|Hrz zsNUe07B6R}LQ6`o=t$s?os#<1$t_%&=+cYC&yoEsUYnW&`BGD!`}x zb;M?nvJrZA_PvBh*T8+GA9tcHsvb@$8q@+h6k>~|WciPMF@;x!b~vB~5!5|QJtLwg z|6|r8o3L!gmf<^vaC=Lk#_A~t(~*L{cDJD&(YyyY+m%6{4l6sZ4;v5dPwG5y zFO*1V>*NX^be);Fg~$((Hz=GgJh{A*f$gDYbm`!hn9nEPm~qu}_z@5~;yD~ZtHU`G zQKpt!gx6}9tl1LW8}oVDY|ZYsNksQ1w^`FSyz6C~uE}%D-VocoYs9KzUxA)(!IAbTg@n9MQRD*!Ce(!RsY;TvJ zz2S@_5x&%fg=mx}tbnarmj3sKa2#Cl^u+f&59J)^8J2A7!zVz9<$MPX;lni#gF@ln z9Bz?^n)jP(7+07#&bnag{@Vife(D#JJRs9O@!L4P__Sll`_w