Redesign of the `docs` as an `mdBook`

This initial commit moves many of the documentation into a `mdBook` format version for ease of readibility.

It includes a github action workflows that publish the book on the default `gh-pages` branch to be publish on github page.

.github/workflows/gh-pages.yml

@@ -0,0 +1,32 @@
+
+name: github pages
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+      
+
+jobs:
+  deploy:
+    runs-on: ubuntu-20.04
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.ref }}
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Setup mdBook
+        uses: peaceiris/actions-mdbook@v1
+        with:
+          mdbook-version: '0.4.10'
+          # mdbook-version: 'latest'
+
+      - run: mdbook build
+
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        if: ${{ github.ref == 'refs/heads/main' }}
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs/book

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-    <img src="./docs/design/logo.svg" width="400">
+    <img src="./docs/src/logo.svg" width="400">
 </p>
 
 <h3 align="center">

docs/.gitignore

@@ -0,0 +1 @@
+book/

docs/book.toml

@@ -0,0 +1,6 @@
+[book]
+authors = ["Haixuan Xavier Tao", "Philipp Oppermann"]
+language = "en"
+multilingual = false
+src = "src"
+title = "dora-rs"

docs/design/README.md

@@ -1,7 +0,0 @@
-# Design
-
-This folder contains our work-in-progress design documents:
-
-- [**Overview:**](overview.md) A high level overview of dora's different components and the differences between operators and custom nodes.
-- [**Dataflow Configuration:**](dataflow-config.md) Specification of dora's YAML-based dataflow configuration format.
-- [**State Management:**](state-management.md) Describes dora's state management capabilities for state sharing and recovery.

docs/src/SUMMARY.md

@@ -0,0 +1,22 @@
+# Summary
+
+[Introduction](./introduction.md)
+
+---
+
+# User Guide
+
+- [installation](./installation.md)
+- [Getting started](./getting-started.md)
+
+# Reference Guide
+
+
+- [Overview](overview.md) A high level overview of dora's different components and the differences between operators and custom nodes.
+- [Dataflow Configuration](dataflow-config.md) Specification of dora's YAML-based dataflow configuration format.
+
+# Brainstorming Ideas
+
+- [State Management](state-management.md) Describes dora's state management capabilities for state sharing and recovery.
+- [Librairy vs Framework](rust-client.md)
+- [Middleware Layer Abstraction](./communication-layer.md)

docs/src/communication-layer.md

@@ -0,0 +1,30 @@
+# Middleware (communication) layer abstraction (MLA)
+
+`dora` needs to implement MLA as a separate crate to provides a middleware abstraction layer that enables scalable, high performance communications for inter async tasks, intra-process (OS threads), interprocess communication on a single computer node or between different nodes in a computer network. MLA needs to support different communication patterns:
+- publish-subscribe push / push pattern - the published message is pushed to subscribers
+- publish-subscribe push / pull pattern - the published message is write to storage and later pulled by subscribers
+- Request / reply pattern
+- Point-to-point pattern
+- Client / Server pattern
+
+The MLA needs to abstract following details: 
+
+- inter-async tasks (e.g., tokio channels), intraprocess (OS threads, e.g., shared memory), interprocess and inter-host / inter-network communication
+- different transport layer implementations (shared memory, UDP, TCP)
+- builtin support for multiple serialization / deserialization protocols, e.g, capnproto, protobuf, flatbuffers etc
+- different language bindings to Rust, Python, C, C++ etc
+- telemetry tools for logs, metrics, distributed tracing, live data monitoring (e.g., tap a live data), recording and replay
+
+Rust eco-system has abundant crates to provide underlaying communications, e.g.,:
+- tokio / crossbeam provides different types of channels serving different purpose: mpsc, oneshot, broadcast, watch etc
+- Tonic provides gRPC services
+- Tower provides request/reply service
+- Zenoh middleware provides many different pub/sub capabilities
+
+MLA also needs to provide high level APIs:
+- publish(topic, value, optional fields):- optional fields may contain senders' identify to help MLA logics to satify above requirements
+- subscriber(topic, optional fields)-> future streams
+- put(key, value, optional fields)
+- get(key, optional fields) -> value
+- send(key, msg, optional fields)
+- recv(key, optional fields)->value

docs/src/getting-started.md

@@ -0,0 +1,17 @@
+### 2. Compile the Rust example operator:
+
+```bash
+cargo build --manifest-path ../examples/example-operator/Cargo.toml --release
+```
+
+### 3. Compile the C example operator:
+```bash
+cd ../../examples/c-operator
+cp ../../apis/c/operator/api.h .
+clang -c operator.c
+clang -shared -v operator.o -o operator.so
+```
+- Run the `mini-dataflow` example using `cargo run --release -- run examples/mini-dataflow.yml`
+  - This spawns a `timer` source, which sends the current time periodically, and a `logger` sink, which prints the incoming data.
+  - The `timer` will exit after 100 iterations. The other nodes/operators will then exit as well because all sources closed.
+

docs/src/installation.md

@@ -0,0 +1,19 @@
+# Installation
+
+This project is in early development, and many features have yet to be implemented with breaking changes. Please don't take for granted the current design. The installation process will be streamlined in the future.
+
+### 1. Compile the dora-coordinator
+
+The `dora-coordinator` is responsible for reading the dataflow descriptor file and launching the operators accordingly. 
+
+Build it using:
+```bash
+cargo build -p dora-coordinator --examples --release
+```
+
+### 2. Compile the dora-runtime for operators
+
+The `dora-runtime` is responsible for managing a set of operators. 
+```bash
+cargo build -p dora-runtime --release
+```

docs/src/introduction.md

@@ -0,0 +1,30 @@
+# Welcome to `dora`!
+
+`dora` goal is to be a low latency, composable, and distributed data flow.
+
+By using `dora`, you can define robotic applications as a graph of nodes that can be easily swapped and replaced. Those nodes can be shared and implemented in different languages such as Rust, Python or C. `dora` will then connect those nodes and try to provide as many features as possible to facilitate the dataflow.
+
+## ✨ Features that we want to provide
+
+Composability as:
+- [x] `YAML` declarative programming
+- [ ] language-agnostic:
+  - [x] Rust
+  - [x] C
+  - [ ] Python
+- [ ] Isolated operator and node that can be reused.
+
+Low latency as:
+- [x] written in  <i>...Cough...blazingly fast ...Cough...</i> Rust.
+- [ ] Minimal abstraction close to the metal.
+
+Distributed as:
+- [x] PubSub communication with [`zenoh`](https://github.com/eclipse-zenoh/zenoh)
+- [x] Distributed telemetry with [`opentelemetry`](https://github.com/open-telemetry/opentelemetry-rust)
+
+
+
+## ⚖️ LICENSE 
+
+This project is licensed under Apache-2.0. Check out [NOTICE.md](NOTICE.md) for more information.
+

docs/design/rust-client.md → docs/src/rust-client.md

@@ -1,10 +1,4 @@
-# Rust Client Design
-
-## Brainstorm
-
-framework vs library
-
-### Framework
+# Framework
 
 - Runtime process
   - Talks with other runtime processes
@@ -27,7 +21,7 @@ framework vs library
     - by runtime -> aggregation specified in config file
     - by operator -> custom handling possible
 
-### Library
+# Library
 
 - All sources/operator/sinks are separate processes that link a runtime library
 - "Orchestrator" process