diff --git a/.github/workflows/gh-pages.yml b/.github/workflows/gh-pages.yml new file mode 100644 index 00000000..7a4417f1 --- /dev/null +++ b/.github/workflows/gh-pages.yml @@ -0,0 +1,32 @@ + +name: github pages + +on: + push: + branches: + - main + pull_request: + + +jobs: + deploy: + runs-on: ubuntu-20.04 + concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + steps: + - uses: actions/checkout@v2 + + - name: Setup mdBook + uses: peaceiris/actions-mdbook@v1 + with: + mdbook-version: '0.4.10' + # mdbook-version: 'latest' + + - run: mdbook build + + - name: Deploy + uses: peaceiris/actions-gh-pages@v3 + if: ${{ github.ref == 'refs/heads/main' }} + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: ./docs/book \ No newline at end of file diff --git a/README.md b/README.md index bfbbccb3..d1b80dc1 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- +

diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 00000000..4e42a1bc --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1 @@ +book/ \ No newline at end of file diff --git a/docs/book.toml b/docs/book.toml new file mode 100644 index 00000000..e9cca438 --- /dev/null +++ b/docs/book.toml @@ -0,0 +1,6 @@ +[book] +authors = ["Haixuan Xavier Tao", "Philipp Oppermann"] +language = "en" +multilingual = false +src = "src" +title = "dora-rs" diff --git a/docs/design/README.md b/docs/design/README.md deleted file mode 100644 index 70cf1e00..00000000 --- a/docs/design/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# Design - -This folder contains our work-in-progress design documents: - -- [**Overview:**](overview.md) A high level overview of dora's different components and the differences between operators and custom nodes. -- [**Dataflow Configuration:**](dataflow-config.md) Specification of dora's YAML-based dataflow configuration format. -- [**State Management:**](state-management.md) Describes dora's state management capabilities for state sharing and recovery. diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md new file mode 100644 index 00000000..2d3fa62a --- /dev/null +++ b/docs/src/SUMMARY.md @@ -0,0 +1,22 @@ +# Summary + +[Introduction](./introduction.md) + +--- + +# User Guide + +- [installation](./installation.md) +- [Getting started](./getting-started.md) + +# Reference Guide + + +- [Overview](overview.md) A high level overview of dora's different components and the differences between operators and custom nodes. +- [Dataflow Configuration](dataflow-config.md) Specification of dora's YAML-based dataflow configuration format. + +# Brainstorming Ideas + +- [State Management](state-management.md) Describes dora's state management capabilities for state sharing and recovery. +- [Librairy vs Framework](rust-client.md) +- [Middleware Layer Abstraction](./communication-layer.md) \ No newline at end of file diff --git a/docs/src/communication-layer.md b/docs/src/communication-layer.md new file mode 100644 index 00000000..593822f0 --- /dev/null +++ b/docs/src/communication-layer.md @@ -0,0 +1,30 @@ +# Middleware (communication) layer abstraction (MLA) + +`dora` needs to implement MLA as a separate crate to provides a middleware abstraction layer that enables scalable, high performance communications for inter async tasks, intra-process (OS threads), interprocess communication on a single computer node or between different nodes in a computer network. MLA needs to support different communication patterns: +- publish-subscribe push / push pattern - the published message is pushed to subscribers +- publish-subscribe push / pull pattern - the published message is write to storage and later pulled by subscribers +- Request / reply pattern +- Point-to-point pattern +- Client / Server pattern + +The MLA needs to abstract following details: + +- inter-async tasks (e.g., tokio channels), intraprocess (OS threads, e.g., shared memory), interprocess and inter-host / inter-network communication +- different transport layer implementations (shared memory, UDP, TCP) +- builtin support for multiple serialization / deserialization protocols, e.g, capnproto, protobuf, flatbuffers etc +- different language bindings to Rust, Python, C, C++ etc +- telemetry tools for logs, metrics, distributed tracing, live data monitoring (e.g., tap a live data), recording and replay + +Rust eco-system has abundant crates to provide underlaying communications, e.g.,: +- tokio / crossbeam provides different types of channels serving different purpose: mpsc, oneshot, broadcast, watch etc +- Tonic provides gRPC services +- Tower provides request/reply service +- Zenoh middleware provides many different pub/sub capabilities + +MLA also needs to provide high level APIs: +- publish(topic, value, optional fields):- optional fields may contain senders' identify to help MLA logics to satify above requirements +- subscriber(topic, optional fields)-> future streams +- put(key, value, optional fields) +- get(key, optional fields) -> value +- send(key, msg, optional fields) +- recv(key, optional fields)->value diff --git a/docs/design/dataflow-config.md b/docs/src/dataflow-config.md similarity index 100% rename from docs/design/dataflow-config.md rename to docs/src/dataflow-config.md diff --git a/docs/src/getting-started.md b/docs/src/getting-started.md new file mode 100644 index 00000000..7d561161 --- /dev/null +++ b/docs/src/getting-started.md @@ -0,0 +1,17 @@ +### 2. Compile the Rust example operator: + +```bash +cargo build --manifest-path ../examples/example-operator/Cargo.toml --release +``` + +### 3. Compile the C example operator: +```bash +cd ../../examples/c-operator +cp ../../apis/c/operator/api.h . +clang -c operator.c +clang -shared -v operator.o -o operator.so +``` +- Run the `mini-dataflow` example using `cargo run --release -- run examples/mini-dataflow.yml` + - This spawns a `timer` source, which sends the current time periodically, and a `logger` sink, which prints the incoming data. + - The `timer` will exit after 100 iterations. The other nodes/operators will then exit as well because all sources closed. + diff --git a/docs/src/installation.md b/docs/src/installation.md new file mode 100644 index 00000000..8a28f502 --- /dev/null +++ b/docs/src/installation.md @@ -0,0 +1,19 @@ +# Installation + +This project is in early development, and many features have yet to be implemented with breaking changes. Please don't take for granted the current design. The installation process will be streamlined in the future. + +### 1. Compile the dora-coordinator + +The `dora-coordinator` is responsible for reading the dataflow descriptor file and launching the operators accordingly. + +Build it using: +```bash +cargo build -p dora-coordinator --examples --release +``` + +### 2. Compile the dora-runtime for operators + +The `dora-runtime` is responsible for managing a set of operators. +```bash +cargo build -p dora-runtime --release +``` diff --git a/docs/src/introduction.md b/docs/src/introduction.md new file mode 100644 index 00000000..93a33dc4 --- /dev/null +++ b/docs/src/introduction.md @@ -0,0 +1,30 @@ +# Welcome to `dora`! + +`dora` goal is to be a low latency, composable, and distributed data flow. + +By using `dora`, you can define robotic applications as a graph of nodes that can be easily swapped and replaced. Those nodes can be shared and implemented in different languages such as Rust, Python or C. `dora` will then connect those nodes and try to provide as many features as possible to facilitate the dataflow. + +## ✨ Features that we want to provide + +Composability as: +- [x] `YAML` declarative programming +- [ ] language-agnostic: + - [x] Rust + - [x] C + - [ ] Python +- [ ] Isolated operator and node that can be reused. + +Low latency as: +- [x] written in ...Cough...blazingly fast ...Cough... Rust. +- [ ] Minimal abstraction close to the metal. + +Distributed as: +- [x] PubSub communication with [`zenoh`](https://github.com/eclipse-zenoh/zenoh) +- [x] Distributed telemetry with [`opentelemetry`](https://github.com/open-telemetry/opentelemetry-rust) + + + +## ⚖️ LICENSE + +This project is licensed under Apache-2.0. Check out [NOTICE.md](NOTICE.md) for more information. + diff --git a/docs/design/logo.svg b/docs/src/logo.svg similarity index 100% rename from docs/design/logo.svg rename to docs/src/logo.svg diff --git a/docs/design/overview.md b/docs/src/overview.md similarity index 100% rename from docs/design/overview.md rename to docs/src/overview.md diff --git a/docs/design/overview.svg b/docs/src/overview.svg similarity index 100% rename from docs/design/overview.svg rename to docs/src/overview.svg diff --git a/docs/design/rust-client.md b/docs/src/rust-client.md similarity index 93% rename from docs/design/rust-client.md rename to docs/src/rust-client.md index ee9b0066..d853e6d3 100644 --- a/docs/design/rust-client.md +++ b/docs/src/rust-client.md @@ -1,10 +1,4 @@ -# Rust Client Design - -## Brainstorm - -framework vs library - -### Framework +# Framework - Runtime process - Talks with other runtime processes @@ -27,7 +21,7 @@ framework vs library - by runtime -> aggregation specified in config file - by operator -> custom handling possible -### Library +# Library - All sources/operator/sinks are separate processes that link a runtime library - "Orchestrator" process diff --git a/docs/design/state-management.md b/docs/src/state-management.md similarity index 100% rename from docs/design/state-management.md rename to docs/src/state-management.md