From 2de867fa69686f568ade85ee66952d7c61c6b272 Mon Sep 17 00:00:00 2001 From: haixuanTao Date: Wed, 10 Aug 2022 11:45:33 +0200 Subject: [PATCH] Adding api documentation --- README.md | 2 +- docs/src/c-api.md | 29 ++++++++ docs/src/communication-layer.md | 4 +- docs/src/dataflow-config.md | 50 +------------ docs/src/getting-started.md | 120 +++++++++++++++++++++++++++++--- docs/src/installation.md | 10 ++- docs/src/python-api.md | 42 +++++++++++ docs/src/rust-api.md | 73 +++++++++++++++++++ 8 files changed, 268 insertions(+), 62 deletions(-) create mode 100644 docs/src/c-api.md create mode 100644 docs/src/python-api.md create mode 100644 docs/src/rust-api.md diff --git a/README.md b/README.md index d1b80dc1..fc89949f 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ Composability as: - [ ] language-agnostic: - [x] Rust - [x] C - - [ ] Python + - [x] Python - [ ] Isolated operator and node that can be reused. Low latency as: diff --git a/docs/src/c-api.md b/docs/src/c-api.md new file mode 100644 index 00000000..5a7f0f7c --- /dev/null +++ b/docs/src/c-api.md @@ -0,0 +1,29 @@ +# C API + +## Operator + +The operator API gives you a framework for operator that is going to be managed by `dora`. This framework enable us to make optimisation and provide advanced features. + +### Try it out! + +- Create an `operator.c` file: +```c +{{#include ../../examples/c-operator/operator.c}} +``` + +- Copy `operator.h` header file: + +```bash +cp apis/c/operator/api.h . +``` + +- And compile your C operator: +```bash +clang -c operator.c +clang -shared -v operator.o -o operator.so -fPIC +``` + +- Link it in your graph as: +```yaml +{{#include ../../binaries/coordinator/examples/mini-dataflow.yml:47:52}} +``` \ No newline at end of file diff --git a/docs/src/communication-layer.md b/docs/src/communication-layer.md index bf06f231..0469270d 100644 --- a/docs/src/communication-layer.md +++ b/docs/src/communication-layer.md @@ -1,4 +1,4 @@ -# Middleware (communication) layer abstraction (MLA) +# [Middleware (communication) layer abstraction (MLA)](https://github.com/dora-rs/dora/discussions/53) `dora` needs to implement MLA as a separate crate to provides a middleware abstraction layer that enables scalable, high performance communications for inter async tasks, intra-process (OS threads), interprocess communication on a single computer node or between different nodes in a computer network. MLA needs to support different communication patterns: - publish-subscribe push / push pattern - the published message is pushed to subscribers @@ -29,4 +29,4 @@ MLA also needs to provide high level APIs: - send(key, msg, optional fields) - recv(key, optional fields)->value -More info here: https://github.com/dora-rs/dora/discussions/53 \ No newline at end of file +More info here: [#53](https://github.com/dora-rs/dora/discussions/53) \ No newline at end of file diff --git a/docs/src/dataflow-config.md b/docs/src/dataflow-config.md index cbf85535..9d881460 100644 --- a/docs/src/dataflow-config.md +++ b/docs/src/dataflow-config.md @@ -96,57 +96,11 @@ Each operator must specify exactly one implementation. The implementation must f ## Example -TODO: - - ```yaml -nodes: - - id: main - name: Main Node - description: Implements the main task - - operators: - # sources (only outputs) - - id: timer - name: Clock timer # optional, human-readable name - description: Send the time. - shared_library: path/to/timer.so - outputs: - - time - - id: camera-uuid-1 - name: Front camera - python: camera.py - outputs: - - image - - metadata - - # actions (inputs and outputs) - - id: timestamp - description: Add a watermark on the camera. - python: timestamp.py - inputs: - timestamp: timer/time - image: camera-uuid-1/image - outputs: - - image # with timestamp watermark - - # sinks (only inputs) - - id: logger - description: Sink the data into the logger. - python: logger.py - inputs: - image: timestamp/image - camera: camera-uuid-2/metadata - - - id: camera-uuid-2 - name: Back camera - run: camera_driver --kind back - outputs: - - image - - metadata - +{{#include ../../binaries/coordinator/examples/mini-dataflow.yml}} ``` + ## TODO: Integration with ROS 1/2 To integrate dora-rs operators with ROS1 or ROS2 operators, we plan to provide special _bridge operators_. These operators act as a sink in one dataflow framework and push all messages to a different dataflow framework, where they act as source. diff --git a/docs/src/getting-started.md b/docs/src/getting-started.md index 7d561161..d0befd85 100644 --- a/docs/src/getting-started.md +++ b/docs/src/getting-started.md @@ -1,17 +1,117 @@ -### 2. Compile the Rust example operator: +### Create a Rust workspace + +- Initiate the workspace with: + +```bash +mkdir my_first_dataflow +cd my_first_dataflow +``` + +- Create the Cargo.toml file that will configure the entire workspace: + +`Cargo.toml` +```toml +[workspace] + +members = [ + "source_timer", +] +``` + +### Write your first node + +Let's write a node which sends the current time periodically. Let's make it after 100 iterations. The other nodes/operators will then exit as well because all sources closed. + +- Generate a new Rust binary (application): ```bash -cargo build --manifest-path ../examples/example-operator/Cargo.toml --release +cargo new source_timer ``` -### 3. Compile the C example operator: +with `Cargo.toml`: +```toml +[package] +name = "rust-node" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" + +[dependencies] +dora-node-api = { git = "https://github.com/dora-rs/dora" } +time = "0.3.9" +``` + +with `src/main.rs`: +```rust +{{#include ../../binaries/coordinator/examples/nodes/rust/source_timer.rs}} +``` + +### Write your second node + +Let's write a `logger` which will print incoming data. + +- Generate a new Rust binary (application): + ```bash -cd ../../examples/c-operator -cp ../../apis/c/operator/api.h . -clang -c operator.c -clang -shared -v operator.o -o operator.so +cargo new sink_logger ``` -- Run the `mini-dataflow` example using `cargo run --release -- run examples/mini-dataflow.yml` - - This spawns a `timer` source, which sends the current time periodically, and a `logger` sink, which prints the incoming data. - - The `timer` will exit after 100 iterations. The other nodes/operators will then exit as well because all sources closed. +with `Cargo.toml`: +```toml +[package] +name = "sink_logger" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" + +[dependencies] +dora-node-api = { git = "https://github.com/dora-rs/dora" } +time = "0.3.9" +``` + +with `src/main.rs`: +```rust +{{#include ../../binaries/coordinator/examples/nodes/rust/sink_logger.rs}} +``` + +- And modify the root `Cargo.toml`: +```toml= +[workspace] + +members = [ + "source_timer", + "sink_logger" +] +``` + + +### Write a graph definition + +Let's write the graph definition so that the nodes know who to communicate with. + +`mini-dataflow.yml` +```yaml +communication: + zenoh: + prefix: /foo + +nodes: + - id: timer + custom: + run: cargo run --release --bin source_timer + outputs: + - time + + - id: logger + custom: + run: cargo run --release --bin sink_logger + inputs: + time: timer/time +``` + +### Run it! + +- Run the `mini-dataflow`: +```bash +dora-coordinator run mini-dataflow.yml +``` diff --git a/docs/src/installation.md b/docs/src/installation.md index 8a28f502..88a21f3b 100644 --- a/docs/src/installation.md +++ b/docs/src/installation.md @@ -8,7 +8,9 @@ The `dora-coordinator` is responsible for reading the dataflow descriptor file a Build it using: ```bash -cargo build -p dora-coordinator --examples --release +git clone https://github.com/dora-rs/dora.git +cd dora +cargo build -p dora-coordinator --release ``` ### 2. Compile the dora-runtime for operators @@ -17,3 +19,9 @@ The `dora-runtime` is responsible for managing a set of operators. ```bash cargo build -p dora-runtime --release ``` + +### 3. Add those binaries to your path + +```bash +export PATH=$PATH:$(pwd)/target/release +``` \ No newline at end of file diff --git a/docs/src/python-api.md b/docs/src/python-api.md new file mode 100644 index 00000000..248ce03f --- /dev/null +++ b/docs/src/python-api.md @@ -0,0 +1,42 @@ +# Python API + +## Cutom Node + +The custom node API allow you to integrate `dora` into your application. It allows you to retrieve input and send output in any fashion you want. + +### Try it out! + +- Install python node API: +```bash +cd apis/python/node +python3 -m venv .env +source .env/bin/activate +pip install maturin +maturin develop +``` + +- Create a python file called `printer.py`: +```python +{{#include ../../binaries/coordinator/examples/nodes/python/printer.py}} +``` + +- Link it in your graph as: +```yaml +{{#include ../../binaries/coordinator/examples/graphs/python_test.yml:12:17}} +``` + +## Operator + +The operator API gives you a framework for operator that is going to be managed by `dora`. This framework enable us to make optimisation and provide advanced features. + +### Try it out! + +- Create an operator python file called `op.py`: +```python +{{#include ../../examples/python-operator/op.py}} +``` + +- Link it in your graph as: +```yaml +{{#include ../../binaries/coordinator/examples/graphs/mini-dataflow.yml:67:73}} +``` \ No newline at end of file diff --git a/docs/src/rust-api.md b/docs/src/rust-api.md new file mode 100644 index 00000000..5680ae60 --- /dev/null +++ b/docs/src/rust-api.md @@ -0,0 +1,73 @@ +# Rust API + +## Node + +The custom node API allow you to integrate `dora` into your application. It allows you to retrieve input and send output in any fashion you want. + +### Try it out! + +- Generate a new Rust binary (application): + +```bash +cargo new source_timer +``` + +```toml +[package] +name = "source_timer" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" + +[dependencies] +dora-node-api = { path = "../../apis/rust/node" } +time = "0.3.9" +``` + +`src/main.rs` +```rust +{{#include ../../binaries/coordinator/examples/nodes/rust/source_timer.rs}} +``` + +- Link it in your graph as: +```yaml + - id: timer + custom: + run: cargo run --release + outputs: + - time +``` + +## Operator + +The operator API gives you a framework for operator that is going to be managed by `dora`. This framework enable us to make optimisation and provide advanced features. + +### Try it out! + +- Generate a new Rust library + +```bash +cargo new example-operator --lib +``` + +`Cargo.toml` +```toml +{{#include ../../examples/example-operator/Cargo.toml}} +``` + +`src/lib.rs` +```rust +{{#include ../../examples/example-operator/src/lib.rs}} +``` + +- Build it: +```bash +cargo build --release +``` + +- Link it in your graph as: +```yaml +{{#include ../../binaries/coordinator/examples/mini-dataflow.yml:38:46}} +``` + +This example can be found in `examples`. \ No newline at end of file