Website
|
Python API
- -
+ |
Rust API
|
Guide
@@ -29,174 +30,155 @@
----
+An extremely fast and simple **dataflow oriented robotic** framework to manage your projects and run complex **apps**, written in Rust.
+
+
+
+
+
+
+
+
+
+
+ Latency benchmark with Python API for both framework, sending 40M of random bytes.
+
-# What is dora-rs?
+## Highlights
-Dataflow-Oriented Robotic Architecture (dora-rs) is a framework that makes creation of robotic applications fast and simple.
+- 🚀 A single CLI to launch your Python and Rust robotic projects.
+- ⚡️ [10-17x faster](https://github.com/dora-rs/dora-benchmark/README.md) than `ros2`.
+- 🐍 Easy and Fast prototyping with a clean [Python API](https://www.dora-rs.ai/docs/api/python-api).
+- 🖥️ Supports macOS, Linux, and Windows.
+- ⏬ Installable without Rust via `curl` or `powershell`.
+- ❇️ Includes a large set of pre-packaged nodes for fast prototyping.
+- 🛠️ Build and Run applications **without compilation step** beyond the native compiler of your favourite language.
+- 🤖 Simplifies building robotic applications by integrating hardware, algorithms, and AI models to facilitate seamless communication.
+- ⚙️ Eases integration of hardware and software by supporting Python, C, C++, and ROS2, while ensuring low-latency communication with zero-copy Arrow messages.
-Building a robotic application can be summed up as bringing together hardwares, algorithms, and AI models, and make them communicate with each others. At dora-rs, we try to:
+`dora-rs` is backed by [1ms.ai](https://1ms.ai).
-- make integration of hardware and software easy by supporting Python, C, C++, and also ROS2.
-- make communication low latency by using zero-copy Arrow messages.
+## Installation
+
+Install dora with our standalone installers, or from [crates.io](https://crates.io/crates/dora-cli):
+
+### On macOS and Linux
+```bash
+curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/dora-rs/dora/main/install.sh | bash
+```
-dora-rs is still experimental and you might experience bugs, but we're working very hard to make it stable as possible.
+### On Windows
+```powershell
+powershell -c "irm https://raw.githubusercontent.com/dora-rs/dora/main/install.ps1 | iex"
+```
-## Performance
+### With cargo
+```bash
+cargo install dora-cli
+```
+
+## Documentation
+
+The full documentation is available on [our website](https://www.dora-rs.ai/docs).
+A lot of guides are available on [this section](https://dora-rs.ai/docs/guides/) of our website.
+
+## Getting Started
+
+1. Run the benchmark example (Rust must be installed):
+
+```bash
+git clone https://github.com/dora-rs/dora
+cd dora
+cargo run --example benchmark --release
+```
+
+2. Run some Python examples (A venv must be activated):
+
+```bash
+# activate your venv here
+cd dora/examples/python-dataflow
+dora up
+dora build dataflow.yml
+dora start dataflow.yml
+```
-dora-rs can show impressive performance, up to 17x faster compared to current status quo ROS2 in Python! This is the result of using our own shared memory server and Apache Arrow to achieve zero copy data passing.
+> Make sure to have a webcam
-
-
-
+To stop your dataflow, you can use ctrl+c
-> See: https://github.com/dora-rs/dora-benchmark/tree/main for reproduction.
+## What is Dora? And what features does Dora offer?
-## Dataflow Paradigm
+**D**ataflow-**O**riented **R**obotic **A**rchitecture (`dora-rs`) is a framework that makes creation of robotic applications fast and simple.
-dora-rs implements a declarative dataflow paradigm where tasks are split between nodes isolated as individual processes.
+`dora-rs` implements a declarative dataflow paradigm where tasks are split between nodes isolated as individual processes.
Each node defines its inputs and outputs to connect with other nodes.
```yaml
nodes:
- id: camera
- build: pip install ../../node-hub/opencv-video-capture
path: opencv-video-capture
inputs:
tick: dora/timer/millis/20
outputs:
- image
- env:
- CAPTURE_PATH: 0
- IMAGE_WIDTH: 640
- IMAGE_HEIGHT: 480
-
- - id: object-detection
- build: pip install ../../node-hub/dora-yolo
- path: dora-yolo
- inputs:
- image:
- source: camera/image
- queue_size: 1
- outputs:
- - bbox
- env:
- MODEL: yolov8n.pt
-
+ -
- id: plot
- build: pip install ../../node-hub/opencv-plot
path: opencv-plot
inputs:
- image:
- source: camera/image
- queue_size: 1
- bbox: object-detection/bbox
+ image: camera/image
```
The dataflow paradigm has the advantage of creating an abstraction layer that makes robotic applications modular and easily configurable.
-
-
-
+
+
+
+
+
+
+
+
+---
-## Communication
+It offers several features, such as:
+
+### TCP Communication and Shared Memory
Communication between nodes is handled with shared memory on a same machine and TCP on distributed machines. Our shared memory implementation tracks messages across processes and discards them when obsolete. Shared memory slots are cached to avoid new memory allocation.
-## Message Format
+### Arrow Message Format
Nodes communicate with Apache Arrow Data Format.
-[Apache Arrow](https://github.com/apache/arrow-rs) is a universal memory format for flat and hierarchical data. The Arrow memory format supports zero-copy reads for lightning-fast data access without serialization overhead. It defines a C data interface without any build-time or link-time dependency requirement, that means that dora-rs has **no compilation step** beyond the native compiler of your favourite language.
-
-
+[Apache Arrow](https://github.com/apache/arrow-rs) is a universal memory format for flat and hierarchical data. The Arrow memory format supports zero-copy reads for lightning-fast data access without serialization overhead. It defines a C data interface without any build-time or link-time dependency requirement, that means that `dora-rs` has **no compilation step** beyond the native compiler of your favourite language.
-## Opentelemetry
+### Opentelemetry
dora-rs uses Opentelemetry to record all your logs, metrics and traces. This means that the data and telemetry can be linked using a shared abstraction.
-[Opentelemetry](https://opentelemetry.io/) is an open source observability standard that makes dora-rs telemetry collectable by most backends such as elasticsearch, prometheus, Datadog..
+[Opentelemetry](https://opentelemetry.io/) is an open source observability standard that makes dora-rs telemetry collectable by most backends such as elasticsearch, prometheus, Datadog...
Opentelemetry is language independent, backend agnostic, and easily collect distributed data, making it perfect for dora-rs applications.
-
-
-## Hot-Reloading
+### Hot-Reloading
dora-rs implements Hot-Reloading for python which means you can change code at runtime in Python while keeping your state intact.
Using the feature flag: `--attach --hot-reload`, dora-rs watch for code change and reload nodes that has been modified.
-You can check fail-safe mechanism at: https://github.com/dora-rs/dora/pull/239
-
-
-
-
-
-## Self-Coding Robot: Code RAG (WIP)
-
-You can easily create a self-coding robot, by combining Hot-reloading with a Retrieval Augmented Generation (RAG) that is going to generate code modification from your prompt.
-See:[examples/python-operator-dataflow](examples/python-operator-dataflow)
-
-
-
-Self-Coding Robot is just the tip of the iceberg of robotics combined with llm, that we hope to power. There is so much more that we haven't explored yet like:
-
-- [self-debugging](https://arxiv.org/pdf/2304.05128.pdf)
-- [memory](https://github.com/cpacker/MemGPT)
-- [function calling](https://github.com/ShishirPatil/gorilla)
-- ...
-
-## Installation
-
-Quickest way:
-
-```bash
-cargo install dora-cli --locked
-
-dora --help
-```
-
-On Unix system, you can install dora without cargo with:
-
-```bash
-curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/dora-rs/dora/main/install.sh | bash
-
-dora --help
-```
-
-For more info on installation, check out [our guide](https://www.dora-rs.ai/docs/guides/Installation/installing).
-## Getting Started
+You can check fail-safe mechanism at: https://github.com/dora-rs/dora/pull/239.
-1. Run the example:
+See [this demo](http://www.youtube.com/watch?v=NvvTEP8Jak8).
-```bash
-git clone https://github.com/dora-rs/dora
-cd examples/python-dataflow
-dora build dataflow.yml
-```
+### ROS2 Bridge
-2. Get some example operators:
-
-```bash
-dora up
-dora start dataflow.yml
-```
-
-> Make sure to have a webcam
-
-To stop your dataflow, you can use ctrl+c
-
-To go further, you can add a yolov8 operator, check out our getting started here: https://www.dora-rs.ai/docs/guides/getting-started/yolov8/
-
-## ROS2 Bridge
+**Note**: this feature is marked as unstable.
- Compilation Free Message passing to ROS 2
- Automatic conversion ROS 2 Message <-> Arrow Array
```python
-import random
import pyarrow as pa
# Configuration Boilerplate...
@@ -218,7 +200,28 @@ turtle_twist_writer.publish(message)
> You might want to use ChatGPT to write the Arrow Formatting: https://chat.openai.com/share/4eec1c6d-dbd2-46dc-b6cd-310d2895ba15
-## Hardwares
+## Showcases
+
+### Self-Coding Robot: Code RAG (WIP)
+
+You can easily create a self-coding robot, by combining Hot-reloading with a Retrieval Augmented Generation (RAG) that is going to generate code modification from your prompt.
+See:[examples/python-operator-dataflow](examples/python-operator-dataflow)
+
+
+
+
+
+
+
+
+
+Self-Coding Robot is just the tip of the iceberg of robotics combined with llm, that we hope to power. There is so much more that we haven't explored yet like:
+
+- [self-debugging](https://arxiv.org/pdf/2304.05128.pdf)
+- [memory](https://github.com/cpacker/MemGPT)
+- [function calling](https://github.com/ShishirPatil/gorilla)
+
+### Cool Hardwares
Cool hardware that we think might be good fit to try out dora-rs 🙋 We are not sponsored by manufacturers:
@@ -235,23 +238,8 @@ Cool hardware that we think might be good fit to try out dora-rs 🙋 We are not
> For more: https://docs.google.com/spreadsheets/d/1YYeW2jfOIWDVgdEgqnMvltonHquQ7K8OZCrnJRELL6o/edit#gid=0
-## Documentation
-
-The full documentation is available on [our website](https://www.dora-rs.ai)
-
-## Discussions
-
-Our main communication channels are:
-
-- [Our Discord server](https://discord.gg/6eMGGutkfE)
-- [Our Github Project Discussion](https://github.com/orgs/dora-rs/discussions)
-
-Feel free to reach out on any topic, issues or ideas.
-
-We also have [a contributing guide](CONTRIBUTING.md).
## Support Matrix
-
| | dora-rs | Hoped for |
| --------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Tier 1 Support** | Python, Rust | C, C++, ROS 2 |
@@ -259,7 +247,7 @@ We also have [a contributing guide](CONTRIBUTING.md).
| **Hot-reloading** | Python | Rust (https://github.com/orgs/dora-rs/discussions/360) |
| **Message Format** | Arrow | Native |
| **Local Communication** | Shared Memory | Custom Middleware, [zero-copy GPU IPC](https://arrow.apache.org/docs/python/api/cuda.html), intra-process `tokio::channel` communication |
-| **Remote Communication** | TCP (See: https://github.com/dora-rs/dora/issues/459) | Custom Middleware, [Zenoh](https://zenoh.io/) |
+| **Remote Communication** | TCP | Custom Middleware, [Zenoh](https://zenoh.io/) |
| **Metrics, Tracing, and Logging** | Opentelemetry | Native logging libraries into Opentelemetry |
| **Data archives** | Parquet ([dora-record](libraries/extensions/dora-record)) |
| **Visualization and annotation** | OpenCV | [rerun.io](rerun.io) |
@@ -267,20 +255,22 @@ We also have [a contributing guide](CONTRIBUTING.md).
| **Supported Platforms (ARM)** | macOS, Linux |
| **Configuration** | YAML |
-### Unstable functionality
+## Contributing
-`dora-rs` Ros2 Bridge is marked as **unstable**.
-There are a number of reasons functionality may be marked as unstable:
+We are passionate about supporting contributors of all levels of experience and would love to see
+you get involved in the project. See the
+[contributing guide](https://github.com/dora-rs/dora/blob/main/CONTRIBUTING.md) to get started.
-- We are unsure about the exact API. The name, function signature, or implementation are likely to change in the future.
-- The functionality is not tested extensively yet. Bugs may pop up when used in real-world scenarios.
-- The functionality does not integrate well with the full dora-rs API. You may find it works in one context but not in another.
+## Discussions
-Releasing functionality as unstable allows us to gather important feedback from users that use dora-rs in real-world scenarios.
-This helps us fine-tune things before giving it the final stamp of approval.
-Users are only interested in solid, well-tested functionality can avoid this part of the API.
+Our main communication channels are:
+
+- [Our Discord server](https://discord.gg/6eMGGutkfE)
+- [Our Github Project Discussion](https://github.com/orgs/dora-rs/discussions)
-Functionality marked as unstable may change at any point without it being considered a breaking change.
+Feel free to reach out on any topic, issues or ideas.
+
+We also have [a contributing guide](CONTRIBUTING.md).
## License
diff --git a/docs/src/Dora-RS.drawio b/docs/src/Dora-RS.drawio
new file mode 100644
index 00000000..c72535e9
--- /dev/null
+++ b/docs/src/Dora-RS.drawio
@@ -0,0 +1,221 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/src/bar_chart_dark.svg b/docs/src/bar_chart_dark.svg
new file mode 100644
index 00000000..a0484ff5
--- /dev/null
+++ b/docs/src/bar_chart_dark.svg
@@ -0,0 +1,613 @@
+
+
+
diff --git a/docs/src/bar_chart_light.svg b/docs/src/bar_chart_light.svg
new file mode 100644
index 00000000..67f17f8c
--- /dev/null
+++ b/docs/src/bar_chart_light.svg
@@ -0,0 +1,613 @@
+
+
+
diff --git a/docs/src/dora_diag_dark.svg b/docs/src/dora_diag_dark.svg
new file mode 100644
index 00000000..d564badf
--- /dev/null
+++ b/docs/src/dora_diag_dark.svg
@@ -0,0 +1,3 @@
+
+
+
\ No newline at end of file
diff --git a/docs/src/dora_diag_light.svg b/docs/src/dora_diag_light.svg
new file mode 100644
index 00000000..b9883f14
--- /dev/null
+++ b/docs/src/dora_diag_light.svg
@@ -0,0 +1,3 @@
+
+
+
\ No newline at end of file
diff --git a/docs/src/latency.png b/docs/src/latency.png
deleted file mode 100644
index 217ea9b4..00000000
Binary files a/docs/src/latency.png and /dev/null differ
diff --git a/docs/src/self_coding.drawio b/docs/src/self_coding.drawio
new file mode 100644
index 00000000..e9e55b2b
--- /dev/null
+++ b/docs/src/self_coding.drawio
@@ -0,0 +1,117 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/src/self_coding_dark.svg b/docs/src/self_coding_dark.svg
new file mode 100644
index 00000000..c113366c
--- /dev/null
+++ b/docs/src/self_coding_dark.svg
@@ -0,0 +1,3 @@
+
+
+
\ No newline at end of file
diff --git a/docs/src/self_coding_light.svg b/docs/src/self_coding_light.svg
new file mode 100644
index 00000000..4cf729ba
--- /dev/null
+++ b/docs/src/self_coding_light.svg
@@ -0,0 +1,3 @@
+
+
+
\ No newline at end of file
+ 
+ 
-
+To stop your dataflow, you can use ctrl+c
-> See: https://github.com/dora-rs/dora-benchmark/tree/main for reproduction.
+## What is Dora? And what features does Dora offer?
-## Dataflow Paradigm
+**D**ataflow-**O**riented **R**obotic **A**rchitecture (`dora-rs`) is a framework that makes creation of robotic applications fast and simple.
-dora-rs implements a declarative dataflow paradigm where tasks are split between nodes isolated as individual processes.
+`dora-rs` implements a declarative dataflow paradigm where tasks are split between nodes isolated as individual processes.
Each node defines its inputs and outputs to connect with other nodes.
```yaml
nodes:
- id: camera
- build: pip install ../../node-hub/opencv-video-capture
path: opencv-video-capture
inputs:
tick: dora/timer/millis/20
outputs:
- image
- env:
- CAPTURE_PATH: 0
- IMAGE_WIDTH: 640
- IMAGE_HEIGHT: 480
-
- - id: object-detection
- build: pip install ../../node-hub/dora-yolo
- path: dora-yolo
- inputs:
- image:
- source: camera/image
- queue_size: 1
- outputs:
- - bbox
- env:
- MODEL: yolov8n.pt
-
+ -
- id: plot
- build: pip install ../../node-hub/opencv-plot
path: opencv-plot
inputs:
- image:
- source: camera/image
- queue_size: 1
- bbox: object-detection/bbox
+ image: camera/image
```
The dataflow paradigm has the advantage of creating an abstraction layer that makes robotic applications modular and easily configurable.
-
-
-
+
+ 
+[Apache Arrow](https://github.com/apache/arrow-rs) is a universal memory format for flat and hierarchical data. The Arrow memory format supports zero-copy reads for lightning-fast data access without serialization overhead. It defines a C data interface without any build-time or link-time dependency requirement, that means that `dora-rs` has **no compilation step** beyond the native compiler of your favourite language.
-## Opentelemetry
+### Opentelemetry
dora-rs uses Opentelemetry to record all your logs, metrics and traces. This means that the data and telemetry can be linked using a shared abstraction.
-[Opentelemetry](https://opentelemetry.io/) is an open source observability standard that makes dora-rs telemetry collectable by most backends such as elasticsearch, prometheus, Datadog..
+[Opentelemetry](https://opentelemetry.io/) is an open source observability standard that makes dora-rs telemetry collectable by most backends such as elasticsearch, prometheus, Datadog...
Opentelemetry is language independent, backend agnostic, and easily collect distributed data, making it perfect for dora-rs applications.
-
-
-## Hot-Reloading
+### Hot-Reloading
dora-rs implements Hot-Reloading for python which means you can change code at runtime in Python while keeping your state intact.
Using the feature flag: `--attach --hot-reload`, dora-rs watch for code change and reload nodes that has been modified.
-You can check fail-safe mechanism at: https://github.com/dora-rs/dora/pull/239
-
-
-
-
-
-## Self-Coding Robot: Code RAG (WIP)
-
-You can easily create a self-coding robot, by combining Hot-reloading with a Retrieval Augmented Generation (RAG) that is going to generate code modification from your prompt.
-See:[examples/python-operator-dataflow](examples/python-operator-dataflow)
-
-
-
-Self-Coding Robot is just the tip of the iceberg of robotics combined with llm, that we hope to power. There is so much more that we haven't explored yet like:
-
-- [self-debugging](https://arxiv.org/pdf/2304.05128.pdf)
-- [memory](https://github.com/cpacker/MemGPT)
-- [function calling](https://github.com/ShishirPatil/gorilla)
-- ...
-
-## Installation
-
-Quickest way:
-
-```bash
-cargo install dora-cli --locked
-
-dora --help
-```
-
-On Unix system, you can install dora without cargo with:
-
-```bash
-curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/dora-rs/dora/main/install.sh | bash
-
-dora --help
-```
-
-For more info on installation, check out [our guide](https://www.dora-rs.ai/docs/guides/Installation/installing).
-## Getting Started
+You can check fail-safe mechanism at: https://github.com/dora-rs/dora/pull/239.
-1. Run the example:
+See [this demo](http://www.youtube.com/watch?v=NvvTEP8Jak8).
-```bash
-git clone https://github.com/dora-rs/dora
-cd examples/python-dataflow
-dora build dataflow.yml
-```
+### ROS2 Bridge
-2. Get some example operators:
-
-```bash
-dora up
-dora start dataflow.yml
-```
-
-> Make sure to have a webcam
-
-To stop your dataflow, you can use ctrl+c
-
-To go further, you can add a yolov8 operator, check out our getting started here: https://www.dora-rs.ai/docs/guides/getting-started/yolov8/
-
-## ROS2 Bridge
+**Note**: this feature is marked as unstable.
- Compilation Free Message passing to ROS 2
- Automatic conversion ROS 2 Message <-> Arrow Array
```python
-import random
import pyarrow as pa
# Configuration Boilerplate...
@@ -218,7 +200,28 @@ turtle_twist_writer.publish(message)
> You might want to use ChatGPT to write the Arrow Formatting: https://chat.openai.com/share/4eec1c6d-dbd2-46dc-b6cd-310d2895ba15
-## Hardwares
+## Showcases
+
+### Self-Coding Robot: Code RAG (WIP)
+
+You can easily create a self-coding robot, by combining Hot-reloading with a Retrieval Augmented Generation (RAG) that is going to generate code modification from your prompt.
+See:[examples/python-operator-dataflow](examples/python-operator-dataflow)
+
+
+