haixuantao
f72991b09c
Add Cause to stpo
1 year ago
Philipp Oppermann
6a3dcecf33
Refactor: Move message definitions to `dora-message` crate
First step towards versioning the message definitions indepedently.
2 years ago
haixuanTao
bc68de3bbd
Change `MetadataParameters` into a `BTreeMap` to allow user defined metadata as well as enable more flexibility in managing metadata
2 years ago
Philipp Oppermann
6140681ab9
Rename and reexport `Data` enums
Makes the distinction between raw data and data messages clearer.
2 years ago
Philipp Oppermann
89f980e753
Simplify arrow array construction through new `IntoArrow` trait
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
haixuanTao
b20ef2391d
Optimise sending of small vector for python
3 years ago
Philipp Oppermann
c4bd9d3edf
Add convenience functions for merging external event streams with `EventStream`
Our upcoming ros2<->dora bridge will allow nodes to subscribe to ROS2 topics, which results in a second stream of events. By providing a merge function, this ROS2 event stream can be merged together with the dora event stream.
3 years ago
haixuanTao
f9d8fac8e5
Adding Rust docstring documentation
3 years ago
Philipp Oppermann
1948a45e6d
Copy outputs directly into shared memory in dora runtime
Instead of doing an additional copy to send them from the operator thread to the runtime thread.
This commit uses the new `allocate_data_sample` and `send_output_sample` methods introduced in d7cd370 .
3 years ago
Philipp Oppermann
c1544f8257
Allow accessing data multiple times on Python event type
3 years ago
Philipp Oppermann
a308c48f3e
Refactor Rust node API to ensure proper stopping
3 years ago
Philipp Oppermann
feca83e309
Refactor: Split Rust node API into smaller submodules
3 years ago
Philipp Oppermann
c21b70ae65
Rename submodule
3 years ago
Philipp Oppermann
11e970e268
Refactor: Move `DoraNode` implementation into submodule
3 years ago
Philipp Oppermann
00421e2bcd
Send small messages directly without shared memory
We currently need an additional request to prepare zero-copy shared memory messages. For small messages, it might be faster to avoid this extra round-trip and send the message directly over TCP. This commit implements support for this. Messages smaller than a threshold (currently set to 4096 bytes) are sent via TCP, while larger messages still use shared memory.
This step also enables future optimizations such as queueing output messages in order to improve the throughput.
3 years ago
Philipp Oppermann
2ba397c644
Fix: Don't wait on event stream thread on drop, as it might be dropped later
It is not guaranteed that the `EventStream` is dropped before the `DoraNode`. If it is dropped later on the same thread, this `join` leads to a deadlock.
3 years ago
Philipp Oppermann
d334c6f243
Make `daemon` module of Rust node API private and reexport symbols
3 years ago
Philipp Oppermann
1553a198ef
Re-export dora-message from dora-core
3 years ago
Philipp Oppermann
ff836d200c
Start adding back support for daemon communication over TCP
3 years ago
Philipp Oppermann
6eb9b46637
Close outputs of operators when they stop
We cannot wait until all operators of the runtime node are finished because operators might be subscribed to other operators.
3 years ago
Philipp Oppermann
6438ef7413
Add special handling for zero-sized messages to avoid one roundtrip
3 years ago
Philipp Oppermann
cf95078ca1
Extract shared memory server/client into separate crate
To allow for individual testing and benchmarking. This will also enable reuse of the library in other projects.
3 years ago
Philipp Oppermann
c9e8d8dfe4
Join event stream thread before exiting
3 years ago
Philipp Oppermann
503a9497fd
Use shared memory for event stream too
3 years ago
Philipp Oppermann
70298d836f
Start migrating the daemon control channel to shared memory
The goal is to avoid the latency of the TCP socket.
3 years ago
Philipp Oppermann
619d0bb34b
Fix: limit slice length to requested length
On Windows, the size of the shared memory region is rounded up to the next multiple of the page size. By slicing the region to the expected size we allow nodes to assume that they get exactly the requsted size.
3 years ago
Philipp Oppermann
1516ee6559
Don't allocate shared memory when `data_len=0` when preparing output
3 years ago
Philipp Oppermann
ce4e5a1616
Send metadata through TCP for now
3 years ago
Philipp Oppermann
0ae5fc2ffa
Add dataflow ID to daemon messages
3 years ago
Philipp Oppermann
9a9394c204
Update Rust node API to parse new `NodeConfig`
3 years ago
Philipp Oppermann
2e7ef8b630
Implement shared memory mapping in node API
3 years ago
Philipp Oppermann
201fd228f5
Remove communication layer from dora-node-api
3 years ago
Philipp Oppermann
c2e4948dd5
Update Rust node API to communicate through dora-daemon
3 years ago
Philipp Oppermann
7d85b92b2a
Connect to dora-daemon from node API and register with node ID
3 years ago
Philipp Oppermann
5375628930
Allow operators to stop the full dataflow
Useful in case of emergencies.
3 years ago
Philipp Oppermann
8fbe60acf0
Merge branch 'main' into uhlc
3 years ago
Philipp Oppermann
c4da2cbdad
Refactor: Move configuration to `core` crate
3 years ago
Philipp Oppermann
1c1891446a
Add `uhlc` timestamp to message metadata
3 years ago
Philipp Oppermann
47896abe81
Fix: Also add metadata to data slice
3 years ago
Philipp Oppermann
6ab39e229b
Don't serialize data to provide true zero-copy reading and writing
3 years ago
Philipp Oppermann
7e9c3050f9
Use `tracing` for logging state of nodes, coordinator, and runtimes
3 years ago
Philipp Oppermann
0df06fce86
Send metadata in messages encoded with capnproto
Changes the message format from raw data bytes to a higher-level `Message` struct serialized with capnproto. In addition to the raw data, which is sent as a byte array as before, the `Message` struct features `metadata` field. This metadata field can be used to pass open telemetry contexts, deadlines, etc.
3 years ago
Philipp Oppermann
848e136a75
Move communication layer implementation into separate library
To make it usable for other projects.
3 years ago
Philipp Oppermann
3997b2d3e3
Remove some println logging in Rust node API
3 years ago
Philipp Oppermann
03348c5be8
Redesign communication layer to be synchronous and add support for iceoryx
3 years ago
Philipp Oppermann
41afaed2c1
Add some more log messages
4 years ago
Philipp Oppermann
3c8ee37795
Implement parsing of new dora timer input keys
4 years ago
haixuanTao
515ae91151
making `CommunicationLayer` Send and Sync
4 years ago