Philipp Oppermann
e48fed3e2d
Merge pull request #202 from dora-rs/fix-dependabot
Remove `remove_dir_all` from `Cargo.lock`as it is vulnerable to a race condition according to dependabot
2 years ago
haixuanTao
447777a14c
Remove `remove_dir_al` as it is vulnerable to a race condition according to dependabot
2 years ago
Philipp Oppermann
99384e3b32
Remove iceoryx dependency
We now have our own shared memory implementation.
2 years ago
haixuanTao
70e6d4ce8f
Remove code duplicate for tracing subscriber and use env variable to manage log level.
The intent of this commit is to remove the quantity of log that is being pushed to user.
This commit removes the redeclaration of a set up tracing methods to centralise
the tokio-tracing subscriber within the extension crate. It also add the
feature to filter information based on Environment variable.
This makes it possible to define the log level for tokio tracing like
this:
```
RUST_LOG=debug dora-daemon --run-dataflow dataflow.yml
```
I have also unified the feature flag to make it easier to manage tracing features among the workspace.
I did not change the default behaviour of tracing in our crates and therefore by
using the command above you should get the same tracing log as before.
fix merge conflict generated
2 years ago
Philipp Oppermann
9bb57ee531
Don't send replies for `SendMessage` requests when using TCP
Allows queueing of multiple messages on the TCP socket, which considerably improves throughput.
2 years ago
Philipp Oppermann
2ccc170649
Send all queued incoming events at once on `NextEvent` request
Sends a list of all queued events in the reply instead of sending them one-by-one. This way, we avoid some communication overhead.
2 years ago
Philipp Oppermann
00421e2bcd
Send small messages directly without shared memory
We currently need an additional request to prepare zero-copy shared memory messages. For small messages, it might be faster to avoid this extra round-trip and send the message directly over TCP. This commit implements support for this. Messages smaller than a threshold (currently set to 4096 bytes) are sent via TCP, while larger messages still use shared memory.
This step also enables future optimizations such as queueing output messages in order to improve the throughput.
2 years ago
Philipp Oppermann
019e411eef
Change CLI control request reply type
2 years ago
Philipp Oppermann
1e07a73a64
Merge branch 'main' into unify-nodes-and-operators
2 years ago
Philipp Oppermann
1553a198ef
Re-export dora-message from dora-core
2 years ago
Philipp Oppermann
63dd2ff03d
Fix `_pthread_rwlock_timedrdlock` link error on macOS
2 years ago
Philipp Oppermann
44d8ee0e6a
Default to safer TCP for daemon control channels for now
2 years ago
haixuanTao
933dadc68b
Expand `env` value when possible
This commit enables users to pass environment variable as composition
of env variable such as:
```yaml
- id: yolov5
operator:
outputs:
- bbox
- ready
inputs:
image: oasis_agent/image
check: oasis_agent/yolov5_check
python: ../../operators/yolov5_op.py
env:
YOLOV5_PATH: $HOME/Documents/dependencies/yolov5
YOLOV5_WEIGHT_PATH: $DORA_DEP_HOME/dependencies/yolov5/yolov5n.pt
```
Note that this is only for the env field. Expanding env variable for
source path might conflict with using url as source path.
2 years ago
Philipp Oppermann
ff836d200c
Start adding back support for daemon communication over TCP
2 years ago
Philipp Oppermann
77c4b11c8d
Fix: adjust input mappings that refer to single operator nodes
3 years ago
dependabot[bot]
b74a524031
Bump tokio from 1.23.1 to 1.24.2
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.23.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
3 years ago
Philipp Oppermann
26def8d26a
Fix parsing of dora timer inputs
3 years ago
Philipp Oppermann
6aaf4b845c
ShmemServer: Set `disconnect` flag on drop and check it on requests
3 years ago
Philipp Oppermann
cac527a382
Add support for spawning runtime nodes with operators
3 years ago
Philipp Oppermann
44b1f77035
Remove first-class operator support from input mappings
Treat them like normal input IDs with a slash in their name in most cases.
3 years ago
Philipp Oppermann
6eb9b46637
Close outputs of operators when they stop
We cannot wait until all operators of the runtime node are finished because operators might be subscribed to other operators.
3 years ago
Philipp Oppermann
492339a687
WIP: Start porting `dora-runtime` to `dora-daemon`
3 years ago
Philipp Oppermann
6438ef7413
Add special handling for zero-sized messages to avoid one roundtrip
3 years ago
Philipp Oppermann
8383c99823
Benchmark latency of ShmemServer/ShmemClient
3 years ago
Philipp Oppermann
c1346894b3
Remove logging of elapsed time
3 years ago
Philipp Oppermann
cf95078ca1
Extract shared memory server/client into separate crate
To allow for individual testing and benchmarking. This will also enable reuse of the library in other projects.
3 years ago
Philipp Oppermann
bf978d9d7d
Always send register message first, also when subscribing
3 years ago
Philipp Oppermann
c6d864b20f
Make shmem server/client typed and report event stream closure
3 years ago
Philipp Oppermann
5eddf1cae9
Remove uneeded serialization methods
3 years ago
Philipp Oppermann
b7f47c3d99
Log an error instead of panicking when ShmemServer is dropped before client
3 years ago
Philipp Oppermann
827fb1b241
Fix: Don't use timeout on EventStream client
3 years ago
Philipp Oppermann
3aa50d2209
Assert that message length is never 0
3 years ago
Philipp Oppermann
5b12ba8428
Panic when shm server is dropped before client
3 years ago
Philipp Oppermann
503a9497fd
Use shared memory for event stream too
3 years ago
dependabot[bot]
ae161e644c
Bump tokio from 1.21.2 to 1.23.1
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.21.2 to 1.23.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.23.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
3 years ago
Philipp Oppermann
670c154e5e
Create a safer server/client API on top of `ShmemChannel`
3 years ago
Philipp Oppermann
8d1c3742a1
Implement disconnect signal for `ShmemChannel` and set it on drop
3 years ago
Philipp Oppermann
6ae75c04bd
Remove some log messages
3 years ago
Philipp Oppermann
f04960d680
Fix event + data offsets for Shmem channel client
3 years ago
Philipp Oppermann
6253d6e5ad
Fix: Initialize events to 'clear' state and length to 0
3 years ago
Philipp Oppermann
70298d836f
Start migrating the daemon control channel to shared memory
The goal is to avoid the latency of the TCP socket.
3 years ago
Philipp Oppermann
218106721d
Add watchdog for checking that coordinator is still reachable
Throw an error if a daemon cannot reach the coordinator anymore.
This is only an interim solution. In the future, we want to make the daemon more robust and ideally even allow restarts of the coordinator.
3 years ago
Philipp Oppermann
2b7d2508f4
Implement watchdog messages for detecting sudden disconnects of daemon
3 years ago
Philipp Oppermann
35e86684d0
Update `up` and `destroy` commands for dora-daemon and ensure proper coordinator exit
3 years ago
Philipp Oppermann
7448594622
Update `dora check` to check whether daemon is running
3 years ago
Philipp Oppermann
778d9d7eb7
Fix: Use correct length for received data
The shared memory region size might be larger because of padding.
3 years ago
Philipp Oppermann
df5dcfb342
Merge branch 'main' into unify-nodes-and-operators
3 years ago
Philipp Oppermann
2ed7db1b0e
Make `dora stop` command working with new daemon design
3 years ago
Philipp Oppermann
377703c325
Report finished dataflows from daemon to coordinator
Makes the `dora list` command work.
3 years ago
Philipp Oppermann
2527ca0e85
Report spawn result from daemon to coordinator
3 years ago