Philipp Oppermann
89f980e753
Simplify arrow array construction through new `IntoArrow` trait
2 years ago
Philipp Oppermann
ddf8035d2f
Adjust multiple-daemons example
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
e13415d532
Send cross-machine outputs directly to target daemon without involving coordinator
The coordinator is our control plane and should not be involved in data plane operations. This way, the dataflow can continue even if the coordinator fails.
2 years ago
Philipp Oppermann
f6219203dc
Fix `multiple-daemons` example: Use new `_unstable_` prefix for deploy keys
2 years ago
Philipp Oppermann
bf12a7b496
Canoncialize dataflow path to determine working dir
The path should also be valid on the receiving node, which might run in a different directory.
2 years ago
Philipp Oppermann
aac5e6515a
Integrate `dora-runtime` into `dora-daemon`
Removes the separate `dora-runtime` binary. The runtime can now be started by passing `--run-dora-runtime` to `dora-daemon`. This change makes setup and deployment easier since it removes one executable that needs to be copied across machines.
2 years ago
Philipp Oppermann
72a57ce1ef
Remove deprecated `communication` options from dataflow examples, templates, and docs
2 years ago
Philipp Oppermann
17c52e3482
Use control messages in `multiple-daemons` example instead of sleeping
2 years ago
Philipp Oppermann
9f76239c71
Run distributed dataflow in `multiple-daemons` example
2 years ago
Philipp Oppermann
4f649e884e
Only set ctrl-c handler in deamon when it's run as a binary
2 years ago
Philipp Oppermann
c515651264
Start working on a example that uses multiple daemons
2 years ago