Michael-J-Ward
3b59b026b4
lint: allow type-complexity for select signatures
2 years ago
haixuanTao
c6948c1644
Only use conda if conda is present
2 years ago
haixuanTao
629a218ddb
`copy_array_into_sample` do not need to return a result
2 years ago
Philipp Oppermann
e8b87f0338
Merge branch 'main' into c++-ros2-bridge
2 years ago
Philipp Oppermann
2eeb40b1ff
Fix operator API: Box optional error string since `Option<String>` is not FFI-safe
2 years ago
haixuanTao
4ca53258ad
Move opentelemetry implementation to `dora metrics`
2 years ago
haixuanTao
0725620eab
Simplyfing the metrics functionality by using meter provider instead of
the global meter.
2 years ago
haixuanTao
4085ebc46b
Make PrimitiveArray only serialize one element at a time
2 years ago
haixuanTao
0568656ef0
Remove unused dependencies for faster compile time
2 years ago
haixuanTao
4ec81cc743
Removing old main files
2 years ago
haixuanTao
fd7c0939df
Run `cargo fmt --all`
2 years ago
haixuanTao
135d6433fb
Trace send_output as it can be a big source of overhead for large messages.
2 years ago
haixuanTao
645c56d7a4
Bump to opentelemetry system metrics 0.1.6 that measure GPU memory usage
2 years ago
haixuanTao
32831cae75
Bump opentelemetry metrics to 0.21
2 years ago
haixuanTao
ab3fde910a
Put reloading flag outside of the loop to keep the state of reloading
2 years ago
haixuanTao
5bff656138
Add an error catch in python `on_event` when using hot-reloading
features
2 years ago
haixuanTao
e45975f65a
Use cleaner `__from_elem` `avec` constructor
2 years ago
haixuanTao
fe065eec6f
replace const cache line with 128 to match arrow
2 years ago
haixuanTao
739cfd57f7
Change type hinting frfom Uint8 only array to any array
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
8cc5837cfa
Update `arrow` and `pyo3` dependencies to latest versions
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
Philipp Oppermann
0a9348f7f0
Only enable `arrow/pyarrow` feature when `python` feature is enabled
We don't want to link to pythonlib when compiling the dora-daemon.
2 years ago
Philipp Oppermann
a882104b93
Fix: Count buffer length instead of owned capacity when sending arrow data
Buffer capacity is always 0 if the buffer is allocated externally. Also, we don't want to allocate unneccessary extra space if only parts of a buffer a used (e.g. when slicing).
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago
haixuanTao
90c28de777
Fix clippy warnings about unnecessary `.clone()`, `.into()`, borrowing and let statement
2 years ago
haixuanTao
b20ef2391d
Optimise sending of small vector for python
3 years ago
haixuanTao
e944b794a3
Adding `data_type` parsing
3 years ago
haixuanTao
0220d1ef65
Adding `data_type` to metadata for typing output
3 years ago
haixuanTao
6b910ab619
Adding `arrow-schema` as dependency
3 years ago
haixuanTao
19172523df
Update `pyo3` and `arrow` version to use latest version
3 years ago
haixuanTao
d55df9a326
Adding documentation for the send method
This PR/commit adds documentation for the send_output method.
Follow up discussion: https://github.com/orgs/dora-rs/discussions/317#discussioncomment-6475816
3 years ago
haixuanTao
23651cd002
Make `pythonize` optional in runtime as it link to libpython
3 years ago
Philipp Oppermann
049e5e1e43
Make runtime pass dataflow descriptor to Python operators
3 years ago
navy
5f64bd2be6
Update mod.rs
add an attribute that disables the dead_code lint
3 years ago
haixuanTao
fc0e12fb3b
Remove `api-python` for isolating daemon from python
3 years ago
haixuanTao
93b7da2b74
Remove pyo3 in runtime and daemon as it generates `libpython` dependencies
Previous implementation of the runtime would link `libpython` even though it
might not be required.
This commit put pyo3 dependency under a feature flag that is only active if
the runtime is called from python for a python operator.
This fix:
- https://github.com/dora-rs/dora/actions/runs/4879188973/jobs/8705515472
- https://github.com/dora-rs/dora/issues/280#issuecomment-1534058207
- and https://github.com/dora-rs/dora-drives/issues/58#event-9162532660
3 years ago
Philipp Oppermann
8bf124a014
Merge branch 'main' into multiple-daemons
3 years ago
Philipp Oppermann
aac5e6515a
Integrate `dora-runtime` into `dora-daemon`
Removes the separate `dora-runtime` binary. The runtime can now be started by passing `--run-dora-runtime` to `dora-daemon`. This change makes setup and deployment easier since it removes one executable that needs to be copied across machines.
3 years ago
haixuanTao
c80a1958cb
Show node name in process and put error after Traceback
In #229 , I renamed `eyre::eyre!("{err}{traceback}")` where it should have `eyre::eyre!("{traceback}\n{err}")` . This PR fix this issue.
This Pull Request also add the name of the node process at the end of the process call to help with debugging.
3 years ago
Philipp Oppermann
8ec45239ce
Send empty map empty arrays to `None` when sending outputs
3 years ago
Philipp Oppermann
1948a45e6d
Copy outputs directly into shared memory in dora runtime
Instead of doing an additional copy to send them from the operator thread to the runtime thread.
This commit uses the new `allocate_data_sample` and `send_output_sample` methods introduced in d7cd370 .
3 years ago
Philipp Oppermann
c1544f8257
Allow accessing data multiple times on Python event type
3 years ago
Philipp Oppermann
235f6da79d
Support sending arrow data from operators
This is mostly about convenience when working with arrow data types, as the data is still being copied once.
3 years ago
Philipp Oppermann
9d4909b880
Adjust log level
3 years ago
Philipp Oppermann
a308c48f3e
Refactor Rust node API to ensure proper stopping
3 years ago
Philipp Oppermann
abd850c81e
Send drop tokens over separate channel
3 years ago
Philipp Oppermann
ef37caa3ed
Add arrow support for Python operator inputs
Reuse same event type as for Python node API.
3 years ago
Philipp Oppermann
4cee14ae04
Restore operator state through `__dict__.update`
The previous approach no longer works on `pyo3` v0.18 because `setattr` now requires a `IntoPy<Py<PyString>>` as key.
3 years ago