haixuanTao
7bcb132075
Retrieve `open_telemetry_context` from metadata parameters.
1 year ago
Philipp Oppermann
9cec8df1cc
Delay dropping of `DoraNode` in Python until all event data is freed
When dropping the `DoraNode`, it waits for the remaining drop tokens. This only works if all the dora events were already dropped before. With the Python GC, this is not guaranteed as some events might still be live on the heap (the user might even use them later). In such cases, we waited until we ran into a timeout, which resulted in very long exit times (see https://github.com/dora-rs/dora/issues/598 ).
This commit fixes this issue by adding a reference-counted copy of the `DoraNode` and `EventStream` to every event given to Python. This way, we can ensure that the underlying `DoraNode` is only dropped after the last event reference has been freed.
1 year ago
haixuanTao
c4263e6c8f
Remove bound from definition of .to_py_dict
1 year ago
haixuanTao
b2eace277d
Transform custom PyEvent into standard python dictionary for easier debuggability.
Currently having a custom PyEvent make debugging very hard as fields are hidden within the class PyEvent that is defined within Rust Code.
Python user are getting really confused about this obscure class.
This PR transforms the class into a standard python dictionary.
1 year ago
Haixuan Xavier Tao
d7be6a49e3
Merge pull request #472 from Michael-J-Ward/pyo3-bounds
Update Pyo3 bounds
1 year ago
Michael-J-Ward
af332c18f7
remove deprecated `py::new_pool`
The deprecation warning states:
> code not using the `GIL Refs` API can safely remove the use of `Py::new_pool`
All GIL Refs usage have been removed, so this should be fine.
1 year ago
Michael-J-Ward
25884b658a
update pyo3 to new bounds api
1 year ago
Michael-J-Ward
cd8ed19620
deps: upgrade pyo3 and arrow deps
Arrow needs to be upgraded alongside pyo3 because they both link to python.
NOTE: Arrow marked `ffi::from_ffi` unsafe.
https://github.com/apache/arrow-rs/pull/5080
1 year ago
EricLBuehler
6c62246e53
Fix typos
1 year ago
Michael-J-Ward
3b59b026b4
lint: allow type-complexity for select signatures
1 year ago
haixuanTao
c6948c1644
Only use conda if conda is present
1 year ago
haixuanTao
629a218ddb
`copy_array_into_sample` do not need to return a result
2 years ago
Philipp Oppermann
e8b87f0338
Merge branch 'main' into c++-ros2-bridge
1 year ago
Philipp Oppermann
2eeb40b1ff
Fix operator API: Box optional error string since `Option<String>` is not FFI-safe
1 year ago
haixuanTao
4ca53258ad
Move opentelemetry implementation to `dora metrics`
2 years ago
haixuanTao
0725620eab
Simplyfing the metrics functionality by using meter provider instead of
the global meter.
2 years ago
haixuanTao
4085ebc46b
Make PrimitiveArray only serialize one element at a time
2 years ago
haixuanTao
0568656ef0
Remove unused dependencies for faster compile time
2 years ago
haixuanTao
4ec81cc743
Removing old main files
2 years ago
haixuanTao
fd7c0939df
Run `cargo fmt --all`
2 years ago
haixuanTao
135d6433fb
Trace send_output as it can be a big source of overhead for large messages.
2 years ago
haixuanTao
645c56d7a4
Bump to opentelemetry system metrics 0.1.6 that measure GPU memory usage
2 years ago
haixuanTao
32831cae75
Bump opentelemetry metrics to 0.21
2 years ago
haixuanTao
ab3fde910a
Put reloading flag outside of the loop to keep the state of reloading
2 years ago
haixuanTao
5bff656138
Add an error catch in python `on_event` when using hot-reloading
features
2 years ago
haixuanTao
e45975f65a
Use cleaner `__from_elem` `avec` constructor
2 years ago
haixuanTao
fe065eec6f
replace const cache line with 128 to match arrow
2 years ago
haixuanTao
739cfd57f7
Change type hinting frfom Uint8 only array to any array
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
8cc5837cfa
Update `arrow` and `pyo3` dependencies to latest versions
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
Philipp Oppermann
0a9348f7f0
Only enable `arrow/pyarrow` feature when `python` feature is enabled
We don't want to link to pythonlib when compiling the dora-daemon.
2 years ago
Philipp Oppermann
a882104b93
Fix: Count buffer length instead of owned capacity when sending arrow data
Buffer capacity is always 0 if the buffer is allocated externally. Also, we don't want to allocate unneccessary extra space if only parts of a buffer a used (e.g. when slicing).
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago
haixuanTao
90c28de777
Fix clippy warnings about unnecessary `.clone()`, `.into()`, borrowing and let statement
2 years ago
haixuanTao
b20ef2391d
Optimise sending of small vector for python
2 years ago
haixuanTao
e944b794a3
Adding `data_type` parsing
2 years ago
haixuanTao
0220d1ef65
Adding `data_type` to metadata for typing output
2 years ago
haixuanTao
6b910ab619
Adding `arrow-schema` as dependency
2 years ago
haixuanTao
19172523df
Update `pyo3` and `arrow` version to use latest version
2 years ago
haixuanTao
d55df9a326
Adding documentation for the send method
This PR/commit adds documentation for the send_output method.
Follow up discussion: https://github.com/orgs/dora-rs/discussions/317#discussioncomment-6475816
2 years ago
haixuanTao
23651cd002
Make `pythonize` optional in runtime as it link to libpython
2 years ago
Philipp Oppermann
049e5e1e43
Make runtime pass dataflow descriptor to Python operators
2 years ago
navy
5f64bd2be6
Update mod.rs
add an attribute that disables the dead_code lint
2 years ago
haixuanTao
fc0e12fb3b
Remove `api-python` for isolating daemon from python
2 years ago
haixuanTao
93b7da2b74
Remove pyo3 in runtime and daemon as it generates `libpython` dependencies
Previous implementation of the runtime would link `libpython` even though it
might not be required.
This commit put pyo3 dependency under a feature flag that is only active if
the runtime is called from python for a python operator.
This fix:
- https://github.com/dora-rs/dora/actions/runs/4879188973/jobs/8705515472
- https://github.com/dora-rs/dora/issues/280#issuecomment-1534058207
- and https://github.com/dora-rs/dora-drives/issues/58#event-9162532660
2 years ago
Philipp Oppermann
8bf124a014
Merge branch 'main' into multiple-daemons
2 years ago
Philipp Oppermann
aac5e6515a
Integrate `dora-runtime` into `dora-daemon`
Removes the separate `dora-runtime` binary. The runtime can now be started by passing `--run-dora-runtime` to `dora-daemon`. This change makes setup and deployment easier since it removes one executable that needs to be copied across machines.
2 years ago
haixuanTao
c80a1958cb
Show node name in process and put error after Traceback
In #229 , I renamed `eyre::eyre!("{err}{traceback}")` where it should have `eyre::eyre!("{traceback}\n{err}")` . This PR fix this issue.
This Pull Request also add the name of the node process at the end of the process call to help with debugging.
2 years ago