Shar-jeel-Sajid
a049e97e7c
clippy linting automatic fix
1 year ago
haixuantao
649902783c
Fix CI/CD
1 year ago
Philipp Oppermann
49aef8bac6
Move more implementation code out of `dora-message` into `dora-core`
Ensures that we need to update `dora-message` less often.
1 year ago
haixuanTao
3775a3c08b
Make downloading file use `content_dosposition` header or the filename within the url to get the path avoiding to create confusion on the name of the file.
1 year ago
haixuanTao
dd10bbf85c
Bump `pyo3` version to `0.22`
1 year ago
Michael-J-Ward
15bbacc3d9
fix clippy lints in binaries
1 year ago
Philipp Oppermann
6a3dcecf33
Refactor: Move message definitions to `dora-message` crate
First step towards versioning the message definitions indepedently.
1 year ago
haixuanTao
7bcb132075
Retrieve `open_telemetry_context` from metadata parameters.
1 year ago
Philipp Oppermann
9cec8df1cc
Delay dropping of `DoraNode` in Python until all event data is freed
When dropping the `DoraNode`, it waits for the remaining drop tokens. This only works if all the dora events were already dropped before. With the Python GC, this is not guaranteed as some events might still be live on the heap (the user might even use them later). In such cases, we waited until we ran into a timeout, which resulted in very long exit times (see https://github.com/dora-rs/dora/issues/598 ).
This commit fixes this issue by adding a reference-counted copy of the `DoraNode` and `EventStream` to every event given to Python. This way, we can ensure that the underlying `DoraNode` is only dropped after the last event reference has been freed.
1 year ago
haixuanTao
c4263e6c8f
Remove bound from definition of .to_py_dict
2 years ago
haixuanTao
b2eace277d
Transform custom PyEvent into standard python dictionary for easier debuggability.
Currently having a custom PyEvent make debugging very hard as fields are hidden within the class PyEvent that is defined within Rust Code.
Python user are getting really confused about this obscure class.
This PR transforms the class into a standard python dictionary.
2 years ago
Haixuan Xavier Tao
d7be6a49e3
Merge pull request #472 from Michael-J-Ward/pyo3-bounds
Update Pyo3 bounds
2 years ago
Michael-J-Ward
af332c18f7
remove deprecated `py::new_pool`
The deprecation warning states:
> code not using the `GIL Refs` API can safely remove the use of `Py::new_pool`
All GIL Refs usage have been removed, so this should be fine.
2 years ago
Michael-J-Ward
25884b658a
update pyo3 to new bounds api
2 years ago
Michael-J-Ward
cd8ed19620
deps: upgrade pyo3 and arrow deps
Arrow needs to be upgraded alongside pyo3 because they both link to python.
NOTE: Arrow marked `ffi::from_ffi` unsafe.
https://github.com/apache/arrow-rs/pull/5080
2 years ago
EricLBuehler
6c62246e53
Fix typos
2 years ago
Michael-J-Ward
3b59b026b4
lint: allow type-complexity for select signatures
2 years ago
haixuanTao
c6948c1644
Only use conda if conda is present
2 years ago
haixuanTao
629a218ddb
`copy_array_into_sample` do not need to return a result
2 years ago
Philipp Oppermann
e8b87f0338
Merge branch 'main' into c++-ros2-bridge
2 years ago
Philipp Oppermann
2eeb40b1ff
Fix operator API: Box optional error string since `Option<String>` is not FFI-safe
2 years ago
haixuanTao
4ca53258ad
Move opentelemetry implementation to `dora metrics`
2 years ago
haixuanTao
0725620eab
Simplyfing the metrics functionality by using meter provider instead of
the global meter.
2 years ago
haixuanTao
4085ebc46b
Make PrimitiveArray only serialize one element at a time
2 years ago
haixuanTao
fd7c0939df
Run `cargo fmt --all`
2 years ago
haixuanTao
135d6433fb
Trace send_output as it can be a big source of overhead for large messages.
2 years ago
haixuanTao
645c56d7a4
Bump to opentelemetry system metrics 0.1.6 that measure GPU memory usage
2 years ago
haixuanTao
32831cae75
Bump opentelemetry metrics to 0.21
2 years ago
haixuanTao
ab3fde910a
Put reloading flag outside of the loop to keep the state of reloading
2 years ago
haixuanTao
5bff656138
Add an error catch in python `on_event` when using hot-reloading
features
2 years ago
haixuanTao
e45975f65a
Use cleaner `__from_elem` `avec` constructor
2 years ago
haixuanTao
fe065eec6f
replace const cache line with 128 to match arrow
2 years ago
haixuanTao
739cfd57f7
Change type hinting frfom Uint8 only array to any array
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
Philipp Oppermann
a882104b93
Fix: Count buffer length instead of owned capacity when sending arrow data
Buffer capacity is always 0 if the buffer is allocated externally. Also, we don't want to allocate unneccessary extra space if only parts of a buffer a used (e.g. when slicing).
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago
haixuanTao
90c28de777
Fix clippy warnings about unnecessary `.clone()`, `.into()`, borrowing and let statement
2 years ago
haixuanTao
b20ef2391d
Optimise sending of small vector for python
2 years ago
haixuanTao
e944b794a3
Adding `data_type` parsing
2 years ago
haixuanTao
0220d1ef65
Adding `data_type` to metadata for typing output
2 years ago
haixuanTao
d55df9a326
Adding documentation for the send method
This PR/commit adds documentation for the send_output method.
Follow up discussion: https://github.com/orgs/dora-rs/discussions/317#discussioncomment-6475816
2 years ago
Philipp Oppermann
049e5e1e43
Make runtime pass dataflow descriptor to Python operators
3 years ago
navy
5f64bd2be6
Update mod.rs
add an attribute that disables the dead_code lint
3 years ago
haixuanTao
93b7da2b74
Remove pyo3 in runtime and daemon as it generates `libpython` dependencies
Previous implementation of the runtime would link `libpython` even though it
might not be required.
This commit put pyo3 dependency under a feature flag that is only active if
the runtime is called from python for a python operator.
This fix:
- https://github.com/dora-rs/dora/actions/runs/4879188973/jobs/8705515472
- https://github.com/dora-rs/dora/issues/280#issuecomment-1534058207
- and https://github.com/dora-rs/dora-drives/issues/58#event-9162532660
3 years ago
Philipp Oppermann
8bf124a014
Merge branch 'main' into multiple-daemons
3 years ago
Philipp Oppermann
aac5e6515a
Integrate `dora-runtime` into `dora-daemon`
Removes the separate `dora-runtime` binary. The runtime can now be started by passing `--run-dora-runtime` to `dora-daemon`. This change makes setup and deployment easier since it removes one executable that needs to be copied across machines.
3 years ago
haixuanTao
c80a1958cb
Show node name in process and put error after Traceback
In #229 , I renamed `eyre::eyre!("{err}{traceback}")` where it should have `eyre::eyre!("{traceback}\n{err}")` . This PR fix this issue.
This Pull Request also add the name of the node process at the end of the process call to help with debugging.
3 years ago
Philipp Oppermann
8ec45239ce
Send empty map empty arrays to `None` when sending outputs
3 years ago