haixuanTao
bc68de3bbd
Change `MetadataParameters` into a `BTreeMap` to allow user defined metadata as well as enable more flexibility in managing metadata
1 year ago
Haixuan Xavier Tao
16ef29ab80
Merge pull request #601 from dora-rs/delay-node-drop
Delay dropping of `DoraNode` in Python until all event data is freed
1 year ago
Philipp Oppermann
9cec8df1cc
Delay dropping of `DoraNode` in Python until all event data is freed
When dropping the `DoraNode`, it waits for the remaining drop tokens. This only works if all the dora events were already dropped before. With the Python GC, this is not guaranteed as some events might still be live on the heap (the user might even use them later). In such cases, we waited until we ran into a timeout, which resulted in very long exit times (see https://github.com/dora-rs/dora/issues/598 ).
This commit fixes this issue by adding a reference-counted copy of the `DoraNode` and `EventStream` to every event given to Python. This way, we can ensure that the underlying `DoraNode` is only dropped after the last event reference has been freed.
1 year ago
chang xu
bccb1ae27d
Add domain unix socket supports ( #594 )
1 year ago
haixuanTao
329844358a
Reduce timeout time and increase grace duration so that node exit gracefully within one second
1 year ago
haixuanTao
5b7a614c14
Reduce drop token time from 30s to 1s as most of the time the drop token is having a race condition with the GIL in python.
1 year ago
Haixuan Xavier Tao
f34cbe2ffc
Merge pull request #517 from dora-rs/detached-python-process
Run dynamic node
1 year ago
haixuanTao
7d3a5de1c5
Make rust flexible logic and make sure to always use env variables DORA_NODE_CONFIG if it exists.
This remove the complexity of having to rename the node id when used within a yaml configuration.
1 year ago
haixuanTao
3f2f274036
Remove unnecessary condition on dynamic node
1 year ago
haixuanTao
a0a95b730c
Rename dynamic node listener -> local listener
1 year ago
haixuanTao
3ad402ce45
Add dynamic node event loop and dynamic node connection
1 year ago
Michael-J-Ward
25884b658a
update pyo3 to new bounds api
1 year ago
Michael-J-Ward
cd8ed19620
deps: upgrade pyo3 and arrow deps
Arrow needs to be upgraded alongside pyo3 because they both link to python.
NOTE: Arrow marked `ffi::from_ffi` unsafe.
https://github.com/apache/arrow-rs/pull/5080
1 year ago
haixuanTao
b5de93e5f5
Pretty Print Rust object when called from Python print
1 year ago
Michael-J-Ward
7cb43ca3cf
warn: allow clippy::missing_safety_doc
1 year ago
Michael-J-Ward
dfe298eb1e
lint: fix clippy lints
1 year ago
Philipp Oppermann
14bb4fbfb7
Merge pull request #429 from dora-rs/out_dir
Send runs artefacts into a dedicated `out` folder
1 year ago
haixuanTao
f4fbd6562f
Sending data recordings into an `out/<DATAFLOW_ID>/input,arrow` folder to make it easier to manage multiple data coming from multiple runs of a dataflow
1 year ago
haixuanTao
629a218ddb
`copy_array_into_sample` do not need to return a result
2 years ago
Philipp Oppermann
e8b87f0338
Merge branch 'main' into c++-ros2-bridge
1 year ago
Philipp Oppermann
2eeb40b1ff
Fix operator API: Box optional error string since `Option<String>` is not FFI-safe
1 year ago
haixuanTao
0568656ef0
Remove unused dependencies for faster compile time
2 years ago
haixuanTao
afae2bdece
Bump to version 0.3.1
2 years ago
haixuanTao
4978f800a6
Fix clippy and rust warnings
2 years ago
haixuanTao
8648ce92ce
Format document
2 years ago
haixuanTao
1ad35fa817
Add `recv_async_timeout`
2 years ago
haixuanTao
0cd5281325
Use duration instead of f32 secs
2 years ago
haixuanTao
b1b69693b7
Adding a timeout method to not block indefinetely
2 years ago
haixuanTao
e45975f65a
Use cleaner `__from_elem` `avec` constructor
2 years ago
haixuanTao
fe065eec6f
replace const cache line with 128 to match arrow
2 years ago
haixuanTao
23d7f7a968
Use `CACHELINE_ALIGN` constant to align data into the right cache size corresponding to the architecture
2 years ago
haixuanTao
4cd9b4d71e
Fix `x86` pip release by type hinting integer
2 years ago
Philipp Oppermann
85338b61e2
Consider alignment when copying array into raw buffer
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
ac7d1740c3
Fix alignment errors when receiving empty arrays
Empty `Vec`s have a dummy pointer that is set to the alignment of the item type. Thus, the pointer for empty `Vec<u8>` data is `0x1`. Using this pointer to construct a `ArrayData` instance can lead to alignment errors if the arrow array is of a type with larger alignment, e.g. float64.
This commit fixes the alignment error by checking for an empty raw buffer and constructing an empty `ArrayData` instance in this case.
Fixes #362
2 years ago
Philipp Oppermann
c6ce45785f
Rework raw data to arrow array conversion
- If input has no data, return an empty `NullArray` instead of an empty byte array.
- If input contains Vec data (instead of shared memory data), use safe `arrow::buffer::Buffer::from_vec` constructor, which is also zero-copy.
2 years ago
Philipp Oppermann
8cc5837cfa
Update `arrow` and `pyo3` dependencies to latest versions
2 years ago
Philipp Oppermann
4acc3c6fd9
Fix name collision
2 years ago
Philipp Oppermann
94c0da61de
Provide higher-level function to send output from C operator
2 years ago
Philipp Oppermann
36a1f7781e
Provide accessor functions for input ID and data in C operator API
2 years ago
Philipp Oppermann
14a5531311
Fix doc test
2 years ago
Philipp Oppermann
6140681ab9
Rename and reexport `Data` enums
Makes the distinction between raw data and data messages clearer.
2 years ago
Philipp Oppermann
9a10280595
Simplify reading of primitive and string types from arrow arrays
- introduce a new `ArrayData` wrapper struct
- implement `TryFrom` to convert to primitive types and strings
- update examples to use the new conversion functions
2 years ago
Philipp Oppermann
89f980e753
Simplify arrow array construction through new `IntoArrow` trait
2 years ago
Philipp Oppermann
afae3b72ce
Merge branch 'main' into rust-typed-input
2 years ago
Philipp Oppermann
d3f9166c36
Make `safer-ffi` use `inventory v0.3`
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
Philipp Oppermann
7451135d34
Fix reconstruction of buffers and child data from received data slices
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago