haixuanTao
3e5a3d7d91
Fix NON_INPUT_EVENT
1 year ago
haixuanTao
acf4d0cb44
Fix typo
1 year ago
haixuanTao
aefb8f02ab
Rewrite the scheduler to avoid overhead
1 year ago
haixuanTao
90752a0bb7
Make the non input event a priority when collecting next event
1 year ago
haixuanTao
f77f400da2
Add a scheduler to the node api to manage fairness among inputs
1 year ago
haixuanTao
411436d3bb
Replace drop token mecanism on closed channel by leaving drop token active on the receiver side and closing sender side shared memory with a lower timeout
1 year ago
haixuanTao
a2d651446b
Dropping remaining drop token on exit so that the origin node does not get stuck
1 year ago
haixuanTao
45a7ee9437
Reduce `eventstream` timeout as it is linked to Python cleanup race condition that is cleaning up node first and then python arrow reference.
1 year ago
haixuanTao
7487ce17c3
Fix clippy warning
1 year ago
haixuanTao
5ccdbc593e
Moving queue size and making node flume queue bigger
1 year ago
Philipp Oppermann
49aef8bac6
Move more implementation code out of `dora-message` into `dora-core`
Ensures that we need to update `dora-message` less often.
1 year ago
haixuanTao
60496e62ea
Better error linting for unknown output
1 year ago
Michael-J-Ward
96c6ba0f1d
fix simple lintst in api
1 year ago
Philipp Oppermann
6a3dcecf33
Refactor: Move message definitions to `dora-message` crate
First step towards versioning the message definitions indepedently.
1 year ago
haixuanTao
bc68de3bbd
Change `MetadataParameters` into a `BTreeMap` to allow user defined metadata as well as enable more flexibility in managing metadata
1 year ago
Haixuan Xavier Tao
16ef29ab80
Merge pull request #601 from dora-rs/delay-node-drop
Delay dropping of `DoraNode` in Python until all event data is freed
1 year ago
Philipp Oppermann
9cec8df1cc
Delay dropping of `DoraNode` in Python until all event data is freed
When dropping the `DoraNode`, it waits for the remaining drop tokens. This only works if all the dora events were already dropped before. With the Python GC, this is not guaranteed as some events might still be live on the heap (the user might even use them later). In such cases, we waited until we ran into a timeout, which resulted in very long exit times (see https://github.com/dora-rs/dora/issues/598 ).
This commit fixes this issue by adding a reference-counted copy of the `DoraNode` and `EventStream` to every event given to Python. This way, we can ensure that the underlying `DoraNode` is only dropped after the last event reference has been freed.
1 year ago
chang xu
bccb1ae27d
Add domain unix socket supports ( #594 )
1 year ago
haixuanTao
329844358a
Reduce timeout time and increase grace duration so that node exit gracefully within one second
1 year ago
haixuanTao
5b7a614c14
Reduce drop token time from 30s to 1s as most of the time the drop token is having a race condition with the GIL in python.
1 year ago
haixuanTao
7d3a5de1c5
Make rust flexible logic and make sure to always use env variables DORA_NODE_CONFIG if it exists.
This remove the complexity of having to rename the node id when used within a yaml configuration.
1 year ago
haixuanTao
3f2f274036
Remove unnecessary condition on dynamic node
2 years ago
haixuanTao
a0a95b730c
Rename dynamic node listener -> local listener
2 years ago
haixuanTao
3ad402ce45
Add dynamic node event loop and dynamic node connection
2 years ago
haixuanTao
b5de93e5f5
Pretty Print Rust object when called from Python print
2 years ago
Michael-J-Ward
dfe298eb1e
lint: fix clippy lints
2 years ago
Philipp Oppermann
14bb4fbfb7
Merge pull request #429 from dora-rs/out_dir
Send runs artefacts into a dedicated `out` folder
2 years ago
haixuanTao
f4fbd6562f
Sending data recordings into an `out/<DATAFLOW_ID>/input,arrow` folder to make it easier to manage multiple data coming from multiple runs of a dataflow
2 years ago
haixuanTao
629a218ddb
`copy_array_into_sample` do not need to return a result
2 years ago
haixuanTao
8648ce92ce
Format document
2 years ago
haixuanTao
1ad35fa817
Add `recv_async_timeout`
2 years ago
haixuanTao
0cd5281325
Use duration instead of f32 secs
2 years ago
haixuanTao
b1b69693b7
Adding a timeout method to not block indefinetely
2 years ago
haixuanTao
e45975f65a
Use cleaner `__from_elem` `avec` constructor
2 years ago
haixuanTao
fe065eec6f
replace const cache line with 128 to match arrow
2 years ago
haixuanTao
23d7f7a968
Use `CACHELINE_ALIGN` constant to align data into the right cache size corresponding to the architecture
2 years ago
haixuanTao
4cd9b4d71e
Fix `x86` pip release by type hinting integer
2 years ago
Philipp Oppermann
85338b61e2
Consider alignment when copying array into raw buffer
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
ac7d1740c3
Fix alignment errors when receiving empty arrays
Empty `Vec`s have a dummy pointer that is set to the alignment of the item type. Thus, the pointer for empty `Vec<u8>` data is `0x1`. Using this pointer to construct a `ArrayData` instance can lead to alignment errors if the arrow array is of a type with larger alignment, e.g. float64.
This commit fixes the alignment error by checking for an empty raw buffer and constructing an empty `ArrayData` instance in this case.
Fixes #362
2 years ago
Philipp Oppermann
c6ce45785f
Rework raw data to arrow array conversion
- If input has no data, return an empty `NullArray` instead of an empty byte array.
- If input contains Vec data (instead of shared memory data), use safe `arrow::buffer::Buffer::from_vec` constructor, which is also zero-copy.
2 years ago
Philipp Oppermann
14a5531311
Fix doc test
2 years ago
Philipp Oppermann
6140681ab9
Rename and reexport `Data` enums
Makes the distinction between raw data and data messages clearer.
2 years ago
Philipp Oppermann
9a10280595
Simplify reading of primitive and string types from arrow arrays
- introduce a new `ArrayData` wrapper struct
- implement `TryFrom` to convert to primitive types and strings
- update examples to use the new conversion functions
2 years ago
Philipp Oppermann
89f980e753
Simplify arrow array construction through new `IntoArrow` trait
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
Philipp Oppermann
7451135d34
Fix reconstruction of buffers and child data from received data slices
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago
Philipp Oppermann
b3e5bf90f5
Add new MergeExternalSend trait for usage in Python
The only difference betwenn the two traits is the additional Send bound. We should unify this again using the trait-level impl trait feature once it's stable.
2 years ago