haixuanTao
b5de93e5f5
Pretty Print Rust object when called from Python print
1 year ago
Michael-J-Ward
dfe298eb1e
lint: fix clippy lints
1 year ago
Philipp Oppermann
14bb4fbfb7
Merge pull request #429 from dora-rs/out_dir
Send runs artefacts into a dedicated `out` folder
1 year ago
haixuanTao
f4fbd6562f
Sending data recordings into an `out/<DATAFLOW_ID>/input,arrow` folder to make it easier to manage multiple data coming from multiple runs of a dataflow
1 year ago
haixuanTao
629a218ddb
`copy_array_into_sample` do not need to return a result
2 years ago
haixuanTao
0568656ef0
Remove unused dependencies for faster compile time
2 years ago
haixuanTao
8648ce92ce
Format document
2 years ago
haixuanTao
1ad35fa817
Add `recv_async_timeout`
2 years ago
haixuanTao
0cd5281325
Use duration instead of f32 secs
2 years ago
haixuanTao
b1b69693b7
Adding a timeout method to not block indefinetely
2 years ago
haixuanTao
e45975f65a
Use cleaner `__from_elem` `avec` constructor
2 years ago
haixuanTao
fe065eec6f
replace const cache line with 128 to match arrow
2 years ago
haixuanTao
23d7f7a968
Use `CACHELINE_ALIGN` constant to align data into the right cache size corresponding to the architecture
2 years ago
haixuanTao
4cd9b4d71e
Fix `x86` pip release by type hinting integer
2 years ago
Philipp Oppermann
85338b61e2
Consider alignment when copying array into raw buffer
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
ac7d1740c3
Fix alignment errors when receiving empty arrays
Empty `Vec`s have a dummy pointer that is set to the alignment of the item type. Thus, the pointer for empty `Vec<u8>` data is `0x1`. Using this pointer to construct a `ArrayData` instance can lead to alignment errors if the arrow array is of a type with larger alignment, e.g. float64.
This commit fixes the alignment error by checking for an empty raw buffer and constructing an empty `ArrayData` instance in this case.
Fixes #362
2 years ago
Philipp Oppermann
c6ce45785f
Rework raw data to arrow array conversion
- If input has no data, return an empty `NullArray` instead of an empty byte array.
- If input contains Vec data (instead of shared memory data), use safe `arrow::buffer::Buffer::from_vec` constructor, which is also zero-copy.
2 years ago
Philipp Oppermann
8cc5837cfa
Update `arrow` and `pyo3` dependencies to latest versions
2 years ago
Philipp Oppermann
14a5531311
Fix doc test
2 years ago
Philipp Oppermann
6140681ab9
Rename and reexport `Data` enums
Makes the distinction between raw data and data messages clearer.
2 years ago
Philipp Oppermann
9a10280595
Simplify reading of primitive and string types from arrow arrays
- introduce a new `ArrayData` wrapper struct
- implement `TryFrom` to convert to primitive types and strings
- update examples to use the new conversion functions
2 years ago
Philipp Oppermann
89f980e753
Simplify arrow array construction through new `IntoArrow` trait
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
7d96e03eed
Start to make Rust node API typed using arrow
2 years ago
Philipp Oppermann
7451135d34
Fix reconstruction of buffers and child data from received data slices
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago
Philipp Oppermann
b3e5bf90f5
Add new MergeExternalSend trait for usage in Python
The only difference betwenn the two traits is the additional Send bound. We should unify this again using the trait-level impl trait feature once it's stable.
2 years ago
Philipp Oppermann
ffaea07e31
Revert new Send requirement for 'merge_external'
2 years ago
Philipp Oppermann
dfa71c77c5
Merge branch 'main' into python-ros2-bridge-example
2 years ago
haixuanTao
b20ef2391d
Optimise sending of small vector for python
2 years ago
haixuanTao
0220d1ef65
Adding `data_type` to metadata for typing output
2 years ago
haixuanTao
6b910ab619
Adding `arrow-schema` as dependency
2 years ago
haixuanTao
eb515c7755
Removing lifetime to the Metadata struct
2 years ago
haixuanTao
19172523df
Update `pyo3` and `arrow` version to use latest version
2 years ago
Philipp Oppermann
7a21d965a9
Implement event stream merging for Python
2 years ago
Philipp Oppermann
e2899c382e
Merge pull request #311 from dora-rs/external-events
Add convenience functions for merging external event streams with `EventStream`
2 years ago
Haixuan Xavier Tao
53198b861b
Merge pull request #248 from dora-rs/shmem-read-only
Map shared memory region as read-only in receiver
2 years ago
Philipp Oppermann
8401d8654e
Merge pull request #310 from dora-rs/send-output-bytes
Add convenience function `send_output_bytes`
2 years ago
Philipp Oppermann
c4bd9d3edf
Add convenience functions for merging external event streams with `EventStream`
Our upcoming ros2<->dora bridge will allow nodes to subscribe to ROS2 topics, which results in a second stream of events. By providing a merge function, this ROS2 event stream can be merged together with the dora event stream.
2 years ago
Philipp Oppermann
d382761e2d
Add convenience function `send_output_bytes`
It's common to send a byte array that already exists, so it makes sense to provide a convenience function for this use case.
2 years ago
Philipp Oppermann
326c650f1f
Implement `Stream` for `EventStream`
2 years ago
Philipp Oppermann
979573689a
Switch to forked `shared_memory_extended` crate
The maintainer of the `shared_memory` crate did not react to our PR, so I decided to create a (temporary) fork.
2 years ago
Philipp Oppermann
f4498a4abf
Map shared memory region as read-only in receiver
This ensures that the data is not modified, even if shared with Python as zero-copy arrow array. Modifying the data could result in undefined behavior since there might be other subscribers that read the data at the same time.
2 years ago
Philipp Oppermann
4da34d82d9
Merge branch 'main' into event-timestamps
2 years ago
Philipp Oppermann
984ee364e4
Fill in timestamps when sending events from node to daemon
2 years ago
Philipp Oppermann
e0559b031e
Add HLC clock to daemon and to all event types
2 years ago
Philipp Oppermann
482b282d92
Document `dataflow_descriptor` methods
2 years ago
Philipp Oppermann
0bf31b750e
WIP: Add `uhlc` timestamps to all events and update HLC clocks
2 years ago
Philipp Oppermann
8a439c3411
Pass dataflow descriptor to Rust nodes via API
2 years ago