Philipp Oppermann
7232a31b57
Send `InputClosed` events to operators too
2 years ago
Philipp Oppermann
9e4f01318a
Fix: Move `InputBuffer` break condition to end of loop
Fixes a deadlock issue when the `incoming` channel is closed first. The problem was that the `send_out_buf` was set to `Fuse::terminated` without breaking the loop, so on the next loop iteration both futures were already terminated.
2 years ago
Philipp Oppermann
598ece765e
Don't panic in runtime when operator panics
2 years ago
Philipp Oppermann
ea537d45f6
Remove resolved TODO
2 years ago
Philipp Oppermann
a84f91c572
Fix: Don't keep on polling incoming event channel after it's closed
This causes the `send_out_buf` future to completely starve.
2 years ago
Philipp Oppermann
a35850cb07
Drop oldest operator inputs when queue becomes too full
2 years ago
Philipp Oppermann
0f29d26fe8
Drop oldest operator inputs when queue becomes too full
2 years ago
Philipp Oppermann
a9203e0274
Add intermediate operator channel buffer to avoid blocking runtime
2 years ago
Philipp Oppermann
de786ef101
Fix some warnings in runtime
2 years ago
Philipp Oppermann
cdc0be1206
Rename `FfiEvent` to `RawEvent`
2 years ago
Philipp Oppermann
62319f1ec1
Add support for Rust operators again
2 years ago
Philipp Oppermann
d334c6f243
Make `daemon` module of Rust node API private and reexport symbols
2 years ago
Philipp Oppermann
83c8ecf3e9
Change Python operator API: `on_event` instead of `on_input`
We now report other event types as well. Right now, the only other event type is `Stop`, but we can extend this in the future.
2 years ago
Philipp Oppermann
1e07a73a64
Merge branch 'main' into unify-nodes-and-operators
2 years ago
Philipp Oppermann
1553a198ef
Re-export dora-message from dora-core
2 years ago
Philipp Oppermann
8dc184a7e8
Improve a log message
2 years ago
Philipp Oppermann
5e4f1b2519
Clean up
2 years ago
Philipp Oppermann
ea2c39a26c
Remove unused runtime dependencies
2 years ago
Philipp Oppermann
67976d2c41
Add some Debug derives
2 years ago
Philipp Oppermann
18a908c579
Close operator event channel after last input is closed
3 years ago
Philipp Oppermann
c27d7d1a34
Propagate panics of runtime main task
3 years ago
Philipp Oppermann
6980636686
Update futures-concurrency to v7.1
3 years ago
Philipp Oppermann
0d1bbc8eb5
Fix: actually send input to runtime instead of dropping future
3 years ago
dependabot[bot]
b74a524031
Bump tokio from 1.23.1 to 1.24.2
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.23.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
3 years ago
Philipp Oppermann
ea136adc2e
Fix: Run Python operator in main thread
3 years ago
Philipp Oppermann
2e67362c15
Fix: Python operator spawning is blocking, so do it in a separate thread
3 years ago
Philipp Oppermann
772fc41d93
Slight improvements to log messages
3 years ago
Philipp Oppermann
cac527a382
Add support for spawning runtime nodes with operators
3 years ago
Philipp Oppermann
27d74a7a71
Resolve some todos
3 years ago
Philipp Oppermann
4bd235afd7
Close event channels after sending Stop
3 years ago
Philipp Oppermann
00b863fdc9
Fix some warnings and remove some unused code
3 years ago
Philipp Oppermann
6eb9b46637
Close outputs of operators when they stop
We cannot wait until all operators of the runtime node are finished because operators might be subscribed to other operators.
3 years ago
Philipp Oppermann
d6470e75a3
Don't use blocking output send function in runtime loop
3 years ago
Philipp Oppermann
492339a687
WIP: Start porting `dora-runtime` to `dora-daemon`
3 years ago
haixuanTao
a841577e10
add documentation of scoped unsage `GILPool`
3 years ago
haixuanTao
d45f9373fd
Use `py.new_pool()` to bound pyo3 variable
To alievate the unbounded memory growth, we're replacing variable dereferencing
with scoped `GILPool` as described in the pyo3 documentation recently updated. See:
https://github.com/PyO3/pyo3/pull/2864
3 years ago
haixuanTao
71078bd9f2
Force removal of Pyo3 Object to avoid memory leak
This commit fix the memory leak happening in the dora API operator. This
is seemingly due to pyo3 leaking memory on object created in Rust.
Using standard `drop` did not drop the memory on the `PyBytes` included in
the `PyDict`.
See: https://github.com/PyO3/pyo3/issues/1801
Fixes https://github.com/dora-rs/dora/issues/163
3 years ago
dependabot[bot]
ae161e644c
Bump tokio from 1.21.2 to 1.23.1
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.21.2 to 1.23.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.23.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
3 years ago
Philipp Oppermann
df5dcfb342
Merge branch 'main' into unify-nodes-and-operators
3 years ago
Philipp Oppermann
433b7b4ed7
Merge branch 'main' into unify-nodes-and-operators
3 years ago
Philipp Oppermann
ce4fd8b027
Set dora subcrate dependencies as workspace dependencies
3 years ago
Philipp Oppermann
d8260d5d2c
Inherit package version from workspace root
This way, we don't need to update the version manually in all sub-crates on every release.
3 years ago
haixuanTao
b15d25b9a3
Fix clippy warnings
3 years ago
haixuanTao
9a0b303e98
Replace all versions to `v0.1.1`
3 years ago
haixuanTao
32e11f538c
Remove `stop_thread` loop after a `StopAll`
3 years ago
haixuanTao
294990e307
Make `stop_thread` not borrow `communication` for static
3 years ago
Philipp Oppermann
1d2d71b15a
Comment out uses of communication layer in coordinator for now
3 years ago
haixuanTao
9ec9d22620
wait for stop thread to finish before stopping runtime
3 years ago
haixuanTao
9f0a0e6957
Fix GIL race condition
By making the stopping loop the first loop, we can avoid using `pyo3/allow_threads`, which
seems buggy.
3 years ago
haixuanTao
992b8251c2
DRAFT: Fixing Python linking error
This commit is an initial draft at fixing #147 . The error is due to the
fact that pyo3 has linked the libpython from the compilation and not
trying to use libpython that is available in `LD_LIBRARY_PATH`.
The current only solution to disable the linking is to use the `extension-module` flag.
This requires to make the python `runtime-node` packaged in a python library.
The python `runtime-node` should also be fully compatible with the other operators in case we want hybrid runtime.
The issue that i'm facing is that. Due to the packaging, I have to deal with the `GIL` that is present from the start of
`dora-runtime` node. This however makes the process single threaded wich is impossible.
So, I have to disable it, but when I do, I have a race condition:
```bash
Exception ignored in: <module 'threading' from '/usr/lib/python3.8/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 1373, in _shutdown
assert tlock.locked()
AssertionError:
```
The issue is the same as https://github.com/PyO3/pyo3/issues/1274
To fix this issue, I'm going to look again at the different step to make this work.
But this is my current investigation.
3 years ago