Pip installations of dora use a small Python wrapper script, which then calls a functions of the `dora_cli` module. This means that the `current_exe()` will be a python binary instead of a standalone `dora` binary. This leads to errors when trying to start a `dora runtime` instance for shared library operators (see #900).
This commit fixes this issue by introducing a new `DoraCommand` struct that is initialized in the respective main function. For normal Rust binaries, this is just initialized with `current_exe()` as before. When invoked from Python, we also include the first argument, which should be the path to the Python wrapper script invoking dora. We also use the 0th argument instead of `current_exe` for invoking the `python` binary because `current_exe` resolves symlinks on some platforms, which affects import statements (different base directory).
Fixes#900
This PR makes it possible to run dataflow from python using:
```python
from dora import run
# Make sure to build it first with the CLI.
# Build step is on the todo list.
run("qwen2.5.yaml", uv=True)
```
Making it easier to run dataflow based on our needs
When dropping the `DoraNode`, it waits for the remaining drop tokens. This only works if all the dora events were already dropped before. With the Python GC, this is not guaranteed as some events might still be live on the heap (the user might even use them later). In such cases, we waited until we ran into a timeout, which resulted in very long exit times (see https://github.com/dora-rs/dora/issues/598).
This commit fixes this issue by adding a reference-counted copy of the `DoraNode` and `EventStream` to every event given to Python. This way, we can ensure that the underlying `DoraNode` is only dropped after the last event reference has been freed.
Currently having a custom PyEvent make debugging very hard as fields are hidden within the class PyEvent that is defined within Rust Code.
Python user are getting really confused about this obscure class.
This PR transforms the class into a standard python dictionary.