When dropping the `DoraNode`, it waits for the remaining drop tokens. This only works if all the dora events were already dropped before. With the Python GC, this is not guaranteed as some events might still be live on the heap (the user might even use them later). In such cases, we waited until we ran into a timeout, which resulted in very long exit times (see https://github.com/dora-rs/dora/issues/598).
This commit fixes this issue by adding a reference-counted copy of the `DoraNode` and `EventStream` to every event given to Python. This way, we can ensure that the underlying `DoraNode` is only dropped after the last event reference has been freed.
Currently having a custom PyEvent make debugging very hard as fields are hidden within the class PyEvent that is defined within Rust Code.
Python user are getting really confused about this obscure class.
This PR transforms the class into a standard python dictionary.
The deprecation warning states:
> code not using the `GIL Refs` API can safely remove the use of `Py::new_pool`
All GIL Refs usage have been removed, so this should be fine.
Buffer capacity is always 0 if the buffer is allocated externally. Also, we don't want to allocate unneccessary extra space if only parts of a buffer a used (e.g. when slicing).
In #229, I renamed `eyre::eyre!("{err}{traceback}")` where it should have `eyre::eyre!("{traceback}\n{err}")` . This PR fix this issue.
This Pull Request also add the name of the node process at the end of the process call to help with debugging.
Instead of doing an additional copy to send them from the operator thread to the runtime thread.
This commit uses the new `allocate_data_sample` and `send_output_sample` methods introduced in d7cd370.
This commit adds the ability to use the `opentelemetry` feature flag at
runtime by using the env variable `DORA_TELEMETRY`. Using environment
variable reduce the need to do some argument drilling between our
different process as well as making configurable in the YAML at no cost.
Usage:
```bash
docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest
DORA_TELEMETRY=true dora-daemon --run-dataflow dataflow.yml
```
Current limitation:
- The telemetry currently only work with Jaeger on the local network with
default port: 6831 and 6832.
After adding #236 , error of initialization is shadowed by bailing on
the sender channel.
Bailing used to happen here: ccec196234/binaries/runtime/src/lib.rs (L136)
This makes it hard to debug.
This PR will push the error into the sender channel, and report any error
of initialization making it easier to debug.
Now the channel is used without error, but any error of initialization
will bail the runtime here: ccec196234/binaries/runtime/src/lib.rs (L137)