haixuanTao
4ca53258ad
Move opentelemetry implementation to `dora metrics`
2 years ago
haixuanTao
0568656ef0
Remove unused dependencies for faster compile time
2 years ago
haixuanTao
4ec81cc743
Removing old main files
2 years ago
haixuanTao
645c56d7a4
Bump to opentelemetry system metrics 0.1.6 that measure GPU memory usage
2 years ago
haixuanTao
32831cae75
Bump opentelemetry metrics to 0.21
2 years ago
Philipp Oppermann
ad3f0482ac
Avoid alignment errors by aligning raw data on deseralization
Ensures that all raw data has at least an alignment of 128.
2 years ago
Philipp Oppermann
8cc5837cfa
Update `arrow` and `pyo3` dependencies to latest versions
2 years ago
Philipp Oppermann
6154415cbc
Update Rust APIs and rust-dataflow-example to use arrow types
2 years ago
Philipp Oppermann
0a9348f7f0
Only enable `arrow/pyarrow` feature when `python` feature is enabled
We don't want to link to pythonlib when compiling the dora-daemon.
2 years ago
Philipp Oppermann
19f9cc7c90
Add support for arbitrary Arrow types in Python API
2 years ago
haixuanTao
6b910ab619
Adding `arrow-schema` as dependency
2 years ago
haixuanTao
19172523df
Update `pyo3` and `arrow` version to use latest version
2 years ago
haixuanTao
23651cd002
Make `pythonize` optional in runtime as it link to libpython
2 years ago
Philipp Oppermann
049e5e1e43
Make runtime pass dataflow descriptor to Python operators
2 years ago
haixuanTao
fc0e12fb3b
Remove `api-python` for isolating daemon from python
3 years ago
haixuanTao
93b7da2b74
Remove pyo3 in runtime and daemon as it generates `libpython` dependencies
Previous implementation of the runtime would link `libpython` even though it
might not be required.
This commit put pyo3 dependency under a feature flag that is only active if
the runtime is called from python for a python operator.
This fix:
- https://github.com/dora-rs/dora/actions/runs/4879188973/jobs/8705515472
- https://github.com/dora-rs/dora/issues/280#issuecomment-1534058207
- and https://github.com/dora-rs/dora-drives/issues/58#event-9162532660
3 years ago
Philipp Oppermann
4fb519dbc1
Update `pyo3` to `v0.18`
3 years ago
haixuanTao
d3b612faf2
Use opentelemetry if env variable `DORA_JAEGER_TRACING` is set
3 years ago
haixuanTao
ff2006f77b
Rewrite `Context` flowing using `tracing` instead of opentelemetry
3 years ago
haixuanTao
1e965452d5
Replace opentelemetry with tokio::tracing
3 years ago
haixuanTao
a61e53f151
Add version print in runtime
3 years ago
Philipp Oppermann
9db09ee930
Enable tracing for dora runtime by default
3 years ago
haixuanTao
7a7a20a298
Add missing licenses and centralise license definition
test new crates version release
3 years ago
haixuanTao
101ddd2092
add documentation and description to all crates
test rc 2 version
fix version typo
Test crates.io new dora tag release
3 years ago
Philipp Oppermann
152076ad87
Use separate `telemetry` feature in `dora-runtime`
The `tracing` feature just enables the tracing subscriber, the `telemetry` feature the `opentelemetry` integration.
3 years ago
haixuanTao
f1cf0864a4
Remove static path from workspace dependency
Removing those path make it easier to move package, reduce path management
reduce complexity and make it easier to export crates to `crates.io`
3 years ago
Philipp Oppermann
a9203e0274
Add intermediate operator channel buffer to avoid blocking runtime
3 years ago
Philipp Oppermann
1e07a73a64
Merge branch 'main' into unify-nodes-and-operators
3 years ago
Philipp Oppermann
1553a198ef
Re-export dora-message from dora-core
3 years ago
Philipp Oppermann
ea2c39a26c
Remove unused runtime dependencies
3 years ago
Philipp Oppermann
6980636686
Update futures-concurrency to v7.1
3 years ago
dependabot[bot]
b74a524031
Bump tokio from 1.23.1 to 1.24.2
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.23.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
3 years ago
dependabot[bot]
ae161e644c
Bump tokio from 1.21.2 to 1.23.1
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.21.2 to 1.23.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.23.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
3 years ago
Philipp Oppermann
df5dcfb342
Merge branch 'main' into unify-nodes-and-operators
3 years ago
Philipp Oppermann
433b7b4ed7
Merge branch 'main' into unify-nodes-and-operators
3 years ago
Philipp Oppermann
ce4fd8b027
Set dora subcrate dependencies as workspace dependencies
3 years ago
Philipp Oppermann
d8260d5d2c
Inherit package version from workspace root
This way, we don't need to update the version manually in all sub-crates on every release.
3 years ago
haixuanTao
9a0b303e98
Replace all versions to `v0.1.1`
3 years ago
Philipp Oppermann
1d2d71b15a
Comment out uses of communication layer in coordinator for now
3 years ago
haixuanTao
992b8251c2
DRAFT: Fixing Python linking error
This commit is an initial draft at fixing #147 . The error is due to the
fact that pyo3 has linked the libpython from the compilation and not
trying to use libpython that is available in `LD_LIBRARY_PATH`.
The current only solution to disable the linking is to use the `extension-module` flag.
This requires to make the python `runtime-node` packaged in a python library.
The python `runtime-node` should also be fully compatible with the other operators in case we want hybrid runtime.
The issue that i'm facing is that. Due to the packaging, I have to deal with the `GIL` that is present from the start of
`dora-runtime` node. This however makes the process single threaded wich is impossible.
So, I have to disable it, but when I do, I have a race condition:
```bash
Exception ignored in: <module 'threading' from '/usr/lib/python3.8/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 1373, in _shutdown
assert tlock.locked()
AssertionError:
```
The issue is the same as https://github.com/PyO3/pyo3/issues/1274
To fix this issue, I'm going to look again at the different step to make this work.
But this is my current investigation.
3 years ago
Xavier Tao
2d1ac25220
Add getting started section in `README` ( #131 )
* Add getting started section in `README`
* Update README.md
Co-authored-by: Philipp Oppermann <dev@phil-opp.com>
* Update README.md
Co-authored-by: Philipp Oppermann <dev@phil-opp.com>
* Update README.md
Co-authored-by: Philipp Oppermann <dev@phil-opp.com>
Co-authored-by: Philipp Oppermann <dev@phil-opp.com>
3 years ago
haixuanTao
ffe686a10d
Pin zenoh git commit
Pinning zenoh git version to fix recompilation of the `dora-api-operator` when it
is a user dependency.
3 years ago
Philipp Oppermann
64cf49703b
Implement download support for custom nodes
3 years ago
Philipp Oppermann
a37d3ee6c8
Don't parse operator source always as URI
Parsing as URI does not work for relative paths, e.g. `../../foo`.
3 years ago
Philipp Oppermann
13d3a91683
Implement support for downloading operator sources
Makes it possible to set an URL as operator source. The `dora-runtime` will then try to download the operator from the given URL when the dataflow is started.
3 years ago
Xavier Tao
44a7c16c21
Add Python Node API metadata ( #101 )
* Add a `dora-python-operator` crate to hold utils functions for dora python
* Remove python serialisation and deserialisation from `dora-runtime`
* Update `python` documentation
3 years ago
haixuanTao
f98859bfc4
Adding `metrics` feature to `dora-runtime`
3 years ago
Xavier Tao
c2bb4c1f67
Runtime tracing ( #95 )
* add opentelemetry feature to python op
* modify `python-dataflow` to test opentelemetry
* change `timer` topic to `tick`
* adding tracing to `shared_libraries`
* example install refactoring
* Rename `input` to `dora_input` for python ex
3 years ago
Philipp Oppermann
7e9c3050f9
Use `tracing` for logging state of nodes, coordinator, and runtimes
3 years ago
Philipp Oppermann
e0cc48a557
Move `dora-message` crate directly under `libraries` (it's not really an extension)
3 years ago