The working directory of nodes with git fields is only decided on `dora build` because that's where we translate the branch/tag name into the commit hash.
Before, we fell back to using the base working directory if no build was done. However, this is confusing since the reason for the different working directory is not communicated in any.
This commit throws an error if no build happened before starting a dataflow with git nodes.
Most important change is https://github.com/GREsau/schemars/pull/310,
which keeps newlines unchanged during schema generation. This is
important to preserve the lists, markdown code blocks, and markdown
headings in our docs.
Note: This PR deliberately does not update the schema file yet in order
to test #1083. So it should be merged after #1083.
Most important change is https://github.com/GREsau/schemars/pull/310, which keeps newlines unchanged during schema generation. This is important to preserve the lists, markdown code blocks, and markdown headings in our docs.
The dataflow descriptor format still changes often, which led to parse errors. By doing the parsing lazily, this should only affect users of the `dataflow_descriptor` function from now on.
The coordinator now sends an immediate `DataflowStartTriggered` reply when receiving a `DataflowStart` command. This enables the CLI to directly attach to the dataflow and observe the build output.
To wait until the build/spawning is done, this commit introduces a new `WaitForSpawn` command, to which the coordinator replies with a `DataflowSpawned` message once the node has been started. We use this command for the `dora build` command.
Instead of running all the build commands directly, run them on their intended target machines through the coordinator.
This commit is a breaking change because a coordinator connection is now required for `dora build`.
This allows using up and start with uv:
## How to use:
```bash
cd examples/python-dataflow
dora up
dora start dataflow.yaml --uv
```
Fixes: https://github.com/dora-rs/dora/issues/780
This would help the CI as well
We use `/` as separator for input mappings in dataflow YAML files (e.g. input: `node/output_id`).
We do support `/` characters in output IDs for runtime nodes. If we allowed `/` characters in node IDs too, splitting a `a/b/c` mapping would no longer be possible (could be node `a` with output `b/c` or node `a/b` with output `c`). Therefore we explicitly error on node IDs containing `/` characters in this commit.