haixuanTao
9e495c5dfd
Disable pylint fixme
1 year ago
haixuanTao
edc1467612
Add next call loop in qwenvl2 to queue fix
1 year ago
Haixuan Xavier Tao
a254d7007c
Enable multiline build for better packaging of dora node. ( #683 )
It is necessary to make build multiline in order to build certain node.
This allow users to do the following:
```bash
nodes:
- id: dora-microphone
build: |
pip install -e ../../node-hub/dora-microphones
pip install -e ../../node-hub/dora-vad
path: dora-microphone
```
or
```bash
nodes:
- id: dora-microphone
build: |
sudo apt install abc
pip install bar
path: dora-microphone
```
1 year ago
Haixuan Xavier Tao
903cb239d2
Enable downloading remote dataflow ( #682 )
This PR makes it possible to start a dataflow from scratch without
having to git clone a repository.
## Example
```bash
dora up
dora build 09f7f2927d/camera.yaml
dora start 09f7f2927d/camera.yaml
```
## Video
[Screencast from 2024-10-09
12-11-57.webm](https://github.com/user-attachments/assets/041ca5a0-7cd2-4dc6-bc34-aa94e6010166 )
## Small breaking change
- This slighlty change the logic of `download_file` function to use the
predefined downloaded file name instead of changing it when downloading.
- It removes the caching of `download_file` as there is now no way to
know if the filename and path is going to be the same before hand. I
think that if users want to have caching, they should link the path
before hand instead of using the url. This can be done in conjunction of
a build `wget` or `curl` command.
1 year ago
Haixuan Xavier Tao
2d8b9ee507
Add jpeg format to qwenvl2 ( #684 )
Small PR to enable jpeg encoded image communication when using qwenvl2.
1 year ago
LyonRust
4dcf2cf572
Fix small bug for jpeg encoding
1 year ago
Haixuan Xavier Tao
e2b469b82b
Add package flake ( #685 )
This means that someone using nix can add this to their flake inputs:
```nix
dora-rs = {
url = "github:dora-rs/dora";
inputs.nixpkgs.follows = "nixpkgs";
};
```
and that allows them to add dora-cli, v0.3.6, as one of the declared
packages of the system.
This is the first step towards adding dora to the nix package repo.
1 year ago
haixuanTao
254e74d4ac
Skip image that cannot be encoded
1 year ago
Ben-PH
2ae712ab9d
Add package flake
1 year ago
haixuanTao
ce408a1282
Adding opencv dependency to qwenvl2
1 year ago
haixuanTao
bf06c5c92f
Add a multiline build example
1 year ago
haixuanTao
c30d9cda91
Enable multi-line build for more explicit packaging
1 year ago
haixuanTao
b494a47d85
use `download_file` when calling build or start for a url dataflow
1 year ago
haixuanTao
3775a3c08b
Make downloading file use `content_dosposition` header or the filename within the url to get the path avoiding to create confusion on the name of the file.
1 year ago
Philipp Oppermann
3d6360ddab
Fix dora list listing twice a stopping dataflow when using multiple daemon. ( #668 )
Previously, When a daemon stop it sends a message to the coordinator
which will log it as an archived dataflow even though not every daemon
has stopped within the dataflow.
This PR should fix this issue.
1 year ago
haixuanTao
07ed728a45
Remove dataflow from appearing twice within `dora list` when stopping
1 year ago
Haixuan Xavier Tao
8e4264e169
Small example improvement using pyarrow assertion ( #669 )
This PR makes it possible to test node using `pyarrow-sender` and
`pyarrow-assert` in order to check the validity of input and output.
1 year ago
haixuanTao
5c0c8c5c3e
Small example improvement using pyarrow assertion
1 year ago
Haixuan Xavier Tao
54d1bc52be
Fix hanged coordinator when failing to connect to the daemon on destroy command ( #664 )
This PR fixes one error that happen when the coordinator fail to connect
to the daemon and ultimately hangs indefinetely.
## To reproduce previous behaviour:
```bash
dora up
kill <DAEMON PID> && dora destroy
dora destroy # <- This hangs
```
Fix: https://github.com/dora-rs/dora/issues/253
1 year ago
Philipp Oppermann
91a245d5e4
Update Rust crate hyper to v0.14.30 ( #680 )
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [hyper](https://hyper.rs )
([source](https://redirect.github.com/hyperium/hyper )) | dependencies |
patch | `0.14.29` -> `0.14.30` |
---
### Release Notes
<details>
<summary>hyperium/hyper (hyper)</summary>
###
[`v0.14.30`](https://redirect.github.com/hyperium/hyper/releases/tag/v0.14.30 )
[Compare
Source](https://redirect.github.com/hyperium/hyper/compare/v0.14.29...v0.14.30 )
#### Bug Fixes
- **http1:** reject final chunked if missing 0
([4a51b2af](4a51b2afef ))
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on monday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/ ).
View the [repository job
log](https://developer.mend.io/github/dora-rs/dora ).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC45Ny4wIiwidXBkYXRlZEluVmVyIjoiMzguOTcuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->
1 year ago
renovate[bot]
ee419203b1
Update dependencies
1 year ago
Philipp Oppermann
9ef3265611
Update dependencies ( #679 )
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [arrow](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [arrow-array](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [arrow-data](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [arrow-schema](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [clap](https://redirect.github.com/clap-rs/clap ) | dependencies |
patch | `4.5.18` -> `4.5.19` |
| [futures](https://rust-lang.github.io/futures-rs )
([source](https://redirect.github.com/rust-lang/futures-rs )) |
dependencies | patch | `0.3.30` -> `0.3.31` |
| [futures](https://rust-lang.github.io/futures-rs )
([source](https://redirect.github.com/rust-lang/futures-rs )) |
dev-dependencies | patch | `0.3.30` -> `0.3.31` |
| [once_cell](https://redirect.github.com/matklad/once_cell ) |
dependencies | minor | `1.19.0` -> `1.20.2` |
| [parquet](https://redirect.github.com/apache/arrow-rs ) | dependencies
| minor | `53.0.0` -> `53.1.0` |
| [reqwest](https://redirect.github.com/seanmonstar/reqwest ) |
dependencies | patch | `0.12.7` -> `0.12.8` |
---
### Release Notes
<details>
<summary>apache/arrow-rs (arrow)</summary>
###
[`v53.1.0`](https://redirect.github.com/apache/arrow-rs/blob/HEAD/CHANGELOG.md#5310-2024-10-02 )
[Compare
Source](https://redirect.github.com/apache/arrow-rs/compare/53.0.0...53.1.0 )
[Full
Changelog](https://redirect.github.com/apache/arrow-rs/compare/53.0.0...53.1.0 )
**Implemented enhancements:**
- Write null counts in Parquet statistics when they are known to be zero
[#​6502](https://redirect.github.com/apache/arrow-rs/issues/6502 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Make it easier to find / work with `ByteView`
[#​6478](https://redirect.github.com/apache/arrow-rs/issues/6478 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Update lexical-core version due to soundness issues with current
version
[#​6468](https://redirect.github.com/apache/arrow-rs/issues/6468 )
- Add builder style API for manipulating `ParquetMetaData`
[#​6465](https://redirect.github.com/apache/arrow-rs/issues/6465 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- `ArrayData.align_buffers` should support `Struct` data type / child
data
[#​6461](https://redirect.github.com/apache/arrow-rs/issues/6461 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add a method to return the number of skipped rows in a `RowSelection`
[#​6428](https://redirect.github.com/apache/arrow-rs/issues/6428 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Bump lexical-core to 1.0
[#​6397](https://redirect.github.com/apache/arrow-rs/issues/6397 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add union_extract kernel
[#​6386](https://redirect.github.com/apache/arrow-rs/issues/6386 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- implement `regexp_is_match_utf8` and `regexp_is_match_utf8_scalar` for
`StringViewArray`
[#​6370](https://redirect.github.com/apache/arrow-rs/issues/6370 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add support for BinaryView in arrow_string::length
[#​6358](https://redirect.github.com/apache/arrow-rs/issues/6358 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add `as_union` to `AsArray`
[#​6351](https://redirect.github.com/apache/arrow-rs/issues/6351 )
- Ability to append non contiguous strings to `StringBuilder`
[#​6347](https://redirect.github.com/apache/arrow-rs/issues/6347 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add Catalog DB Schema subcommands to `flight_sql_client`
[#​6331](https://redirect.github.com/apache/arrow-rs/issues/6331 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- Add support for Utf8View in arrow_string::length
[#​6305](https://redirect.github.com/apache/arrow-rs/issues/6305 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Reading FIXED_LEN_BYTE_ARRAY columns with nulls is inefficient
[#​6296](https://redirect.github.com/apache/arrow-rs/issues/6296 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Optionally verify 32-bit CRC checksum when decoding parquet pages
[#​6289](https://redirect.github.com/apache/arrow-rs/issues/6289 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Speed up `pad_nulls` for `FixedLenByteArrayBuffer`
[#​6297](https://redirect.github.com/apache/arrow-rs/pull/6297 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Improve performance of set_bits by avoiding to set individual bits
[#​6288](https://redirect.github.com/apache/arrow-rs/pull/6288 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([kazuyukitanimura](https://redirect.github.com/kazuyukitanimura ))
**Fixed bugs:**
- BitIterator panics when retrieving length
[#​6480](https://redirect.github.com/apache/arrow-rs/issues/6480 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Flight data retrieved via Python client (wrapping C++) cannot be used
by Rust Arrow
[#​6471](https://redirect.github.com/apache/arrow-rs/issues/6471 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- CI integration test failing: Archery test With other arrows
[#​6448](https://redirect.github.com/apache/arrow-rs/issues/6448 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- IPC not respecting not preserving dict ID
[#​6443](https://redirect.github.com/apache/arrow-rs/issues/6443 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- Failing CI: Prost requires Rust 1.71.1
[#​6436](https://redirect.github.com/apache/arrow-rs/issues/6436 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- Invalid struct arrays in IPC data causes panic during read
[#​6416](https://redirect.github.com/apache/arrow-rs/issues/6416 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- REE Dicts cannot be encoded/decoded with streaming IPC
[#​6398](https://redirect.github.com/apache/arrow-rs/issues/6398 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Reading json `map` with non-nullable value schema doesn't error if
values are actually null
[#​6391](https://redirect.github.com/apache/arrow-rs/issues/6391 )
- StringViewBuilder with deduplication does not clear observed values
[#​6384](https://redirect.github.com/apache/arrow-rs/issues/6384 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Cast from Decimal(p, s) to dictionary-encoded Decimal(p, s) loses
precision and scale
[#​6381](https://redirect.github.com/apache/arrow-rs/issues/6381 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- LocalFileSystem `list` operation returns objects in wrong order
[#​6375](https://redirect.github.com/apache/arrow-rs/issues/6375 )
- `compute::binary_mut` returns `Err(PrimitiveArray<T>)` only with
certain arrays
[#​6374](https://redirect.github.com/apache/arrow-rs/issues/6374 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Exporting Binary/Utf8View from arrow-rs to pyarrow fails
[#​6366](https://redirect.github.com/apache/arrow-rs/issues/6366 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- warning: methods `as_any` and `next_batch` are never used in `parquet`
crate
[#​6143](https://redirect.github.com/apache/arrow-rs/issues/6143 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
**Documentation updates:**
- chore: add docs, part of
[#​37](https://redirect.github.com/apache/arrow-rs/issues/37 )
[#​6496](https://redirect.github.com/apache/arrow-rs/pull/6496 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([ByteBaker](https://redirect.github.com/ByteBaker ))
- Minor: improve `ChunkedReader` docs
[#​6477](https://redirect.github.com/apache/arrow-rs/pull/6477 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([alamb](https://redirect.github.com/alamb ))
- Minor: Add some missing documentation to fix CI errors
[#​6445](https://redirect.github.com/apache/arrow-rs/pull/6445 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([etseidl](https://redirect.github.com/etseidl ))
- Fix doc "bit width" to "byte width"
[#​6434](https://redirect.github.com/apache/arrow-rs/pull/6434 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([kylebarron](https://redirect.github.com/kylebarron ))
- chore: add docs, part of
[#​37](https://redirect.github.com/apache/arrow-rs/issues/37 )
[#​6433](https://redirect.github.com/apache/arrow-rs/pull/6433 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([ByteBaker](https://redirect.github.com/ByteBaker ))
- chore: add docs, part of
[#​37](https://redirect.github.com/apache/arrow-rs/issues/37 )
[#​6424](https://redirect.github.com/apache/arrow-rs/pull/6424 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([ByteBaker](https://redirect.github.com/ByteBaker ))
- Rephrase doc comment
[#​6421](https://redirect.github.com/apache/arrow-rs/pull/6421 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([waynexia](https://redirect.github.com/waynexia ))
- Remove "NOT YET FULLY SUPPORTED" comment from
DataType::Utf8View/BinaryView
[#​6380](https://redirect.github.com/apache/arrow-rs/pull/6380 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- Improve `GenericStringBuilder` documentation
[#​6372](https://redirect.github.com/apache/arrow-rs/pull/6372 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
**Closed issues:**
- Columnar json writer for arrow-json
[#​6411](https://redirect.github.com/apache/arrow-rs/issues/6411 )
- Primitive `binary`/`unary` are not as fast as they could be
[#​6364](https://redirect.github.com/apache/arrow-rs/issues/6364 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Different numeric type may be able to compare
[#​6357](https://redirect.github.com/apache/arrow-rs/issues/6357 )
**Merged pull requests:**
- fix: override `size_hint` for `BitIterator` to return the exact
remaining size
[#​6495](https://redirect.github.com/apache/arrow-rs/pull/6495 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([Beihao-Zhou](https://redirect.github.com/Beihao-Zhou ))
- Minor: Fix path in format command in CONTRIBUTING.md
[#​6494](https://redirect.github.com/apache/arrow-rs/pull/6494 )
([etseidl](https://redirect.github.com/etseidl ))
- Write null counts in Parquet statistics when they are known
[#​6490](https://redirect.github.com/apache/arrow-rs/pull/6490 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Add configuration option to `StatisticsConverter` to control
interpretation of missing null counts in Parquet statistics
[#​6485](https://redirect.github.com/apache/arrow-rs/pull/6485 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- fix: check overflow numbers while inferring type for csv files
[#​6481](https://redirect.github.com/apache/arrow-rs/pull/6481 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([CookiePieWw](https://redirect.github.com/CookiePieWw ))
- Add better documentation, examples and builer-style API to `ByteView`
[#​6479](https://redirect.github.com/apache/arrow-rs/pull/6479 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- Add take_arrays util for getting entries from 2d arrays
[#​6475](https://redirect.github.com/apache/arrow-rs/pull/6475 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([akurmustafa](https://redirect.github.com/akurmustafa ))
- Deprecate `MetadataLoader`
[#​6474](https://redirect.github.com/apache/arrow-rs/pull/6474 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Update tonic-build requirement from =0.12.2 to =0.12.3
[#​6473](https://redirect.github.com/apache/arrow-rs/pull/6473 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- Align buffers from Python (FFI)
[#​6472](https://redirect.github.com/apache/arrow-rs/pull/6472 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([EnricoMi](https://redirect.github.com/EnricoMi ))
- Add `ParquetMetaDataBuilder`
[#​6466](https://redirect.github.com/apache/arrow-rs/pull/6466 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([alamb](https://redirect.github.com/alamb ))
- Make `ArrayData.align_buffers` align child data buffers recursively
[#​6462](https://redirect.github.com/apache/arrow-rs/pull/6462 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([EnricoMi](https://redirect.github.com/EnricoMi ))
- Minor: Silence compiler warnings for `parquet::file::metadata::reader`
[#​6457](https://redirect.github.com/apache/arrow-rs/pull/6457 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Minor: Error rather than panic for unsupported for dictionary
`cast`ing
[#​6456](https://redirect.github.com/apache/arrow-rs/pull/6456 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([goldmedal](https://redirect.github.com/goldmedal ))
- Support cast between Durations + between Durations all numeric types
[#​6452](https://redirect.github.com/apache/arrow-rs/pull/6452 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([tisonkun](https://redirect.github.com/tisonkun ))
- Deprecate methods from footer.rs in favor of `ParquetMetaDataReader`
[#​6451](https://redirect.github.com/apache/arrow-rs/pull/6451 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Workaround for missing Parquet page indexes in `ParquetMetadaReader`
[#​6450](https://redirect.github.com/apache/arrow-rs/pull/6450 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Fix CI by disabling newly failing rust <> nanoarrow integration test
in CI
[#​6449](https://redirect.github.com/apache/arrow-rs/pull/6449 )
([alamb](https://redirect.github.com/alamb ))
- Add `IpcSchemaEncoder`, deprecate ipc schema functions, Fix IPC not
respecting not preserving dict ID
[#​6444](https://redirect.github.com/apache/arrow-rs/pull/6444 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([brancz](https://redirect.github.com/brancz ))
- Add additional documentation and builder APIs to `SortOptions`
[#​6441](https://redirect.github.com/apache/arrow-rs/pull/6441 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- Update prost-build requirement from =0.13.2 to =0.13.3
[#​6440](https://redirect.github.com/apache/arrow-rs/pull/6440 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- Bump arrow-flight MSRV to 1.71.1
[#​6437](https://redirect.github.com/apache/arrow-rs/pull/6437 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([gstvg](https://redirect.github.com/gstvg ))
- Silence warnings that `as_any` and `next_batch` are never used
[#​6432](https://redirect.github.com/apache/arrow-rs/pull/6432 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Add `ParquetMetaDataReader`
[#​6431](https://redirect.github.com/apache/arrow-rs/pull/6431 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Add RowSelection::skipped_row_count
[#​6429](https://redirect.github.com/apache/arrow-rs/pull/6429 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([progval](https://redirect.github.com/progval ))
- perf: Faster decimal precision overflow checks
[#​6419](https://redirect.github.com/apache/arrow-rs/pull/6419 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([andygrove](https://redirect.github.com/andygrove ))
- fix: don't panic in IPC reader if struct child arrays have different
lengths
[#​6417](https://redirect.github.com/apache/arrow-rs/pull/6417 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alexwilcoxson-rel](https://redirect.github.com/alexwilcoxson-rel ))
- Reduce integration test matrix
[#​6407](https://redirect.github.com/apache/arrow-rs/pull/6407 )
([kou](https://redirect.github.com/kou ))
- Move lifetime of `take_iter` from iterator to its items
[#​6403](https://redirect.github.com/apache/arrow-rs/pull/6403 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([dariocurr](https://redirect.github.com/dariocurr ))
- Update lexical-core requirement from 0.8 to 1.0 (to resolve
RUSTSEC-2023-0086)
[#​6402](https://redirect.github.com/apache/arrow-rs/pull/6402 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([dariocurr](https://redirect.github.com/dariocurr ))
- Fix encoding/decoding REE Dicts when using streaming IPC
[#​6399](https://redirect.github.com/apache/arrow-rs/pull/6399 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([brancz](https://redirect.github.com/brancz ))
- fix: binary_mut should work if only one input array has null buffer
[#​6396](https://redirect.github.com/apache/arrow-rs/pull/6396 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([viirya](https://redirect.github.com/viirya ))
- Add `set_bits` fuzz test
[#​6394](https://redirect.github.com/apache/arrow-rs/pull/6394 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- impl `From<ScalarBuffer<T>>` for `Buffer`
[#​6389](https://redirect.github.com/apache/arrow-rs/pull/6389 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([mbrobbel](https://redirect.github.com/mbrobbel ))
- Add `union_extract` kernel
[#​6387](https://redirect.github.com/apache/arrow-rs/pull/6387 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([gstvg](https://redirect.github.com/gstvg ))
- Clear string-tracking hash table when ByteView deduplication is
enabled
[#​6385](https://redirect.github.com/apache/arrow-rs/pull/6385 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([shanesveller](https://redirect.github.com/shanesveller ))
- fix: Stop losing precision and scale when casting decimal to
dictionary
[#​6383](https://redirect.github.com/apache/arrow-rs/pull/6383 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([andygrove](https://redirect.github.com/andygrove ))
- Add `ARROW_VERSION` const
[#​6379](https://redirect.github.com/apache/arrow-rs/pull/6379 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([samuelcolvin](https://redirect.github.com/samuelcolvin ))
- parquet writer: Raise an error when the row_group_index overflows i16
[#​6378](https://redirect.github.com/apache/arrow-rs/pull/6378 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([progval](https://redirect.github.com/progval ))
- Implement native support StringViewArray for `regexp_is_match` and
`regexp_is_match_scalar` function, deprecate `regexp_is_match_utf8` and
`regexp_is_match_utf8_scalar`
[#​6376](https://redirect.github.com/apache/arrow-rs/pull/6376 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([tlm365](https://redirect.github.com/tlm365 ))
- Update chrono-tz requirement from 0.9 to 0.10
[#​6371](https://redirect.github.com/apache/arrow-rs/pull/6371 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- Support StringViewArray interop with python: fix lingering C Data
Interface issues for \*ViewArray
[#​6368](https://redirect.github.com/apache/arrow-rs/pull/6368 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([a10y](https://redirect.github.com/a10y ))
- stop panic in `MetadataLoader` on invalid data
[#​6367](https://redirect.github.com/apache/arrow-rs/pull/6367 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([samuelcolvin](https://redirect.github.com/samuelcolvin ))
- Add support for BinaryView in arrow_string::length
[#​6359](https://redirect.github.com/apache/arrow-rs/pull/6359 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([Omega359](https://redirect.github.com/Omega359 ))
- impl `From<Vec<T>>` for `Buffer`
[#​6355](https://redirect.github.com/apache/arrow-rs/pull/6355 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([mbrobbel](https://redirect.github.com/mbrobbel ))
- Add breaking change from
[#​6043](https://redirect.github.com/apache/arrow-rs/issues/6043 )
to `CHANGELOG`
[#​6354](https://redirect.github.com/apache/arrow-rs/pull/6354 )
([mbrobbel](https://redirect.github.com/mbrobbel ))
- Benchmark for bit_mask (set_bits)
[#​6353](https://redirect.github.com/apache/arrow-rs/pull/6353 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([kazuyukitanimura](https://redirect.github.com/kazuyukitanimura ))
- Update prost-build requirement from =0.13.1 to =0.13.2
[#​6350](https://redirect.github.com/apache/arrow-rs/pull/6350 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- fix: clippy warnings from nightly rust 1.82
[#​6348](https://redirect.github.com/apache/arrow-rs/pull/6348 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([waynexia](https://redirect.github.com/waynexia ))
- Add support for Utf8View in arrow_string::length
[#​6345](https://redirect.github.com/apache/arrow-rs/pull/6345 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([Omega359](https://redirect.github.com/Omega359 ))
- feat: add catalog/schema subcommands to flight_sql_client.
[#​6332](https://redirect.github.com/apache/arrow-rs/pull/6332 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([nathanielc](https://redirect.github.com/nathanielc ))
- Manually run fmt on all files under parquet
[#​6328](https://redirect.github.com/apache/arrow-rs/pull/6328 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Implement UnionArray logical_nulls
[#​6303](https://redirect.github.com/apache/arrow-rs/pull/6303 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([gstvg](https://redirect.github.com/gstvg ))
- Parquet: Verify 32-bit CRC checksum when decoding pages
[#​6290](https://redirect.github.com/apache/arrow-rs/pull/6290 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([xmakro](https://redirect.github.com/xmakro ))
\* *This Changelog was automatically generated by
[github_changelog_generator](https://redirect.github.com/github-changelog-generator/github-changelog-generator )*
</details>
<details>
<summary>clap-rs/clap (clap)</summary>
###
[`v4.5.19`](https://redirect.github.com/clap-rs/clap/blob/HEAD/CHANGELOG.md#4519---2024-10-01 )
[Compare
Source](https://redirect.github.com/clap-rs/clap/compare/v4.5.18...v4.5.19 )
##### Internal
- Update dependencies
</details>
<details>
<summary>rust-lang/futures-rs (futures)</summary>
###
[`v0.3.31`](https://redirect.github.com/rust-lang/futures-rs/blob/HEAD/CHANGELOG.md#0331---2024-10-05 )
[Compare
Source](https://redirect.github.com/rust-lang/futures-rs/compare/0.3.30...0.3.31 )
- Fix use after free of task in `FuturesUnordered` when dropped future
panics
([#​2886](https://redirect.github.com/rust-lang/futures-rs/issues/2886 ))
- Fix soundness bug in `task::waker_ref`
([#​2830](https://redirect.github.com/rust-lang/futures-rs/issues/2830 ))
This is a breaking change but allowed because it is soundness bug fix.
- Fix bugs in `AsyncBufRead::read_line` and `AsyncBufReadExt::lines`
([#​2884](https://redirect.github.com/rust-lang/futures-rs/issues/2884 ))
- Fix parsing issue in `select!`/`select_biased!`
([#​2832](https://redirect.github.com/rust-lang/futures-rs/issues/2832 ))
This is technically a breaking change as it will now reject a very odd
undocumented syntax that was previously accidentally accepted.
- Work around issue due to upstream `Waker::will_wake` change
([#​2865](https://redirect.github.com/rust-lang/futures-rs/issues/2865 ))
- Add `stream::Iter::{get_ref,get_mut,into_inner}`
([#​2875](https://redirect.github.com/rust-lang/futures-rs/issues/2875 ))
- Add `future::AlwaysReady`
([#​2825](https://redirect.github.com/rust-lang/futures-rs/issues/2825 ))
- Relax trait bound on non-constructor methods of
`io::{BufReader,BufWriter}`
([#​2848](https://redirect.github.com/rust-lang/futures-rs/issues/2848 ))
</details>
<details>
<summary>matklad/once_cell (once_cell)</summary>
###
[`v1.20.2`](https://redirect.github.com/matklad/once_cell/blob/HEAD/CHANGELOG.md#1202 )
[Compare
Source](https://redirect.github.com/matklad/once_cell/compare/v1.20.1...v1.20.2 )
- Remove `portable_atomic` from Cargo.lock if it is not, in fact, used:
[#​267](https://redirect.github.com/matklad/once_cell/pull/267 )
This is a work-around for this cargo
b[https://github.com/rust-lang/cargo/issues/10801 ](https://redirect.github.com/rust-lang/cargo/issues/10801 )0801.
###
[`v1.20.1`](https://redirect.github.com/matklad/once_cell/blob/HEAD/CHANGELOG.md#1201 )
[Compare
Source](https://redirect.github.com/matklad/once_cell/compare/v1.20.0...v1.20.1 )
- Allow using `race` module using just `portable_atomic`, without
`critical_section` and provide
better error messages on targets without atomic CAS instruction,
[#​265](https://redirect.github.com/matklad/once_cell/pull/265 ).
###
[`v1.20.0`](https://redirect.github.com/matklad/once_cell/compare/v1.19.0...v1.20.0 )
[Compare
Source](https://redirect.github.com/matklad/once_cell/compare/v1.19.0...v1.20.0 )
</details>
<details>
<summary>seanmonstar/reqwest (reqwest)</summary>
###
[`v0.12.8`](https://redirect.github.com/seanmonstar/reqwest/blob/HEAD/CHANGELOG.md#v0128 )
[Compare
Source](https://redirect.github.com/seanmonstar/reqwest/compare/v0.12.7...v0.12.8 )
- Add support for SOCKS4 proxies.
- Add `multipart::Form::file()` method for adding files easily.
- Add `Body::wrap()` to wrap any `http_body::Body` type.
- Fix the pool configuration to use a timer to remove expired
connections.
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on monday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions ) if
that's undesired.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/ ).
View the [repository job
log](https://developer.mend.io/github/dora-rs/dora ).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC45Ny4wIiwidXBkYXRlZEluVmVyIjoiMzguOTcuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->
1 year ago
Philipp Oppermann
c8a890a0c8
Create Rust-based openai api proxy server in node hub ( #678 )
This is a Rust-based version of https://github.com/dora-rs/dora/pull/676
based on
https://github.com/LlamaEdge/LlamaEdge/tree/main/llama-api-server . It
includes `serde` structs to fully deserialize chat completion requests.
The server replies to requests asynchronously, which makes it possible
to serve multiple clients at the same time. Note that replies are mapped
to requests purely based on their order right now. So the first reply is
assigned to the first request and so on. If the order of replies is
different (e.g. because some completions take longer), some form of ID
needs to be added to the code to enable a proper `request<->reply`
mapping.
Like #676 , this PR only implements basic chat completion requests. You
can try it through these steps:
```bash
cd examples/openai-server
dora build dataflow-rust.yml
dora start dataflow-rust.yml
# In a separate terminal
python openai_api_client.py
```
I didn't implement the `/v1/models` endpoint, so the _"Testing API
endpoints..."_ part of `openai_api_client.py` is expected to fail with
_"Error listing models: 404"_.
1 year ago
renovate[bot]
ea1353520e
Update dependencies
1 year ago
Philipp Oppermann
b7ae767de0
Merge branch 'main' into openai-api-proxy
1 year ago
Philipp Oppermann
0299f4bd38
Update dependencies ( #674 )
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [anyhow](https://redirect.github.com/dtolnay/anyhow ) | dependencies |
patch | `1.0.86` -> `1.0.89` |
| [arrow](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [arrow-array](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [arrow-data](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [arrow-schema](https://redirect.github.com/apache/arrow-rs ) |
workspace.dependencies | minor | `53.0.0` -> `53.1.0` |
| [clap](https://redirect.github.com/clap-rs/clap ) | dependencies |
patch | `4.5.18` -> `4.5.19` |
| [futures](https://rust-lang.github.io/futures-rs )
([source](https://redirect.github.com/rust-lang/futures-rs )) |
dependencies | patch | `0.3.30` -> `0.3.31` |
| [futures](https://rust-lang.github.io/futures-rs )
([source](https://redirect.github.com/rust-lang/futures-rs )) |
dev-dependencies | patch | `0.3.30` -> `0.3.31` |
| [once_cell](https://redirect.github.com/matklad/once_cell ) |
dependencies | minor | `1.19.0` -> `1.20.2` |
| [parquet](https://redirect.github.com/apache/arrow-rs ) | dependencies
| minor | `53.0.0` -> `53.1.0` |
| [regex](https://redirect.github.com/rust-lang/regex ) | dependencies |
minor | `1.10.6` -> `1.11.0` |
| [reqwest](https://redirect.github.com/seanmonstar/reqwest ) |
dependencies | patch | `0.12.7` -> `0.12.8` |
| [safer-ffi](https://redirect.github.com/getditto/safer_ffi ) |
dependencies | patch | `0.1.12` -> `0.1.13` |
| [thiserror](https://redirect.github.com/dtolnay/thiserror ) |
dependencies | patch | `1.0.63` -> `1.0.64` |
---
### Release Notes
<details>
<summary>dtolnay/anyhow (anyhow)</summary>
###
[`v1.0.89`](https://redirect.github.com/dtolnay/anyhow/releases/tag/1.0.89 )
[Compare
Source](https://redirect.github.com/dtolnay/anyhow/compare/1.0.88...1.0.89 )
- Make anyhow::Error's `UnwindSafe` and `RefUnwindSafe` impl
consistently available between versions of Rust newer and older than
1.72
([#​386](https://redirect.github.com/dtolnay/anyhow/issues/386 ))
###
[`v1.0.88`](https://redirect.github.com/dtolnay/anyhow/releases/tag/1.0.88 )
[Compare
Source](https://redirect.github.com/dtolnay/anyhow/compare/1.0.87...1.0.88 )
- Documentation improvements
###
[`v1.0.87`](https://redirect.github.com/dtolnay/anyhow/releases/tag/1.0.87 )
[Compare
Source](https://redirect.github.com/dtolnay/anyhow/compare/1.0.86...1.0.87 )
- Support more APIs, including `Error::new` and `Error::chain`, in
no-std mode on Rust 1.81+
([#​383](https://redirect.github.com/dtolnay/anyhow/issues/383 ))
</details>
<details>
<summary>apache/arrow-rs (arrow)</summary>
###
[`v53.1.0`](https://redirect.github.com/apache/arrow-rs/blob/HEAD/CHANGELOG.md#5310-2024-10-02 )
[Compare
Source](https://redirect.github.com/apache/arrow-rs/compare/53.0.0...53.1.0 )
[Full
Changelog](https://redirect.github.com/apache/arrow-rs/compare/53.0.0...53.1.0 )
**Implemented enhancements:**
- Write null counts in Parquet statistics when they are known to be zero
[#​6502](https://redirect.github.com/apache/arrow-rs/issues/6502 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Make it easier to find / work with `ByteView`
[#​6478](https://redirect.github.com/apache/arrow-rs/issues/6478 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Update lexical-core version due to soundness issues with current
version
[#​6468](https://redirect.github.com/apache/arrow-rs/issues/6468 )
- Add builder style API for manipulating `ParquetMetaData`
[#​6465](https://redirect.github.com/apache/arrow-rs/issues/6465 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- `ArrayData.align_buffers` should support `Struct` data type / child
data
[#​6461](https://redirect.github.com/apache/arrow-rs/issues/6461 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add a method to return the number of skipped rows in a `RowSelection`
[#​6428](https://redirect.github.com/apache/arrow-rs/issues/6428 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Bump lexical-core to 1.0
[#​6397](https://redirect.github.com/apache/arrow-rs/issues/6397 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add union_extract kernel
[#​6386](https://redirect.github.com/apache/arrow-rs/issues/6386 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- implement `regexp_is_match_utf8` and `regexp_is_match_utf8_scalar` for
`StringViewArray`
[#​6370](https://redirect.github.com/apache/arrow-rs/issues/6370 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add support for BinaryView in arrow_string::length
[#​6358](https://redirect.github.com/apache/arrow-rs/issues/6358 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add `as_union` to `AsArray`
[#​6351](https://redirect.github.com/apache/arrow-rs/issues/6351 )
- Ability to append non contiguous strings to `StringBuilder`
[#​6347](https://redirect.github.com/apache/arrow-rs/issues/6347 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Add Catalog DB Schema subcommands to `flight_sql_client`
[#​6331](https://redirect.github.com/apache/arrow-rs/issues/6331 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- Add support for Utf8View in arrow_string::length
[#​6305](https://redirect.github.com/apache/arrow-rs/issues/6305 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Reading FIXED_LEN_BYTE_ARRAY columns with nulls is inefficient
[#​6296](https://redirect.github.com/apache/arrow-rs/issues/6296 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Optionally verify 32-bit CRC checksum when decoding parquet pages
[#​6289](https://redirect.github.com/apache/arrow-rs/issues/6289 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
- Speed up `pad_nulls` for `FixedLenByteArrayBuffer`
[#​6297](https://redirect.github.com/apache/arrow-rs/pull/6297 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Improve performance of set_bits by avoiding to set individual bits
[#​6288](https://redirect.github.com/apache/arrow-rs/pull/6288 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([kazuyukitanimura](https://redirect.github.com/kazuyukitanimura ))
**Fixed bugs:**
- BitIterator panics when retrieving length
[#​6480](https://redirect.github.com/apache/arrow-rs/issues/6480 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Flight data retrieved via Python client (wrapping C++) cannot be used
by Rust Arrow
[#​6471](https://redirect.github.com/apache/arrow-rs/issues/6471 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- CI integration test failing: Archery test With other arrows
[#​6448](https://redirect.github.com/apache/arrow-rs/issues/6448 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- IPC not respecting not preserving dict ID
[#​6443](https://redirect.github.com/apache/arrow-rs/issues/6443 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- Failing CI: Prost requires Rust 1.71.1
[#​6436](https://redirect.github.com/apache/arrow-rs/issues/6436 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
- Invalid struct arrays in IPC data causes panic during read
[#​6416](https://redirect.github.com/apache/arrow-rs/issues/6416 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- REE Dicts cannot be encoded/decoded with streaming IPC
[#​6398](https://redirect.github.com/apache/arrow-rs/issues/6398 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Reading json `map` with non-nullable value schema doesn't error if
values are actually null
[#​6391](https://redirect.github.com/apache/arrow-rs/issues/6391 )
- StringViewBuilder with deduplication does not clear observed values
[#​6384](https://redirect.github.com/apache/arrow-rs/issues/6384 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Cast from Decimal(p, s) to dictionary-encoded Decimal(p, s) loses
precision and scale
[#​6381](https://redirect.github.com/apache/arrow-rs/issues/6381 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- LocalFileSystem `list` operation returns objects in wrong order
[#​6375](https://redirect.github.com/apache/arrow-rs/issues/6375 )
- `compute::binary_mut` returns `Err(PrimitiveArray<T>)` only with
certain arrays
[#​6374](https://redirect.github.com/apache/arrow-rs/issues/6374 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Exporting Binary/Utf8View from arrow-rs to pyarrow fails
[#​6366](https://redirect.github.com/apache/arrow-rs/issues/6366 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- warning: methods `as_any` and `next_batch` are never used in `parquet`
crate
[#​6143](https://redirect.github.com/apache/arrow-rs/issues/6143 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
**Documentation updates:**
- chore: add docs, part of
[#​37](https://redirect.github.com/apache/arrow-rs/issues/37 )
[#​6496](https://redirect.github.com/apache/arrow-rs/pull/6496 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([ByteBaker](https://redirect.github.com/ByteBaker ))
- Minor: improve `ChunkedReader` docs
[#​6477](https://redirect.github.com/apache/arrow-rs/pull/6477 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([alamb](https://redirect.github.com/alamb ))
- Minor: Add some missing documentation to fix CI errors
[#​6445](https://redirect.github.com/apache/arrow-rs/pull/6445 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([etseidl](https://redirect.github.com/etseidl ))
- Fix doc "bit width" to "byte width"
[#​6434](https://redirect.github.com/apache/arrow-rs/pull/6434 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([kylebarron](https://redirect.github.com/kylebarron ))
- chore: add docs, part of
[#​37](https://redirect.github.com/apache/arrow-rs/issues/37 )
[#​6433](https://redirect.github.com/apache/arrow-rs/pull/6433 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([ByteBaker](https://redirect.github.com/ByteBaker ))
- chore: add docs, part of
[#​37](https://redirect.github.com/apache/arrow-rs/issues/37 )
[#​6424](https://redirect.github.com/apache/arrow-rs/pull/6424 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([ByteBaker](https://redirect.github.com/ByteBaker ))
- Rephrase doc comment
[#​6421](https://redirect.github.com/apache/arrow-rs/pull/6421 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([waynexia](https://redirect.github.com/waynexia ))
- Remove "NOT YET FULLY SUPPORTED" comment from
DataType::Utf8View/BinaryView
[#​6380](https://redirect.github.com/apache/arrow-rs/pull/6380 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- Improve `GenericStringBuilder` documentation
[#​6372](https://redirect.github.com/apache/arrow-rs/pull/6372 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
**Closed issues:**
- Columnar json writer for arrow-json
[#​6411](https://redirect.github.com/apache/arrow-rs/issues/6411 )
- Primitive `binary`/`unary` are not as fast as they could be
[#​6364](https://redirect.github.com/apache/arrow-rs/issues/6364 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
- Different numeric type may be able to compare
[#​6357](https://redirect.github.com/apache/arrow-rs/issues/6357 )
**Merged pull requests:**
- fix: override `size_hint` for `BitIterator` to return the exact
remaining size
[#​6495](https://redirect.github.com/apache/arrow-rs/pull/6495 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([Beihao-Zhou](https://redirect.github.com/Beihao-Zhou ))
- Minor: Fix path in format command in CONTRIBUTING.md
[#​6494](https://redirect.github.com/apache/arrow-rs/pull/6494 )
([etseidl](https://redirect.github.com/etseidl ))
- Write null counts in Parquet statistics when they are known
[#​6490](https://redirect.github.com/apache/arrow-rs/pull/6490 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Add configuration option to `StatisticsConverter` to control
interpretation of missing null counts in Parquet statistics
[#​6485](https://redirect.github.com/apache/arrow-rs/pull/6485 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- fix: check overflow numbers while inferring type for csv files
[#​6481](https://redirect.github.com/apache/arrow-rs/pull/6481 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([CookiePieWw](https://redirect.github.com/CookiePieWw ))
- Add better documentation, examples and builer-style API to `ByteView`
[#​6479](https://redirect.github.com/apache/arrow-rs/pull/6479 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- Add take_arrays util for getting entries from 2d arrays
[#​6475](https://redirect.github.com/apache/arrow-rs/pull/6475 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([akurmustafa](https://redirect.github.com/akurmustafa ))
- Deprecate `MetadataLoader`
[#​6474](https://redirect.github.com/apache/arrow-rs/pull/6474 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Update tonic-build requirement from =0.12.2 to =0.12.3
[#​6473](https://redirect.github.com/apache/arrow-rs/pull/6473 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- Align buffers from Python (FFI)
[#​6472](https://redirect.github.com/apache/arrow-rs/pull/6472 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([EnricoMi](https://redirect.github.com/EnricoMi ))
- Add `ParquetMetaDataBuilder`
[#​6466](https://redirect.github.com/apache/arrow-rs/pull/6466 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([alamb](https://redirect.github.com/alamb ))
- Make `ArrayData.align_buffers` align child data buffers recursively
[#​6462](https://redirect.github.com/apache/arrow-rs/pull/6462 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([EnricoMi](https://redirect.github.com/EnricoMi ))
- Minor: Silence compiler warnings for `parquet::file::metadata::reader`
[#​6457](https://redirect.github.com/apache/arrow-rs/pull/6457 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Minor: Error rather than panic for unsupported for dictionary
`cast`ing
[#​6456](https://redirect.github.com/apache/arrow-rs/pull/6456 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([goldmedal](https://redirect.github.com/goldmedal ))
- Support cast between Durations + between Durations all numeric types
[#​6452](https://redirect.github.com/apache/arrow-rs/pull/6452 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([tisonkun](https://redirect.github.com/tisonkun ))
- Deprecate methods from footer.rs in favor of `ParquetMetaDataReader`
[#​6451](https://redirect.github.com/apache/arrow-rs/pull/6451 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Workaround for missing Parquet page indexes in `ParquetMetadaReader`
[#​6450](https://redirect.github.com/apache/arrow-rs/pull/6450 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Fix CI by disabling newly failing rust <> nanoarrow integration test
in CI
[#​6449](https://redirect.github.com/apache/arrow-rs/pull/6449 )
([alamb](https://redirect.github.com/alamb ))
- Add `IpcSchemaEncoder`, deprecate ipc schema functions, Fix IPC not
respecting not preserving dict ID
[#​6444](https://redirect.github.com/apache/arrow-rs/pull/6444 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([brancz](https://redirect.github.com/brancz ))
- Add additional documentation and builder APIs to `SortOptions`
[#​6441](https://redirect.github.com/apache/arrow-rs/pull/6441 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- Update prost-build requirement from =0.13.2 to =0.13.3
[#​6440](https://redirect.github.com/apache/arrow-rs/pull/6440 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- Bump arrow-flight MSRV to 1.71.1
[#​6437](https://redirect.github.com/apache/arrow-rs/pull/6437 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([gstvg](https://redirect.github.com/gstvg ))
- Silence warnings that `as_any` and `next_batch` are never used
[#​6432](https://redirect.github.com/apache/arrow-rs/pull/6432 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Add `ParquetMetaDataReader`
[#​6431](https://redirect.github.com/apache/arrow-rs/pull/6431 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Add RowSelection::skipped_row_count
[#​6429](https://redirect.github.com/apache/arrow-rs/pull/6429 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([progval](https://redirect.github.com/progval ))
- perf: Faster decimal precision overflow checks
[#​6419](https://redirect.github.com/apache/arrow-rs/pull/6419 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([andygrove](https://redirect.github.com/andygrove ))
- fix: don't panic in IPC reader if struct child arrays have different
lengths
[#​6417](https://redirect.github.com/apache/arrow-rs/pull/6417 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alexwilcoxson-rel](https://redirect.github.com/alexwilcoxson-rel ))
- Reduce integration test matrix
[#​6407](https://redirect.github.com/apache/arrow-rs/pull/6407 )
([kou](https://redirect.github.com/kou ))
- Move lifetime of `take_iter` from iterator to its items
[#​6403](https://redirect.github.com/apache/arrow-rs/pull/6403 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([dariocurr](https://redirect.github.com/dariocurr ))
- Update lexical-core requirement from 0.8 to 1.0 (to resolve
RUSTSEC-2023-0086)
[#​6402](https://redirect.github.com/apache/arrow-rs/pull/6402 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([dariocurr](https://redirect.github.com/dariocurr ))
- Fix encoding/decoding REE Dicts when using streaming IPC
[#​6399](https://redirect.github.com/apache/arrow-rs/pull/6399 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([brancz](https://redirect.github.com/brancz ))
- fix: binary_mut should work if only one input array has null buffer
[#​6396](https://redirect.github.com/apache/arrow-rs/pull/6396 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([viirya](https://redirect.github.com/viirya ))
- Add `set_bits` fuzz test
[#​6394](https://redirect.github.com/apache/arrow-rs/pull/6394 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([alamb](https://redirect.github.com/alamb ))
- impl `From<ScalarBuffer<T>>` for `Buffer`
[#​6389](https://redirect.github.com/apache/arrow-rs/pull/6389 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([mbrobbel](https://redirect.github.com/mbrobbel ))
- Add `union_extract` kernel
[#​6387](https://redirect.github.com/apache/arrow-rs/pull/6387 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([gstvg](https://redirect.github.com/gstvg ))
- Clear string-tracking hash table when ByteView deduplication is
enabled
[#​6385](https://redirect.github.com/apache/arrow-rs/pull/6385 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([shanesveller](https://redirect.github.com/shanesveller ))
- fix: Stop losing precision and scale when casting decimal to
dictionary
[#​6383](https://redirect.github.com/apache/arrow-rs/pull/6383 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([andygrove](https://redirect.github.com/andygrove ))
- Add `ARROW_VERSION` const
[#​6379](https://redirect.github.com/apache/arrow-rs/pull/6379 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([samuelcolvin](https://redirect.github.com/samuelcolvin ))
- parquet writer: Raise an error when the row_group_index overflows i16
[#​6378](https://redirect.github.com/apache/arrow-rs/pull/6378 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([progval](https://redirect.github.com/progval ))
- Implement native support StringViewArray for `regexp_is_match` and
`regexp_is_match_scalar` function, deprecate `regexp_is_match_utf8` and
`regexp_is_match_utf8_scalar`
[#​6376](https://redirect.github.com/apache/arrow-rs/pull/6376 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([tlm365](https://redirect.github.com/tlm365 ))
- Update chrono-tz requirement from 0.9 to 0.10
[#​6371](https://redirect.github.com/apache/arrow-rs/pull/6371 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- Support StringViewArray interop with python: fix lingering C Data
Interface issues for \*ViewArray
[#​6368](https://redirect.github.com/apache/arrow-rs/pull/6368 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([a10y](https://redirect.github.com/a10y ))
- stop panic in `MetadataLoader` on invalid data
[#​6367](https://redirect.github.com/apache/arrow-rs/pull/6367 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([samuelcolvin](https://redirect.github.com/samuelcolvin ))
- Add support for BinaryView in arrow_string::length
[#​6359](https://redirect.github.com/apache/arrow-rs/pull/6359 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([Omega359](https://redirect.github.com/Omega359 ))
- impl `From<Vec<T>>` for `Buffer`
[#​6355](https://redirect.github.com/apache/arrow-rs/pull/6355 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([mbrobbel](https://redirect.github.com/mbrobbel ))
- Add breaking change from
[#​6043](https://redirect.github.com/apache/arrow-rs/issues/6043 )
to `CHANGELOG`
[#​6354](https://redirect.github.com/apache/arrow-rs/pull/6354 )
([mbrobbel](https://redirect.github.com/mbrobbel ))
- Benchmark for bit_mask (set_bits)
[#​6353](https://redirect.github.com/apache/arrow-rs/pull/6353 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([kazuyukitanimura](https://redirect.github.com/kazuyukitanimura ))
- Update prost-build requirement from =0.13.1 to =0.13.2
[#​6350](https://redirect.github.com/apache/arrow-rs/pull/6350 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([dependabot\[bot\]](https://redirect.github.com/apps/dependabot ))
- fix: clippy warnings from nightly rust 1.82
[#​6348](https://redirect.github.com/apache/arrow-rs/pull/6348 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([waynexia](https://redirect.github.com/waynexia ))
- Add support for Utf8View in arrow_string::length
[#​6345](https://redirect.github.com/apache/arrow-rs/pull/6345 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([Omega359](https://redirect.github.com/Omega359 ))
- feat: add catalog/schema subcommands to flight_sql_client.
[#​6332](https://redirect.github.com/apache/arrow-rs/pull/6332 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
\[[arrow-flight](https://redirect.github.com/apache/arrow-rs/labels/arrow-flight )]
([nathanielc](https://redirect.github.com/nathanielc ))
- Manually run fmt on all files under parquet
[#​6328](https://redirect.github.com/apache/arrow-rs/pull/6328 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([etseidl](https://redirect.github.com/etseidl ))
- Implement UnionArray logical_nulls
[#​6303](https://redirect.github.com/apache/arrow-rs/pull/6303 )
\[[arrow](https://redirect.github.com/apache/arrow-rs/labels/arrow )]
([gstvg](https://redirect.github.com/gstvg ))
- Parquet: Verify 32-bit CRC checksum when decoding pages
[#​6290](https://redirect.github.com/apache/arrow-rs/pull/6290 )
\[[parquet](https://redirect.github.com/apache/arrow-rs/labels/parquet )]
([xmakro](https://redirect.github.com/xmakro ))
\* *This Changelog was automatically generated by
[github_changelog_generator](https://redirect.github.com/github-changelog-generator/github-changelog-generator )*
</details>
<details>
<summary>clap-rs/clap (clap)</summary>
###
[`v4.5.19`](https://redirect.github.com/clap-rs/clap/blob/HEAD/CHANGELOG.md#4519---2024-10-01 )
[Compare
Source](https://redirect.github.com/clap-rs/clap/compare/v4.5.18...v4.5.19 )
##### Internal
- Update dependencies
</details>
<details>
<summary>rust-lang/futures-rs (futures)</summary>
###
[`v0.3.31`](https://redirect.github.com/rust-lang/futures-rs/blob/HEAD/CHANGELOG.md#0331---2024-10-05 )
[Compare
Source](https://redirect.github.com/rust-lang/futures-rs/compare/0.3.30...0.3.31 )
- Fix use after free of task in `FuturesUnordered` when dropped future
panics
([#​2886](https://redirect.github.com/rust-lang/futures-rs/issues/2886 ))
- Fix soundness bug in `task::waker_ref`
([#​2830](https://redirect.github.com/rust-lang/futures-rs/issues/2830 ))
This is a breaking change but allowed because it is soundness bug fix.
- Fix bugs in `AsyncBufRead::read_line` and `AsyncBufReadExt::lines`
([#​2884](https://redirect.github.com/rust-lang/futures-rs/issues/2884 ))
- Fix parsing issue in `select!`/`select_biased!`
([#​2832](https://redirect.github.com/rust-lang/futures-rs/issues/2832 ))
This is technically a breaking change as it will now reject a very odd
undocumented syntax that was previously accidentally accepted.
- Work around issue due to upstream `Waker::will_wake` change
([#​2865](https://redirect.github.com/rust-lang/futures-rs/issues/2865 ))
- Add `stream::Iter::{get_ref,get_mut,into_inner}`
([#​2875](https://redirect.github.com/rust-lang/futures-rs/issues/2875 ))
- Add `future::AlwaysReady`
([#​2825](https://redirect.github.com/rust-lang/futures-rs/issues/2825 ))
- Relax trait bound on non-constructor methods of
`io::{BufReader,BufWriter}`
([#​2848](https://redirect.github.com/rust-lang/futures-rs/issues/2848 ))
</details>
<details>
<summary>matklad/once_cell (once_cell)</summary>
###
[`v1.20.2`](https://redirect.github.com/matklad/once_cell/blob/HEAD/CHANGELOG.md#1202 )
[Compare
Source](https://redirect.github.com/matklad/once_cell/compare/v1.20.1...v1.20.2 )
- Remove `portable_atomic` from Cargo.lock if it is not, in fact, used:
[#​267](https://redirect.github.com/matklad/once_cell/pull/267 )
This is a work-around for this cargo
b[https://github.com/rust-lang/cargo/issues/10801 ](https://redirect.github.com/rust-lang/cargo/issues/10801 )0801.
###
[`v1.20.1`](https://redirect.github.com/matklad/once_cell/blob/HEAD/CHANGELOG.md#1201 )
[Compare
Source](https://redirect.github.com/matklad/once_cell/compare/v1.20.0...v1.20.1 )
- Allow using `race` module using just `portable_atomic`, without
`critical_section` and provide
better error messages on targets without atomic CAS instruction,
[#​265](https://redirect.github.com/matklad/once_cell/pull/265 ).
###
[`v1.20.0`](https://redirect.github.com/matklad/once_cell/compare/v1.19.0...v1.20.0 )
[Compare
Source](https://redirect.github.com/matklad/once_cell/compare/v1.19.0...v1.20.0 )
</details>
<details>
<summary>rust-lang/regex (regex)</summary>
###
[`v1.11.0`](https://redirect.github.com/rust-lang/regex/blob/HEAD/CHANGELOG.md#1110-2024-09-29 )
[Compare
Source](https://redirect.github.com/rust-lang/regex/compare/1.10.6...1.11.0 )
\===================
This is a new minor release of `regex` that brings in an update to the
Unicode Character Database. Specifically, this updates the Unicode data
used by `regex` internally to the version 16 release.
New features:
- [FEATURE
#​1228](https://redirect.github.com/rust-lang/regex/pull/1228 ):
Add new `regex::SetMatches::matched_all` method.
- [FEATURE
#​1229](https://redirect.github.com/rust-lang/regex/pull/1229 ):
Update to Unicode Character Database (UCD) version 16.
</details>
<details>
<summary>seanmonstar/reqwest (reqwest)</summary>
###
[`v0.12.8`](https://redirect.github.com/seanmonstar/reqwest/blob/HEAD/CHANGELOG.md#v0128 )
[Compare
Source](https://redirect.github.com/seanmonstar/reqwest/compare/v0.12.7...v0.12.8 )
- Add support for SOCKS4 proxies.
- Add `multipart::Form::file()` method for adding files easily.
- Add `Body::wrap()` to wrap any `http_body::Body` type.
- Fix the pool configuration to use a timer to remove expired
connections.
</details>
<details>
<summary>dtolnay/thiserror (thiserror)</summary>
###
[`v1.0.64`](https://redirect.github.com/dtolnay/thiserror/releases/tag/1.0.64 )
[Compare
Source](https://redirect.github.com/dtolnay/thiserror/compare/1.0.63...1.0.64 )
- Exclude derived impls from coverage instrumentation
([#​322](https://redirect.github.com/dtolnay/thiserror/issues/322 ),
thanks [@​oxalica](https://redirect.github.com/oxalica ))
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on monday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions ) if
that's undesired.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/ ).
View the [repository job
log](https://developer.mend.io/github/dora-rs/dora ).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC45Ny4wIiwidXBkYXRlZEluVmVyIjoiMzguOTcuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->
1 year ago
Philipp Oppermann
f2793f5a88
Add basic support for stream-based completion interface
1 year ago
Philipp Oppermann
10f17a1cd2
Add dataflow file for Rust-based openai-server to `openai-server` example
1 year ago
Philipp Oppermann
d6f766e9bc
Create Rust-based openai api proxy server in node hub
1 year ago
renovate[bot]
9fd4e3550f
Update dependencies
1 year ago
Haixuan Xavier Tao
e1c9d67bee
Dora openai server example ( #676 )
You will find in this pull request a quick implementation of an openAI
server makes it possible to make request from an openai endpoint client
and receive it within the dataflow.
Note that I have only implemented chat completion and this is still
experimental.
Get started with:
```bash
cd examples/openai-server
dora build dataflow.yml
dora start dataflow.yml
# In a separate terminal
python openai_api_client.py
```
1 year ago
Haixuan Xavier Tao
4b7be45a7c
Fix llama recorder multi image recorder ( #677 )
This PR fixes one problem with recording multiple image at once using
llama recorder.
1 year ago
haixuanTao
9e9c4bab0b
Make openai server resilient to unusual input and pin response message topic
1 year ago
haixuanTao
7dbcff8863
Fix linting
1 year ago
haixuanTao
c8cc2f8e1d
Fix ungraceful stop for dora-openai-server
1 year ago
haixuanTao
6f3bdb179c
Fix README.md in openai server example
1 year ago
haixuanTao
05056f7fce
Fix small typo and add dependency for dora openai server
1 year ago
haixuanTao
b4b202596a
Simple example of a dora openai server
1 year ago
Haixuan Xavier Tao
7d93fe65db
Better error handling for unknown output ( #675 )
This PR makes it easier to debug when an output id is used but not
present within the dataflow output list of a node.
1 year ago
haixuanTao
5917b9ea27
Fix badly formatted llama factory recorder main.py
1 year ago
haixuanTao
33307b4e8a
Fix linting
1 year ago
haixuanTao
60496e62ea
Better error linting for unknown output
1 year ago
Haixuan Xavier Tao
6f7b030b64
Remove cli dataflow path check ( #662 )
Validating dataflow within the CLI is very hard as the CLI does not
record enough information to implement the validation of the dataflow,
creating false error of not valid dataflows.
1 year ago
Haixuan Xavier Tao
4e428ee57c
Add transformers version pinning for qwenvl2 ( #665 )
Follwing
https://github.com/huggingface/transformers/releases/tag/v4.45.0 , this
will makes it possible to push `dora-qwenvl2` within pip and let people
use dora build to install qwenvl2.
1 year ago
haixuanTao
7109226c8a
Add transformers version pinning for qwenvl2
1 year ago
Haixuan Xavier Tao
156b08a108
Fix ros2 bridge incompatibility with CI Ubuntu 24 and with pyo3 22 ( #670 )
This fix the CI that I unfortunately broke at #667
1 year ago
LyonRust
d383a486ae
Add adapter config path within qwenvl2
1 year ago
haixuanTao
d98bab8f31
Fix CI by lowering the time spent running the python example
1 year ago
haixuanTao
64fcc5b6ae
Add more time for the dataflow to stop
1 year ago
haixuanTao
e63d95ce8b
CI: Adding time before stopping python dataflow so that we don't need to interrupt the dataflow
1 year ago