Modified LLamaBatch to not share tokens with other sequences if logits is true. This ensures that the logit span at the end in used by exactly one sequence - therefore it's safe to mutate. This removes the need for copying _very_ large arrays (vocab size) and simplifies sampling pipelines.
* Added a `Guidance` method to `LLamaTokenDataArray` which applies classifier free guidance
* Factored out a safer `llama_sample_apply_guidance` method based on spans
* Created a guided sampling demo using the batched executor
* fixed comment, "classifier free" not "context free"
* Rebased onto master and fixed breakage due to changes in `BaseSamplingPipeline`
* Asking user for guidance weight
* Progress bar in batched fork demo
* Improved fork example (using tree display)
* Added proper disposal of resources in batched examples
* Added some more comments in BatchedExecutorGuidance
- Re-implmented `Rewind` as an extension method using `Modify` internally
- Implemented `ShiftLeft`, which shifts everything over except for some starting tokens. This is the same as the `StatelessExecutor` out-of-context handling.
- Starting batch at epoch 1, this ensures that conversations (starting at zero) are below the current epoch. It also means `0` can always be used as a value guaranteed to be below the current epoch.
Conversations can be "forked", to create a copy of a conversation at a given point. This allows e.g. prompting a conversation with a system prefix just once and then forking it again and again for each individual conversation. Conversations can also be "rewound" to an earlier state.
Added two new examples, demonstrating forking and rewinding.
This results in windows users not needing to rename the DLL. This allows native llama builds to be dropped in, even on windows.
I also took the time to update the documentation, removing references to renaming the files, since the names now match.
Fixes#463
- Added a `DecodeAsync` overload which runs the work in a task
- Replaced some `NativeHandle` usage in `BatchedDecoding` with higher level equivalents.
- Made the `LLamaBatch` grow when token capacity is exceeded, removing the need to manage token capacity externally.
- Properly displaying `LLamaToken`
- Removed all tokenisation code in `SafeLLamaContextHandle` - just pass it all through to the `SafeLlamaModelHandle`
- Improved `SafeLlamaModelHandle` tokenisation:
- Renting an array, for one less allocation
- Not using `&tokens[0]` to take a pointer to an array, this is redundant and doesn't work on empty arrays
- made `llama_backend_init` private. This is automatically called, there is no way it can correctly be used externally.
- made `llama_token_to_piece` safe (Span instead of pointer)
- Made `NativeApi` into a `static class` (it's not intended to be instantiated)
- Moved `LLamaTokenType` enum out into a separate file
- Made `LLamaSeqId` and `LLamaPos` into `record struct`, convenient to have equality etc
- Enabled more features in build process (VBMI and VNNI)
- Added runtime checking for this features
- Improved runtime checking to no longer require dotnet8.0
Clovis Henrique Ribeiro
Martin Evans
Scott W Harden
Jason Couture