LLamaSharp

672 Commits

8 Branches

333 MB

358 Download

Tree: df310e15da

Commit Graph

Author	SHA1	Message	Date
Martin Evans	a03fe003de	Fixed decoding of text "accumulating" over time (never properly clearing buffer)	2 years ago
Martin Evans	51d4411a58	Added two new classes for detokenization tasks: - `AntipromptProcessor` accepts chunks of text and returns a value indicating if any antiprompt has been detected. - `StreamingTokenDecoder` decodes tokens into text, maintaining some internal state to handle single characters which are encoded as multiple tokens. Added tests for these classes and updated StatelessExecutor to use them. Removed most DeTokenize methods, marked the rest as obsolete (should always use a `StreamingTokenDecoder`).	2 years ago

Author

SHA1

Message

Date

Martin Evans

a03fe003de

Fixed decoding of text "accumulating" over time (never properly clearing buffer)

2 years ago

Martin Evans

51d4411a58

Added two new classes for detokenization tasks:

- `AntipromptProcessor` accepts chunks of text and returns a value indicating if any antiprompt has been detected.
 - `StreamingTokenDecoder` decodes tokens into text, maintaining some internal state to handle single characters which are encoded as multiple tokens.

Added tests for these classes and updated StatelessExecutor to use them.

Removed most DeTokenize methods, marked the rest as obsolete (should always use a `StreamingTokenDecoder`).

2 years ago

2 Commits (df310e15da331647ba1aff2dfffe22df51cb0880)