Add Text Fitter plugin v0.0.1#2452
Open
AlexMultiAgent wants to merge 6 commits into
Open
Conversation
AlexMultiAgent
force-pushed
the
add-text-fitter-0.0.1
branch
3 times, most recently
from
8373ea4 to
a76c8bf
Compare
AlexMultiAgent
force-pushed
the
add-text-fitter-0.0.1
branch
from
4365de4 to
30f8e83
Compare
- Fix length score discontinuities across all boundaries - Expand tokenizer CJK character ranges - Improve exception handling with error visibility - Increase compression_ratio precision - Use CJK ellipsis for CJK text truncation - Add .difyignore for clean packaging
- Fix length score piecewise continuity at len=20 boundary - Add multi-dot abbreviation protection (i.e., e.g.) - Pre-compile sentence-split regex at module level - Extract shared CJK Unicode ranges into _is_cjk_char helper - Fix incomplete CJK detection in boundary truncation fallback - Guard MMR prefilter empty-selection index mapping
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Submission Type
Checklist
Documentation Checklist
Privacy Protection
Guidelines
Description
Text Fitter is a Dify tool plugin that ensures text fits within LLM context window limits via intelligent extractive summarization. Supports Chinese, Japanese, and English text.
Source Code
https://github.com/AlexMultiAgent/dify-plugin-text-fitter