Add nodes enhancement by raptor (#111)

* add raptor * add raptor ui support * fix logger bug * add node_enhancement class and modify test * fix node_enhancement setting bug * lint adjustment * poetry lock * fix poetry.lock * fix poetry issues * add a param * add token calculation for Chinese and adjust context_window * update tokenization_qwen * update file_path * merge feature and update poetry.lock * exclude pytest since no vocab file in the test env * exclude qwen.tiktoken * delete assert
aigc-apps · Jul 26, 2024 · fa271d8 · fa271d8
1 parent b4864c4
commit fa271d8
Show file tree

Hide file tree

Showing 27 changed files with 152,635 additions and 47 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -85,6 +85,8 @@ repos:
           (\/.*?\.[\w:]+)/poetry.lock
         args:
           [
+            "--exclude",
+            "./data/tokenization/qwen.tiktoken",
             "--ignore-words-list",
             "astroid,gallary,momento,narl,ot,rouge,nin,gere",
           ]