You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
main
deploy/rag-dev
feat/rag-2
deploy/dev
chore/plugin-error
feat/change-user-email-freezes-limit
feat/support-bool-variable-fe
feat/hitl-frontend
deploy/enterprise
feat/enchance-prompt-and-code-fe
build/oauth
feat/oauth
feature/smtp-oauth2-support
feat/human-input
build/plugin-auto-upgrade
test/performance
feat/tool-plugin-oauth
feat/tool-oauth
feat/email-update-frontend
temp-feat-owner-transfer-enterprise-frontend
feat/ownership-transfer-frontend
release/e-3.0.2
feat/plugin-auto-upgrade-fe
refactor/tanstack-form
chore/offline-the-sys-files
feat/rag-pipeline
feat/r2
release/e-3.2.0
wtw/rag-pipeline
feat/add-retrival-field-in-dataset
feat/datasource
e-260
revert-20786-fix/add_postgres_user
feat/webapp-verified-sso-main
fix/app-not-published-error
feat/webapp-verified-sso-260
release/0.15-support
fix/20421
v141-hotfix
fix/20326-tool-invoke-error-instance-account
fix/19933-account-object-has-no-attribute-_current_tenant-when-workflow-uses-workflow-as-tool-node
hotfix/0519
fix/e-legacy-clean-up
QuantumGhost-patch-1
fix/admin-permission
fix/e-admin-permission
feat/model-memory
feat/webapp-auth-api
1.0.0-fix
revert-17133-fix/i16990-text-to-language-settings
e-0156
release/1.1.3-fix1
hotfix/translation-fix
fix/slider-style-error
e-0154
release/0.15.6-alpha-1
cohere/cleanup-free-tenants-logs
release/1.0.1-fix1
release/1.0.0-fix1
chore/auto-dify
fix/rag-infinity-spaces-loop
fix/dataset-admin
fix/expose-debugging-port
dev/plugin-deploy
fix/tool-info
fix/aws-s3-r2-compatible
feat/knowledge-metabase
fix/fail-branch-stream-output-error
release/0.15.3-fix1
feat/14009-feature-request-support-for-retrieving-historical-conversations-without-from-enduser-id
fix/build-error
chore/infrastructure-upgrade
feat/upgrade-unstructured-version-10.2
feat/classnames-sort
feat/upgrade-unstructured-version
fix/version-check
fix/handle-agent-none-value
fix/agent-params-pasring
fix/agent-parallel
feat/knowledge-dark-mode
feat/custom-tool-input
release/0.15.2-fix1
0.15.1-admin-apis
feat/support-docx-image-view
chore/upgrade-next
hotfix/get-property-of-string-type-cause-page-crash
feat/agent-stream-output
feat/workflow-node-dark-mode
fix/not-show-strategy-type
fix/switch-strategy-clean-param
fix/chore-fix
opik-monitoring
provider-gpustack
feat/mecab-japanese-keywords
feat/retry-single-step-debug
revert-12086-feat/parent-child-retrieval
fix/remove-the-retry-index-field
fix/11839
feat/node-execution-retry
feat/support-extractor-tools-update-wip
feat/update-tidb-batchget-endpoint
fix/iteration-thread-pool-error
feat/support-multi-token-count
feat/parent-child-retrieval-api
fix/app-icon-is-missing
feat/support-extractor-tools-update
test/disable_site
build/add-dependabot
feat/support-extractor-tools
fix/docx-extract-image-ssrf
fix/redis-slow-in-gevent
fix/9772-internal-server-error-when-custom-disclaimer-longer-than-64-characters
alert-autofix-89
build/eslint-react-refresh-plugin
build/switch-to-pnpm
revert-9424-feat/update-dataset-clean-rule
fix/external-knowledge-retrieval-issues
feat/login-type-coontrol
feat/external-knowledge-api
feat/rag-external-knowledge-with-api
feat/external-knowledge
fix/notion-table-extract
fix/retrieval-test
fix/tooltip
feat/workflow-add-block-shortcut
feat/update-docs
fix/note-node-zoom-issue
fix/model-runtime-quato-issue
fix/db-lock-timeout
feat/new-tooltip
feat/web-app-sso
chore/optimize-app-workflow-deletion-slow-sql
fix/trace_app_config_app_id_idx
fix/extra-table-tracing-app-config
feat/update-beat-job-time-to-env
fix/index-estimate-error
chore/enterprise-license-status
fix/remove-tsne_position
feat/2p
feat/aws-iam-auth-check
feat/add-flashrank
feat/support-milvus-2.4
feat/add-resource-from-tools
bai
1.7.0
1.6.0
1.5.1
1.5.0
0.15.8
1.3.1
0.15.7
1.3.0
0.15.6
1.2.0
0.15.6-alpha.1
1.1.3
0.15.5
0.10.2-fix1
0.10.0
0.10.0-beta1
0.10.0-beta2
0.10.0-beta3
0.10.1
0.10.2
0.11.0
0.11.1
0.11.2
0.12.0
0.12.1
0.13.0
0.13.1
0.13.2
0.14.0
0.14.1
0.14.2
0.15.0
0.15.1
0.15.2
0.15.3
0.15.4
0.2.1
0.2.2
0.3.0
0.3.1
0.3.10
0.3.11
0.3.12
0.3.13
0.3.14
0.3.15
0.3.16
0.3.17
0.3.18
0.3.19
0.3.2
0.3.20
0.3.21
0.3.22
0.3.23
0.3.24
0.3.25
0.3.26
0.3.27
0.3.28
0.3.29
0.3.3
0.3.30
0.3.31
0.3.31-fix1
0.3.31-fix2
0.3.31-fix3
0.3.32
0.3.33
0.3.34
0.3.4
0.3.5
0.3.6
0.3.7
0.3.8
0.3.9
0.4.0
0.4.1
0.4.2
0.4.3
0.4.4
0.4.5
0.4.6
0.4.7
0.4.8
0.4.9
0.5.0
0.5.1
0.5.10
0.5.11
0.5.11-fix1
0.5.2
0.5.3
0.5.4
0.5.5
0.5.6
0.5.7
0.5.8
0.5.9
0.6.0
0.6.0-fix1
0.6.0-preview-workflow.1
0.6.0-preview-workflow.2
0.6.1
0.6.10
0.6.11
0.6.12
0.6.12-fix1
0.6.13
0.6.14
0.6.15
0.6.16
0.6.2
0.6.3
0.6.4
0.6.5
0.6.6
0.6.7
0.6.8
0.6.9
0.7.0
0.7.1
0.7.2
0.7.3
0.8.0
0.8.0-beta1
0.8.1
0.8.2
0.8.3
0.9.0
0.9.1
0.9.1-fix1
0.9.2
1.0.0
1.0.0-beta.1
1.0.1
1.1.0
1.1.1
1.1.2
1.4.0
1.4.1
1.4.2
1.4.3
v0.8.3-fix1
${ noResults }
12 lines
320 B
Python
12 lines
320 B
Python
import os
|
|
|
|
import tiktoken
|
|
|
|
from core.model_runtime.model_providers.__base.tokenizers.gpt2_tokenzier import GPT2Tokenizer
|
|
|
|
|
|
def test_tiktoken():
|
|
os.environ["TIKTOKEN_CACHE_DIR"] = "/tmp/.tiktoken_cache"
|
|
GPT2Tokenizer.get_num_tokens("Hello, world!")
|
|
assert tiktoken.registry.ENCODING_CONSTRUCTORS is not None
|