383 Commits (deploy/dev)

Author	SHA1	Message	Date
Bowen Liang	4d9160ca9f	refactor: use dify_config to replace legacy usage of flask app's config (#9089)	1 year ago
-LAN-	e61752bd3a	feat/enhance the multi-modal support (#8818)	1 year ago
ice yao	2155bba5b0	fix: update mismatch vector type (#9462)	1 year ago
zhuhao	b90ad587c2	refactor: move the embedding to the rag module and abstract the rerank runner for extension (#9423)	1 year ago
zhuhao	86594851cb	refactor: update the default values of top-k parameter in vdb to be consistent (#9367)	1 year ago
Jyong	50635e9c15	Fix/economical knowledge retrieval (#9396)	1 year ago
zhuhao	cd7ab6231f	refactor: Add an enumeration type and use the factory pattern to obtain the corresponding class (#9356)	1 year ago
ice yao	d15ba3939d	Add Volcengine VikingDB as new vector provider (#9287)	1 year ago
zhuhao	d97d3ff5fc	chore: add abstract decorator and output log when query embedding fails (#9264)	1 year ago
Shili Cao	2ec6ffe478	feat:support baidu vector db (#9185)	1 year ago
Jyong	42b02b3a5f	Fix/agent external knowledge retrieval (#9241)	2 years ago
Jyong	80b62d50f5	Fix/add es num_candidates (#9225)	2 years ago
Jyong	cabdb4ef17	fix retrieval resource positon missed (#9155) ... Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>	2 years ago
Bowen Liang	240b66d737	chore: avoid implicit optional in type annotations of method (#8727)	2 years ago
Aurelius Huang	4585cffce1	fix: Compatible with special characters in pg full-text search. (#8921) ... Co-authored-by: Aurelius Huang <cm.huang@aftership.com>	2 years ago
Jyong	9d221a5e19	external knowledge api (#8913) ... Co-authored-by: Yi <yxiaoisme@gmail.com>	2 years ago
Zhaofeng Miao	369e1e6f58	feat(website-crawl): add jina reader as additional alternative for website crawling (#8761)	2 years ago
Bowen Liang	74f58f29f9	chore: bump ruff to 0.6.8 for fixing violation in SIM910 (#8869)	2 years ago
ice yao	27e33fb15c	chore: fix wrong VectorType match case (#8857)	2 years ago
zhuhao	55e6123db9	feat: add min-connection and max-connection for pgvector (#8841)	2 years ago
8bitpd	4c1063e1c5	fix: AnalyticdbVector retrieval scores (#8803)	2 years ago
zhuhao	008e0efeb0	refactor: update delete method as an abstract method (#8794)	2 years ago
crazywoola	bf64ff215b	fix: . is missing in file_extension (#8736)	2 years ago
omr	8fd297f8b4	fix: redundant check for available_document_count (#8491)	2 years ago
Jyong	7e611ffbf3	multi-retrival use dataset's top-k (#8416)	2 years ago
Bowen Liang	a1104ab97e	chore: refurish python code by applying Pylint linter rules (#8322)	2 years ago
Bowen Liang	6613b8f2e0	chore: fix unnecessary string concatation in single line (#8311)	2 years ago
-LAN-	08c486452f	fix: score_threshold handling in vector search methods (#8356)	2 years ago
Jyong	49cee773c5	fixed score threshold is none (#8342)	2 years ago
Bowen Liang	40fb4d16ef	chore: refurbish Python code by applying refurb linter rules (#8296)	2 years ago
Bowen Liang	c69f5b07ba	chore: apply ruff E501 line-too-long linter rule (#8275) ... Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
takatost	56c90e212a	fix(workflow): missing content in the answer node stream output during iterations (#8292) ... Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
Bowen Liang	0f14873255	chore: cleanup ruff flake8-simplify linter rules (#8286) ... Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
Bowen Liang	292220c596	chore: apply pep8-naming rules for naming convention (#8261)	2 years ago
Jyong	bb3002b173	revert page column (#8217)	2 years ago
Bowen Liang	2cf1187b32	chore(api/core): apply ruff reformatting (#7624)	2 years ago
Jyong	af92f19291	filter excel empty sheet (#8194)	2 years ago
Nam Vu	2d7954c7da	Fix variable typo (#8084)	2 years ago
ChengZi	2060db8e11	fix: change milvus init args from (host, port) to (url, token) (#8019) ... Signed-off-by: ChengZi <chen.zhang@zilliz.com>	2 years ago
Byeongjin Kang	d489b8b3e0	feat: return page number of pdf documents upon retrieval (#7749)	2 years ago
Jyong	0e71f6db84	fix spliter length missed (#7987)	2 years ago
KVOJJJin	14af87527f	Feat:remove estimation of embedding cost (#7950) ... Co-authored-by: jyong <718720800@qq.com>	2 years ago
Sumkor	571415d1a4	fix: split text keep separator (#7930)	2 years ago
Jyong	d8b6c053a2	fix rerank model value is empty string (#7937)	2 years ago
Jyong	01581dd35f	improve the notion table extract (#7925)	2 years ago
Jyong	6f33351eb3	ignore linked images when image id is none (#7890)	2 years ago
Jyong	60001a62c4	fixed chunk_overlap is None (#7703)	2 years ago
Kenn	122ce41020	feat: rewrite Elasticsearch index and search code to achieve Elasticsearch vector and full-text search (#7641) ... Co-authored-by: haokai <haokai@shuwen.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com> Co-authored-by: Bowen Liang <bowenliang@apache.org> Co-authored-by: wellCh4n <wellCh4n@foxmail.com>	2 years ago
Qin Liu	162faee4f2	fix: set score_threshold to zero if it is None for MyScale vectordb (#7640) ... Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2 years ago
Jyong	7ae728a9a3	fix nltk averaged_perceptron_tagger download and fix score limit is none (#7582) ... Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2 years ago
张皮皮	f29685f8a1	fix score_threshold is none, return all top K documents (#7581)	2 years ago
Byeongjin Kang	0223fc6fd5	feat: add pgvector full_text_search (#7396)	2 years ago
mochi	ba79088ffc	Fix SQL parser Error in MyScale vdb. (#7255)	2 years ago
miendinh	f104b930cf	feat: support elasticsearch vector database (#3558) ... Co-authored-by: miendinh <miendinh@users.noreply.github.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com> Co-authored-by: crazywoola <427733928@qq.com>	2 years ago
Bowen Liang	ccb6ddd840	chore: bump Ruff to 0.5.7 (#7174)	2 years ago
Joe	425174e82f	feat: update ops trace (#7102)	2 years ago
Jyong	12095f8cd6	extract docx filter comment element (#7092)	2 years ago
Jyong	169cde6c3c	add nltk punkt resource (#7063)	2 years ago
8bitpd	40c6f3c724	fix: add redis lock to AnalyticdbVector init (#6859) ... Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>	2 years ago
-LAN-	df8f8c9a2c	feat(api/core/rag/datasource/vdb/analyticdb/analyticdb_vector.py): Checking config before init analyticdb (#7050)	2 years ago
Jyong	80c94f02e9	add vector field for other vectordb (#7051)	2 years ago
chenxu9741	72c75b75cf	feat: Add hyperlink parsing to the DOCX document. (#7017)	2 years ago
Bryan	eece50acec	fix: tran list issue (#7009) ... Co-authored-by: libing <libing@healink.cn>	2 years ago
Jyong	28d4e5b045	Fix/reranking mode is null (#7012)	2 years ago
yanghx	c53875ce8c	fix #6902 .docx handles images within tables and handles cross-column tables (#6951)	2 years ago
Joe	f97a51ce24	fix: reranking disable timer error (#6910)	2 years ago
Bowen Liang	6aa02f8c63	dep: bump pgvecto-rs client from 0.1.x to 0.2.x (#6891)	2 years ago
Jyong	44801df8f8	fix score threshold limit be None (#6900)	2 years ago
Jyong	8157fccf6d	delete weight_type (#6865)	2 years ago
-LAN-	a98284b1ef	refactor(api): Switch to `dify_config` (#6750) ... Signed-off-by: -LAN- <laipz8200@outlook.com>	2 years ago
Bowen Liang	b414ea41d6	dep: initial support for Milvus 2.4.x (#6084)	2 years ago
Jyong	cf258b7a67	add xlsx support hyperlink extract (#6722)	2 years ago
Yeuoly	79cb23e8ac	security/SSRF vulns (#6682)	2 years ago
灰灰	5e4ac11df3	fix: code block segmentation problem of markdown document (#6465)	2 years ago
tmuife	16b4f560cd	fix bugs(when using Oracle23ai as Vector DB) (#6658)	2 years ago
Jyong	e4bb943fe5	Feat/delete single dataset retrival (#6570)	2 years ago
崔亮	0569c547ee	fix the issue of MILVUS_DATABASE has no effect. (#6424)	2 years ago
tmuife	06fc1bce9e	Add search by full text when using Oracle23ai as vector DB (#6559)	2 years ago
Sangmin Ahn	093b8ca475	fix: escape double quotation marks in the vector DB search query (#6506)	2 years ago
-LAN-	cd7fa8027a	fix(api/core/model_manager.py): Avoid mutation during iteration. (#6536)	2 years ago
-LAN-	5e6fc58db3	Feat/environment variables in workflow (#6515) ... Co-authored-by: JzoNg <jzongcode@gmail.com>	2 years ago
Jyong	f38034e455	clean vector collection redis cache (#6494)	2 years ago
Waffle	2ba05b041f	refactor(myscale):Set the default value of the myscale vector db in DifyConfig. (#6441)	2 years ago
leoterry	dc847ba145	Fix the vector retrieval sorting issue (#6431) ... Co-authored-by: weifj <“weifj@tuyuansu.com.cn”>	2 years ago
Poorandy	c8f5dfcf17	refactor(rag): switch to dify_config. (#6410) ... Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
Jyong	0de224b153	fix wrong using of RetrievalMethod Enum (#6345)	2 years ago
tangyoha	0cbbaf3f68	fix: markdown proc will remove image (#5855)	2 years ago
Matri	a9ee52f2d7	Fix/firecrawl parameters issue (#6213)	2 years ago
Waffle	7b225a5ab0	refactor(services/tasks): Swtich to dify_config witch Pydantic (#6203)	2 years ago
Zhuo Qiu	63e34e5227	feat: support MyScale vector database (#6092)	2 years ago
8bitpd	7c70eb87bc	feat: support AnalyticDB vector store (#5586) ... Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>	2 years ago
Aurelius Huang	f546db5437	fix: document truncation and loss in notion document sync (#5631) ... Co-authored-by: Aurelius Huang <cm.huang@aftership.com>	2 years ago
Joe	e8b8f6c6dd	Feat/fix ops trace (#5672) ... Co-authored-by: takatost <takatost@gmail.com>	2 years ago
Bowen Liang	dcb72e0067	chore: apply flake8-comprehensions Ruff rules to improve collection comprehensions (#5652) ... Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
Joe	4e2de638af	feat: add ops trace (#5483) ... Co-authored-by: takatost <takatost@gmail.com>	2 years ago
Jyong	43335b5c87	delete the deprecated method (#5612)	2 years ago
Xiaoguang Sun	13fcd7a901	feat: Add program_name attribute to TiDB connection (#5499) ... Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com>	2 years ago
tmuife	6a09409ec9	Add Oracle23ai as a vector datasource (#5342) ... Co-authored-by: walter from vm <walter.jin@oracle.com>	2 years ago
Bowen Liang	39c14ec7c1	improve: unify Excel files parsing in either xls or xlsx file format by Pandas (#4965)	2 years ago
Bowen Liang	aed56b1a8f	fix: Revert "feat: initial support for Milvus 2.4.x (#3795)" downgrading to 2.3.x for Linux arm64 installation failure (#5414)	2 years ago
quicksand	0105129fa8	fix bug: tencent vdb #5378 (#5408)	2 years ago
Bowen Liang	c923684edd	chore: extract retrival method literal values into enum (#5060)	2 years ago
Bowen Liang	bb33ffc332	feat: initial support for Milvus 2.4.x (#3795)	2 years ago
baojingyu	d160d1ed02	feat: support opensearch approximate k-NN (#5322)	2 years ago
sino	be3512aa57	fix: unable to reindex documents (#5276)	2 years ago
takatost	12c815c597	fix: ExtractSetting optional value missing None as default val (#5238)	2 years ago
Jyong	ba5f8afaa8	Feat/firecrawl data source (#5232) ... Co-authored-by: Nicolas <nicolascamara29@gmail.com> Co-authored-by: chenhe <guchenhe@gmail.com> Co-authored-by: takatost <takatost@gmail.com>	2 years ago
quicksand	4080f7b8ad	feat: support tencent vector db (#3568)	2 years ago
saga.rey	edf2047f04	fix: milvus_vector default dataset index_struct type from weaviate to milvus (#5098)	2 years ago
Bowen Liang	f976740b57	improve: mordernizing validation by migrating pydantic from 1.x to 2.x (#4592)	2 years ago
Bowen Liang	cdc08a434f	feat: support Chroma vector store (#5015)	2 years ago
Ikko Eltociear Ashimine	a40f68cf94	chore: update qdrant_vector.py (#5128)	2 years ago
Bowen Liang	bdad993901	improve: generalize vector factory classes and vector type (#5033)	2 years ago
Jyong	3b60c28b3a	deal the external image when extract docx image (#5024)	2 years ago
Jyong	237b8fe3d9	add meta.doc_id index for tidb (#4963)	2 years ago
Jyong	02e4de5166	fix some tidb bugs (#4960)	2 years ago
Weaxs	0797f9bc05	feat: support tidb vector (#4588)	2 years ago
YC	9f8ca75a81	fixing a bug of handling header row when parsing xls file, and tune xls/xlsx parsing result to be more structured (#3600)	2 years ago
Bowen Liang	f32b440c4a	chore: fix indention violations by applying E111 to E117 ruff rules (#4925)	2 years ago
JasonVV	7749b71fff	Optimize knowledge retrieval performance by batching dataset quries. (#4917)	2 years ago
takatost	d1dbbc1e33	feat: backend model load balancing support (#4927)	2 years ago
Jyong	b6631cd878	modify rerank and splitter code directory (#4924)	2 years ago
Bowen Liang	58db719a2c	dep: bump pandas from 1.x to 2.x (#4820)	2 years ago
Oliver Lee	176d91937d	fix 'NoneType' and new ContentType supported. (#4818)	2 years ago
Jyong	1b2d862973	add error msg for hit test (#4704)	2 years ago
yalei	026175c8f7	feat: update notion extractor (#3898) ... Co-authored-by: duyalei <>	2 years ago
Jyong	24624491cd	add qdrant metadata.doc_id index when create qdrant collection (#4570)	2 years ago
Jyong	233c4150d1	support images and tables extract from docx (#4619)	2 years ago
Rain Chen	c255a20d7c	allow to config max segmentation tokens length for RAG document using environment variable (#4375)	2 years ago
majian	b5204111da	Add UNSTRUCTURED_API_KEY env support (#4369)	2 years ago
Bowen Liang	04ad46dd31	chore: skip unnecessary key checks prior to accessing a dictionary (#4497)	2 years ago
Charlie.Wei	97b65f9b4b	Optimize webscraper (#4392) ... Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM> Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2 years ago
kerlion	98140ae5d9	fix the issue of MILVUS_DATABASE has no effect. (#4353)	2 years ago
LiuVaayne	875249eb00	Feat/vector db pgvector (#3879)	2 years ago
Bowen Liang	142814d451	chore: skip deprecated field_schema param in creating payload index on Qdrant (#3903)	2 years ago
Jyong	3e9dbe3e0a	add pgvecto_rs support and upgrade SQLAlchemy (#3833)	2 years ago
呆萌闷油瓶	0940f01634	enhancement:support Qdrant gRPC mode (#3929)	2 years ago
Bowen Liang	045827043d	test: improve vector store tests (#3855)	2 years ago
Bowen Liang	7919596a21	fix: UP031 style rule violation (#3866)	2 years ago
Bowen Liang	45dd1683fd	test: add tests covering all methods of vector store (#3849)	2 years ago
Jingpan Xiong	1be222af2e	fix: using api can not execute relyt vector database (#3766) ... Co-authored-by: jingsi <jingsi@leadincloud.com>	2 years ago
呆萌闷油瓶	78988ed60e	fix:still enable SSL verification when using qdrant based on HTTP protocol (#3805)	2 years ago
Bowen Liang	9cec8c1750	test: add unit tests for vector stores of Milvus, Qdrant and Weaviate (#3688)	2 years ago
Jyong	f257f2c396	Knowledge optimization (#3755) ... Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: JzoNg <jzongcode@gmail.com>	2 years ago
Bowen Liang	2867d29021	fix: milvus usage with create_collection (#3683)	2 years ago
Jingpan Xiong	33397836a5	feat: support relyt vector database (#3367) ... Co-authored-by: jingsi <jingsi@leadincloud.com>	2 years ago
Jyong	0737e930cb	chore: remove Langchain tools import (#3407)	2 years ago
chenxu9741	ad65c891e7	add xls file suport (#3321)	2 years ago
LiuVaayne	b00466f025	feat:api Add support for extracting EPUB files in ExtractProcessor (#3254) ... Co-authored-by: crazywoola <427733928@qq.com>	2 years ago
Jyong	1f302990c6	add segment with keyword issue (#3351) ... Co-authored-by: StyleZhang <jasonapring2015@outlook.com>	2 years ago
Jyong	6164604462	fix dataset retrival in dataset mode (#3334)	2 years ago
Jyong	b6de97ad53	Remove langchain dataset retrival agent logic (#3311)	2 years ago
Jyong	8fcf459285	fix milvus database name parameter missed (#3229)	2 years ago
Leo Q	9c01bcb3e5	feat: support setting database used in Milvus (#3003)	2 years ago
Jyong	283979fc46	fix keyword index error when storage source is S3 (#3182)	2 years ago
takatost	7753ba2d37	FEAT: NEW WORKFLOW ENGINE (#3160) ... Co-authored-by: Joel <iamjoel007@gmail.com> Co-authored-by: Yeuoly <admin@srmxy.cn> Co-authored-by: JzoNg <jzongcode@gmail.com> Co-authored-by: StyleZhang <jasonapring2015@outlook.com> Co-authored-by: jyong <jyong@dify.ai> Co-authored-by: nite-knite <nkCoding@gmail.com> Co-authored-by: jyong <718720800@qq.com>	2 years ago
Jyong	9eba6ffdd4	Optimize csv and excel extract (#3155) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Vikey Chen	e4f686deb7	fix unstructured api,remove unused parameters (#3056)	2 years ago
Jyong	a94d86da6d	add keyword table s3 storage support (#3065) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	84d118de07	add redis lock on create collection in multiple thread mode (#3054) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	a6cd0f0e73	fix add segment when dataset and document is empty (#3021) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	b0b0cc045f	add mutil-thread document embedding (#3016) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Qiwen Tong	180775a0ec	fix: init qdrant vector max recursion (#2909)	2 years ago
listeng	696efe494e	fix: Ignore some emtpy page_content when append to split_documents (#2898)	2 years ago
Weaxs	20bd49285b	excel: get keys from every sheet (#2796)	2 years ago
Jyong	8ba38e8e74	fix overlap and splitter optimization (#2742) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Bowen Liang	b163545771	Use `python-docx` to extract docx files (#2654)	2 years ago
Jyong	31070ffbca	fix qa index processor tenant id is None error (#2713) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Charlie.Wei	fa7ba30ba3	Fix rebuild index&csv parsing (#2705) ... Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM> Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2 years ago
Jyong	3631e53ff0	Feat/add annotation migrate (#2675) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
waltcow	f322d9bddb	Fix vdb merge error (#2650)	2 years ago
Bowen Liang	801d135390	generalize the generation of new collection name by dataset id (#2620)	2 years ago
takatost	a4d86496e1	fix: notion extractor raise 'NoneType' object has no attribute 'curre… (#2608)	2 years ago
Jyong	920b2c2b40	Fix/hit test tsne issue (#2581) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	fc64cdee64	fix mivlus delete by ids error (#2573) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	5b953c1ef2	Fix some RAG bugs (#2570) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Bowen Liang	562ca45e07	fix weaviate delete_by_ids (#2565)	2 years ago
Bowen Liang	e352a8ed1b	chore: remove redundant casting flask app config into dict (#2564)	2 years ago
Bowen Liang	e55225e2bc	fix typo in error message of supported keyword store (#2560)	2 years ago
Jyong	0620fa3094	Feat/vdb migrate command (#2562) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	4be3087642	Fix/new RAG bugs (#2547) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	91ea6fe4ee	Fix/langchain document schema (#2539) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	6c4e6bf1d6	Feat/dify rag (#2528) ... Co-authored-by: jyong <jyong@dify.ai>	2 years ago