gcgj-dify-1.7.0

Commit Graph

Author	SHA1	Message	Date
张皮皮	f29685f8a1	fix score_threshold is none, return all top K documents (#7581 )	2 years ago
Byeongjin Kang	0223fc6fd5	feat: add pgvector full_text_search (#7396 )	2 years ago
mochi	ba79088ffc	Fix SQL parser Error in MyScale vdb. (#7255 )	2 years ago
miendinh	f104b930cf	feat: support elasticsearch vector database (#3558 ) Co-authored-by: miendinh <miendinh@users.noreply.github.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com> Co-authored-by: crazywoola <427733928@qq.com>	2 years ago
Bowen Liang	ccb6ddd840	chore: bump Ruff to 0.5.7 (#7174 )	2 years ago
Joe	425174e82f	feat: update ops trace (#7102 )	2 years ago
Jyong	12095f8cd6	extract docx filter comment element (#7092 )	2 years ago
Jyong	169cde6c3c	add nltk punkt resource (#7063 )	2 years ago
8bitpd	40c6f3c724	fix: add redis lock to AnalyticdbVector init (#6859 ) Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>	2 years ago
-LAN-	df8f8c9a2c	feat(api/core/rag/datasource/vdb/analyticdb/analyticdb_vector.py): Checking config before init analyticdb (#7050 )	2 years ago
Jyong	80c94f02e9	add vector field for other vectordb (#7051 )	2 years ago
chenxu9741	72c75b75cf	feat: Add hyperlink parsing to the DOCX document. (#7017 )	2 years ago
Bryan	eece50acec	fix: tran list issue (#7009 ) Co-authored-by: libing <libing@healink.cn>	2 years ago
Jyong	28d4e5b045	Fix/reranking mode is null (#7012 )	2 years ago
yanghx	c53875ce8c	fix #6902 .docx handles images within tables and handles cross-column tables (#6951 )	2 years ago
Joe	f97a51ce24	fix: reranking disable timer error (#6910 )	2 years ago
Bowen Liang	6aa02f8c63	dep: bump pgvecto-rs client from 0.1.x to 0.2.x (#6891 )	2 years ago
Jyong	44801df8f8	fix score threshold limit be None (#6900 )	2 years ago
Jyong	8157fccf6d	delete weight_type (#6865 )	2 years ago
-LAN-	a98284b1ef	refactor(api): Switch to `dify_config` (#6750 ) Signed-off-by: -LAN- <laipz8200@outlook.com>	2 years ago
Bowen Liang	b414ea41d6	dep: initial support for Milvus 2.4.x (#6084 )	2 years ago
Jyong	cf258b7a67	add xlsx support hyperlink extract (#6722 )	2 years ago
Yeuoly	79cb23e8ac	security/SSRF vulns (#6682 )	2 years ago
灰灰	5e4ac11df3	fix: code block segmentation problem of markdown document (#6465 )	2 years ago
tmuife	16b4f560cd	fix bugs(when using Oracle23ai as Vector DB) (#6658 )	2 years ago
Jyong	e4bb943fe5	Feat/delete single dataset retrival (#6570 )	2 years ago
崔亮	0569c547ee	fix the issue of MILVUS_DATABASE has no effect. (#6424 )	2 years ago
tmuife	06fc1bce9e	Add search by full text when using Oracle23ai as vector DB (#6559 )	2 years ago
Sangmin Ahn	093b8ca475	fix: escape double quotation marks in the vector DB search query (#6506 )	2 years ago
-LAN-	cd7fa8027a	fix(api/core/model_manager.py): Avoid mutation during iteration. (#6536 )	2 years ago
-LAN-	5e6fc58db3	Feat/environment variables in workflow (#6515 ) Co-authored-by: JzoNg <jzongcode@gmail.com>	2 years ago
Jyong	f38034e455	clean vector collection redis cache (#6494 )	2 years ago
Waffle	2ba05b041f	refactor(myscale):Set the default value of the myscale vector db in DifyConfig. (#6441 )	2 years ago
leoterry	dc847ba145	Fix the vector retrieval sorting issue (#6431 ) Co-authored-by: weifj <“weifj@tuyuansu.com.cn”>	2 years ago
Poorandy	c8f5dfcf17	refactor(rag): switch to dify_config. (#6410 ) Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
Jyong	0de224b153	fix wrong using of RetrievalMethod Enum (#6345 )	2 years ago
tangyoha	0cbbaf3f68	fix: markdown proc will remove image (#5855 )	2 years ago
Matri	a9ee52f2d7	Fix/firecrawl parameters issue (#6213 )	2 years ago
Waffle	7b225a5ab0	refactor(services/tasks): Swtich to dify_config witch Pydantic (#6203 )	2 years ago
Zhuo Qiu	63e34e5227	feat: support MyScale vector database (#6092 )	2 years ago
8bitpd	7c70eb87bc	feat: support AnalyticDB vector store (#5586 ) Co-authored-by: xiaozeyu <xiaozeyu.xzy@alibaba-inc.com>	2 years ago
Aurelius Huang	f546db5437	fix: document truncation and loss in notion document sync (#5631 ) Co-authored-by: Aurelius Huang <cm.huang@aftership.com>	2 years ago
Joe	e8b8f6c6dd	Feat/fix ops trace (#5672 ) Co-authored-by: takatost <takatost@gmail.com>	2 years ago
Bowen Liang	dcb72e0067	chore: apply flake8-comprehensions Ruff rules to improve collection comprehensions (#5652 ) Co-authored-by: -LAN- <laipz8200@outlook.com>	2 years ago
Joe	4e2de638af	feat: add ops trace (#5483 ) Co-authored-by: takatost <takatost@gmail.com>	2 years ago
Jyong	43335b5c87	delete the deprecated method (#5612 )	2 years ago
Xiaoguang Sun	13fcd7a901	feat: Add program_name attribute to TiDB connection (#5499 ) Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com>	2 years ago
tmuife	6a09409ec9	Add Oracle23ai as a vector datasource (#5342 ) Co-authored-by: walter from vm <walter.jin@oracle.com>	2 years ago
Bowen Liang	39c14ec7c1	improve: unify Excel files parsing in either xls or xlsx file format by Pandas (#4965 )	2 years ago
Bowen Liang	aed56b1a8f	fix: Revert "feat: initial support for Milvus 2.4.x (#3795 )" downgrading to 2.3.x for Linux arm64 installation failure (#5414 )	2 years ago
quicksand	0105129fa8	fix bug: tencent vdb #5378 (#5408 )	2 years ago
Bowen Liang	c923684edd	chore: extract retrival method literal values into enum (#5060 )	2 years ago
Bowen Liang	bb33ffc332	feat: initial support for Milvus 2.4.x (#3795 )	2 years ago
baojingyu	d160d1ed02	feat: support opensearch approximate k-NN (#5322 )	2 years ago
sino	be3512aa57	fix: unable to reindex documents (#5276 )	2 years ago
takatost	12c815c597	fix: ExtractSetting optional value missing None as default val (#5238 )	2 years ago
Jyong	ba5f8afaa8	Feat/firecrawl data source (#5232 ) Co-authored-by: Nicolas <nicolascamara29@gmail.com> Co-authored-by: chenhe <guchenhe@gmail.com> Co-authored-by: takatost <takatost@gmail.com>	2 years ago
quicksand	4080f7b8ad	feat: support tencent vector db (#3568 )	2 years ago
saga.rey	edf2047f04	fix: milvus_vector default dataset index_struct type from weaviate to milvus (#5098 )	2 years ago
Bowen Liang	f976740b57	improve: mordernizing validation by migrating pydantic from 1.x to 2.x (#4592 )	2 years ago
Bowen Liang	cdc08a434f	feat: support Chroma vector store (#5015 )	2 years ago
Ikko Eltociear Ashimine	a40f68cf94	chore: update qdrant_vector.py (#5128 )	2 years ago
Bowen Liang	bdad993901	improve: generalize vector factory classes and vector type (#5033 )	2 years ago
Jyong	3b60c28b3a	deal the external image when extract docx image (#5024 )	2 years ago
Jyong	237b8fe3d9	add meta.doc_id index for tidb (#4963 )	2 years ago
Jyong	02e4de5166	fix some tidb bugs (#4960 )	2 years ago
Weaxs	0797f9bc05	feat: support tidb vector (#4588 )	2 years ago
YC	9f8ca75a81	fixing a bug of handling header row when parsing xls file, and tune xls/xlsx parsing result to be more structured (#3600 )	2 years ago
Bowen Liang	f32b440c4a	chore: fix indention violations by applying E111 to E117 ruff rules (#4925 )	2 years ago
JasonVV	7749b71fff	Optimize knowledge retrieval performance by batching dataset quries. (#4917 )	2 years ago
takatost	d1dbbc1e33	feat: backend model load balancing support (#4927 )	2 years ago
Jyong	b6631cd878	modify rerank and splitter code directory (#4924 )	2 years ago
Bowen Liang	58db719a2c	dep: bump pandas from 1.x to 2.x (#4820 )	2 years ago
Oliver Lee	176d91937d	fix 'NoneType' and new ContentType supported. (#4818 )	2 years ago
Jyong	1b2d862973	add error msg for hit test (#4704 )	2 years ago
yalei	026175c8f7	feat: update notion extractor (#3898 ) Co-authored-by: duyalei <>	2 years ago
Jyong	24624491cd	add qdrant metadata.doc_id index when create qdrant collection (#4570 )	2 years ago
Jyong	233c4150d1	support images and tables extract from docx (#4619 )	2 years ago
Rain Chen	c255a20d7c	allow to config max segmentation tokens length for RAG document using environment variable (#4375 )	2 years ago
majian	b5204111da	Add UNSTRUCTURED_API_KEY env support (#4369 )	2 years ago
Bowen Liang	04ad46dd31	chore: skip unnecessary key checks prior to accessing a dictionary (#4497 )	2 years ago
Charlie.Wei	97b65f9b4b	Optimize webscraper (#4392 ) Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM> Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2 years ago
kerlion	98140ae5d9	fix the issue of MILVUS_DATABASE has no effect. (#4353 )	2 years ago
LiuVaayne	875249eb00	Feat/vector db pgvector (#3879 )	2 years ago
Bowen Liang	142814d451	chore: skip deprecated field_schema param in creating payload index on Qdrant (#3903 )	2 years ago
Jyong	3e9dbe3e0a	add pgvecto_rs support and upgrade SQLAlchemy (#3833 )	2 years ago
呆萌闷油瓶	0940f01634	enhancement:support Qdrant gRPC mode (#3929 )	2 years ago
Bowen Liang	045827043d	test: improve vector store tests (#3855 )	2 years ago
Bowen Liang	7919596a21	fix: UP031 style rule violation (#3866 )	2 years ago
Bowen Liang	45dd1683fd	test: add tests covering all methods of vector store (#3849 )	2 years ago
Jingpan Xiong	1be222af2e	fix: using api can not execute relyt vector database (#3766 ) Co-authored-by: jingsi <jingsi@leadincloud.com>	2 years ago
呆萌闷油瓶	78988ed60e	fix:still enable SSL verification when using qdrant based on HTTP protocol (#3805 )	2 years ago
Bowen Liang	9cec8c1750	test: add unit tests for vector stores of Milvus, Qdrant and Weaviate (#3688 )	2 years ago
Jyong	f257f2c396	Knowledge optimization (#3755 ) Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: JzoNg <jzongcode@gmail.com>	2 years ago
Bowen Liang	2867d29021	fix: milvus usage with create_collection (#3683 )	2 years ago
Jingpan Xiong	33397836a5	feat: support relyt vector database (#3367 ) Co-authored-by: jingsi <jingsi@leadincloud.com>	2 years ago
Jyong	0737e930cb	chore: remove Langchain tools import (#3407 )	2 years ago
chenxu9741	ad65c891e7	add xls file suport (#3321 )	2 years ago
LiuVaayne	b00466f025	feat:api Add support for extracting EPUB files in ExtractProcessor (#3254 ) Co-authored-by: crazywoola <427733928@qq.com>	2 years ago
Jyong	1f302990c6	add segment with keyword issue (#3351 ) Co-authored-by: StyleZhang <jasonapring2015@outlook.com>	2 years ago
Jyong	6164604462	fix dataset retrival in dataset mode (#3334 )	2 years ago
Jyong	b6de97ad53	Remove langchain dataset retrival agent logic (#3311 )	2 years ago
Jyong	8fcf459285	fix milvus database name parameter missed (#3229 )	2 years ago
Leo Q	9c01bcb3e5	feat: support setting database used in Milvus (#3003 )	2 years ago
Jyong	283979fc46	fix keyword index error when storage source is S3 (#3182 )	2 years ago
takatost	7753ba2d37	FEAT: NEW WORKFLOW ENGINE (#3160 ) Co-authored-by: Joel <iamjoel007@gmail.com> Co-authored-by: Yeuoly <admin@srmxy.cn> Co-authored-by: JzoNg <jzongcode@gmail.com> Co-authored-by: StyleZhang <jasonapring2015@outlook.com> Co-authored-by: jyong <jyong@dify.ai> Co-authored-by: nite-knite <nkCoding@gmail.com> Co-authored-by: jyong <718720800@qq.com>	2 years ago
Jyong	9eba6ffdd4	Optimize csv and excel extract (#3155 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Vikey Chen	e4f686deb7	fix unstructured api,remove unused parameters (#3056 )	2 years ago
Jyong	a94d86da6d	add keyword table s3 storage support (#3065 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	84d118de07	add redis lock on create collection in multiple thread mode (#3054 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	a6cd0f0e73	fix add segment when dataset and document is empty (#3021 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	b0b0cc045f	add mutil-thread document embedding (#3016 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Qiwen Tong	180775a0ec	fix: init qdrant vector max recursion (#2909 )	2 years ago
listeng	696efe494e	fix: Ignore some emtpy page_content when append to split_documents (#2898 )	2 years ago
Weaxs	20bd49285b	excel: get keys from every sheet (#2796 )	2 years ago
Jyong	8ba38e8e74	fix overlap and splitter optimization (#2742 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Bowen Liang	b163545771	Use `python-docx` to extract docx files (#2654 )	2 years ago
Jyong	31070ffbca	fix qa index processor tenant id is None error (#2713 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Charlie.Wei	fa7ba30ba3	Fix rebuild index&csv parsing (#2705 ) Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM> Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2 years ago
Jyong	3631e53ff0	Feat/add annotation migrate (#2675 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
waltcow	f322d9bddb	Fix vdb merge error (#2650 )	2 years ago
Bowen Liang	801d135390	generalize the generation of new collection name by dataset id (#2620 )	2 years ago
takatost	a4d86496e1	fix: notion extractor raise 'NoneType' object has no attribute 'curre… (#2608 )	2 years ago
Jyong	920b2c2b40	Fix/hit test tsne issue (#2581 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	fc64cdee64	fix mivlus delete by ids error (#2573 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	5b953c1ef2	Fix some RAG bugs (#2570 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Bowen Liang	562ca45e07	fix weaviate delete_by_ids (#2565 )	2 years ago
Bowen Liang	e352a8ed1b	chore: remove redundant casting flask app config into dict (#2564 )	2 years ago
Bowen Liang	e55225e2bc	fix typo in error message of supported keyword store (#2560 )	2 years ago
Jyong	0620fa3094	Feat/vdb migrate command (#2562 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	4be3087642	Fix/new RAG bugs (#2547 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	91ea6fe4ee	Fix/langchain document schema (#2539 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago
Jyong	6c4e6bf1d6	Feat/dify rag (#2528 ) Co-authored-by: jyong <jyong@dify.ai>	2 years ago

1 2 3 4 5

233 Commits (56e15d09a9e94de963dd5fc38c7455a7aaee99f3)