feat: add rerank top-k configuration and integrate into retrieval pro…

[SCMforever01]

…cess

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Screenshots

Before	After
...	...

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint and make type-check (backend) and cd web && npx lint-staged (frontend) to appease the lint gods

[gemini-code-assist]

Summary of Changes

Hello @SCMforever01, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the document retrieval system by implementing a 'rerank top-k relaxation' mechanism. The primary goal is to improve the quality of results when a reranking model is used. Instead of directly retrieving the final top_k documents, the system will now fetch a larger, configurable set of candidates. This expanded set is then passed to the reranker, which can more effectively identify the most relevant top_k documents from a broader pool, leading to potentially more accurate and comprehensive search outcomes.

Highlights

• Rerank Top-K Configuration: Introduced new configuration parameters, RERANK_TOPK_MULTIPLIER and RERANK_TOPK_CAP, in both .env.example and DataSetConfig. These settings control how the initial top_k is expanded when reranking is enabled, allowing for a broader set of documents to be considered before final selection.
• Expanded Retrieval for Reranking: The retrieval service now dynamically calculates an expanded_top_k based on the new configuration. When reranking is active and supported by the retrieval method, the system will fetch more documents than the originally requested top_k (up to the defined cap) to provide a richer pool for the reranker.
• Integration into Retrieval Process: The core _retrieve function has been updated to manage the original_top_k and expanded_top_k. Initial search calls (semantic, full-text, hybrid) now use the expanded_top_k, while the final reranking step correctly uses the original_top_k to select the ultimate set of documents.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a configurable top_k expansion for the reranking process, which is a valuable feature for improving retrieval quality. The implementation is well-structured, adding new configurations and correctly applying them to expand the number of candidates for reranking. I have a couple of suggestions to enhance code conciseness and maintainability. Overall, this is a good addition.

[gemini-code-assist]

This check can be written more concisely and scalably using all() with a generator expression. This is especially useful if more keys need to be checked in the future.

Suggested change

	enable_rerank = bool(
	reranking_model
	and reranking_model.get("reranking_provider_name")
	and reranking_model.get("reranking_model_name")
	)
	enable_rerank = bool(reranking_model and all(
	reranking_model.get(key) for key in [
	"reranking_provider_name", "reranking_model_name"
	]
	))

[gemini-code-assist]

To improve consistency and reuse existing logic, you can use the static methods RetrievalMethod.is_support_semantic_search and RetrievalMethod.is_support_fulltext_search to define support_expand. This makes the code more maintainable as the logic for supported methods is centralized in the RetrievalMethod enum, and it's consistent with how these checks are performed elsewhere in this function.

Suggested change

	support_expand = retrieval_method in {
	RetrievalMethod.SEMANTIC_SEARCH,
	RetrievalMethod.FULL_TEXT_SEARCH,
	RetrievalMethod.HYBRID_SEARCH,
	}
	support_expand = (
	RetrievalMethod.is_support_semantic_search(retrieval_method)
	or RetrievalMethod.is_support_fulltext_search(retrieval_method)
	)

[SCMforever01]

When a user sets top_k = 2, they naturally expect “the two most relevant chunks” to come back. But with rerank enabled, both the initial vector/full-text retrieval and the reranker are capped at 2 candidates. If the first-stage retrieval misses a semantically relevant chunk due to BM25/ts_rank quirks, it
never reaches the reranker, so the final results deviate from the user’s expectation of “top_k means the number of answers returned.” By expanding the first-stage top_k only when rerank is active—while still trimming the final output back to the original top_k—we feed the reranker a richer candidate pool,
improve accuracy, and keep the user-facing semantics of top_k intact.

[crazywoola]

Please link an issue in the description :)

[Thebinary110]

@QuantumGhost @crazywoola @SCMforever01 can I look into the issue too or Contribute as well.
@QuantumGhost @crazywoola can you assign an Issue or task to do on the main default subjugated ahead in commit branch.

[crazywoola]

Once @SCMforever01 create an issue, you can leave a message there, I will assign it to you as well.