bf_tree migration away from diskann-providers by JordanMaples · Pull Request #1020 · microsoft/DiskANN

JordanMaples · 2026-05-05T18:45:45Z

Migrate bf_tree provider to dedicated crate and modernize

Motivation

The long-term goal is to remove diskann-providers entirely. The bf_tree provider was one of several components keeping that crate alive — it depended on diskann-providers for generic delete infrastructure (DeletionCheck/AsDeletionCheck/RemoveDeletedIdsAndCopy) and PQ quantization types. This PR extracts bf_tree into a standalone diskann-bf_tree crate, cuts those dependencies, simplifies the generics, and switches from PQ to spherical quantization — all to reduce the surface area of diskann-providers and move closer to removing it.

What Changed

1. Crate Extraction (`bf7d33fd`)

Moved the bf_tree provider from diskann-providers/src/model/graph/provider/async_/bf_tree/ into a standalone diskann-bf_tree crate. Updated workspace Cargo.toml and CI configuration.

2. PQ → Spherical Quantization (`74149bbc`)

QuantVectorProvider: Stores Arc<Poly<dyn Quantizer>> (spherical) instead of Arc<FixedChunkPQTable>.
QuantQueryComputer: Newtype wrapper adapting spherical's Opaque-based API to PreprocessedDistanceFunction<&[u8], f32>.
Serialization: Uses Quantizer::serialize / iface::try_deserialize (flatbuffers format).
CreateQuantProvider: Now implemented for Poly<dyn Quantizer> instead of FixedChunkPQTable.

3. Remove `D` (Delete Provider) Parameter (`14f504e8`)

BfTreeProvider<T, Q, D> → BfTreeProvider<T, Q>.
Hard deletes via bf_tree's native delete(key) API replace bitmap-based soft deletes.
delete() removes from all three trees (neighbors, full_vectors, quant_vectors).
status_by_internal_id() checks vector presence (Ok = valid, Err = deleted).
Removed delete_bitmap_serde.rs, NoDeletes/TableBasedDeletes from constructors.

4. Simplify to Core Strategies, Then Re-add (`02d355fb` → `c598d71d`)

First removed the old Hybrid/QuantAccessor/hybrid_computer code to start fresh, then re-added clean implementations:

QuantAccessor (e1ac1011): New accessor that reads quantized vectors from the bf_tree and uses QuantQueryComputer for distance. Implements all Quantized strategy traits.
Hybrid strategies with Rerank (c598d71d): Re-introduced Hybrid as a struct with max_fp_vecs_per_prune: Option<usize>. Rerank post-processor re-ranks quantized search results using full-precision vectors. Implements SearchPostProcess, SearchPreprocess, InsertPreprocess, and ExpandBeam for Hybrid.
Removed criterion/iai benchmarks (benches/ directory).

5. Use `CopyIds` Instead of `RemoveDeletedIdsAndCopy` (`8a3b6742`)

Since bf_tree uses hard deletes, deleted IDs never appear in neighbor lists. The AsDeletionCheck / DeletionCheck / RemoveDeletedIdsAndCopy machinery is unnecessary. Replaced with glue::CopyIds and removed the AsDeletionCheck impl from FullAccessor.

6. Move `DeletionCheck` Traits Back to `diskann-providers` (`38f1e614`)

These traits were temporarily moved into diskann/src/graph/glue.rs during the extraction as a stop-gap. Now that bf_tree no longer depends on them, they are restored to their original location in diskann-providers/src/model/graph/provider/async_/postprocess.rs with pub(crate) visibility.

7. Scope `diskann-providers` API Back to `pub(crate)` (`ee595e2c`)

Reverted methods on TableDeleteProviderAsync (is_deleted, delete, undelete, clear) from pub back to pub(crate). Removed unused RemoveDeletedIdsAndCopy re-export.

8. Tests and Delete-Panic Fix (`cbea923f`)

Bug fix: FullAccessor::on_elements_unordered and QuantAccessor::on_elements_unordered now silently skip deleted entries instead of propagating hard errors. bf_tree does hard deletes but graph edges still reference deleted nodes — when search traverses those edges, get_vector_into returns an ANNError (always-escalate), which panicked in expand_beam. The fix skips entries that fail to read.
7 new tests: test_quantized_index_search, test_quantized_index_multi_insert_search, test_quantized_delete_and_search, test_hybrid_index_search, test_hybrid_delete_and_search, test_full_precision_index_search, test_full_precision_delete_and_search.
Added doc comments for OwnedOpaque and indexing strategy descriptions on BfTreeProvider.

⚠️ Breaking Changes

Serialization format: Persisted indexes using the PQ format will not load. Full reindex required.
Training API: Callers must provide Poly<dyn Quantizer> instead of FixedChunkPQTable.
Constructor signatures: D parameter removed, no NoDeletes/TableBasedDeletes needed.
Delete semantics: Deletes are immediate and permanent (hard delete from bf_tree).

Net Impact

+230 net lines (+1461 / -1231) across the workspace
diskann-bf_tree is now a self-contained crate with two generic parameters (T, Q) instead of three
Three indexing strategies: FullPrecision, Quantized, Hybrid (with Rerank post-processor)
No dependency on DeletionCheck / AsDeletionCheck / RemoveDeletedIdsAndCopy
7 new integration-style tests covering all three strategies (insert, search, delete)
diskann-providers API surface reduced (methods scoped back to pub(crate))

codecov-commenter · 2026-05-05T21:17:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.52%. Comparing base (c7dfae6) to head (a1e5dbd).

Additional details and impacted files

@@            Coverage Diff            @@
##             main    #1020     +/-   ##
=========================================
  Coverage   89.51%   89.52%             
=========================================
  Files         461      476     +15     
  Lines       85920    90193   +4273     
=========================================
+ Hits        76912    80742   +3830     
- Misses       9008     9451    +443

Flag	Coverage Δ
miri	`89.52% <100.00%> (+<0.01%)`	⬆️
unittests	`89.16% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-bf_tree/src/provider/bf_tree_provider.rs	`91.45% <ø> (ø)`
diskann-bf_tree/src/provider/mod.rs	`0.00% <ø> (ø)`
diskann-bf_tree/src/provider/neighbor_provider.rs	`92.21% <ø> (ø)`
...kann-bf_tree/src/provider/quant_vector_provider.rs	`92.67% <100.00%> (ø)`
diskann-bf_tree/src/provider/vector_provider.rs	`93.44% <100.00%> (ø)`
...del/graph/provider/async_/table_delete_provider.rs	`97.41% <ø> (ø)`

... and 22 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Migrates the async bf_tree provider away from diskann-providers by extracting it into a dedicated diskann-bf_tree-provider crate, while also modernizing quantization (PQ → spherical) and simplifying deletion semantics (hard deletes).

Changes:

Extracts bf_tree + caching into diskann-bf_tree-provider and updates workspace/CI accordingly.
Replaces PQ-based quant vector store with spherical quantization (Poly<dyn Quantizer>) + updated serialization.
Simplifies delete handling by removing the delete-provider generic and switching to hard deletes.

Reviewed changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
diskann-providers/src/model/graph/provider/async_/table_delete_provider.rs	Makes delete-table operations public and removes bf_tree-specific bitmap (de)serialization helpers/tests.
diskann-providers/src/model/graph/provider/async_/mod.rs	Adjusts module exports, removes bf_tree/caching modules, re-exports postprocess utilities.
diskann-providers/src/model/graph/provider/async_/caching/utils.rs	Updates import path and minor formatting in tests.
diskann-providers/src/model/graph/provider/async_/caching/provider.rs	Refactors imports/bounds formatting; adds `Future` import.
diskann-providers/src/model/graph/provider/async_/caching/mod.rs	Removes caching module root from `diskann-providers`.
diskann-providers/src/model/graph/provider/async_/caching/example.rs	Updates test imports to use `diskann_providers` / local module paths.
diskann-providers/src/model/graph/provider/async_/caching/bf_cache.rs	Updates ConfigError import path and test formatting.
diskann-providers/src/model/graph/provider/async_/bf_tree/vector_provider.rs	Adds hard-delete helper and adjusts TestCallCount assertions.
diskann-providers/src/model/graph/provider/async_/bf_tree/quant_vector_provider.rs	Switches PQ → spherical quantization + new query/distance computer adapter + test updates.
diskann-providers/src/model/graph/provider/async_/bf_tree/provider.rs	Removes delete-provider generic, implements hard delete, switches postprocess to `CopyIds`, updates quantizer serialization.
diskann-providers/src/model/graph/provider/async_/bf_tree/neighbor_provider.rs	Updates TestCallCount import path.
diskann-providers/src/model/graph/provider/async_/bf_tree/mod.rs	Switches re-exports to new `bf_tree_provider` module and removes QuantAccessor re-export.
diskann-providers/Cargo.toml	Removes `bf_tree` feature/deps from `diskann-providers`.
diskann-bf_tree-provider/src/lib.rs	Introduces new crate entry point with provider + caching modules.
diskann-bf_tree-provider/src/caching/mod.rs	Adds caching module root (example tests disabled with TODO).
diskann-bf_tree-provider/Cargo.toml	Adds new crate manifest + dependencies for bf_tree provider and spherical quantization.
Cargo.toml	Adds `diskann-bf_tree-provider` to workspace members.
.github/workflows/ci.yml	Removes `bf_tree` from explicitly tested feature list.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

JordanMaples · 2026-05-12T21:08:44Z

@copilot would you mind giving this a review

Copilot · 2026-05-12T21:11:30Z

@copilot would you mind giving this a review

Reviewed the current branch at cbea923: cargo test -p diskann-bf_tree and cargo clippy -p diskann-bf_tree --all-targets -- -D warnings both pass locally, and CI run 25761279810 shows no failed jobs. I don’t see additional blockers right now.

Add bf_tree insert benchmarks (Criterion + IAI) Add pre-migration bf_tree provider benchmarks mirroring the existing inmem insert benchmark. Both Criterion (wall-clock) and IAI-Callgrind (instruction-count) benchmarks use the same sift-256 dataset and PQ training as the inmem benchmarks for apples-to-apples comparison. Gated behind #[cfg(feature = "bf_tree")] so existing benchmarks continue to work without the feature flag. Run with: cargo bench --bench bench_main --features virtual_storage,bf_tree Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Add bf_tree search benchmarks for migration parity Add search benchmarks (Criterion + IAI-Callgrind) alongside existing insert benchmarks. Search exercises the read path through BfTreeProvider (vector lookups, neighbor access, greedy search traversal) which is critical for detecting regressions post-migration. The search benchmark builds the index once during setup, then measures 10 KNN queries (k=10, l=20) against the populated index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Extract bf_tree provider into diskann-bf_tree-provider crate Move the bf_tree provider and caching modules from diskann-providers into the dedicated diskann-bf_tree-provider crate. This makes the bf-tree dependency explicit at the crate level rather than hidden behind a feature flag. Changes: - Move bf_tree/ and caching/ modules to diskann-bf_tree-provider/src/ - Set up Cargo.toml with proper dependencies (bf-tree, diskann, diskann-providers, diskann-utils, diskann-vector, etc.) - Convert all crate:: imports to diskann_providers:: imports - Bump visibility of postprocess module, AsDeletionCheck/DeletionCheck traits, and TableDeleteProviderAsync methods to pub - Remove bf_tree feature flag and optional deps from diskann-providers - Remove cfg(feature = "bf_tree") gates from table_delete_provider.rs - Rename provider/provider.rs to provider/bf_tree_provider.rs (clippy) - Disable caching example.rs tests (cross-crate cfg(test) limitation) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Move bf_tree benchmarks to diskann-bf_tree-provider Migrate Criterion and IAI-Callgrind benchmark targets from diskann-providers to the new diskann-bf_tree-provider crate. Update imports to use the new crate's module paths. Remove bf_tree benchmark remnants (cfg gates, modules) from diskann-providers benches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> cargo.lock got missed by last commit Move delete bitmap serialization to diskann-bf_tree-provider Move to_bytes/from_bytes logic out of TableDeleteProviderAsync (where it was bf_tree-scoped) into a dedicated delete_bitmap_serde module in the new crate. The reimplementation works through the public API and produces byte-identical output (LE u32 words). Revert the methods in diskann-providers to pub(crate) + #[cfg(test)]. Adds format-level compatibility tests including known fixtures and padding bit rejection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Move DeletionCheck traits and RemoveDeletedIdsAndCopy to diskann::graph::glue These are core to the inplace delete algorithm's search post-processing and belong in the diskann crate alongside SearchPostProcess, CopyIds, FilterStartPoints, and Pipeline. This removes the need to widen diskann-providers' postprocess module to pub for external crates. Also removes the now-dead to_bytes/from_bytes methods and their tests from TableDeleteProviderAsync in diskann-providers, since that serialization logic has been moved to diskann-bf_tree-provider. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Use workspace_root() for benchmark test data paths Replace env!("CARGO_MANIFEST_DIR")-relative paths with diskann_utils::workspace_root() so benchmarks find test_data/ without requiring symlinks in each crate directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The bf_tree feature was removed from diskann-providers when the provider was extracted into its own crate (diskann-bf_tree-provider). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Rewrite QuantVectorProvider to use Arc<Poly<dyn Quantizer>> - Add local HybridComputer and QuantQueryComputer wrapper - Update CreateQuantProvider impl for Poly<dyn Quantizer> - Update serialization to use Quantizer::serialize/try_deserialize - Rewrite all tests and doc examples for spherical quantizer - Update benchmarks to use SphericalQuantizer::train with ReciprocalMeanNorm - Move rand to regular dependencies (needed by doc tests) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Delete HybridAccessor, HybridComputer, and all Hybrid strategy impls (Search, PostProcess, Prune, Insert, MultiInsert, InplaceDelete) - Delete Rerank post-processor (only used by Hybrid) - Remove QuantAccessor (only constructed by Hybrid search path) - Move QuantQueryComputer from hybrid_computer.rs to quant_vector_provider.rs - Delete hybrid_computer.rs - Remove metric field from QuantVectorProvider (dead after Hybrid removal) - Add SaveWith/LoadWith serialization impls back (accidentally lost during D removal) - Fix test compilation: remove deleted_ids references, fix turbofish generics, insert vectors before status checks, skip hard-deleted IDs in verification loops - Update doc examples to remove NoDeletes/TableBasedDeletes from constructors - Clean up unused imports (distances::pq, Hybrid, DistanceFunction, etc.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ree impl now uses hard deletion)

Restore postprocess.rs in diskann-providers with AsDeletionCheck, DeletionCheck, and RemoveDeletedIdsAndCopy. These were temporarily moved into diskann/src/graph/glue.rs during bf_tree extraction but belong alongside the inmem providers that use them. - Create diskann-providers/src/model/graph/provider/async_/postprocess.rs - Remove traits and struct from diskann/src/graph/glue.rs - Update all inmem provider imports to use local postprocess module - Update DeletionCheck impls on NoDeletes and TableDeleteProviderAsync Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Restore QuantAccessor with Opaque-based element types for spherical quantization compatibility - Add Quantized strategy impls: SearchStrategy, DefaultPostProcessor, PruneStrategy, InsertStrategy, MultiInsertStrategy, InplaceDeleteStrategy - Use PassThrough working set pattern (matching inmem spherical approach) - Add OwnedOpaque newtype for workingset::View (bf_tree copies vs zero-copy) - Add BuildDistanceComputer impl using UnwrapErr<DistanceComputer> - Remove #[cfg(test)] from get_vector_into and get_vector_sync Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Hybrid searches in quantized space (same as Quantized), then reranks final candidates using full-precision distances. Pruning stays in quantized space. This replaces the old HybridComputer with its 4 distance modes (full×full, quant×quant, cross-type) with a simpler search-quant → rerank-full → prune-quant pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reverts accidental pub visibility on TableDeleteProviderAsync methods (is_deleted, delete, undelete, clear) and removes unused RemoveDeletedIdsAndCopy re-export. Nothing outside diskann-providers uses these, so widening the public API was unnecessary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Fix FullAccessor::on_elements_unordered to silently skip deleted entries - Fix QuantAccessor::on_elements_unordered to silently skip deleted entries - Add create_quant_index and create_full_precision_index test helpers - Add tests: quantized insert+search, multi_insert, delete+search - Add tests: hybrid insert+search, hybrid delete+search - Add tests: full precision insert+search, full precision delete+search - Add doc comments for OwnedOpaque, strategy descriptions, and BfTreeProvider Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rerank::post_process used .expect() on get_vector_sync, which could panic if a concurrent delete removed a vector between the quantized search phase and the reranking phase. Replace with .filter_map() to silently skip unreadable entries, consistent with the pattern used in QuantAccessor::on_elements_unordered and FullAccessor. Caught during local code review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

JordanMaples force-pushed the jordanmaples/bf_tree_migration branch from b94f4df to 44be934 Compare May 5, 2026 20:58

hildebrandmw reviewed May 8, 2026

View reviewed changes

Comment thread diskann-bf_tree-provider/src/provider/delete_bitmap_serde.rs Outdated

Comment thread diskann-bf_tree-provider/src/provider/quant_vector_provider.rs Outdated

JordanMaples force-pushed the jordanmaples/bf_tree_migration branch from ca413d4 to 1051152 Compare May 11, 2026 18:26

JordanMaples marked this pull request as ready for review May 11, 2026 21:00

JordanMaples requested review from a team and Copilot May 11, 2026 21:00

Copilot AI reviewed May 11, 2026

View reviewed changes

Copilot started reviewing on behalf of JordanMaples May 11, 2026 21:29 View session

JordanMaples marked this pull request as draft May 11, 2026 22:51

JordanMaples linked an issue May 12, 2026 that may be closed by this pull request

Extract bftree provider into its own crate with appropriate tests. #971

Open

JordanMaples force-pushed the jordanmaples/bf_tree_migration branch from 1219bcc to cbea923 Compare May 12, 2026 20:47

JordanMaples marked this pull request as ready for review May 12, 2026 20:49

Copilot started work on behalf of JordanMaples May 12, 2026 21:08 View session

Copilot finished work on behalf of JordanMaples May 12, 2026 21:12

JordanMaples and others added 13 commits May 12, 2026 14:58

Remove bf_tree from CI feature list

0c590c8

The bf_tree feature was removed from diskann-providers when the provider was extracted into its own crate (diskann-bf_tree-provider). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fmt

566c7a9

fix cargo.lock

22df656

move product logic from using pq to spherical

6936ab6

remove D from BfTreeProvider

72f1497

fmt

d17f06e

remove RemoveDeletedIdsAndCopy and use CopyIds instead since the bf_t…

d4f53ba

…ree impl now uses hard deletion)

cleanup

f3ee105

JordanMaples and others added 4 commits May 12, 2026 14:58

JordanMaples force-pushed the jordanmaples/bf_tree_migration branch from 0ad39b8 to 83e6231 Compare May 12, 2026 21:59

bump

a1e5dbd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bf_tree migration away from diskann-providers#1020

bf_tree migration away from diskann-providers#1020
JordanMaples wants to merge 18 commits into
mainfrom
jordanmaples/bf_tree_migration

JordanMaples commented May 5, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JordanMaples commented May 12, 2026

Uh oh!

Copilot AI commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

JordanMaples commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migrate bf_tree provider to dedicated crate and modernize

Motivation

What Changed

1. Crate Extraction (bf7d33fd)

2. PQ → Spherical Quantization (74149bbc)

3. Remove D (Delete Provider) Parameter (14f504e8)

4. Simplify to Core Strategies, Then Re-add (02d355fb → c598d71d)

5. Use CopyIds Instead of RemoveDeletedIdsAndCopy (8a3b6742)

6. Move DeletionCheck Traits Back to diskann-providers (38f1e614)

7. Scope diskann-providers API Back to pub(crate) (ee595e2c)

8. Tests and Delete-Panic Fix (cbea923f)

⚠️ Breaking Changes

Net Impact

Uh oh!

codecov-commenter commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JordanMaples commented May 12, 2026

Uh oh!

Copilot AI commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JordanMaples commented May 5, 2026 •

edited

Loading

1. Crate Extraction (`bf7d33fd`)

2. PQ → Spherical Quantization (`74149bbc`)

3. Remove `D` (Delete Provider) Parameter (`14f504e8`)

4. Simplify to Core Strategies, Then Re-add (`02d355fb` → `c598d71d`)

5. Use `CopyIds` Instead of `RemoveDeletedIdsAndCopy` (`8a3b6742`)

6. Move `DeletionCheck` Traits Back to `diskann-providers` (`38f1e614`)

7. Scope `diskann-providers` API Back to `pub(crate)` (`ee595e2c`)

8. Tests and Delete-Panic Fix (`cbea923f`)

codecov-commenter commented May 5, 2026 •

edited

Loading