Add Microsoft.Extensions.DataRetrieval abstractions and pipeline

[luisquintanilla]

PR: Add Microsoft.Extensions.DataRetrieval packages

Resolves #7507

What this PR adds

Two new packages providing retrieval pipeline abstractions for RAG (Retrieval-Augmented Generation) applications:

Microsoft.Extensions.DataRetrieval.Abstractions

Thin abstraction layer defining the data types and processor contracts:

• RetrievalQuery — query with variant expansion and metadata
• RetrievalChunk — scored content chunk from vector search
• RetrievalResults — result container with pipeline metadata
• RetrievalQueryProcessor — abstract base for pre-search processors
• RetrievalResultProcessor — abstract base for post-search processors
• IReranker — interface for re-ranking strategies
• IRetriever — data-source-agnostic retrieval contract for DI and testability

Microsoft.Extensions.DataRetrieval

Pipeline implementation:

• RetrievalPipeline — orchestrates query processing → vector search → result processing
• VectorStoreRetriever<TKey, TRecord> — IRetriever implementation binding pipeline + collection
• RetrievalPipelineExtensions.AsRetriever() — convenience extension for pipeline → retriever conversion
• RetrievalPipelineOptions — configuration (ActivitySource name)
• Built-in: Reciprocal Rank Fusion for multi-query deduplication
• Built-in: Tree-traversal search paradigm for hierarchical indices
• OpenTelemetry tracing via System.Diagnostics.ActivitySource
• Structured logging via ILoggerFactory

Design

These packages are the read-side counterpart to Microsoft.Extensions.DataIngestion (the write-side). Together with Microsoft.Extensions.AI and Microsoft.Extensions.VectorData, they provide a complete composable RAG stack.

The pipeline follows the same "zero-cost when empty" philosophy as DataIngestion: with no processors registered, ProcessAsync performs a raw vector search. Each processor adds exactly one capability.

Dependencies

• Microsoft.Extensions.VectorData.Abstractions (for VectorStoreCollection<TKey, TRecord>)
• Microsoft.Extensions.Logging.Abstractions
• Microsoft.Extensions.Options

No dependency on Microsoft.Extensions.AI in the abstractions package — processor implementations bring their own AI client dependencies.

Testing

• Unit tests covering pipeline orchestration, RRF deduplication, tree traversal
• Processor contract tests (mock implementations)
• Integration verified against Qdrant, Azure AI Search, and in-memory vector stores via the advanced-rag reference application

API diff summary

+ Microsoft.Extensions.DataRetrieval.Abstractions (new package)
  + RetrievalQuery
  + RetrievalChunk
  + RetrievalResults
  + RetrievalQueryProcessor
  + RetrievalResultProcessor
  + IReranker
  + IRetriever

+ Microsoft.Extensions.DataRetrieval (new package)
  + RetrievalPipeline
  + RetrievalPipelineOptions
  + VectorStoreRetriever<TKey, TRecord>
  + RetrievalPipelineExtensions (AsRetriever)

Checklist

• XML documentation on all public types
• Exception documentation (<exception>) on throwing methods
• ActivitySource telemetry
• Structured logging
• README for both packages
• Follows existing DataIngestion packaging pattern
• Audited against dotnet/runtime Framework Design Guidelines

Microsoft Reviewers: Open in CodeFlow

[CZEMacLeod]

DataRetrieval is a kind of generic name/activity which may or may not indicate that this is AI related (RAG).
As it is directly under the very generic Microsoft.Extensions.* packages list, it would be nice if these sort of packages were better grouped/named.
Other areas/features that might match ME.DataRetrieval

• agnostic archived/'cold data retrieval (e.g. from blob storage, azure backup or another source).
• database or nosql storage systems
• (corrupt) filesystem recovery tools