Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat: Document-level GraphRAG — markdown nodes + cross-reference relationships #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
de-snake wants to merge 9 commits into
base: master
Choose a base branch
from
Open

feat: Document-level GraphRAG — markdown nodes + cross-reference relationships #39

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b5ce920
feat(graphrag): add References relationship type for document cross-l…
de-snake Mar 24, 2026
21224d0
feat(markdown): extract cross-reference links as imports for GraphRAG
de-snake Mar 24, 2026
0544c78
feat(indexer): include markdown files in GraphRAG pipeline
de-snake Mar 24, 2026
c3740d1
feat(graphrag): use References type for markdown cross-links
de-snake Mar 24, 2026
13d1335
fix(graphrag): include markdown files from document_blocks in build_f…
de-snake Mar 24, 2026
9d023bd
fix(graphrag): route markdown nodes through path-based import resolut…
de-snake Mar 25, 2026
ab83278
chore: remove debug logging from relationship discovery
de-snake Mar 26, 2026
69319e6
fix(store): concatenate all LanceDB batches when reading relationships
de-snake Mar 26, 2026
efd1ed7
perf(graphrag): deduplicate relationships on load (575K → 288K unique)
de-snake Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
perf(graphrag): deduplicate relationships on load (575K → 288K unique) 
Batch writes produce duplicate relationships across incremental flushes.
Added HashSet-based dedup by (source, target, type) triple when loading
relationships from database. Prints dedup stats when not in quiet mode.
  • Loading branch information
@de-snake
de-snake committed Mar 26, 2026
commit efd1ed76cf4e72f4ef0fc044e2b06795b60c62b0
31 changes: 23 additions & 8 deletions src/indexer/graphrag/database.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,23 +192,38 @@ impl<'a> DatabaseOperations<'a> {
let desc_array = extract_string_column(&rel_batch, "description")?;
let conf_array = extract_f32_column(&rel_batch, "confidence")?;

// Process each relationship
// Deduplicate relationships by (source, target, type) triple —
// batch writes can produce duplicates across incremental flushes
let mut seen = std::collections::HashSet::new();
let mut dedup_count = 0usize;
for i in 0..rel_batch.num_rows() {
let source = source_array.value(i);
let target = target_array.value(i);
let rel_type = type_array.value(i);
let key = (source.to_string(), target.to_string(), rel_type.to_string());
if !seen.insert(key) {
dedup_count += 1;
continue;
}
let relationship = CodeRelationship {
source: source_array.value(i).to_string(),
target: target_array.value(i).to_string(),
relation_type: type_array
.value(i)
source: source.to_string(),
target: target.to_string(),
relation_type: rel_type
.parse()
.unwrap_or(crate::indexer::graphrag::types::RelationType::Imports),
description: desc_array.value(i).to_string(),
confidence: conf_array.value(i),
weight: 1.0, // Default weight for legacy relationships
weight: 1.0,
};

// Add to graph
graph.relationships.push(relationship);
}
if dedup_count > 0 && !quiet {
println!(
" Deduplicated {} → {} unique relationships",
rel_batch.num_rows(),
graph.relationships.len()
);
}
}

if !graph.nodes.is_empty() && !quiet {
Expand Down
Loading