invoices/sql: fix full table scans on HTLC settlement hot path#10619
invoices/sql: fix full table scans on HTLC settlement hot path#10619ziggie1984 wants to merge 5 commits intolightningnetwork:masterfrom
Conversation
Remove four indexes from the invoices table that either duplicate existing UNIQUE constraint indexes or are never used in WHERE clauses: - invoices_hash_idx: redundant, UNIQUE constraint on hash already creates an implicit index - invoices_payment_addr_idx: redundant, UNIQUE constraint on payment_addr already creates an implicit index - invoices_preimage_idx: unused, preimage is NULL on all new invoices and is never used as a query filter - invoices_settled_at_idx: unused, settled_at is NULL on all pending invoices and is never used as a query filter (settle_index is used instead) Dropping these reduces B-tree working set size, which improves page cache utilization as the invoice table grows.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the performance of invoice lookups and updates, particularly in the HTLC settlement hot path. By moving away from a single, inefficient query that caused full table scans, and instead implementing targeted queries that utilize unique indexes, the system's database interaction for invoices becomes much more efficient. Additionally, the removal of unnecessary indexes further streamlines database operations, contributing to overall system health and speed. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
🔴 PR Severity: CRITICAL
🔴 Critical (5 files)
🟠 High (1 file)
⚪ Excluded (auto-generated) (1 file)
AnalysisThis PR adds a new SQL migration ( The migration drops indexes, which could have performance implications for queries against the invoice table. Reviewers should verify:
Severity bump check: 6 non-excluded files (< 20 threshold), 139 non-excluded lines changed (< 500 threshold) — no bump applied. To override, add a |
There was a problem hiding this comment.
Code Review
This pull request effectively addresses a performance bottleneck by replacing a generic, slow database query with more specific, indexed queries for fetching invoices. The removal of redundant and unused database indexes is also a good cleanup. The overall approach is sound. My review includes a couple of suggestions to improve error handling consistency and to restore a safety check for ambiguous invoice references that was present in the previous implementation.
The GetInvoice query used an OR IS NULL pattern for each filter
parameter:
WHERE (i.hash = $1 OR $1 IS NULL)
AND (i.payment_addr = $2 OR $2 IS NULL)
SQLite's query planner decides on an execution plan at prepare time,
before seeing any parameter values. Because either condition can be
trivially true when its parameter is NULL, the planner conservatively
falls back to a full table scan rather than using the unique indexes on
hash and payment_addr. This caused every invoice lookup and update on
the hot path (HTLC settlement) to scan the entire invoices table.
Replace the single catch-all query with three dedicated queries, each
using a direct equality on a uniquely constrained column:
- GetInvoiceByHash: WHERE hash = $1
- GetInvoiceByAddr: WHERE payment_addr = $1 (new, covers AMP path)
- GetInvoiceBySetID: existing, unchanged
Update getInvoiceByRef to route to the appropriate query based on which
fields are present in the InvoiceRef. When both hash and payment address
are provided, we look up by hash and then verify the returned invoice's
payment address matches.
GetInvoice is now unused and removed from the SQLInvoiceQueries
interface.
4ad19a9 to
3ace0b4
Compare
After upgrading Bob or Dave, the test only waited for the upgraded node's own channels to become active. However, for alice to route a payment to dave she needs her local channel to Bob to be active from her own perspective, which is only the case once the p2p connection is re-established.
Summary
Resulted from #10535, where we observed SQLITE slowdowns during performance testing.
Every invoice lookup and update on the HTLC settlement hot path was
causing a full table scan of the invoices table. The root cause was the
catch-all
GetInvoicequery, which used anOR $1 IS NULLpattern foreach filter parameter:
WHERE (i.hash = $1 OR $1 IS NULL)
AND (i.payment_addr = $2 OR $2 IS NULL)
SQLite's query planner resolves execution plans at prepare time, before
seeing actual parameter values. Because either condition can be trivially
true when its parameter is NULL, the planner conservatively falls back to
a full table scan rather than using the unique indexes on
hashandpayment_addr.This PR fixes the issue in two steps:
Replace the single catch-all query with three dedicated queries, each
filtering on a uniquely-constrained column:
GetInvoiceByHash: WHERE hash = $1GetInvoiceByAddr: WHERE payment_addr = $1 (new, covers AMP path)GetInvoiceBySetID: existing, unchangedDrop four redundant or unused indexes that were adding write overhead
and B-tree bloat without benefiting any query:
invoices_hash_idxandinvoices_payment_addr_idx: duplicated bythe UNIQUE constraints on those columns
invoices_preimage_idxandinvoices_settled_at_idx: never usedas query filters