feat: add TF-IDF weighting for name matching (#597) #689
Conversation
Summary of ChangesHello @MorganaFuture, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the name matching accuracy by integrating TF-IDF (Term Frequency-Inverse Document Frequency) weighting. This approach assigns higher importance to rare and distinctive terms in entity names while down-weighting common business terms like "Trading" or "Company". The goal is to reduce false positives in search results and improve the ranking of truly unique matches, leading to more precise and relevant search outcomes. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces TF-IDF weighting for name matching, which is a great enhancement for improving search accuracy. The implementation is well-structured within a new internal/tfidf package and is cleanly integrated into the existing search pipeline. The addition of comprehensive unit and integration tests, including those using real OFAC data, is particularly commendable and provides high confidence in the new functionality. My main feedback is a suggestion to refactor some duplicated logic in the string scoring functions to improve long-term maintainability.
This comment was marked as off-topic.
This comment was marked as off-topic.
Implement TF-IDF (Term Frequency - Inverse Document Frequency) weighting to improve name matching accuracy. Rare terms now receive higher weight than common business terms like "Trading", "Ltd", "Company". Key changes: - Add internal/tfidf package with Index, Config, and weighted scoring - Build TF-IDF index from entity names during data refresh - Add BestPairsJaroWinklerWeighted() for TF-IDF weighted scoring - Thread TF-IDF index through search service and similarity functions This helps reduce false positives for common names and improves ranking for distinctive/rare names in search results.
6ead602 to
29de083
Compare
Implement TF-IDF (Term Frequency - Inverse Document Frequency) weighting to improve name matching accuracy. Rare terms now receive higher weight than common business terms like "Trading", "Ltd", "Company".
Key changes:
This helps reduce false positives for common names and improves ranking for distinctive/rare names in search results.