🚢 Data Toolkit for Sailor Language Models
-
Updated
Feb 24, 2025 - Python
#
data-deduplication
Here are 7 public repositories matching this topic...
PolyDeDupe: Multi-Lingual Data Deduplication
-
Updated
Oct 27, 2025 - Python
Этот проект представляет собой мощный инструмент для поиска и анализа дублирующихся файлов в указанной директории. Программа позволяет эффективно выявлять одинаковые файлы на основе их содержимого, используя алгоритм хеширования SHA-256. Она поддерживает настройку параметров, таких как минимальный размер файла для проверки и игнорирование определен
python hashing productivity multithreading data-deduplication file-system sha256 file-management system-utility cli-tool dev-tools file-deduplication file-comparison disk-cleanup command-line-utility duplicate-file-finder
-
Updated
Feb 14, 2025 - Python
-
Updated
Jan 7, 2023 - Python
ETL workflow for stock data processing using Mage and PostgreSQL
python etl docker-compose postgresql data-deduplication data-engineering stock-market data-processing financial-data data-modeling data-cleaning data-aggregation api-integration alpha-vantage mage-ai
-
Updated
Jan 17, 2025 - Python
The HR Roster Change Detection Pipeline is an automated solution for processing HR roster data. Leveraging Apache Airflow and PostgreSQL, it enables seamless data ingestion, deduplication, and change detection, streamlining HR operations.
python open-source automation etl postgresql data-deduplication data-engineering data-pipelines apache-airflow roster-management workforce-analytics scalable-solutions hr-technology delta-detection hr-data-processing
-
Updated
Dec 4, 2024 - Python
A Python-based tool for preprocessing, cleaning, and analyzing text datasets, designed to filter, deduplicate, sort data, and generate statistical insights.
machine-learning natural-language-processing data-validation data-deduplication data-preprocessing data-sorting data-automation dataset-cleaning text-data-analysis dataset-boundaries data-statistics-generation
-
Updated
Oct 31, 2025 - Python
Improve this page
Add a description, image, and links to the data-deduplication topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-deduplication topic, visit your repo's landing page and select "manage topics."