Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@sjzar
Copy link
Owner

@sjzar sjzar commented Nov 16, 2023

  • 新增了 dumper 的并发读取功能。
  • ips dumpips pack 命令现在支持 reader_jobs 参数,可用于调整读取操作的并发数。
  • 实现了基于数据密度的 IP 数据库分段功能,有效解决了不同 IP 段数据密度差异的问题。
  • 更新了 IPv6 正则匹配规则,移除 IPv6 Link-local 规格的匹配。

示例

读取 GeoLite2-City.mmdb 数据,重新打包生成 ipdb 数据库,1 个并发与 8 个并发的耗时对比,平台是 MBP M1 Max (8 cores)

$ time ./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 1
./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 1  596.91s user 54.99s system 132% cpu 8:13.25 total

$ time ./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 8
./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 8  698.84s user 25.46s system 777% cpu 1:33.19 total

  • Added concurrent reading functionality to the dumper.
  • The ips dump and ips pack commands have been enhanced to support the reader_jobs parameter, allowing adjustment of the concurrency level for reading operations.
  • Implemented segmentation of the IP database based on data density, effectively addressing the issue of data density variations across different IP segments.
  • Updated IPv6 regular matching rules to remove IPv6 Link-local specification matching.

Example

Reading GeoLite2-City.mmdb data and repackaging it into an ipdb database, comparing the time taken with 1 and 8 concurrent jobs, the platform used is MBP M1 Max (8 cores)

$ time ./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 1
./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 1  596.91s user 54.99s system 132% cpu 8:13.25 total

$ time ./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 8
./ips pack -i ~/.ips/GeoLite2-City.mmdb -o 1.ipdb --reader-jobs 8  698.84s user 25.46s system 777% cpu 1:33.19 total

@sjzar sjzar merged commit e6b2afe into main Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants