Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Jul 25, 2025

• update packages

  acidus99

  committed
  69093ab

  Copy full SHA for 69093ab

• Actually display invalid rules in lint tool

  acidus99

  committed
  79749aa

  Copy full SHA for 79749aa

• Add ignore rules for repeated crawling

  acidus99

  committed
  cca82de

  Copy full SHA for cca82de

• Update block-list.txt

  acidus99

  committed
  c2ac70f

  Copy full SHA for c2ac70f

• Keep processing even if WARC is truncated

  acidus99

  committed
  7b83cd0

  Copy full SHA for 7b83cd0

• bug: not processing global rules

  acidus99

  committed
  19f5e35

  Copy full SHA for 19f5e35

Commits on Jan 2, 2025

• Merge pull request #12 from acidus99/robots-tool

  Show description for 1bc232d
  acidus99

  authored
  1bc232d

  Copy full SHA for 1bc232d

• robots.txt testing tool

  acidus99

  committed
  fda233a

  Copy full SHA for fda233a

• prepping foer Robots.txt Checking Tool

  acidus99

  committed
  dc13e26

  Copy full SHA for dc13e26

Commits on Dec 20, 2024

• Bug Fix: unhandled null with improper input

  acidus99

  committed
  924be0d

  Copy full SHA for 924be0d

Commits on Dec 19, 2024

• Delete appsettings.PROD.json

  acidus99

  committed
  f5d6bc6

  Copy full SHA for f5d6bc6

• updating packages

  acidus99

  committed
  adb4e7c

  Copy full SHA for adb4e7c

Commits on Dec 18, 2024

• hack around malformed robots.txt

  acidus99

  committed
  2a2e35f

  Copy full SHA for 2a2e35f

Commits on Dec 14, 2024

• adding Url Testing tool

  acidus99

  committed
  2a38771

  Copy full SHA for 2a38771

• fix bug

  acidus99

  committed
  91a181f

  Copy full SHA for 91a181f

• reorder

  acidus99

  committed
  1c52cff

  Copy full SHA for 1c52cff

Commits on Dec 7, 2024

• Use new certificate utils

  acidus99

  committed
  9b96fb1

  Copy full SHA for 9b96fb1

Commits on Nov 10, 2024

• update to 8.0

  acidus99

  committed
  a8edabc

  Copy full SHA for a8edabc

• musicbrainz

  acidus99

  committed
  210cd7b

  Copy full SHA for 210cd7b

• Rider exclude

  acidus99

  committed
  9cb83ee

  Copy full SHA for 9cb83ee

Commits on Sep 1, 2024

• Move URL draining to workers, check for robots and connectivity

  acidus99

  committed
  f9438af

  Copy full SHA for f9438af

• ability to check robots.txt files offline

  acidus99

  committed
  273145b

  Copy full SHA for 273145b

• spacing

  acidus99

  committed
  e2af816

  Copy full SHA for e2af816

• Update block-list.txt

  acidus99

  committed
  7b847c4

  Copy full SHA for 7b847c4

Commits on Aug 31, 2024

• check if URL already seen before adding seed.

  Show description for a6a8562
  acidus99

  committed
  a6a8562

  Copy full SHA for a6a8562

Commits on Aug 25, 2024

• Seed URLS should checked against ROBOTS, because of how we add them

  acidus99

  committed
  7a7a80f

  Copy full SHA for 7a7a80f

• update dependency

  acidus99

  committed
  cd44ae7

  Copy full SHA for cd44ae7

Commits on Aug 18, 2024

• Further simplify priority used for probable non-text URLS

  acidus99

  committed
  1117cf2

  Copy full SHA for 1117cf2

• Log whether Seeds are added or not

  acidus99

  committed
  09e8406

  Copy full SHA for 09e8406

• bug fix - response log not properly flushed/closed

  acidus99

  committed
  8e04fa3

  Copy full SHA for 8e04fa3

Commits on Jul 9, 2024

• big update!

  Show description for 11f7225
  acidus99

  committed
  11f7225

  Copy full SHA for 11f7225

Commits on Jul 8, 2024

• Update block-list.txt

  acidus99

  committed
  e53a4e4

  Copy full SHA for e53a4e4

Commits on May 29, 2024

• Optimize indexing for meta data and FTS

  acidus99

  committed
  52425ae

  Copy full SHA for 52425ae

• track proactive requests and only non-proactive requests are indexable

  acidus99

  committed
  97ba94c

  Copy full SHA for 97ba94c

Commits on May 27, 2024

• Speed optimization - batch work

  acidus99

  committed
  5863854

  Copy full SHA for 5863854

Pagination

Previous

Next