Support crash-tolerant crawls #7978

JOBDIR only survives a clean stop. On a crash, a double Ctrl-C or a power loss you can lose the whole scheduler queue or corrupt it, lose every in-flight request, and lose or overwrite exported items.

I do not think we should support crash-tolerant crawls by default: crash tolerance means a durable write on every state change plus tracking every in-flight request and item, and most crawls have no reason to pay for that. But it comes up often enough that I think we should make it possible to opt into, with a documented guarantee: after the process dies for any reason, re-running the same job loses no data.

What I think it takes:

• Crash-safe scheduler queues. Support SQLite-backed queues #7877 is the foundation here.
• A crash-safe duplicate filter.
• Durable scheduler metadata. active.json is only written at close.
• Tracking in-flight requests, i.e. those in the downloader or being parsed, so that they are rescheduled on resume.
• Tracking in-flight items until every feed they belong to has flushed them.
• Resumable feeds: batch IDs surviving a resume, and partially written files being truncated to the last complete record or continued. Maybe limited to specific storages.
• A durable spider.state.

I think SQLite is the right disk storage approach in most cases.

In-flight requests and items would be recovered as inputs, not as partial progress: on resume they go through the whole middleware and pipeline chain again, like a retry does. So this requires idempotent item pipelines, and that is on the user. I would rather document that than try to checkpoint mid-chain.

Since enabling it means lining up a matching scheduler, queues, duplicate filter, feed settings and extensions, I would also ship an add-on that configures the whole set, and document that as the way to turn it on.

What I am unsure about:

• Which feed storages and formats can be made recoverable at all. Appending to a local .jl file is easy; a single JSON array, a compressed stream or a multipart upload much less so. Some may just be unsupported.
• Same for the files and images pipelines: a half-written file in the store is indistinguishable from a complete one.
• The durability granularity. One fsync per request is safe and probably unusable; a bounded window of N requests or T seconds is cheaper, but then the guarantee is no longer zero data loss.
• How to test this. Killing a real process at many different points seems like the only honest way, and that is a new kind of test for us.
• Whether the HTTP cache stays the better answer for people who only want to avoid re-downloading after a crash.

Related:

• Scrapy resume last batch after crash #2399 — resuming after a crash
• Persistence is not fully persistent when using JOBDIR setting #4106 — the disk queue is only written on a clean close, and the memory queue never is
• Spider Unhandled Error : f.seek(-size-self.SIZE_SIZE, os.SEEK_END) exceptions.IOError #845, The size of requests.queue may be wrong when resuming crawl from unclean shutdown. #3333, Resuming crawls gives the error: self.size, = struct.unpack(self.SIZE_FORMAT, qsize) #3413, PickleFifoDiskQueue and FifoMemoryQueue throw EOF when used to persist a crawl #1346 — corrupt or wrong-sized requests.queue after an unclean shutdown
• Restarting a crawl using with JOBDIR overwrites feed files that use batch_id #5153 — resuming overwrites %(batch_id)s feed files
• Exported item batches only finalized/uploaded at spider closing #7730 — feed batches only finalized at close
• Allow to stop Crawler non-gracefully, without completing all requests in the Downloader #4749 — a public API for non-graceful stops
• The spider doesn't wait for pipelines of items produced from start() #7029 — items from start() are not awaited through pipelines
• dbm-based cache can become corrupted when spider is killed #500, Download Middleware HTTPCache pickle error #3295 — the dbm HTTP cache backend gets corrupted when the spider is killed
• Add line buffering to file requests.seen of RFPDupeFilter #6019 — durability of requests.seen
• Sitemap spider does not resume though JOBDIR is set #4479 — SitemapSpider does not resume
• General Message Queues as Storage for Requests #4326, Improve support for external request queues #4783 — external request queues
• Request priority not guaranteed with the default scheduler configurations due to different queues (memory/disk) used. #3666 — priority is not honored across the memory and disk queues