Can't restore where blocks are missing #166
Description
Describe the bug
Can't restore invalid version after deep-scrub, where metadata/blocks are missing. Benji bails out with a file not found error, even though I've marked the blocks as invalid with a deep-scrub
INFO: Restoring 10100/80813 blocks (12.5%)
INFO: Restoring 10504/80813 blocks (13.0%)
ERROR: Storage backend read failed: Object metadata or data of block 13528 (UID 229a-34d9) not found.
WARNING: Job executor "Storage-Read" is being shutdown with 70112 outstanding jobs, cancelling them.
ERROR: InvalidBlockException: Object metadata or data of block 13528 (UID 229a-34d9) not found.
Traceback (most recent call last):
File "/srv/benji/lib/python3.11/site-packages/benji/storage/base.py", line 241, in _read
data = self._read_object(key)
^^^^^^^^^^^^^^^^^^^^^^
File "/srv/benji/lib/python3.11/site-packages/benji/storage/file.py", line 46, in _read_object
raise FileNotFoundError('File {} not found.'.format(filename))
FileNotFoundError: File /mnt/benji/blocks/6d/94/000000000000229a-00000000000034d9 not found.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/srv/benji/lib/python3.11/site-packages/benji/scripts/benji.py", line 357, in main
func(**func_args)
File "/srv/benji/lib/python3.11/site-packages/benji/commands.py", line 81, in restore
benji_obj.restore(version_uid_obj, destination, sparse, force)
File "/srv/benji/lib/python3.11/site-packages/benji/benji.py", line 656, in restore
raise entry
File "/srv/benji/lib/python3.11/site-packages/benji/jobexecutor.py", line 54, in get_completed
result = future.result()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/benji/lib/python3.11/site-packages/benji/jobexecutor.py", line 42, in execute_with_acquire
return function()
^^^^^^^^^^
File "/srv/benji/lib/python3.11/site-packages/benji/storage/base.py", line 279, in job
return self._read(block_deref, metadata_only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/benji/lib/python3.11/site-packages/benji/storage/base.py", line 249, in _read
raise InvalidBlockException(
benji.storage.base.InvalidBlockException: Object metadata or data of block 13528 (UID 229a-34d9) not found.
To Reproduce
Benji Version 0.17.0
Steps to reproduce the behavior.
- delete some blocks (not what we did, but you asked for reproduction steps)
- deep-scrub the backup
- try to restore
Expected behavior
Restore works, skipping over the missing blocks.