Open
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Once submit-job is called batch state is corrupted and can no longer be saved or loaded, even after localstack has been stopped and restarted.
Expected Behavior
submitting a job should not corrupt localstack batch state.
How are you starting LocalStack?
With the localstack script
Steps To Reproduce
How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)
localstack start
Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
This can be reproduced using the batch docs example found at https://docs.localstack.cloud/user-guide/aws/batch/
# create service role
awslocal iam create-role \
--role-name myrole \
--assume-role-policy-document "{}"
# create compute environment
awslocal batch create-compute-environment \
--compute-environment-name myenv \
--type UNMANAGED \
--service-role arn:aws:iam::000000000000:role/myrole
# create job-queue
awslocal batch create-job-queue \
--job-queue-name myqueue \
--priority 1 \
--compute-environment-order order=0,computeEnvironment=arn:aws:batch:us-east-1:000000000000:compute-environment/myenv \
--state ENABLED
# create job definition
awslocal batch register-job-definition \
--job-definition-name myjobdefn \
--type container \
--container-properties '{"image":"busybox","vcpus":1,"memory":128,"command":["sleep","30"]}'
At this point batch state is still fine with no issues. However after submitting a job, the next state save will generate an error
# submit batch job
awslocal batch submit-job \
--job-name myjob \
--job-queue myqueue \
--job-definition myjobdefn \
--container-overrides '{"command":["sh", "-c", "sleep 5; pwd"]}'
Now looking at the logs, the next time state is saved, the following error occurs
2025-01-02T18:44:39.669 DEBUG --- [read-4 (run)] l.p.c.persistence.manager : Serializing state of service batch
2025-01-02T18:44:39.671 ERROR --- [read-4 (run)] localstack.utils.functions : error calling function save
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/functions.py", line 42, in call_safe
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/persistence/manager.py.enc", line 12, in save
B.lifecycle_hook.on_before_state_save();B.accept_state_visitor(visitor);B.lifecycle_hook.on_after_state_save();LOG.info('Saving state for %s took %.0f ms',A,(time.perf_counter()-D)*1000)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/plugins.py", line 138, in accept_state_visitor
ReflectionStateLocator(service=self.name()).accept_state_visitor(visitor)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/state/inspect.py", line 99, in accept_state_visitor
visitor.visit(attribute)
File "/usr/local/lib/python3.11/functools.py", line 946, in _method
return method.__get__(obj, cls)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/persistence/snapshot/save.py.enc", line 19, in _
def _(self,state_container):A=state_container;B=os.path.join(self.data_dir,A.service_name,constants.MOTO_BACKEND_STATE_FILE);self._encode(A,B)
^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/persistence/snapshot/save.py.enc", line 24, in _encode
with open(file,'wb')as B:A.encoder.encode(state_container,B)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/state/pickle.py", line 240, in encode
return self.pickler_class(file).dump(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 394, in dump
StockPickler.dump(self, obj)
File "/usr/local/lib/python3.11/pickle.py", line 487, in dump
self.save(obj)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/lib/python3.11/pickle.py", line 713, in save_reduce
self._batch_setitems(dictitems)
File "/usr/local/lib/python3.11/pickle.py", line 1003, in _batch_setitems
save(v)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/lib/python3.11/pickle.py", line 713, in save_reduce
self._batch_setitems(dictitems)
File "/usr/local/lib/python3.11/pickle.py", line 998, in _batch_setitems
save(v)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/lib/python3.11/pickle.py", line 717, in save_reduce
save(state)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/patch.py", line 61, in proxy
return new(target, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/persistence.py.enc", line 17, in save_dict
return fn(self,A,*B,**C)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/lib/python3.11/pickle.py", line 998, in _batch_setitems
save(v)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/patch.py", line 61, in proxy
return new(target, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/persistence.py.enc", line 17, in save_dict
return fn(self,A,*B,**C)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/lib/python3.11/pickle.py", line 1003, in _batch_setitems
save(v)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/lib/python3.11/pickle.py", line 717, in save_reduce
save(state)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/patch.py", line 61, in proxy
return new(target, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/persistence.py.enc", line 17, in save_dict
return fn(self,A,*B,**C)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/lib/python3.11/pickle.py", line 998, in _batch_setitems
save(v)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/pickle.py", line 932, in save_list
self._batch_appends(obj)
File "/usr/local/lib/python3.11/pickle.py", line 959, in _batch_appends
save(tmp[0])
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/local/lib/python3.11/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/state/pickle.py", line 305, in _pickle
self.prepare(obj, state)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/batch/models.py.enc", line 11, in prepare
if A:A.seek(0);B[_B]=A.read()
^^^^^^^^^
io.UnsupportedOperation: underlying stream is not seekable
If you stop and restart localstack, there is now an issue with loading the batch state
2025-01-02T18:48:13.402 DEBUG --- [ady_monitor)] l.p.c.persistence.manager : Loading state of service batch
2025-01-02T18:48:13.425 ERROR --- [ady_monitor)] localstack.utils.functions : error calling function load
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/functions.py", line 42, in call_safe
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/persistence/manager.py.enc", line 16, in load
B.lifecycle_hook.on_before_state_load();B.accept_state_visitor(visitor);B.lifecycle_hook.on_after_state_load();LOG.info('Loading state for %s took %.0f ms',A,(time.perf_counter()-D)*1000)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/plugins.py", line 138, in accept_state_visitor
ReflectionStateLocator(service=self.name()).accept_state_visitor(visitor)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/state/inspect.py", line 99, in accept_state_visitor
visitor.visit(attribute)
File "/usr/local/lib/python3.11/functools.py", line 946, in _method
return method.__get__(obj, cls)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/persistence/snapshot/load.py.enc", line 23, in _
A=state_container;C=os.path.join(self.data_dir,A.service_name,MOTO_BACKEND_STATE_FILE);B=self._deserialize_file(C)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/persistence/snapshot/load.py.enc", line 33, in _deserialize_file
with open(A,'rb')as C:return B.decoder.decode(C)
^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/state/pickle.py", line 255, in decode
return self.unpickler_class(file).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/dill/_dill.py", line 419, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: Ran out of input
After getting to this point, you can't even reset the state using localstack state reset or localstack state reset --services=batch. The only way to recover from this is manually deleting the saved state in docker volume and starting over.
rm -rf /Users/efrench/Library/Caches/localstack/volume
Environment
- OS: macOS Sequoia 15.2
- LocalStack Pro:
LocalStack version: 4.0.3
LocalStack build date: 2024-11-29
LocalStack build git hash: 425d18c9b
LocalStack Docker image: localstack/localstack-pro:4.0.3
LocalStack Docker image sha: sha256:062e6335d102649dbb7c455be2bee492414c6ac9c64a5f4d9589b843037d4f7cAnything else?
No response