Run metrics are empty for runs copied via aim runs cp #2255
Description
🐛 Bug
Aim run metrics at http://localhost:43800/runs/*/metrics show the metric names, but their plot contents are empty after running aim runs cp to existing repo destination.
To reproduce
mkdir new_repo && cd new_repo
aim init
cd ../old_repo
aim runs cp $(aim runs ls | head -n -1) --destination=../new_repo
cd ../new_repo
aim upNavigate to http://localhost:43800/runs/e4e6d4d5e5c59c69f3bd7be2/metrics
Also, just for fun:
import aim
from aim.storage.context import Context
repo = aim.Repo(".")
run = repo.get_run("e4e6d4d5e5c59c69f3bd7be2")
context = Context({"loader": "infer", "scope": "epoch"})
m = run.get_metric("loss", context)
print(m)Output for source repo:
<Sequence#7942733726269182380 name=`loss` context=`<Context#5740049561794408557 {'loader': 'infer', 'scope': 'epoch'}>` run=`<Run#2410547331679290040 name=e4e6d4d5e5c59c69f3bd7be2 repo=<Repo#3870764907936859003 path=./.aim read_only=None>>`>Output for destination repo:
NoneExpected behavior
Non-empty metrics.
Environment
- Aim Version: 3.13.2
- Python version: 3.8
- pip version: 22.1
- OS: Linux
- Any other relevant information
Additional context
The .aim/*/chunks/$RUN_HASH directories are not actually copied exactly. The destination repo is much smaller than the source repo.
Source repo:
λ ls **/e4e6d4d5e5c59c69f3bd7be2 -l
0 2022-09-21 04:58:00 meta/locks/e4e6d4d5e5c59c69f3bd7be2
0 2022-09-21 04:57:57 seqs/locks/e4e6d4d5e5c59c69f3bd7be2
0 2022-09-21 04:57:57 seqs/progress/e4e6d4d5e5c59c69f3bd7be2
meta/chunks/e4e6d4d5e5c59c69f3bd7be2:
total 15M
14K 2022-09-21 04:58:01 003135.sst
29K 2022-09-21 04:58:01 003137.sst
7.6K 2022-09-21 04:58:01 003139.sst
7.6K 2022-09-21 04:58:01 003141.sst
7.6K 2022-09-21 04:58:01 003143.sst
0 2022-09-21 04:58:01 003144.log
30K 2022-09-21 04:58:01 003145.sst
16 2022-09-21 04:58:01 CURRENT
36 2022-09-21 04:58:01 IDENTITY
0 2022-09-21 04:58:01 LOCK
14M 2022-09-21 04:58:01 LOG
425K 2022-09-21 04:58:01 MANIFEST-000004
6.4K 2022-09-21 04:58:01 OPTIONS-000007
seqs/chunks/e4e6d4d5e5c59c69f3bd7be2:
total 96M
58M 2022-09-21 04:58:00 011941.sst
265K 2022-09-21 04:58:00 011943.sst
265K 2022-09-21 04:58:00 011945.sst
262K 2022-09-21 04:58:00 011947.sst
260K 2022-09-21 04:58:00 011949.sst
262K 2022-09-21 04:58:00 011951.sst
261K 2022-09-21 04:58:00 011953.sst
267K 2022-09-21 04:58:00 011955.sst
269K 2022-09-21 04:58:00 011957.sst
0 2022-09-21 04:58:00 011959.log
31K 2022-09-21 04:58:00 011960.sst
16 2022-09-21 04:58:00 CURRENT
36 2022-09-21 04:58:00 IDENTITY
0 2022-09-21 04:58:00 LOCK
35M 2022-09-21 04:58:00 LOG
1.6M 2022-09-21 04:58:00 MANIFEST-000004
6.4K 2022-09-21 04:58:00 OPTIONS-000007Destination repo:
λ ls **/e4e6d4d5e5c59c69f3bd7be2 -l
0 2022-10-07 03:40:33 seqs/progress/e4e6d4d5e5c59c69f3bd7be2
meta/chunks/e4e6d4d5e5c59c69f3bd7be2:
total 64K
0 2022-10-07 03:40:33 000008.log
16K 2022-10-07 03:40:33 000009.sst
16 2022-10-07 03:40:33 CURRENT
36 2022-10-07 03:40:33 IDENTITY
0 2022-10-07 03:40:33 LOCK
28K 2022-10-07 03:40:33 LOG
218 2022-10-07 03:40:33 MANIFEST-000004
6.4K 2022-10-07 03:40:33 OPTIONS-000007
seqs/chunks/e4e6d4d5e5c59c69f3bd7be2:
total 2.2M
0 2022-10-07 03:40:33 000008.log
2.2M 2022-10-07 03:40:34 000009.sst
16 2022-10-07 03:40:33 CURRENT
36 2022-10-07 03:40:33 IDENTITY
0 2022-10-07 03:40:33 LOCK
24K 2022-10-07 03:40:34 LOG
222 2022-10-07 03:40:34 MANIFEST-000004
6.4K 2022-10-07 03:40:33 OPTIONS-000007Possibly related: #2188
Workaround
rsync -ah --delete old_repo/.aim/meta/chunks/e4e6d4d5e5c59c69f3bd7be2/ new_repo/.aim/meta/chunks/e4e6d4d5e5c59c69f3bd7be2/
rsync -ah --delete old_repo/.aim/seqs/chunks/e4e6d4d5e5c59c69f3bd7be2/ new_repo/.aim/seqs/chunks/e4e6d4d5e5c59c69f3bd7be2/This workaround mostly works, except there are a few occasional runs whose metrics are still unreadable, though the Runs explorer (i.e. meta/) seems to be fine... weird.