Problem with inference on video for `SingleImageVideo` backend

# Bug description

Inference cannot run on `SingleImageVideo` backend through GUI if select any options other than "Predict On:" `nothing`, `user labeled frames`, or `suggested frames`. The workaround is to "Predict On:" `nothing`, `user labeled frames`, or `suggested frames`.

## Expected behaviour

Inference would run and output predictions.

## Actual behaviour

Error during inference.

## Your personal set up



- OS:

- Version(s):

- SLEAP installation method (listed [here](https://sleap.ai/installation.html#)):
  - [x] [Conda from package](https://sleap.ai/installation.html#conda-package)
  - [x] [Conda from source](https://sleap.ai/installation.html#conda-from-source)
  - [x] [pip package](https://sleap.ai/installation.html#pip-package)
  - [x] [M1 Macs](https://sleap.ai/installation.html#m1-macs)

<details><summary>Environment packages</summary>


```
# paste output of `pip freeze` or `conda list` here
```

</details>

<details><summary>Logs</summary>


```
Epoch 34: ReduceLROnPlateau reducing learning rate to 3.12499992105586e-06.
213/213 - 89s - loss: 3.9481e-05 - val_loss: 3.9871e-05 - lr: 6.2500e-06 - 89s/epoch - 416ms/step
Epoch 35/200
Polling: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/viz/validation.*.png
213/213 - 88s - loss: 3.9465e-05 - val_loss: 3.9647e-05 - lr: 3.1250e-06 - 88s/epoch - 415ms/step
Epoch 35: early stopping
INFO:sleap.nn.training:Finished training loop. [62.7 min]
INFO:sleap.nn.training:Deleting visualization directory: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Polling: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/viz/validation.*.png
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2022-08-03 16:59:24.275291: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 3090" frequency: 1695 num_cores: 82 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22417833984 bandwidth: 936096000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 29.9 FPS
/home/lenovo/sleap/sleap/nn/evals.py:506: RuntimeWarning: Mean of empty slice
  "dist.avg": np.nanmean(dists),
/home/lenovo/sleap/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice.
  mPCK = mPCK_parts.mean()
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/lenovo/sleap/sleap/nn/evals.py:633: RuntimeWarning: Mean of empty slice.
  pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide
  ret, rcount, out=ret, casting='unsafe', subok=False)
/home/lenovo/sleap/sleap/nn/evals.py:635: RuntimeWarning: Mean of empty slice.
  metrics["oks.mOKS"] = pair_oks.mean()
WARNING:sleap.nn.evals:Failed to compute metrics.
INFO:sleap.nn.evals:Saved predictions: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/labels_pr.train.slp
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2022-08-03 16:59:52.833114: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 3090" frequency: 1695 num_cores: 82 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22417833984 bandwidth: 936096000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━  97% ETA: 0:00:01 33.3 FPS2022-08-03 16:59:55.902539: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 3090" frequency: 1695 num_cores: 82 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22417833984 bandwidth: 936096000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 23.6 FPS
/home/lenovo/sleap/sleap/nn/evals.py:506: RuntimeWarning: Mean of empty slice
  "dist.avg": np.nanmean(dists),
/home/lenovo/sleap/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice.
  mPCK = mPCK_parts.mean()
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/lenovo/sleap/sleap/nn/evals.py:633: RuntimeWarning: Mean of empty slice.
  pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide
  ret, rcount, out=ret, casting='unsafe', subok=False)
/home/lenovo/sleap/sleap/nn/evals.py:635: RuntimeWarning: Mean of empty slice.
  metrics["oks.mOKS"] = pair_oks.mean()
WARNING:sleap.nn.evals:Failed to compute metrics.
INFO:sleap.nn.evals:Saved predictions: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/labels_pr.val.slp
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947
Finished training single_instance.
Command line call:
sleap-track /home/lenovo/EstellaGuo/deeplabcut/deeplabcut-2.0/7points-Estella-resnet101/labeled-data/single-rat-exp/img00005.png --frames 912 -m /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947 -o /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/predictions/img00005.png.220803_165958.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2022-08-03 17:00:02.320557
Traceback (most recent call last):
  File "/home/lenovo/.conda/envs/sleap/bin/sleap-track", line 33, in <module>
    sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
  File "/home/lenovo/sleap/sleap/nn/inference.py", line 4285, in main
    provider, data_path = _make_provider_from_cli(args)
  File "/home/lenovo/sleap/sleap/nn/inference.py", line 4170, in _make_provider_from_cli
    filename=data_path, example_indices=frame_list(args.frames), **video_kwargs
  File "/home/lenovo/sleap/sleap/nn/data/providers.py", line 344, in from_filepath
    video = sleap.Video.from_filename(filename, **kwargs)
  File "/home/lenovo/sleap/sleap/io/video.py", line 1258, in from_filename
    raise ValueError("Could not detect backend for specified filename.")
ValueError: Could not detect backend for specified filename.
```

</details>

## Screenshots



## How to reproduce



1. Go to Inference Pipeline
2. Select "Predict On:" `entire current video`
3. Click Run button
4. See error

### Discussed in https://github.com/talmolab/sleap/discussions/887

<div type='discussions-op-text'>

<sup>Originally posted by **Estella23** August  3, 2022</sup>
After I imported a deeplabcut datasets, the training went well at first. After several epochs , it stopped automatically. 
I really don't know why this problem happened.
and the terminal : 

```
Epoch 34: ReduceLROnPlateau reducing learning rate to 3.12499992105586e-06.
213/213 - 89s - loss: 3.9481e-05 - val_loss: 3.9871e-05 - lr: 6.2500e-06 - 89s/epoch - 416ms/step
Epoch 35/200
Polling: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/viz/validation.*.png
213/213 - 88s - loss: 3.9465e-05 - val_loss: 3.9647e-05 - lr: 3.1250e-06 - 88s/epoch - 415ms/step
Epoch 35: early stopping
INFO:sleap.nn.training:Finished training loop. [62.7 min]
INFO:sleap.nn.training:Deleting visualization directory: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Polling: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/viz/validation.*.png
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2022-08-03 16:59:24.275291: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 3090" frequency: 1695 num_cores: 82 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22417833984 bandwidth: 936096000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 29.9 FPS
/home/lenovo/sleap/sleap/nn/evals.py:506: RuntimeWarning: Mean of empty slice
  "dist.avg": np.nanmean(dists),
/home/lenovo/sleap/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice.
  mPCK = mPCK_parts.mean()
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/lenovo/sleap/sleap/nn/evals.py:633: RuntimeWarning: Mean of empty slice.
  pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide
  ret, rcount, out=ret, casting='unsafe', subok=False)
/home/lenovo/sleap/sleap/nn/evals.py:635: RuntimeWarning: Mean of empty slice.
  metrics["oks.mOKS"] = pair_oks.mean()
WARNING:sleap.nn.evals:Failed to compute metrics.
INFO:sleap.nn.evals:Saved predictions: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/labels_pr.train.slp
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2022-08-03 16:59:52.833114: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 3090" frequency: 1695 num_cores: 82 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22417833984 bandwidth: 936096000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━  97% ETA: 0:00:01 33.3 FPS2022-08-03 16:59:55.902539: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -41 } dim { size: -42 } dim { size: -43 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -5 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 3090" frequency: 1695 num_cores: 82 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22417833984 bandwidth: 936096000 } outputs { dtype: DT_FLOAT shape { dim { size: -5 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 23.6 FPS
/home/lenovo/sleap/sleap/nn/evals.py:506: RuntimeWarning: Mean of empty slice
  "dist.avg": np.nanmean(dists),
/home/lenovo/sleap/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice.
  mPCK = mPCK_parts.mean()
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/lenovo/sleap/sleap/nn/evals.py:633: RuntimeWarning: Mean of empty slice.
  pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1)
/home/lenovo/.conda/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide
  ret, rcount, out=ret, casting='unsafe', subok=False)
/home/lenovo/sleap/sleap/nn/evals.py:635: RuntimeWarning: Mean of empty slice.
  metrics["oks.mOKS"] = pair_oks.mean()
WARNING:sleap.nn.evals:Failed to compute metrics.
INFO:sleap.nn.evals:Saved predictions: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947/labels_pr.val.slp
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947
Finished training single_instance.
Command line call:
sleap-track /home/lenovo/EstellaGuo/deeplabcut/deeplabcut-2.0/7points-Estella-resnet101/labeled-data/single-rat-exp/img00005.png --frames 912 -m /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/models/220803_155540.single_instance.n=947 -o /home/lenovo/EstellaGuo/Sleap/Sleap-seven-points-dlc/predictions/img00005.png.220803_165958.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2022-08-03 17:00:02.320557
Traceback (most recent call last):
  File "/home/lenovo/.conda/envs/sleap/bin/sleap-track", line 33, in <module>
    sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
  File "/home/lenovo/sleap/sleap/nn/inference.py", line 4285, in main
    provider, data_path = _make_provider_from_cli(args)
  File "/home/lenovo/sleap/sleap/nn/inference.py", line 4170, in _make_provider_from_cli
    filename=data_path, example_indices=frame_list(args.frames), **video_kwargs
  File "/home/lenovo/sleap/sleap/nn/data/providers.py", line 344, in from_filepath
    video = sleap.Video.from_filename(filename, **kwargs)
  File "/home/lenovo/sleap/sleap/io/video.py", line 1258, in from_filename
    raise ValueError("Could not detect backend for specified filename.")
ValueError: Could not detect backend for specified filename.
```</div>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Problem with inference on video for `SingleImageVideo` backend #891

Bug description

Expected behaviour

Actual behaviour

Your personal set up

Screenshots

How to reproduce

Discussed in #887

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Problem with inference on video for SingleImageVideo backend #891

Description

Bug description

Expected behaviour

Actual behaviour

Your personal set up

Screenshots

How to reproduce

Discussed in #887

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Problem with inference on video for `SingleImageVideo` backend #891