ToggleRef exception: "Could not toggle reference to strong"

This was encountered in Pinta: https://bugs.launchpad.net/pinta/+bug/2072309
```
System.Exception: Could not toggle reference to strong. It is garbage collected.
   at GObject.Internal.ObjectMapper.ToggleRef.ToggleReference(IntPtr data, IntPtr object, Boolean isLastRef)
....
```