I recently switched to a mouse which has an 8000hz polling rate, and this causes my game to drop to single digits FPS.

While one solution could be to get everything out of the main thread (which I will probably do in the end), this still seems unreasonable.

As a matter of fact it seems that GLFW uses GetRawInputData which is the cause of the issue, as it was previously an issue in SDL, [SDL #8756](see libsdl-org/SDL#8756).
The fix would be to use GetRawInputBuffer as per the previously mention issue in SDL.