Moved mouse interaction with Unity Controllers over to a custom implementation, which solved all the problems we were facing with the uGUI based approach.