That's super weird and I have no idea why that could be happening. Seems odd a controller would cause literal framerate lag depending on if it's plugged in or not. All I had to test with was an Xbox 360-esque controller, so I'm not sure how other controllers affect the game. Could it be some of your drivers being possibly outdated?