Well your CPU shouldn't be bottlenecking you, you could probably see Youtube videos with your CPU+GPU for reference fps.
Windows Aero, are you still using Windows 7? That Windows was terrible with RAM usage and it always switches off Windows Aero unless you tell it not to. That's not a big issue. Try using a different display adapter see if the flickering still happens.
And as mentioned before, did you use DDU to do a clean install after upgrading your GPU? That's very important.
It shouldn't - at least not in up-to-4-players-coop/singleplayer games - but that was the only culprit I was able to think of.
As for the RAM usage, I'm not sure. I've been using Win7 for a long time and don't recall a single time when I ran out of RAM. Moreover, with the previous 770 GPU I was able to run Chrome and torrent client in background (while playing) without Win7 switching to basic theme - and both these apps are known for their RAM drain.
No, it wasn't the DDU method, I used the clean install option when installed the new nvidia driver. Thanks for the tip, just used the DDU, will see how the things are going now.
Forgot to add - after I've created this topic I've limited FPS to ~72 as a workaround and there weren't any spikes.