Hi there, I have a 480 system and I hated the severe stuttering I had!!!
TL;DR: Your CPU is not enough to play this title. No amount of minor tinkering or overclocking will make this title playable at smooth FPS, sadly.
The problem (assuming your drivers are good and your clocks are running normal) is your CPU. I had a watercooled, very stable i5 6500 system running with a 480 (remember it is practically the same card) and it would stutter it's way through the game. If you type in the following after pressing the "~" key perfoverlay.drawfps 1 you will notice that your card is pumping out 100+ FPS @ Ultra 1080p and your CPU is lagging behind running at 70FPS (yeah it's a weird way to measure, but so be it) then drop to 0, 1, 2, or 3 FPS, then jump back to 70s-80s, then drop again, and this is holding you back.
For personal futureproofing reasons and my available resources lying around, I switched my RX 480 over to a Ryzen 7 1700x system, and the game could not be any better. The CPU requirements for that game and many new AAA titles are redicioulous and seem way overkill. Who knew people would start making minimums i5 6600ks!!! That's crazy when you need an i7 or Ryzen 5 to play a game at high settings! Anyway, in short you need a better CPU and no amount of tinkering or overclocking anything is going to change that for this title at least. Sorry