EDIT(SOLVED): The culprit the entire time was "Smooth Motion Frames" being enabled. Had no idea that was a performance killer. https://www.3dmark.com/3dm/143448852
From day 1 on my fresh build I have not been able to get a decent score with my 5080. I feel like I'm running a pretty aggressive OC and I'm getting lower scores that non OC guys are getting out of a 5080. I can't even finish a Time Spy run because it freezes while loading between the 3rd and 4th test even with no OC at all. Latest drivers 5080, latest chipset, and latest BIOS. Granted I've had these issues across all nvidia drivers since getting the 5080. My previous build I made for a buddy with a 5070 was peaking top 1% in both SN and TS.
ROG Astral OC 5080
R7 9800x3D
Aorus x870e Pro
Samsung 990 Pro 4TB
2x16GB gskill 6000MT/s CL30
BeQuiet Straight Power 1500w(I got it for less than a 1000w so I said why not)
Playing on a Samsung Odyssey G8 32" 4K OLED if that makes in difference. I didn't get this monitor until the same day I finished this build.