So I spent an hour and a half fixing this, so hopefully someone comes across this post and it helps them.

My issue was that the OpenVPN adapter on my Windows 7 computer was an Unidentified network and I couldn’t change it. Because of this, it didn’t follow the right firewall rules, making it impossible to RDP in over the VPN.

So, after much searching high and low, the fix is very simple, just add the following lines to your client config file:

# NLA issues
route-metric 512
route 0.0.0.0 0.0.0.0

And restart the VPN connection.

Thanks to this site which was very difficult to find.