Can't RDP over Point to Site VPN

I have a Point to Site VPN connection established from my computer to my vnet in Azure. However, I cannot RDP to either one of my VM's (both on the same subnet). My client VPN subnet is 10.0.3.0 and the VM subnet is 10.0.1.0. I can see a route on my computer for 10.0.0.0/16 going through the VPN interface. 

So why can't I RDP to my VM's? Am I supposed to create an endpoint or something?