Using Traffic Manager to switch between Azure IaaS RDP endpoint

We have two Azure VMs in an availability set. The users RDP into the VM using remote desktop connection.

This Azure article says that we can use Traffic Manager with IaaS VM endpoints:

https://azure.microsoft.com/en-in/documentation/articles/traffic-manager-endpoint-types/

Is there any way where we can put Traffic manager ahead of the two VMs and users use Traffic Manager DNS and endpoint the TM can switch between two RDP endpoints in case one VM is down. (Just as traffic manager can switch between webapp DNS and it seems seamless to users)

What is the probable solution for this situation?

Thanks

Bhavya