Azure Load Balancing if VM (Backend pool) port fails

Hi There,

We have an Azure Load Balancer (LB) which is connected to Two VMs (VM1 and VM2) over the same backend pool port 8443(using Rule type #1: No backend port reuse). The load balancer works fine and redirects the traffic to the either of the VMs according to its topology. If the VM2 goes down, all the connections will be redirected by the LB to the VM1.  
In the current scenario, if the backup pool port of any one of the VMs fails(port 8443 is dropped on VM2), the connections that are redirected by the LB to the VM2 throw an error. How can we redirect the connections to VM1 in such cases. 



Thanks & Regards
Venu