ADFS - Azure Internal and External Load Balancers

We are currently deploying an ADFS Farm within Azure as follows:

VNet1 - VPN Tunnel to head Office

10.0.0.61 - Internal Load Balancer VIP(No ACLs)

10.0.0.62 - ADFS Server 1

10.0.0.63 - ADFS Server 2

Vnet2 - DMZ VNet

10.10.0.62 - ADFS Proxy 1

10.10.0.63 - ADFS Proxy 2

137.x.x.61 - External VIP on Cloud Service

We plan on utilizing a VNET for the DMZ, and setting an ACL on an External endpoint on Backend ADFS Farm to allow all traffic from the proxies(from Vnet2) to communicate back to the 443 endpoint.  The problem I am running into is that it does not seem to allow and internal load balancer to utilize the same port(443) as an external load balanced set listening on the external IP Address.

Let me know if anyone has had success with this,

Thanks,