Azure to on-premise connectivity

Hi,

I have my ADFS just built in Azure in this config:

Azure Site 1:

1 x DC

1 x ADFS (Primary)

1 x WAP

Azure Site 2:

1 x ADFS (Secondary)

1 x WAP

I have also two VPN site-to-site tunnels one between my site and Azure Site 1 and one between my site and Azure Site 2.

Also i have a vnet-to-vnet tunnel between the two Azure sites.

If all three are up and running no issues.

Here is the problem.

If one of my site-to-site tunnel is down (for example the tunnel to Azure Site 1) the servers from Azure Site 1 can't reach my on-premise servers via the other tunnel going through the vnet-to-vnet tunel.

What am I missing here?

Thanks for help