Site to Site Tunnel Issue

I built a site to site tunnel from our ASA to out Azure virtual network. I can connect fine from our network to a server in Azure. I can even ping that server. I can't connect back to our network from that same server.  I've check firewalls on servers and tried to RDP back to a server on out internal network and nothing seems to work.  So basically only one direction is working.

Any suggestions.

Thanks,