Steve
The asymmetric routing is not to do with the actual IPs used on the WAN and tunnel interfaces. It's to do with the routing. So site 4 (as in all the other new sites) has -
1) a static default route pointing to site 5
2) routes received from BGP for all the new sites subnets but it does not receive a default route from BGP because that default is used for your existing sites and internet.
So if site 4 sends a packet with an destination IP that is not one of the subnets for the other sites ie. an internet IP address, the WAN router has to use the default route because it does not have a more specfic route for that destination. So it sends it down the GRE tunnel to site 5 and site 5 forwards it on to the internet.
When the packet comes back from the internet to site 5 the WAN router at site 5 looks up the destination IP which belongs to one of the subnets in site 4. Because site 5 is receiving these routes from BGP it then sends them via the AVPN cloud and not back down the GRE tunnel.
Site 5's routing was what caused all the trouble when we were discussing how to route traffic between the new site. The solution you implemented meant only return traffic from the internet was asymmetric because if site 4 wants to talk to a device in site 5 they both use the AVPN cloud because site 4's WAN router has routes for site 5's subnets via BGP.
If we had used statics at site 5 for each new sites subnets then internet traffic would have used the GRE tunnel both ways but site 4's (and sites 1, 2, 3) traffic to site 5's subnets would have gone via the AVPN cloud but returned by the GRE tunnel because of the statics at site 5. So we just chose the one with the least amount of config. .
What we may have been able to do was run EIGRP down the tunnel and change the AD of either BGP or EIGRP in all sites so EIGRP routes were preferred so there was no asymmetric traffic and all traffic went via the GRE tunnel but i'm not sure this was the actual requirement ie. keep the new sites off the AVPN cloud altogether and use the GRE tunnel for all traffic.
And we definitely couldn't have gone the other way because you needed the default route for internet to use the GRE tunnel.
So there is a bit of asymmetric traffic, which like i say, you could probably get rid of if you needed to but is probably doing no harm.
Jon
