Hi All,
We worked with a customer to update their 2110 from 6.6.4 to 7.0.4 (both FMC and FTD) They have 2 2110s in the rack, they are independent of each other (was going to be HA, but never eventuated)
Pre Upgrade:
- 2110-A is production unit, with working RA-VPN (Anyconnect)
- 2110-B is spare unit, with basic vanilla config applied (basic Nat, that's all)
- Both units on 6.6.4
Steps Taken:
- Full backup of 2110-A
- cloned policies of 2110-A and applied them to 2110-B (we did need to modify them to remove some of the licenced features as they only have licencing for one) These are the only changed made to the cloned policies.
- Moved Anyconnect licence from 2110-A -> 2110-B
- deployed and tested everything working (traffic and VPN) - This was achieved by swapping the WAN and LAN cables from 2110-A -> 2110-B. After verification was complete, cables were reverted from 2110-B -> 2110-A
- updated 2110-B to 7.0.4. This left 2110-A in production state as failover / fall back / DR unit if there were upgrade issues
Post upgrade:
- swapped cabling 2110-A -> 2110-B to verify.
- Traffic is passing
- RA-VPN (Anyconnect) IS NOT working at all. There are no logs shown in Devices -> VPN -> Troubleshooting
- Traffic is reaching the unit, as when we ping it, the traffic is blocked (this indicates that it is at least reaching the endpoint) We can see the DENY in the event logs on FMC
- Tracert fails to resolve. ( Included below, however IP's removed for security)
- we swapped the cables back from 2110-B -> 2110-A for fall back / DR, and Anyconnect is STILL NOT working.
The 2110-A config was never changed or modified, and yet the VPN failed to work when we rolled back the cables. There was also an issue when we swapped back from 2110-B -> 2110-A where the interfaces came up, but no traffic passed. We discovered this was mostly due to the ARP cache on the downstream router, and clearing it earlier had resolved this, but for some reason at this point, it would no longer work.
Due to needing a production unit, we then decided as a best option, to re-deploy the original PRODICTION policies and licences to 2110-B.
Prior to our emergency break / fix of redeploying 2110-A policies to 2110-B, we checked over all the policies, objects, NAT rules etc.. and it all looked exactly the same on both units.
Current state:
- 2110-B is in production with original 2110-A policies and licences
- 2110-A has had the cloned policies applied to it.
- Anyconnect VPN is not working. We suspect that it is not listening on 443.
- Internet traffic is working as expected.
- No logs at all in VPN trouble-shooter (confirmed that logging is enabled and also set to debug)
Tracert
PS C:\Users\Jason> tracert <FQDN>
Tracing route to <FQDN> [W.X.Y.Z]
over a maximum of 30 hops:
1 1 ms 2 ms 2 ms 10.227.1.2
2 15 ms 13 ms 14 ms 10.22.0.149
3 14 ms 17 ms 14 ms 10.22.0.148
4 20 ms 15 ms 25 ms 103.3.217.157
5 22 ms 15 ms 15 ms 103.3.217.164
6 18 ms 15 ms 16 ms 45.115.20.0
7 * 16 ms * ip-137.83.254.124.vocus.net.au [124.254.83.137]
8 15 ms 15 ms 15 ms be117.cor01.syd11.nsw.vocus.network [114.31.192.68]
9 15 ms 16 ms 16 ms be100.bdr01.syd11.nsw.vocus.network [114.31.192.81]
10 15 ms 16 ms 16 ms bundle-ether12.ken-edge903.sydney.telstra.net [203.27.185.57]
11 17 ms 18 ms 16 ms bundle-ether2.chw-edge903.sydney.telstra.net [203.50.11.175]
12 16 ms 26 ms 16 ms bundle-ether12.stl-core30.sydney.telstra.net [203.50.11.176]
13 21 ms 18 ms 17 ms bundle-ether19.chw-core10.sydney.telstra.net [203.50.13.144]
14 27 ms 27 ms 29 ms bundle-ether20.woo-core10.brisbane.telstra.net [203.50.11.181]
15 27 ms 26 ms 26 ms bundle-ether1.woo-edge902.brisbane.telstra.net [203.50.11.139]
16 * * * Request timed out.
Thanks all.
Jason
