Hi experts, I am in trouble with PfRv3 loadbalancing implementation.  Given the three independent cellular uplink, with three DMVPN cloud in one spoke router. What bandwidth parameter should I apply under the cellular interfaces on the branch devices? Based on the mobile endpoint location, the WAN throughput is changing, and cannot be granted (any of them might be EDGE, HSDPA/HSUPA or LTE speed) Can PfRv3 adapt the actual throughput or relies only on the interface specified, fixed value?? Any ideas? Thanks jonagy ... View more

Hi I am asking for the community wisdom on the following customer reported problem. Given the following simplified topology (I neither has have detailed running-config on this...) Outside -- default route Inside -- some 10/8 networks, might have overlapping subnets with networks behind DMZ if!! DMZ -- Many networks behind this if, in 10.x.x.x, but NOT routed on the ASA! Customer wants _not_ to appear the 10.x.x.x/8 in ASA routing table towards DMZ, therefore configures PBR on specific inside sources to be transmitted in this direction. In the route-map they use "set ip next-hop". Inside -> (PBR + sNAT) -> DMZ: works fine DMZ -> staticNAT -> Inside:  returning reply fails:  "Routing failed to locate next hop for .." The first case, sNAT is like accessing any outside: nat (any,DMZ-networks) after-auto source dynamic any pat-pool pat-DMZ-range extended Works fine. The second (DMZ->Inside) case packets reaches inside host/rserver, but replies got routing failed. This case is rather like offering inside service for the public internet, hoping that the xlate enrty or the PBR redirects the reply package towards the DMZ where it came from, instead of default Outside. We tried to enforce the return by identity sNAT: nat (Inside,DMZ) source static REAL-INSIDE-ADDR GLOBAL-DMZ-ADDR destination static DMZ-10_x_x_x DMZ-10_x_x_x service SVC-GLOBAL SVC-LOCAL on this last NAT entry we got the hitcount increasing, hen testing (initiating traffic) from the DMZ, but still getting the "Routing failed for" syslog. Additional, probably relevant detail: asa-1(config)# sh nat divert-table interface Inside Divert Table id=0x7ffc80e22240, domain=twice-nat section=1 ignore=no     type=static, hits=0, flags=0x9, protocol=6     src ip/id=10.a.b.c, mask=255.255.255.255, port=1111-1111       dst ip/id=10.0.0.0, mask=255.0.0.0, port=0-0     input_ifc=Inside, output_ifc=DMZ id=0x7ffcbdf1a300, domain=auto-nat section=2 ignore=yes (port 1111 is on local, real server, 10.0.0.0/8 is on inf DMZ, PBR should forward that) This last statement, "ignore=yes" confusing: Anyway, if there is a route lookup, PBR shoud direct the packet towards DMZ, if not, the existing xlate entry should force the returning packet to exit on the DMZ interface. Both should forward reply packets towards the DMZ if. Still geting the "routing failed" message. Any advise experts? thanks, jonagy ... View more

Hi please help resolving the following issue. We are facing the problem with the following: -IKEv2 -PSK -dVTI  tunnel mode ipsec - tunnel src in vrf On the far end non-cisco (DIGI Transport WR44) devices are establishing the IPsec successfully, and the following happens:   - IPsec establishes successfully, traffic is passing perfectly - before 8hrs default IPsec lifetime DIGI initiates the IPsec rekeying about 20 minutes before the expiry - the rekeying seems successful, all SPI-s renewed at both sides - no decaps counter increases at both sides, until - 1-2 minutes before the 8 hrs cisco resets the COMPLETE CRYPTO SESSION, and the connection establishes fine again I have tried to - decrease Cisco's IPsec lifetime, no success, cisco initiated IPsec rekeying success, but the result is the same: SPI-s negotiated, but neither side shows any decaps - increase DIGI side IPsec lifetime over 8 hours, no success, same result.   I caught some undocumented syslogs, but nothing useful. Please help me to get rid of this 20 minute blackout, three times daily. Other details, configs, debugs: !!!!CONFIG crypto ikev2 keyring Digi-key      peer digi-routers-ADSL   address 10.191.240.0 255.255.252.0   pre-shared-key xxxxxxxxxxxxxxxxxx crypto ikev2 proposal Digi-ikev2-proposal  encryption aes-cbc-256  integrity sha1  group 5 crypto ikev2 policy Digi-ikev2-policy  match fvrf any  match address local 10.191.254.89  proposal Digi-ikev2-proposal      crypto ikev2 profile Digi-ADSL-T-ikev2-profile   match fvrf T   match address local interface GigabitEthernet0/0/4   match identity remote email domain digi.digi   identity local email asr@asr   authentication remote pre-share   authentication local pre-share   keyring local Digi-key   dpd 10 2 periodic   virtual-template 101   crypto ipsec profile Digi-ADSL-T-ipsec-profile  description Ipsec profile for VTI tunnels to Digi routers - ADSL -T  set security-association lifetime seconds 28800  set transform-set Digi-ipsec-transform-set  set reverse-route distance 10  set reverse-route tag 101  set ikev2-profile Digi-ADSL-T-ikev2-profile  tunnel protection ipsec profile Digi-ADSL-T-ipsec-profile     interface Virtual-Template101 type tunnel  description DIGI router ADSL VPN T  ip unnumbered Loopback100  ip mtu 1422  ip tcp adjust-mss 1350  tunnel source GigabitEthernet0/0/4  tunnel mode ipsec ipv4  tunnel vrf T  tunnel protection ipsec profile Digi-ADSL-T-ipsec-profile   !!!!!!!! SHOW OUTPUTS before and after DIGI has completed the IPsec rekeying: ASR1#sh cry sess rem  10.191.243.83 det Crypto session current status Code: C - IKE Configuration mode, D - Dead Peer Detection      K - Keepalives, N - NAT-traversal, T - cTCP encapsulation      X - IKE Extended Authentication, F - IKE Fragmentation Interface: Virtual-Access20 Profile: Digi-ADSL-T-ikev2-profile Uptime: 07:44:08 Session status: UP-ACTIVE      Peer: 10.191.243.83 port 500 fvrf: T ivrf: (none)       Phase1_id: 4626@digi.digi       Desc: (none)   Session ID: 20252     IKEv2 SA: local 10.191.254.89/500 remote 10.191.243.83/500 Active           Capabilities:D connid:26 lifetime:16:15:52   IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 10.181.196.0/255.255.255.224         Active SAs: 2, origin: crypto map         Inbound:  #pkts dec'ed 15521 drop 0 life (KB/Sec) 4608000/7 hours, 55 mins         Outbound: #pkts enc'ed 16467 drop 0 life (KB/Sec) 4607993/7 hours, 55 mins ASR# ASR# ASR# ASR# ASR# ASR#sh cry sess rem  10.191.243.83 det Crypto session current status Code: C - IKE Configuration mode, D - Dead Peer Detection      K - Keepalives, N - NAT-traversal, T - cTCP encapsulation      X - IKE Extended Authentication, F - IKE Fragmentation Interface: Virtual-Access20 Profile: Digi-ADSL-T-ikev2-profile Uptime: 07:46:01 Session status: UP-ACTIVE      Peer: 10.191.243.83 port 500 fvrf: T ivrf: (none)       Phase1_id: 4626@digi.digi       Desc: (none)   Session ID: 20252     IKEv2 SA: local 10.191.254.89/500 remote 10.191.243.83/500 Active           Capabilities:D connid:26 lifetime:16:13:59   IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 10.181.196.0/255.255.255.224         Active SAs: 2, origin: crypto map         Inbound:  #pkts dec'ed 15522 drop 0 life (KB/Sec) 0/7 hours, 54 mins         Outbound: #pkts enc'ed 16499 drop 0 life (KB/Sec) 4607992/7 hours, 54 mins   During the 20 minutes lockup, in the "show platform hardware qfp active feature ipsec datapath drops" output, the "IN_UNEXP_OCT_EXCEPTION" counter is increasing! ASR#show platform hardware qfp active feature ipsec datapath drops ------------------------------------------------------------------------ Drop Type  Name                                     Packets              ------------------------------------------------------------------------         3  IN_US_V4_PKT_FOUND_IPSEC_NOT_ENABLED                     9697         4  IN_US_V4_PKT_SA_NOT_FOUND_SPI                               8        19  IN_OCT_ANTI_REPLAY_FAIL                                    31        20  IN_UNEXP_OCT_EXCEPTION                                3007839          62  IN_OCT_MAC_EXCEPTION                                      280 ASR#show platform hardware qfp active feature ipsec datapath drops ------------------------------------------------------------------------ Drop Type  Name                                     Packets              ------------------------------------------------------------------------         3  IN_US_V4_PKT_FOUND_IPSEC_NOT_ENABLED                     9697         4  IN_US_V4_PKT_SA_NOT_FOUND_SPI                               8        19  IN_OCT_ANTI_REPLAY_FAIL                                    31        20  IN_UNEXP_OCT_EXCEPTION                                3007841   Some similar DIGI clients's key renewal IKEv2 and IPsec debug from the Cisco ASR (might be some duplicated lines): The unknown errors in bold Apr 23 11:46:46.033: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending DPD/liveness query Apr 23 11:46:46.033: IKEv2:(SESSION ID = 19748,SA ID = 4):Building packet for encryption. Apr 23 11:46:46.035: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking if request will fit in peer window Apr 23 11:46:46.035: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending Packet [To 10.191.243.71:500/From 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2756 IKEv2 INFORMATIONAL Exchange REQUEST Payload contents:  ENCR   Apr 23 11:46:46.059: IKEv2:(SESSION ID = 19748,SA ID = 4):Received Packet [From 10.191.243.71:500/To 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2756 IKEv2 INFORMATIONAL Exchange RESPONSE Payload contents:   Apr 23 11:46:46.059: IKEv2:(SESSION ID = 19748,SA ID = 4):Processing ACK to informational exchange Apr 23 11:46:56.059: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending DPD/liveness query Apr 23 11:46:56.059: IKEv2:(SESSION ID = 19748,SA ID = 4):Building packet for encryption. Apr 23 11:46:56.060: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking if request will fit in peer window Apr 23 11:46:56.060: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending Packet [To 10.191.243.71:500/From 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2757 IKEv2 INFORMATIONAL Exchange REQUEST Payload contents:  ENCR   Apr 23 11:46:56.082: IKEv2:(SESSION ID = 19748,SA ID = 4):Received Packet [From 10.191.243.71:500/To 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2757 IKEv2 INFORMATIONAL Exchange RESPONSE Payload contents:   Apr 23 11:46:56.082: IKEv2:(SESSION ID = 19748,SA ID = 4):Processing ACK to informational exchange Apr 23 11:47:06.082: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending DPD/liveness query Apr 23 11:47:06.082: IKEv2:(SESSION ID = 19748,SA ID = 4):Building packet for encryption. Apr 23 11:47:06.084: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking if request will fit in peer window Apr 23 11:47:06.084: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending Packet [To 10.191.243.71:500/From 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2758 IKEv2 INFORMATIONAL Exchange REQUEST Payload contents:  ENCR   Apr 23 11:47:06.106: IKEv2:(SESSION ID = 19748,SA ID = 4):Received Packet [From 10.191.243.71:500/To 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2758 IKEv2 INFORMATIONAL Exchange RESPONSE Payload contents:   Apr 23 11:47:06.106: IKEv2:(SESSION ID = 19748,SA ID = 4):Processing ACK to informational exchange Apr 23 11:47:07.696: IPSEC(sa_request): ,   (key eng. msg.) OUTBOUND local= 10.191.254.89:500, remote= 10.191.243.71:500,     local_proxy= 0.0.0.0/0.0.0.0/256/0,     remote_proxy= 10.181.194.128/255.255.255.224/256/0,     protocol= ESP, transform= esp-aes 256 esp-sha-hmac  (Tunnel),     lifedur= 28000s and 4608000kb,     spi= 0x1068819F(275284383), conn_id= 0, keysize= 256, flags= 0x0 Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Check for IPSEC rekey Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Beginning IPSec Rekey as Initiator Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Set IPSEC DH group Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking for PFS configuration Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):PFS not configured Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Generating CREATE_CHILD_SA exchange Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):ESP Proposal: 1, SPI size: 4 (IPSec negotiation), Num. transforms: 3    AES-CBC   SHA96   Don't use ESN Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Building packet for encryption.   Payload contents:  NOTIFY(REKEY_SA) SA N TSi TSr Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking if request will fit in peer window Apr 23 11:47:07.696: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending Packet [To 10.191.243.71:500/From 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2759 IKEv2 CREATE_CHILD_SA Exchange REQUEST Payload contents:  ENCR   Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Received Packet [From 10.191.243.71:500/To 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2759 IKEv2 CREATE_CHILD_SA Exchange RESPONSE Payload contents:  SA N TSi TSr Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Processing any notify-messages in child SA exchange Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Validating create child message Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Processing CREATE_CHILD_SA exchange Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking for PFS configuration Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):PFS not configured Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking if IKE SA rekey Apr 23 11:47:07.721: IKEv2:(SESSION ID = 19748,SA ID = 4):Load IPSEC key material Apr 23 11:47:07.726: IPSEC:(SESSION ID = 19748) (create_sa) sa created,   (sa) sa_dest= 10.191.254.89, sa_proto= 50,     sa_spi= 0x416AF6F0(1097529072),     sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 24986     sa_lifetime(k/sec)= (4608000/28000),   (identity) local= 10.191.254.89:0, remote= 10.191.243.71:0,     local_proxy= 0.0.0.0/0.0.0.0/256/0,     remote_proxy= 10.181.194.128/255.255.255.224/256/0 Apr 23 11:47:07.731: IPSEC:(SESSION ID = 19748) (create_sa) sa created,   (sa) sa_dest= 10.191.243.71, sa_proto= 50,     sa_spi= 0x492F5006(1227837446),     sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 24985     sa_lifetime(k/sec)= (4608000/28000),   (identity) local= 10.191.254.89:0, remote= 10.191.243.71:0,     local_proxy= 0.0.0.0/0.0.0.0/256/0,     remote_proxy= 10.181.194.128/255.255.255.224/256/0 Apr 23 11:47:07.737: IPSEC:(SESSION ID = 19748) (update_current_outbound_sa) updated peer 10.191.243.71 current outbound sa to SPI 492F5006 Apr 23 11:47:07.743: IKEv2:(SESSION ID = 19748,SA ID = 4):IKEV2 SA created; inserting SA into database. SA lifetime timer (86400 sec) started Apr 23 11:47:07.743: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending DELETE INFO message for IPsec SA [SPI: 0x1068819F] Apr 23 11:47:07.743: IKEv2:(SESSION ID = 19748,SA ID = 4):Building packet for encryption.   Payload contents:  DELETE Apr 23 11:47:07.743: IKEv2:(SESSION ID = 19748,SA ID = 4):Checking if request will fit in peer window Apr 23 11:47:07.743: IKEv2:(SESSION ID = 19748,SA ID = 4):Sending Packet [To 10.191.243.71:500/From 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2760 IKEv2 INFORMATIONAL Exchange REQUEST Payload contents:  ENCR Apr 23 11:47:07.743: IKEv2:(SESSION ID = 19748,SA ID = 4):Check for existing IPSEC SA Apr 23 11:47:07.764: IKEv2:(SESSION ID = 19748,SA ID = 4):Received Packet [From 10.191.243.71:500/To 10.191.254.89:500/VRF i3:f3] Initiator SPI : 00000060FFFFFF9F - Responder SPI : 882705463FF54AD2 Message id: 2760 IKEv2 INFORMATIONAL Exchange RESPONSE Payload contents:  DELETE Apr 23 11:47:07.764: IKEv2:(SESSION ID = 19748,SA ID = 4):Processing ACK to informational exchange Apr 23 11:47:07.764: IKEv2:(SESSION ID = 19748,SA ID = 4):Check for existing IPSEC SA Apr 23 11:47:07.765: IPSEC: still in use sa: 0x0 Apr 23 11:47:07.765: IPSEC: sa null Apr 23 11:47:07.765: IPSEC:(SESSION ID = 19748) still in use sa: 0x7F483D68EB50 Apr 23 11:47:07.766: IPSEC:(SESSION ID = 19748) (key_engine_delete_sas) delete SA with spi 0x1068819F proto 50 for 10.191.254.89 Apr 23 11:47:07.770: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Apr 23 11:57:49.384: IPSEC:(SESSION ID = 19747) still in use sa: 0x7F48378D2BF0 Apr 23 11:57:49.391: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Apr 23 11:57:49.391: IPSEC: still in use sa: 0x0 Apr 23 11:57:49.391: IPSEC: sa null Apr 23 11:57:49.393 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access32, changed state to down Apr 23 11:57:49.393 CEST: %LINK-3-UPDOWN: Interface Virtual-Access32, changed state to down Apr 23 11:57:49.395: IPSEC(ERROR): ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x7F483CC19BE0 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 11:57:49.395: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_DESTROY_YOURSELF message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 11:57:49.395: IPSEC(MESSAGE): INTF_SM (interface_sm_create_aux_msg_data) called Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_msg_check_policy) Crypto config being deleted Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_sadb_check) called Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_delete_policy) called Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_delete_sadb) called for interface Virtual-Access32 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_not_inuse) SADB 0x7F483DC9DB08 inuse_count 0 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_not_inuse) SADB 0x7F483DC9DB08 inuse_count 0 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_destroy) called Apr 23 11:57:49.396: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_end) called Apr 23 11:57:49.396: IPSEC(ERROR): Failed to remove SADB 0x7F483DC9DB08 from sadb_root_avl_tree Apr 23 11:57:49.396: IPSEC(STATES): INTF_SM (interface_action_end) killing state machine 0x7F483BDB1E10 Apr 23 11:57:49.551 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access32, changed state to down Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (ipsec_pal_create_interface_state_machine) called for interface Virtual-Access32 Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (ipsec_pal_create_interface_state_machine): Created INTF SM 0x7F483BDB1E10 Apr 23 11:57:49.558: IPSEC(MESSAGE): INTF_SM (interface_sm_create_aux_msg_data) called Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_msg_check_policy) Crypto config being added Apr 23 11:57:49.558: IPSEC(MESSAGE): INTF_SM (interface_sm_extract_data) idb:Virtual-Access32 map:Virtual-Access32-head-0 protection:PROTECTION_CLASS_TUNNEL Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_sadb_check) called Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_new_sadb_exists) called Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_sadb_create) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (crypto_create_acl_struct) ACL address:0.0.0.0 mask:0.0.0.0 port:0 Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (ipsec_pal_create_sadb_root_state_machine) called for SADB 0x7F483DCE55D0 Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_initialize) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_initialize) interface Virtual-Access32 hint CRYPTO_SADB_HINT_VTI map Virtual-Access32-head-0 Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_download) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_download_complete) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_inuse) SADB 0x7F483DCE55D0 inuse_count 1 Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_attach_intf) called idb:Virtual-Access32 map:Virtual-Access32-head-0 Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_sadb_update) called Apr 23 11:57:49.559: IPSEC(STATES): INTF_SM (interface_action_remove_old_sadb) called Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 11:57:49.560: IPSEC(STATES): (insert_crypto_map) convert secure socket acl to crypto aces seq_no 65537 acl_name  acl_num 0 Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (sadb_root_create_acl_struct) ACL static:65537 dynamic:0 Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (ipsec_sadb_root_create_acl_state_machine) Calling to create bypass ACL SM Apr 23 11:57:49.560: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_CHANGED message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 11:57:49.560: IPSEC(recalculate_mtu): reset sadb_root 7F483DCE55D0 mtu to 1500 Apr 23 11:57:49.561: IPSEC:(SESSION ID = 19797) (STATES) SADB_ROOT_SM (sadb_root_process_kmi_message) called static seqno 65537 dynamic seqno 0 Apr 23 11:57:49.574: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_KMI_CREATE_SA message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 11:57:49.578: IPSEC:(SESSION ID = 19797) (STATES) ident_set_flow_installed_action Sending crypto_ss_connection_open Apr 23 11:57:49.578: IPSEC(rte_mgr): VPN Route Added 10.181.198.64 255.255.255.224 via Virtual-Access32 in IP DEFAULT TABLE with tag 101 distance 10 Apr 23 11:57:49.578 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access32, changed state to up Apr 23 11:57:49.384: IPSEC:(SESSION ID = 19747) still in use sa: 0x7F48378D2BF0 Apr 23 11:57:49.391: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Apr 23 11:57:49.391: IPSEC: still in use sa: 0x0 Apr 23 11:57:49.391: IPSEC: sa null Apr 23 11:57:49.393 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access32, changed state to down Apr 23 11:57:49.393 CEST: %LINK-3-UPDOWN: Interface Virtual-Access32, changed state to down Apr 23 11:57:49.395: IPSEC(ERROR): ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x7F483CC19BE0 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 11:57:49.395: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_DESTROY_YOURSELF message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 11:57:49.395: IPSEC(MESSAGE): INTF_SM (interface_sm_create_aux_msg_data) called Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_msg_check_policy) Crypto config being deleted Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_sadb_check) called Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_delete_policy) called Apr 23 11:57:49.395: IPSEC(STATES): INTF_SM (interface_action_delete_sadb) called for interface Virtual-Access32 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_not_inuse) SADB 0x7F483DC9DB08 inuse_count 0 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_not_inuse) SADB 0x7F483DC9DB08 inuse_count 0 Apr 23 11:57:49.395: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_destroy) called Apr 23 11:57:49.396: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_end) called Apr 23 11:57:49.396: IPSEC(ERROR): Failed to remove SADB 0x7F483DC9DB08 from sadb_root_avl_tree Apr 23 11:57:49.396: IPSEC(STATES): INTF_SM (interface_action_end) killing state machine 0x7F483BDB1E10 Apr 23 11:57:49.551 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access32, changed state to down Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (ipsec_pal_create_interface_state_machine) called for interface Virtual-Access32 Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (ipsec_pal_create_interface_state_machine): Created INTF SM 0x7F483BDB1E10 Apr 23 11:57:49.558: IPSEC(MESSAGE): INTF_SM (interface_sm_create_aux_msg_data) called Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_msg_check_policy) Crypto config being added Apr 23 11:57:49.558: IPSEC(MESSAGE): INTF_SM (interface_sm_extract_data) idb:Virtual-Access32 map:Virtual-Access32-head-0 protection:PROTECTION_CLASS_TUNNEL Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_sadb_check) called Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_new_sadb_exists) called Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_sadb_create) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (crypto_create_acl_struct) ACL address:0.0.0.0 mask:0.0.0.0 port:0 Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (ipsec_pal_create_sadb_root_state_machine) called for SADB 0x7F483DCE55D0 Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_initialize) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_initialize) interface Virtual-Access32 hint CRYPTO_SADB_HINT_VTI map Virtual-Access32-head-0 Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_download) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_download_complete) called Apr 23 11:57:49.558: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_inuse) SADB 0x7F483DCE55D0 inuse_count 1 Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_attach_intf) called idb:Virtual-Access32 map:Virtual-Access32-head-0 Apr 23 11:57:49.558: IPSEC(STATES): INTF_SM (interface_action_sadb_update) called Apr 23 11:57:49.559: IPSEC(STATES): INTF_SM (interface_action_remove_old_sadb) called Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 11:57:49.560: IPSEC(STATES): (insert_crypto_map) convert secure socket acl to crypto aces seq_no 65537 acl_name  acl_num 0 Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (sadb_root_create_acl_struct) ACL static:65537 dynamic:0 Apr 23 11:57:49.560: IPSEC(STATES): SADB_ROOT_SM (ipsec_sadb_root_create_acl_state_machine) Calling to create bypass ACL SM Apr 23 11:57:49.560: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_CHANGED message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 11:57:49.560: IPSEC(recalculate_mtu): reset sadb_root 7F483DCE55D0 mtu to 1500 Apr 23 11:57:49.561: IPSEC:(SESSION ID = 19797) (STATES) SADB_ROOT_SM (sadb_root_process_kmi_message) called static seqno 65537 dynamic seqno 0 Apr 23 11:57:49.574: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_KMI_CREATE_SA message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 11:57:49.578: IPSEC:(SESSION ID = 19797) (STATES) ident_set_flow_installed_action Sending crypto_ss_connection_open Apr 23 11:57:49.578: IPSEC(rte_mgr): VPN Route Added 10.181.198.64 255.255.255.224 via Virtual-Access32 in IP DEFAULT TABLE with tag 101 distance 10 Apr 23 11:57:49.578 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access32, changed state to up Apr 23 12:04:18.137: IPSEC:(SESSION ID = 19748) still in use sa: 0x7F483D68F5F8 Apr 23 12:04:18.139: IPSEC:(SESSION ID = 19748) (key_engine_delete_sas) delete SA with spi 0x416AF6F0 proto 50 for 10.191.254.89 Apr 23 12:04:18.143: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Apr 23 12:04:18.144: IPSEC: still in use sa: 0x0 Apr 23 12:04:18.144: IPSEC: sa null Apr 23 12:04:18.146 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access24, changed state to down Apr 23 12:04:18.146 CEST: %LINK-3-UPDOWN: Interface Virtual-Access24, changed state to down Apr 23 12:04:18.148: IPSEC(ERROR): ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x7F48378CEBB0 Apr 23 12:04:18.148: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 12:04:18.148: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_DESTROY_YOURSELF message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 12:04:18.148: IPSEC(MESSAGE): INTF_SM (interface_sm_create_aux_msg_data) called Apr 23 12:04:18.148: IPSEC(STATES): INTF_SM (interface_action_msg_check_policy) Crypto config being deleted Apr 23 12:04:18.148: IPSEC(STATES): INTF_SM (interface_action_sadb_check) called Apr 23 12:04:18.148: IPSEC(STATES): INTF_SM (interface_action_delete_policy) called Apr 23 12:04:18.148: IPSEC(STATES): INTF_SM (interface_action_delete_sadb) called for interface Virtual-Access24 Apr 23 12:04:18.149: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_not_inuse) SADB 0x7F48378C6C10 inuse_count 0 Apr 23 12:04:18.149: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_not_inuse) SADB 0x7F48378C6C10 inuse_count 0 Apr 23 12:04:18.149: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_destroy) called Apr 23 12:04:18.149: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_end) called Apr 23 12:04:18.149: IPSEC(ERROR): Failed to remove SADB 0x7F48378C6C10 from sadb_root_avl_tree Apr 23 12:04:18.149: IPSEC(STATES): INTF_SM (interface_action_end) killing state machine 0x7F483C4ADBC8 Apr 23 12:04:18.305 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access24, changed state to down Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (ipsec_pal_create_interface_state_machine) called for interface Virtual-Access24 Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (ipsec_pal_create_interface_state_machine): Created INTF SM 0x7F483C4ADBC8 Apr 23 12:04:18.312: IPSEC(MESSAGE): INTF_SM (interface_sm_create_aux_msg_data) called Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_action_msg_check_policy) Crypto config being added Apr 23 12:04:18.312: IPSEC(MESSAGE): INTF_SM (interface_sm_extract_data) idb:Virtual-Access24 map:Virtual-Access24-head-0 protection:PROTECTION_CLASS_TUNNEL Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_action_sadb_check) called Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_new_sadb_exists) called Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_action_sadb_create) called Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (crypto_create_acl_struct) ACL address:0.0.0.0 mask:0.0.0.0 port:0 Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (ipsec_pal_create_sadb_root_state_machine) called for SADB 0x7F48378C6C10 Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_initialize) called Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_initialize) interface Virtual-Access24 hint CRYPTO_SADB_HINT_VTI map Virtual-Access24-head-0 Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_download) called Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_download_complete) called Apr 23 12:04:18.312: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_inuse) SADB 0x7F48378C6C10 inuse_count 1 Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_action_attach_intf) called idb:Virtual-Access24 map:Virtual-Access24-head-0 Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_action_sadb_update) called Apr 23 12:04:18.312: IPSEC(STATES): INTF_SM (interface_action_remove_old_sadb) called Apr 23 12:04:18.314: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 12:04:18.314: IPSEC(STATES): SADB_ROOT_SM (sadb_root_action_cm_state) static_seqno 65537 Apr 23 12:04:18.314: IPSEC(STATES): (insert_crypto_map) convert secure socket acl to crypto aces seq_no 65537 acl_name  acl_num 0 Apr 23 12:04:18.314: IPSEC(STATES): SADB_ROOT_SM (sadb_root_create_acl_struct) ACL static:65537 dynamic:0 Apr 23 12:04:18.314: IPSEC(STATES): SADB_ROOT_SM (ipsec_sadb_root_create_acl_state_machine) Calling to create bypass ACL SM Apr 23 12:04:18.314: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_CHANGED message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 12:04:18.314: IPSEC(recalculate_mtu): reset sadb_root 7F48378C6C10 mtu to 1500 Apr 23 12:04:18.315: IPSEC:(SESSION ID = 19798) (STATES) SADB_ROOT_SM (sadb_root_process_kmi_message) called static seqno 65537 dynamic seqno 0 Apr 23 12:04:18.317: IPSEC:(SESSION ID = 19798) (crypto_ipsec_sa_find_ident_head) reconnecting with the same proxies and peer 10.191.243.71 Apr 23 12:04:18.321: IPSEC:(SESSION ID = 19798) (create_sa) sa created,   (sa) sa_dest= 10.191.254.89, sa_proto= 50,     sa_spi= 0x8F37419D(2402763165),     sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 24989     sa_lifetime(k/sec)= (4608000/28000),   (identity) local= 10.191.254.89:0, remote= 10.191.243.71:0,     local_proxy= 0.0.0.0/0.0.0.0/256/0,     remote_proxy= 10.181.194.128/255.255.255.224/256/0 Apr 23 12:04:18.326: IPSEC:(SESSION ID = 19798) (create_sa) sa created,   (sa) sa_dest= 10.191.243.71, sa_proto= 50,     sa_spi= 0x492F5007(1227837447),     sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 24990     sa_lifetime(k/sec)= (4608000/28000),   (identity) local= 10.191.254.89:0, remote= 10.191.243.71:0,     local_proxy= 0.0.0.0/0.0.0.0/256/0,     remote_proxy= 10.181.194.128/255.255.255.224/256/0 Apr 23 12:04:18.327: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_KMI_CREATE_SA message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 12:04:18.332: IPSEC:(SESSION ID = 19798) (STATES) ident_set_flow_installed_action Sending crypto_ss_connection_open Apr 23 12:04:18.332: IPSEC(rte_mgr): VPN Route Added 10.181.194.128 255.255.255.224 via Virtual-Access24 in IP DEFAULT TABLE with tag 101 distance 10 Apr 23 12:04:18.332 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access24, changed state to up Apr 23 12:09:59.272: IPSEC(STATES): SADB_ROOT_SM (sadb_root_process_kmi_message) called static seqno 65537 dynamic seqno 0 Apr 23 12:09:59.272: IPSEC(MESSAGE): SADB_ROOT_SM (print_message_to_acl_state_machine) Sent MSG_ACL_KMI_CREATE_SA message to ACL, static seqno 65537, dynamic seqno 0 Apr 23 12:09:59.772: IPSEC(key_engine_enable_outbound): rec'd enable notify from ISAKMP Apr 23 12:09:59.775: IPSEC(STATES): ident_set_flow_installed_action Sending crypto_ss_connection_open Apr 23 12:11:31.733: IPSEC(delete_sa): SA found saving DEL kmi   Thanks, Gyula         ... View more

Hi al,      we are facing with some difficulty. ACE configured for simple SSL offload, L4.pmap dst port mapping (VIP: 1.1.1.1:9015 rserver 2.2.2.2:8011)   The client establishes the SSL successfully, but the next should be, that the server starts the conversation after the conn. establishment. The issue is, that after the SSL handshake, with the lack of further data from the client side, the ACE does not creates the server backend (clear) TCP session.    Without SSL this works flawlessly, coz ACE does not act as TCP endpoint, just forwards (NAT/PAT), and the server starts the conversation.   Could the ACE forced to open the backend TCP to the rserver, without any triggering input from the client side? This shoud be sw update, binary transfer, no further details of the tcp content.   TIA Regards: jonagy     ... View more

Hi   recently in reporter appears some weird table at the end of the "Multi Package Bandwidth per Service" report  (RPT_PUR).    The symptom is, that some of the services are named correctly, some of them are denoted with numbers, like this: Names are OK, numbers are NOT OK. Investigating the issue I found the following: Named services are found in CM MySql/apricot/INI_VALUES  table with matching name and "VALUE" with VALUE_TYPE="1". This VALUE is found, and matching the name in PQB/classification on services (or within unzipped xml) +---------------------+-----------+------------+-------------------+-------+ | TIME_STAMP          | SE_IP     | VALUE_TYPE | VALUE_KEY         | VALUE | +---------------------+-----------+------------+-------------------+-------+ | 2013-10-16 12:30:07 | 172.x.y.z |          1 | WEB               |   400 | | 2013-10-16 12:30:07 | 172.x.y.z |          1 | STREAMING         |   250 | | 2013-10-16 12:30:07 | 172.x.y.z |          1 | INSTANT MESSENGER |   150 | | 2013-10-16 12:30:07 | 172.x.y.z |          1 | GAMES             |   100 | | 2013-10-16 12:30:07 | 172.x.y.z |          1 | FILE-TRANSFERING  |    50 | +---------------------+-----------+------------+-------------------+-------+ in PQB: <service name="WEB" index="400"/> <service name="STREAMING" index="250"/> <service name="INSTANT MESSENGER" index="150"/> <service name="GAMES" index="100"/> <service name="FILE-TRANSFERING" index="50"/> Services only with numbers in the report: those numbers, treating them as "VALUE" field of the INI_VALUES table, does not hold any row with VALUE_TYPE="1", neither can be found such service number in PQB classification (or unzipped xml). I.e. number "64" mysql> select * from INI_VALUES where SE_IP like "172.x.y.z" AND VALUE like "64"; +---------------------+-----------+------------+------------+-------+ | TIME_STAMP          | SE_IP     | VALUE_TYPE | VALUE_KEY  | VALUE | +---------------------+-----------+------------+------------+-------+ | 2013-10-16 12:30:07 | 172.x.y.z |         15 | 64 (covia) |    64 | | 2013-10-16 12:30:07 | 172.x.y.z |         11 | LinkedIN   |    64 | | 2013-10-16 12:30:07 | 172.x.y.z |          6 | SAT-EXPAK  |    64 | | 2013-10-16 12:30:07 | 172.x.y.z |          3 | 64 (covia) |    64 | +---------------------+-----------+------------+------------+-------+ How to determine what services SCE are trying to report on those numbers? In other words, how to configure SCE to report only such services which are classified and named in PQB/Reporter/INI_VALUES? I tried to upgrade latest PP36, reapply PQB (yes it successfully updates CM as well). SCE8000 3.7.2 CM  3.7.5 this combination is compatible !!? (RPT_PUR, INI_TABLES does not changes betveen 3.7.2 -> 3.7.5 by this Changes in Scripts, Tables, and RDRs ) thanks for your help. Regards: gyula ... View more