Background:

rahassan · ‎09-19-2013

Background:

When applying SBC-B2B, as we know, the configuration dependence between SBC, interface SBC, traffic interface and vrf looks very like the figure below.

                 /\
                /  \
               /SBC \
              /.adj  \
             /.med-add\
            /----------\
           /  intf SBC  \
          / Traffic intf \
         /----------------\
        /       VRF        \
       /--------------------\
  Fig 1. Pyramid of SBC-B2B config

VRF config is the foundation of other modules'.

Problem:

Due to the defect CSCuf39344, when doing "no attach" then "attach" on an adjacency, the "VPN-ID" of this adjacency is updated.
As we know there may be two VRF Entries for a certain vrf in SBC B2B redundancy environment, one created locally and one synced from peer.
If an adjacency was configured on A-Box when it was ACTIVE, after switch-over, although calls through that adjacency are still OK, "no attach/attach" the adjacency on the new ACTIVE B-Box can cause the "VPN-ID" updated referring to the VRF Entry that locally configured on B-Box other than that synced from A-Box.

Incorrect config order can also lead to this problem:
If "no vrf definition <vrf-name>" was done before removing "media-address ipv4 <ip-addr> vrf <vrf-name>" and "vrf <vrf-name>" under adjacency, and later that vrf was readded and adjacency's "vrf <vrf-name>" was reconfigured.

Both method will cause adjacency's and media-address's "VPN-ID" different from each other.

The symptom is that:
1. calls are rejected with "503 Service Unavaliable" even if all configurations seem correct,
2. "media-address ipv4 <ip-addr> vrf <vrf-name>" may be displayed as "media-address ipv4 <ip-addr>" or "media-address ipv4 <ip-addr> vrf <some-other-vrf-name>" and cannot be removed.

Trouble shooting (Technical detail):

This problem can be revealed with the following show commands:
ASR# show redundancy application group 1

...
RF state: ACTIVE      // make sure the following commands will be executed on ACTIVE box!!!
Peer RF state: STANDBY HOT
...

ASR# show sbc <sbc-name> rg detail

...
VRF Entry(vrf123): localid=5, tableid=0, vrf_id=65535, active=1, afi=0
VRF Entry(vrf123): localid=6, tableid=3, vrf_id=3, active=0, afi=0
...

Configure the following command to use the following show commands.
ASR(config)# service internal

ASR# show sbc <sbc-name> sbe mib vpssadjtable

...
vpssAdjVrfName = vrf123
vsppAdjVpnId   = 6
...

ASR# show sbc <sbc-name> sbe mib mgmmediaaddresstable // never run this on Standby

...
vpn_id                   5
vpn_name:                vrf123
...

We can see that adjacency refers to VRF Entry whose localid = 6 while media-address refers to 5.

If you found some media-address cannot be removed, then you can check the vrf table:
ASR# show vrf tableid

VRF Name                         Tableid              Address Family
...
vrf123                           0x00000003           ipv4
vrf456                           0x00000004           ipv4
...

media-address refers to 5 which is the VRF Entry whose localid is 5.
However that VRF Entry's tableid is 0, not the value "0x00000003", or even worse there is no "vrf123" in the vrf table,
then you cannot remove that media-address record.

Best practice:

When configuring, please follow these steps (building pyramid from bottom up):

VRF  ->  intf SBC & Traffic intf  ->  SBC(adjacency & media-address)

When removing, it should be done in the reverse order (tearing down pyramid from top down):

SBC(adjacency & media-address)  ->  intf SBC & Traffic intf  ->  VRF

And make sure all configuring is done on the same box.

Or, we may meet problem.

For detailed steps, see below:
Configuring

ACTIVE
     vrf definition
     intf SBC
     Traffic intf
STANDBY
     vrf definition
     intf SBC
     Traffic intf
ACTIVE
     adjacency's vrf&signaling-address and media-address
          // make sure to complete adjacency & media-address config on the same box
     ASR(conf-sbc)# sync
Wait some time until SBC config fully synced to STANDBY

Tearing down

ACTIVE
     remove adjacency(which to be removed) related config under cac-policy-set and call-policy-set
     remove adjacency
          // "no attach force abort" before removing adjacency
     remove media-address
          // make sure that "media-address ... vrf ..." is NOed on the box where it was configured
     ASR(conf-sbc)# sync
Wait some time until SBC config fully synced to STANDBY
STANDBY
     if Traffic intf or intf SBC was tracked, remove track first
          ASR(config)#redundancy
          ASR(config-red)#application redundancy
          ASR(config-red-app)#group <group-number>
          ASR(config-red-app-grp)#no track <track-number>

          ASR(config)#no track <track-number>
     "no redundancy group <number>" under Traffic intf first!!!
          // show run again to make sure it was removed
          // or will trigger an unnecessary re-init!
     remove Traffic intf
     remove intf SBC
     remove vrf definition
ACTIVE
     if Traffic intf or intf SBC was tracked, remove track first
          ASR(config)#redundancy
          ASR(config-red)#application redundancy
          ASR(config-red-app)#group <group-number>
          ASR(config-red-app-grp)#no track <track-number>

          ASR(config)#no track <track-number>
     "no redundancy group <number>" under Traffic intf first!!!
          // show run again to make sure it was removed
          // or will trigger an unnecessary switch-over!
     remove Traffic intf
     remove intf SBC
     remove vrf definition

If it is not possible for you to always make configuration changes on the same box, then at least configure/remove a certain group of VRF, interface SBC, adjacency and media-address which all refer to that VRF on the same box.

How to identify on which side were those configurations added:

Assume there are two ASRs: ASR-A and ASR-B and currently ASR-A is Active one.

ASR-A# show sbc <sbc-name> sbe mib vpssadjtable

...
vpssAdjVrfName = vrf123
vsppAdjVpnId   = 5
...

ASR-A# show sbc <sbc-name> sbe mib mgmmediaaddresstable // never run this on Standby

...
vpn_id                   5
vpn_name:                vrf123
...

ASR-A# show sbc <sbc-name> rg detail

...
VRF Entry(vrf123): localid=5, tableid=0, vrf_id=65535, active=1, afi=0
VRF Entry(vrf123): localid=6, tableid=3, vrf_id=3, active=0, afi=0
...

ASR-B# show sbc <sbc-name> rg detail

...
VRF Entry(vrf123): localid=5, tableid=4, vrf_id=4, active=0, afi=0
VRF Entry(vrf123): localid=6, tableid=0, vrf_id=65535, active=1, afi=0
...

Adjacency and media-address both refer to 5.
On ASR-A VRF Entry(localid=5)'s tableid is 0 but on ASR-B VRF Entry(localid=5)'s tableid is not 0.
Then the adjacency and media-address must be configured on ASR-B.
If you want to change or remove those configurations, you should first switch-over to ASR-B.

One possible workaround:

If we have already met the problem, and we can't reload the box immediately, have a try the workaround below, but with the notice that the workaround was developed in lab, no promise that always works.

Introduction:
The main goal of this workaround is to remove and reconfig vrf related CLIs where adjacency and media-address under that vrf do not serve calls (returning 503).
This workaround needs a switch-over, because without a switch-over, after removed one vrf, interface SBC cannot be reconfigured due to been locked by media-address which itself also cannot be removed.
First of all, make sure configurations on both active and standby sides are the same.

remove stale config

BOX-2 (STANDBY)
     ASR(conf)# no vrf definition vrf123
     ASR(conf)# no interface SBC123

BOX-1 (ACTIVE)
     ASR(conf)# no vrf definition vrf123
          // in most case, this will cause BOX-1 switched to be STANDBY, and BOX-2 to be ACTIVE
          // check its state with this CLI: "ASR# show redundancy application group 1"
          // if not switched over, do switching over manually
          ASR# redundancy application reload group 1 self
     ASR(conf)# no interface SBC123

wait some time, till BOX-2 be ACTIVE, and BOX-1 be STANDBY-HOT

ASR# show redundancy application group 1    // use this command to check their states

reconfig

BOX-2 (ACTIVE)
     reconfig vrf vrf123
     reconfig interface SBC123
     reconfig traffic interface (like gig0/0/0.123)
     reconfig adjacency(vrf & signaling-address) and media-address related to vrf123

BOX-1 (STANDBY)
     reconfig vrf vrf123
     reconfig interface SBC123
     reconfig traffic interface (like gig0/0/0.123)

check

BOX-2 (ACTIVE)
     ASR1004-2-DEV# show sbc test sbe mib vpssadjtable
          ...
          vsppAdjVpnId   = 5
          ...
     ASR1004-2-DEV# show sbc test sbe mib mgmmediaaddresstable  // never run this on Standby
          ...
          vpn_id                   5
          ...
     // make sure the values of vsppAdjVpnId and vpn_id are the same

VPN-ID mismatch issue in SBC B2B

Background:

Problem:

Trouble shooting (Technical detail):

Best practice:

How to identify on which side were those configurations added:

One possible workaround: