C9300 as MPLS PHP

[dervain.legbou-prestataire]

Hello dear,

Recently, while debugging an wireshark packet capture from a C9300 used as MPLS PHP, we noticed a double mpls label (transport and service label) in the frame even though in the mpls frowarding table the transport label is empty with instruction "to be POPed". Any other device like ASR1001x or C8300 was behaving differently in the same position and function as MPLS PHP: just the service label was seen. Did someone experience such behavior or is there any explanation that is unknown form my humble experiences?

[MHM Cisco World]

this depend if SW receive two or three MPLS label, the SW will POP only one upper label not all label

[dervain.legbou-prestataire]

Thks. Yes i know. The SW in this position as many other PE routers receive the same two label: the same Service Label for the VRF and the Implicit-null label since the next router towards the subnet is the last PE router. So the SW is the hop before the last hop before the subnet. The SW shows really that it will pop the outer label but in the packet capture this upper label is present and has value.

[filopeter]

Did you use the embedded wireshark on C9300 to get the capture? It might be an UADP ASIC feature, the capture might be done before the transport label is popped. 

[dervain.legbou-prestataire]

No. we used the classic "monitor capture" on the C9300 cli to get the capture. Maybe you're right...i need to confirm that

[MHM Cisco World]

https://www.packetmischief.ca/2015/05/27/mpls-no-label-vs-pop-label/

there is two modes 
no-label 
pop-label 
the link above talk about the different 

[dervain.legbou-prestataire]

Hi MHM, Thanks. We are in "pop-label" mode. we can see it in the cli "show mpls forwarding-table x.x.x.x detail"

[MHM Cisco World]

can you make double check are the traffic you debug is list in show mpls forwarding-table xxx detail as pop-label or no-label
if you can share 

show mpls forwarding-table detail

[dervain.legbou-prestataire]

Find it below

MPLS NH:x.x.x.x

sh mpls forwarding-table x.x.x.x detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
111 Pop Label x.x.x.x/32 20539033645 Te1/1/3 a.b.c.d
MAC/Encaps=14/14, MRU=1558, Label Stack{}
A8B456DE8E6AC014FEA092E68847
No output feature configured

VPN MOTO Subnets:y.y.y.0
sh mpls forwarding-table vrf MOTO y.y.y.0 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
None 39378 y.y.y.0/24[V] \
0 Te1/1/3 a.b.c.d
MAC/Encaps=14/18, MRU=1554, Label Stack{39378}
A8B456DE8E6AC014FEA092E68847 099D2000
VPN route: MOTO
No output feature configured

[MHM Cisco World · Accepted solution]

I make lab and use 
debug mpls packet 
you can see in R2 & R3 only show debug Rx but only R1 (P of MPLS SP) show Rx & Tx with two label. 
so I think the debug happened before data forwarding not after. 

[dervain.legbou-prestataire]

Hi MHM,

Thanks for taking your time to debug my concern. So we agree with @filopeter saying that this happen before real data forwarding. One more question: are all your MPLS Routers like C9300 ? because in my real world, i did the same packet capture on ASR and c8300 and the packet is built with only one label.

[MHM Cisco World]

in my lab as you can see there is one label but in wireshark there are two labels 
R1 POP the upper label (label using inside the SP Core) and keep only one Label which is VPN label. 
so it normal to see one label if the packet is L3 routed in this router/SW not SWAP/PUSH MPLS Label

[filopeter · Accepted solution]

From the ASIC point of view Cat9K is based on UADP, while ASR or Cat8K use QFP processor.  From the MPLS QoS point of view Cat9K uses by default MPLS pipe mode, while ASR1k and Cat8k use by default MPLS short-pipe mode.

I think that in case of Cat9K the labelled packet must be first internally processed and than popped, while in case of ASR1K the packet must be first popped and than internally processed. Therefore you may see different results in the packet capture from both platforms.

[dervain.legbou-prestataire · Accepted solution]

Finally we did a debug session in real non-prod environment  with true cat9k and went to the same result. But while looking in diverse CLI "platform software fed switch active" and "platform hardware fed switch active fwd-asic" we saw that the packet is processed like other platform: the topmost label has got flags [POP,PHP] because there is the label 3 (imp-null) and then the rewrite_type is POP2MPLS. All of above is confirming that there is a difference from cat9k perspective of such packet processing. There was no outage neither service disruption. Thanks to @filopeter and @MHM Cisco World for your comments and implication.