Introduction
Fabric Port of Exit (FPOE) is stored in the result received from the supervisor forwarding engine. FPOE is an 18-bit field which is placed in the Packet header. Fabric forwarding is based on this value. When a packet is forwarded by the line card, it is received by the fabric and stored in the fabric ports input buffer. The input buffer on the ingress fabric port has 2 queues high priority and low priority queues and depending on the priority value, the packet is demultiplexed in to one of these 2 queues. FPOE determines which fabric queue (high priority fabric or low priority fabric) to receive the packet. The switch fabric port of exit is indicated in the FPOE index field and is used by the fabric to forward a packet to the right destination. It is also used to identify which VOQ (Virtual Ouput Queuing) to put the packet into. For example, if we have many 64b frames in the VOQ, one FPOE header is placed on the larger block of 64b frames.
Requirements
There are no specific requirements for this document.
Troubleshooting tips
1. Test for TestLtlFpoeMemoryConsistency
Online Diagnostic test shows a failure for TestLtlFpoeMemoryConsistency. This is an FPOE/LTL memory consistency check and runs periodically every 15 seconds.
CAT6K# show diagnostic result switch 1 module 2
Current bootup diagnostic level: complete
Switch 1 Module 2: CEF720 48 port 1000mb SFP SerialNo : AAAAAAAHMD7
Overall Diagnostic Result for Switch 1 Module 2 : MAJOR ERROR
Diagnostic level at card bootup: complete
Test results: (. = Pass, F = Fail, U = Untested)
TestLtlFpoeMemoryConsistency ----> F
If self-correction fails, corrective action is triggered via GOLD which is to reset the module. The module is powered-down on third consecutive module reset. If self-correction passes, no action is taken. But if too many self-corrections occur within a short period of time, the module is reset.
A notification is seen as
%CONST_DIAG-SP-X-HM_MOD_RESET: Resetting Module < mod#> for software recovery, Reason: Failed TestLtlFpoeMemoryConsistency
2. FPOE channel to slot assignment
To find the FPOE channel to slot assignment use this command:
Syracuse#sh fabric fpoe map
slot channel fpoe
1 0 0
1 1 9
2 0 1
2 1 10
3 0 2
3 1 11
4 0 3
4 1 12
5 0 4
6 0 5
6 1 14
Troubleshooting examples
Example 1
Problem description:
Following error message was observed:
Feb 2 10:21:17.950 EST: SP: Linecard endpoint of Channel 7 lost Sync. To Lower fabric and trying to recover now!
Explanation:
This message is often caused by a line card not being properly seated. To identify the particular line card use the "show fpoe map" command.
Here is the output from that command,
Switch#sh fabric fpoe map
slot channel fpoe
1 0 0
1 1 9
2 0 1
2 1 10
3 0 2
3 1 11
4 0 3
4 1 12
5 0 4
6 0 5
6 1 14
7 0 6
7 1 15
8 0 7
8 1 16
9 0 8
9 1 17
The fpoe will be mapped to a specific line card slot. This is where Troubleshooting begins, Once the suspect linecard is identified your next action should be to schedule a removal and re-insert of that card to try to eliminate this message from re-occuring.
It will be necessary to enable "service Internal" globally before issuing the "show fabric fpoe map" command. Also, the "fpoe map" portion of that command is "hidden" and is not supported by the question mark. It will not show up as an option when using the question mark after "show fabric". But it should be accepted and return data. (with service internal configured).
In above case, the line card in slot 8 needs to be reseated in order to avoid the error messages. From the output of show fabric fpoe map, fpoe 7 points to the line card in slot 8 and that is card that is causing the error messages.
Recommended action:
Re-seat line card in slot 8.
Example 2
Problem description:
VLAN drops observed on switch at all corners in IOS only.
Explanation:
Packet drops are seen when running L3 traffic on switch in IOS. Same number of packets is dropped per port regardless of corner.
System Details:
- Cisco WS-C6509-E chassis
- Switch with B0 phys running firmware v2.7 in slot 1
- Sup4 in slot 5 running IOS
Packet drops occur on a linecard when the LtlFpoeMemoryConsistency online diag test is enabled on the Supervisor. Packets are dropped along a VLAN or VRF path where the packets are being lost across the fabric.
The problem occurs at room temp when traffic is running across the fabric between ports, while the LtlFpoeMemoryConsistency test is enabled. The packets will drop regardless of whether the traffic is layer 2 or 3.
Recommended action:
On Catalyst 6k with VS-S720-10G:
Disable the following diagnostic test for Active and Standby Supervisors where applicable, in the running configuration. Use the “no diagnostic monitor module <mod#> test TestLtlFpoeMemoryConsistency” command to disable the diagnostic test.
For VSS:
Use the “no diagnostic monitor switch <switch#> module <mod#> test TestLtlFpoeMemoryConsistency” command to disable the diagnostic test.
Related Information
Switch Fabric Architecture
Cisco Catalyst 6500 Series Switches
