NCS devices not passing RFC 2544 100G test

[DGIxcoy]

RFC 2544 tests fail on all frame sizes below 9000 bytes. The results are also worse the smaller the frame size. I have tried adding a policy-map on the egress interface per the post below with no succes:

https://community.cisco.com/t5/xr-os-and-platforms/ncs540-is-not-passing-the-benchmark-10g-rfc-test/td-p/4674301

Current configuration is using evpn as this is what we have deployed in the field but I have also tried p2p xconnect configuration. I have also tested with different NCS devices and all appear to have the same issue. 

Topology:

TestBox <100G> N540 <100G> ASR9906 <100G> TestBox2

Also tested:

TestBox <100G> NCS5501<100G> ASR9906 <100G> TestBox2

TestBox <100G> N540 <100G> NCS5501<100G> TestBox2

Has anyone had success with running a 100G RFC 2544 test? 

[smilstea]

I apologize I must have missed you note on the other thread.

Since you have the same topology just different speeds of links on access side this should be essentially the same with the 22 byte offset. See https://community.cisco.com/t5/xr-os-and-platforms/ncs540-is-not-passing-the-benchmark-10g-rfc-test/td-p/4674301?attachment-id=217116

This post in general details the account overhead problem on these chipsets that you see https://community.cisco.com/t5/xr-os-and-platforms/ncs540-is-not-passing-the-benchmark-10g-rfc-test/m-p/4679988/highlight/true#M13347

What I am thinking is because you are sending at linerate already you might need to make the change on both imposition and disposition routers! Not just disposition like in the other thread, otherwise I am thinking you have output/tail drops on imposition.

We can try running this command in combination with and without the user-defined 22 defined to see which router is dropping traffic, I would expect not to see drops on the disposition NCS when the user-defined is configured (unless you really are sending at linerate, and then we need to remember ethernet overhead of 14B on all packets). We can also check 'monitor interface' on the imposition and disposition routers to see the real traffic rates (may take a minute to saturate the link, you can use load-interval 30 on the interface to get stats faster).

If enabling on the imposition and disposition PEs doesn't help I'll lab this up for you or jump on a webex with you to investigate.

Sam

Sam

[DGIxcoy]

I applied the service-policy on the both routers and I am still failing the RFC test. I added with and without the user-defined statement. 

RP/0/RP0/CPU0:N540-01#show controller npu stats voq ingress interface all instan$
Fri Apr 21 21:58:12.323 UTC

Interface Name = Hu0/0/1/0
Interface Handle = 8
Location = 0/0/CPU0
Asic Instance = 0
VOQ Base = 1280
Port Speed(kbps) = 100000000
Local Port = local
ReceivedPkts ReceivedBytes DroppedPkts DroppedBytes
-------------------------------------------------------------------
Core-0:
TC_0 = 121613404 371714101262 8502085 24708251000
TC_1 = 0 0 0 0
TC_2 = 0 0 0 0
TC_3 = 0 0 0 0
TC_4 = 0 0 0 0
TC_5 = 0 0 0 0
TC_6 = 47173 3632321 0 0
TC_7 = 414 57075 0 0

RP/0/RP0/CPU0:N540-01#show controller npu stats voq ingress interface all instan$
Fri Apr 21 22:01:27.610 UTC

Interface Name = Hu0/0/1/0
Interface Handle = 8
Location = 0/0/CPU0
Asic Instance = 0
VOQ Base = 1280
Port Speed(kbps) = 100000000
Local Port = local
ReceivedPkts ReceivedBytes DroppedPkts DroppedBytes
-------------------------------------------------------------------
Core-0:
TC_0 = 140252219 447306216413 11010562 29725205000
TC_1 = 0 0 0 0
TC_2 = 0 0 0 0
TC_3 = 0 0 0 0
TC_4 = 0 0 0 0
TC_5 = 0 0 0 0
TC_6 = 61145 4708165 0 0
TC_7 = 543 74838 0 0

[smilstea]

In which direction and on which box are we seeing the drops, ingress PE or egress PE?

Can we get a TAC case open so we can get a recreate started internally to follow up with development? I can track the SR with you and TAC.

Sam

[DGIxcoy]

Drops are on egress PE. I have a TAC case open, I will IM you with case number. 

[writersera21]

Based on the information you provided, it seems like there might be a problem with the configuration of your network devices or the hardware itself. Here are some things you could try to troubleshoot the issue:

1. Check the configuration of your devices to make sure that they are configured correctly for 100G traffic. You may want to consult the documentation or reach out to the vendor for assistance.

2. Check the hardware specifications of your devices to ensure that they are capable of handling 100G traffic and that they meet the requirements of RFC 2544. You may want to run some diagnostics on your devices to verify their functionality.

3. Try different frame sizes and see if the results improve. Sometimes certain frame sizes may be problematic, so experimenting with different sizes may help isolate the issue.

4. Check the connectivity between your devices to make sure that there are no issues with the cabling or the interfaces.

5. If you have access to a different set of devices, try to reproduce the issue in a lab environment to isolate the problem.

6. If the issue persists, you may want to consider reaching out to the vendor for support or engaging a professional services team to assist with troubleshooting.

I also recommend you to give these a read. These will help you greatly.

https://community.cisco.com/t5/xr-os-and-platforms/ncs540-is-not-passing-the-benchmark-10g-rfc-test/td-p/4674301

action

https://community.cisco.com/t5/network-management/rfc-2544/td-p/1304995

[jencia]

In Cisco ASR9000 series routers, you typically don't format hard disks or perform similar operations directly from the ROM Monitor (ROMMON) mode. ROMMON is a low-level diagnostic mode used for disaster recovery and does not provide the same level of functionality as the router's regular operating system (IOS XR in the case of ASR9000 series routers).

To format a hard disk or perform disk-related operations on an ASR9001 router, you would typically do this from the IOS XR operating system. Here are the general steps to format a hard disk on an ASR9001:

1. Access the router: Connect to the router using a terminal emulator or through a console connection.

2. Enter privileged EXEC mode: You'll need to be in privileged EXEC mode to perform this task. Use the enable command and enter the enable password if required.

3. Access the disk submode: Use the disk command to access the disk submode. This is where you can format, create, and manage disk partitions.

bashCopy code

ASR9001# configure terminal ASR9001(config)# disk ASR9001(config-disk)#

1. Format the disk: To format a disk, use the format command followed by the disk device name. For example, if your disk is labeled disk0::

bashCopy code

ASR9001(config-disk)# format disk0:

1. Confirm the operation: The router will prompt you to confirm the format operation. Be aware that this will erase all data on the disk, so make sure you have a backup if needed.

2. Wait for the format to complete: The time it takes to format a disk can vary depending on the size and type of the disk.

3. Exit the disk submode and save the configuration:

bashCopy code

ASR9001(config-disk)# exit ASR9001(config)# exit ASR9001# write memory ASR9001# exit

This will save the configuration and exit back to privileged EXEC mode.

Please be extremely cautious when formatting disks on a production router, as it will result in data loss. Ensure that you have proper backups and take the necessary precautions to avoid disrupting network services. Additionally, the specific commands and procedures may vary slightly depending on the IOS XR version running on your ASR9001 router, so consult the documentation for your specific software version for the most accurate instructions.

you can also refer to this for more information,

https://community.cisco.com/t5/xr-os-and-platforms/ncs-5501-se-reboot-after-upgrade-from-6-6-3-to-7-1-2/td-p/451872

 https://newcapcutmod.pro 

https://community.cisco.com/t5/xr-os-and-platforms/ncs-5501-se-upgrade-from-7-0-2-to-7-5-2/td-p/4709287