topic Re: Connect 100G DCI link in BGW in Nexus Dashboard

Connect 100G DCI link in BGW

Deepak Bansal — Mon, 14 Oct 2024 00:41:26 GMT

Hi experts,

My customer wanted to connect 2*100G link in Inter-DCI link. we are using vxlan multisite setup. Hence the 100G link will be terminate to BGW leafs in both the DC.

My question is Cisco nexus switches supports 100G only in uplink ports. It means I need to convert some of the uplink port to downlink ports. I didnt find the configuration in NDFC. Can someone please guide me how to do it.

Please see the topology attached in attachments

Thanks,

Deepak Bansal

Re: Connect 100G DCI link in BGW

M02@rt37 — Mon, 14 Oct 2024 08:01:36 GMT

Hello @Deepak Bansal

When converting uplink ports to downlink ports, the oversubscription may become a factor depending on the switch architecture and the number of ports used. Nexus switches often have fixed bandwidth limits for port groups, so multiple high-speed connections (like 100G) on the same ASIC can cause oversubscription, leading to potential performance bottlenecks. It's important to check the switch's hardware architecture to understand how many ports share the same ASIC and to plan the port allocation accordingly, ensuring critical traffic isn't affected by contention. To manage this, distribute high-bandwidth links across different ASICs or port groups if possible to minimize oversubscription.

Re: Connect 100G DCI link in BGW

balaji.bandi — Mon, 14 Oct 2024 08:16:53 GMT

I was not sure on your diagram short and Long, can you elaborate to get more clarity ?

Its all depends on the Delivery of the provider handover presentation ? and what Model you have Nexus 9K as part of Leaf/Spine ?

If you are using 100GB then you do not need any converter (like SFP Module + QSFP ) - you can directly plug the SFP on the Uplink ports ( for the between DCI to DCI connection)

Make sure you choose the right optics depends on Service Provider Delivery -

Example SM (OS2) cable then you need LR Modules, OM4 - then you need MM Modules.

check the datasheet :

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/datasheet-c78-736282.html

It means I need to convert some of the uplink port to downlink ports

You do not need to do anything this, as Long as you have Configured the uplinks.

Note : in Most case DCI to DCI use Spine to switches to connect using Layer 3, depends on your design.

check some good presentation for understanding :

https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2024/pdf/BRKDCN-2988.pdf

Re: Connect 100G DCI link in BGW

Deepak Bansal — Mon, 14 Oct 2024 10:56:46 GMT

Thanks Balaji for your response.

I was not sure on your diagram short and Long, can you elaborate to get more clarity ? - Its DWDM links one long and other small to connect the inter DC l, we are going to built BGW square box topology and will use 100G LR-S SFP .

Leaf - N9k-C93180YC-FX3

Spine - N9K-C9364C-GX

If i understood correctly, i dont need to convert uplink to downlink the way we do in ACI . we just need to configure the uplink port as it is ?

Re: Connect 100G DCI link in BGW

balaji.bandi — Mon, 14 Oct 2024 12:00:21 GMT

If i understood correctly, i dont need to convert uplink to downlink the way we do in ACI . we just need to configure the uplink port as it is ?

For the connectiviy you do not need to do anything for the Links to work, how you want that port to work Layer 2 or Layer3 depends on your design and need. either its Leaf or Spine.

N9K-C9364C-GX - You can use any ports or Asic based on your choice since all 100GB ports.

N9k-C93180YC-FX3 - use uplink ports they are 100GB (6 10/25/40/50/100-Gigabit)

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/nb-06-nexus-9000-series-switches-so-cte-en.html

Re: Connect 100G DCI link in BGW

Deepak Bansal — Tue, 15 Oct 2024 02:19:01 GMT

Thanks for replying, we are using N9k-C93180YC-FX3, all the downlink and uplink ports falls under same asic 0. I guess it uses different type of architecture

BL1# show interface hardware-mapping
Legends:
SMod - Source Mod. 0 is N/A
Unit - Unit on which port resides. N/A for port channels
HPort - Hardware Port Number or Hardware Trunk Id:
HName - Hardware port name. None means N/A
FPort - Fabric facing port number. 255 means N/A
NPort - Front panel port number
VPort - Virtual Port Number. -1 means N/A
Slice - Slice Number. N/A for BCM systems
SPort - Port Number wrt Slice. N/A for BCM systems
SrcId - Source Id Number. N/A for BCM systems
MacIdx - Mac index. N/A for BCM systems
MacSubPort - Mac sub port. N/A for BCM systems
Ifg - Interface Group
InFPort - Internal Front port number.

-----------------------------------------------------------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
-----------------------------------------------------------------------------------------------------------

Eth1/49 1a006000 1 0 16 255 192 -1 0 16 32 4 0 193 0 32
Eth1/50 1a006200 1 0 20 255 196 -1 0 20 40 5 0 197 0 40
Eth1/51 1a006400 1 0 8 255 200 -1 0 8 16 2 0 201 0 16
Eth1/52 1a006600 1 0 12 255 204 -1 0 12 24 3 0 205 0 24
Eth1/53 1a006800 1 0 0 255 208 -1 0 0 0 0 0 209 0 0
Eth1/54 1a006a00 1 0 4 255 212 -1 0 4 8 1 0 213 0 8

Thanks,

Deepak Bansal

Re: Connect 100G DCI link in BGW

Deepak Bansal — Wed, 11 Jun 2025 00:02:50 GMT

Thanks Balaji, Indeed nothing needs to be done. Just connect the SFP and cable and port comes up, its takes some time for NDFC and needs two deployment trigger.