Re: Not enough bandwidth for ports

stephen2615 · ‎07-16-2009

Greetings,

I have some 24 port (4 Gpbs) cards in my Gen 2 line card 9509's and while trying to add 4 Gbps ISL's, I can't seem to get some ports to accept 4 Gpbs dedicated E port settings. Some ports do but some ports don't. I keep getting a popup saying not enough bandwidth.

This happens on ports 13 to 24. I already have 2 x 4 Gpbs E ports assigned and I can create E ports in some of the other ports which are all free.

I have configured ports 15 and 19 and I can't make port 23 into an E Port. I can make port 22 into an E port.

What would cause this error? The first 12 ports have 9 enabled and they are all running 4 Gbps shared mode to servers that are not pushing the bandwidth.

Stephen

sidbartle · ‎07-16-2009

Hi Stephen,

Use the "show port-resources module x" to see what resources are available to you. I got caught out with this and ISL's just the other day on a MDS9222i.

I found the sales pitch on these products to be very misleading â€œX times 4GB portsâ€ not true, as the Port-Group does not support the total bandwidth for the ports in the group.

Hope this helps

Sid

Michael Brown · ‎07-17-2009

There are 2 modes a port on the 24 port gen 2 line card can operate in. The default is shared, the other is dedicated. ISLs must be dedicated.

There are 6 port groups on the 24 port line card. As Sid noted, you can see how they line up with the show port-resources command.

The key to remember is that there is 12.8 GB of bandwidth for each port group.

If I run 1 port in dedicated mode at 4, that leave 8.8 GB for the other 5.

The line card design is such that if a port is in dedicated mode, and currently using all the BW it is configured for, that the other ports in the port group can use that bandwidth. If how ever the port in dedicated mode need the BW, the other ports in the port group in shared mode...would be throttled back so that the dedicated port gets what ever BW is dedicated to it.

If you needed to dedicate 3 ports in a group at 4GB, the other 3 ports would need to be put 'out of service' so that there was enough bandwidth to support the 3 x 4GB ports.

There is an automatic over-subscription checking that takes place, and it will not let you take the .8 GB left (assuming 3 x 4GB dedicated) and use it across the 3 remaining ports in shared mode...this can be turn off if needed.

The key to remember is that ISLs must on ports in dedicated mode...and dedicated ports will not let the BW sit unused if other ports in the group in shared mode need the bandwidth.

Hope this helps,

Mike

David McFarland · ‎07-17-2009

If you wish to disable the oversubsrcription limits you can do the following. This is disruptive to all ports on the specific line card.

Out-of service all ports on the line card

switch(config-if)# interface fc2/1-18

switch(config-if)# out-of-service

Putting an interface out-of-service will cause its shared resource configuration to revert to default

Do you wish to continue(y/n)? [n] y

switch(config-if)# sho port-resources module 2

Module 2

Available dedicated buffers are 4509

Port-Group 1

Total bandwidth is 12.8 Gbps

Total shared bandwidth is 12.8 Gbps

Allocated dedicated bandwidth is 0.0 Gbps

--------------------------------------------------------------------

Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode

Buffers (Gbps)

--------------------------------------------------------------------

fc2/1 (out-of-service)

fc2/2 (out-of-service)

fc2/3 (out-of-service)

fc2/4 (out-of-service)

fc2/5 (out-of-service)

fc2/6 (out-of-service)

switch(config-if)# no rate-mode oversubscription-limit module 2

switch(config)# interface fc2/1-18

switch(config-if)# no out-of-service

switch(config-if)# show port-resources module 2

Module 2

Available dedicated buffers are 3987

Port-Group 1

Total bandwidth is 12.8 Gbps

Total shared bandwidth is 12.8 Gbps

Allocated dedicated bandwidth is 0.0 Gbps

--------------------------------------------------------------------

Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode

Buffers (Gbps)

--------------------------------------------------------------------

fc2/1 16 4.0 shared

fc2/2 16 4.0 shared

fc2/3 16 4.0 shared

fc2/4 16 4.0 shared

fc2/5 16 4.0 shared

fc2/6 16 4.0 shared

switch(config-if)# interface fc2/1-3

switch(config-if)# switchport speed 4000

switch(config-if)# switchport rate-mode dedicated

switch(config-if)# show port-resources module 2Module 2 Available dedicated buffers are 3978

Port-Group 1

Total bandwidth is 12.8 Gbps

Total shared bandwidth is 0.8 Gbps

Allocated dedicated bandwidth is 12.0 Gbps

--------------------------------------------------------------------

Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode

Buffers (Gbps)

--------------------------------------------------------------------

fc2/1 16 4.0 dedicated

fc2/2 16 4.0 dedicated

fc2/3 16 4.0 dedicated

fc2/4 16 4.0 shared

fc2/5 16 4.0 shared

fc2/6 16 4.0 shared

Interfaces fc 2/4-6 are sharing 0.8GB of bandwidth.

Michael Brown · ‎07-17-2009

Just to be clear about the 0.8....if each of the dedicated ports are using 4GB, then the 3 shared ports are sharing 0.8. If those ports are using a combined 8 (of the 12 dedicated) there is 4.8 GB for the shared ports. We do not reserve BW, dedicated just means those ports are guaranteed BE if they need it. If they don't, the BW is available for the shared ports.

switch(config-if)# show port-resources module 2Module 2 Available dedicated buffers are 3978

Port-Group 1

Total bandwidth is 12.8 Gbps

Total shared bandwidth is 0.8 Gbps

Allocated dedicated bandwidth is 12.0 Gbps

--------------------------------------------------------------------

Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode

Buffers (Gbps)

--------------------------------------------------------------------

fc2/1 16 4.0 dedicated

fc2/2 16 4.0 dedicated

fc2/3 16 4.0 dedicated

fc2/4 16 4.0 shared

fc2/5 16 4.0 shared

fc2/6 16 4.0 shared

Interfaces fc 2/4-6 are sharing 0.8GB of bandwidth.

stephen2615 · ‎07-18-2009

Thanks to all who replied and offered excellent advice. I found the strangest thing when doing a show port-resources module x (which was a new command to me). Someone had set a port to 4 Gb dedicated and even though it was disabled, it counted for the bandwidth.

Stephen

Michael Brown · ‎07-20-2009

That correct, if a port is configured as dedicated...we allocate the BW even if the port is disabled. The only way to keep a port from being allocated any BW is to configure it for 'out of service'. As noted, if the port is disabled, the 4GB is still available for other shared ports to use. We do not reserve the 4GB because the port is down.