Solved: Re: 6748 vs 6148A line card

vdadlaney · ‎11-24-2009

Hi, I was almost sure i had seen this discussion on the specific difference in the buffering on these 2 cards however I am unable to find it so would appreciate some help.

1. Per documentation it is well known that the 6748 has a lower per-port tx (1.2 or 1.3 depending on the document) buffer than the 6148A (5.2 Mb). I would like to know how this affects this module and why is it better to use the 6748 in a high bandwidth environment (Assuming we take out the DFC). The 6148A is a classic line card and as such would use the classic bus wheras the 6748 would use the 2 x 20Gb connection to the fabric but doesn't the per port buffering become critical if there is a large amount of traffic rate.

2. Wanted to confirm something regarding drops and buffers. Since these buffers are Tx buffers wouldn't they be critical only when the port is an outgoing port (i.e traffic leaving the switch) since in most cases the Rx buffer will only be used to hold the frames until we get access to the fabric but in the case of the 6748 it has dedicated connection to the fabric so it shouldn't be a problem. Am I correct in my assumption?

3. When would the Tx buffers become critical either for the 6748 or the 6148.

Thx

Jon Marshall · ‎11-25-2009

vdadlaney wrote:

Hi Jon,

Thx for replying. I had a few follow questions regarding the buffers

1. Wouldn't the higher per port buffer on the 6148 make it better for high throughput connections since it can hold more in memory hence suffer less drops? the connection to the fabric is dedicated on the 6748 hence it can switch traffic out faster however if the egress port is getting getting a lot more traffic than it can handle it would have to store the frames in the buffer. Wouldn't that be more critical specially in a situation where the egress port is a uplink hence all ingress ports are sending traffic to the egress port.

2. With regards to the drops if the Tx-buffer is full than would that be indicated by the higher number of output drops on the counters?

3. If the output buffers become full on the coil asic and than the buffers begin to fillup on the pinnacle hence causing Head of line blocking would this be reflected in the input drops on the counters?

4. If a system has both 6748 without DFC and 6148 line cards than how is the traffic going to be sent from the 6748 input port to the 6148 output port for example being that the 6748 does not send traffic over the shared bus and the 6148 only connects to the shared bus.

5. Would the traffic between Fabric only line cards be affected as a result of the classic line card being present or would it only affect traffic that goes between the fabric only and classic line card. I always thought that traffic between fabric only line cards would not be affected even if a classic line card was present. Pls confirm

Thx for your help.

Okay, i'll answer the ones i can but i don't know the answer to all questions. Hopefully someone else can add to this -

1) Not really no. A 6148A cannot get anywhere near the throughput performance of a 6748 for the reasons i covered. So lets imnagine you have 40 servers you want to attach to a 48 port module. With a 6748 all 40 servers can in theory receive their full gigabit allowance from the module. With a 6148A they would get nowhere near.

Now about the egress port being an uplink that aggregates all the ingress ports throughputs. Again, the 6748 would be able to overload the buffer quicker but that doesn't mean the 6148A is better. And if you are aggregating many into one or two for example then you would assume the traffic patterns had been taken into account in the design and this oversubscription was acceptable.

Again it is important to note you are talking about congestion situations, but these don't happen all the time and if the network has been designed properly in terms of server location, layout then you can mitigate some of these.

2) Should be yes.

3) Don't know.

4) Because the supervisor has both and hence can transmit packets between the 2.

5) Yes it would. Have a look at this doc and search for "truncated".

Basically if you are only doing centralised forwarding and all your line cards are fabric then the switch can achieve up to 30Mpps using compact mode switching.

If you add just one classic line card then the switch has to go to truncated mode. In truncated mode with centralised forwarding the switch can only achieve 15Mpps.

6500 architecture

Jon

View solution in original post

Edison Ortiz · ‎11-27-2009

Hi Edison,

Thx for replying. Could you please elaborate a bit on your reply in point 1. I was referring to this document "http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html" and per my interpretation of what is written here in the second paragraph of section "Overview of Buffers, Queues, and Thresholds" it says

"In the Catalyst 6500 architecture, access into the switch fabric itself is almost never the bottleneck. Rather, on the transmit side, one or several ports are the likely destination for a majority of the packets entering the switch. As such, the receive-side port buffers on the Ethernet modules are relatively small compared to the transmit-side port buffers."

You need to read between the lines. The 6148 does not have a connection to the switch fabric, it connects to the Supervisor via the bus (32Gbps shared connection). The 67xx modules have access to the switch fabric.

With regards to the 6748 would it possibly help to make use of the buffers on the Janus Asic. Is that even possible? Per my understanding if its bursty traffic than this could help however am very hesitant as this might affect all the ports on the Rohini asic which I believe is the equivalent of the Coil asic

Buffers add latency into the data flow. You don't want latency in a switched network so large buffers in a line card can be counterproductive.

While large buffers in the classic line card can be a marketing ploy for competitive reasons, classic line cards are often targeted for workstation connections. You don't want large buffers & latency while connecting to servers and interswitch links - you want the packet to have the same latency and speed entering and exiting a switch hardware. To mitigate the lack of buffers, it is often recommended to configure flow-control.

Regards

Edison

Please remember to rate helpful posts, thanks.

View solution in original post

Jon Marshall · ‎11-24-2009

vdadlaney wrote:

Hi, I was almost sure i had seen this discussion on the specific difference in the buffering on these 2 cards however I am unable to find it so would appreciate some help.

1. Per documentation it is well known that the 6748 has a lower per-port tx (1.2 or 1.3 depending on the document) buffer than the 6148A (5.2 Mb). I would like to know how this affects this module and why is it better to use the 6748 in a high bandwidth environment (Assuming we take out the DFC). The 6148A is a classic line card and as such would use the classic bus wheras the 6748 would use the 2 x 20Gb connection to the fabric but doesn't the per port buffering become critical if there is a large amount of traffic rate.

2. Wanted to confirm something regarding drops and buffers. Since these buffers are Tx buffers wouldn't they be critical only when the port is an outgoing port (i.e traffic leaving the switch) since in most cases the Rx buffer will only be used to hold the frames until we get access to the fabric but in the case of the 6748 it has dedicated connection to the fabric so it shouldn't be a problem. Am I correct in my assumption?

3. When would the Tx buffers become critical either for the 6748 or the 6148.

Thx

1) It's better because as you noted the 6748 has 2 x 20Gbps dedicated connections to the switch fabric whereas the 6148A is a classic line card and therefore must share the 32Gbps bus with every other classic line card in the chassis. Note if you don't have any other classic line cards in the chassis then this single classic line card can degrade the overall performance of your switch.

The tx buffers are much bigger than the rx buffer because most modules experience congestion when traffic leaves the switch whereas access to the switch fabric is usually not the issue. The rx buffer is primarily to store packets whilst the forwarding decision is made for the packet. The tx buffer is used to queue frames whilst they are placed onto the physical medium.

So the 6748 can run 40 of it's 48 ports with no oversubscription whereas with the 6148A it's not actually possible to say really as it does not have dedicated connections.That is why the 6748 should be used as server aggregation etc. and the 6148A is for use in the access-layer where oversubscription of the module is acceptable.

So yes the port buffers can become important but it's really not as simple as just looking at the actual figure in Mb.

2) Pretty much yes. It should be noted however that without a DFC frames might still be queued in the rx queue on a 6748 whilst the forwarding decision is made.

3) When there is congestion and packets need queueing before being transmitted from the switch.

Jon

vdadlaney · ‎11-25-2009

Hi Jon,

Thx for replying. I had a few follow questions regarding the buffers

1. Wouldn't the higher per port buffer on the 6148 make it better for high throughput connections since it can hold more in memory hence suffer less drops? the connection to the fabric is dedicated on the 6748 hence it can switch traffic out faster however if the egress port is getting getting a lot more traffic than it can handle it would have to store the frames in the buffer. Wouldn't that be more critical specially in a situation where the egress port is a uplink hence all ingress ports are sending traffic to the egress port.

2. With regards to the drops if the Tx-buffer is full than would that be indicated by the higher number of output drops on the counters?

3. If the output buffers become full on the coil asic and than the buffers begin to fillup on the pinnacle hence causing Head of line blocking would this be reflected in the input drops on the counters?

4. If a system has both 6748 without DFC and 6148 line cards than how is the traffic going to be sent from the 6748 input port to the 6148 output port for example being that the 6748 does not send traffic over the shared bus and the 6148 only connects to the shared bus.

5. Would the traffic between Fabric only line cards be affected as a result of the classic line card being present or would it only affect traffic that goes between the fabric only and classic line card. I always thought that traffic between fabric only line cards would not be affected even if a classic line card was present. Pls confirm

Thx for your help.

Jon Marshall · ‎11-25-2009

vdadlaney wrote:

Hi Jon,

Thx for replying. I had a few follow questions regarding the buffers

1. Wouldn't the higher per port buffer on the 6148 make it better for high throughput connections since it can hold more in memory hence suffer less drops? the connection to the fabric is dedicated on the 6748 hence it can switch traffic out faster however if the egress port is getting getting a lot more traffic than it can handle it would have to store the frames in the buffer. Wouldn't that be more critical specially in a situation where the egress port is a uplink hence all ingress ports are sending traffic to the egress port.

2. With regards to the drops if the Tx-buffer is full than would that be indicated by the higher number of output drops on the counters?

3. If the output buffers become full on the coil asic and than the buffers begin to fillup on the pinnacle hence causing Head of line blocking would this be reflected in the input drops on the counters?

4. If a system has both 6748 without DFC and 6148 line cards than how is the traffic going to be sent from the 6748 input port to the 6148 output port for example being that the 6748 does not send traffic over the shared bus and the 6148 only connects to the shared bus.

5. Would the traffic between Fabric only line cards be affected as a result of the classic line card being present or would it only affect traffic that goes between the fabric only and classic line card. I always thought that traffic between fabric only line cards would not be affected even if a classic line card was present. Pls confirm

Thx for your help.

Okay, i'll answer the ones i can but i don't know the answer to all questions. Hopefully someone else can add to this -

1) Not really no. A 6148A cannot get anywhere near the throughput performance of a 6748 for the reasons i covered. So lets imnagine you have 40 servers you want to attach to a 48 port module. With a 6748 all 40 servers can in theory receive their full gigabit allowance from the module. With a 6148A they would get nowhere near.

Now about the egress port being an uplink that aggregates all the ingress ports throughputs. Again, the 6748 would be able to overload the buffer quicker but that doesn't mean the 6148A is better. And if you are aggregating many into one or two for example then you would assume the traffic patterns had been taken into account in the design and this oversubscription was acceptable.

Again it is important to note you are talking about congestion situations, but these don't happen all the time and if the network has been designed properly in terms of server location, layout then you can mitigate some of these.

2) Should be yes.

3) Don't know.

4) Because the supervisor has both and hence can transmit packets between the 2.

5) Yes it would. Have a look at this doc and search for "truncated".

Basically if you are only doing centralised forwarding and all your line cards are fabric then the switch can achieve up to 30Mpps using compact mode switching.

If you add just one classic line card then the switch has to go to truncated mode. In truncated mode with centralised forwarding the switch can only achieve 15Mpps.

6500 architecture

Jon

wsmith@ca.ibm.com · ‎11-25-2009

6148 is signifgantly over-subscribed on its asics and backplane interface. Best for user-facing or as a landfill device these days. We throw them out.

6748 is a full CEF720 enabled card and is capable of 40Gbs

TX buffers are signifigant in any high or spikey load.. In particular if you are using these for uplinks to other switches.

Edison Ortiz · ‎11-25-2009

vdadlaney wrote:

Hi, I was almost sure i had seen this discussion on the specific difference in the buffering on these 2 cards however I am unable to find it so would appreciate some help.

1. Per documentation it is well known that the 6748 has a lower per-port tx (1.2 or 1.3 depending on the document) buffer than the 6148A (5.2 Mb). I would like to know how this affects this module and why is it better to use the 6748 in a high bandwidth environment (Assuming we take out the DFC). The 6148A is a classic line card and as such would use the classic bus wheras the 6748 would use the 2 x 20Gb connection to the fabric but doesn't the per port buffering become critical if there is a large amount of traffic rate.

2. Wanted to confirm something regarding drops and buffers. Since these buffers are Tx buffers wouldn't they be critical only when the port is an outgoing port (i.e traffic leaving the switch) since in most cases the Rx buffer will only be used to hold the frames until we get access to the fabric but in the case of the 6748 it has dedicated connection to the fabric so it shouldn't be a problem. Am I correct in my assumption?

3. When would the Tx buffers become critical either for the 6748 or the 6148.

Thx

1. No, the data contention will occur on the backplane, not on egress. On egress, you will be connected to a device that will perform line rate data thus buffering will have minimum use. That's the reason you see this values being so low. As you stated, if you have a Sup720, there is no reason for purchasing a 6148 over a 6748. Now, if you have a Sup32 then the 6148 will be your only choice.

2. It's very rare to see output drops on a line card if the connected device is able to sustain line rate speed. If the connected device is unable to handle the line rate speed, then you can implement flow-control. I mean, even if you had 5.2Mb on the 6748 and you are sending at 1Gbps - how much that buffer can help? You also need to take into account the 6748 is a 10/100/1000 module while the 6148 is only 10/100.

Regards

Edison.

vdadlaney · ‎11-27-2009

ediortiz wrote:

1. No, the data contention will occur on the backplane, not on egress. On egress, you will be connected to a device that will perform line rate data thus buffering will have minimum use. That's the reason you see this values being so low. As you stated, if you have a Sup720, there is no reason for purchasing a 6148 over a 6748. Now, if you have a Sup32 then the 6148 will be your only choice.

2. It's very rare to see output drops on a line card if the connected device is able to sustain line rate speed. If the connected device is unable to handle the line rate speed, then you can implement flow-control. I mean, even if you had 5.2Mb on the 6748 and you are sending at 1Gbps - how much that buffer can help? You also need to take into account the 6748 is a 10/100/1000 module while the 6148 is only 10/100.

Regards

Edison.

Hi Edison,

Thx for replying. Could you please elaborate a bit on your reply in point 1. I was referring to this document "http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html" and per my interpretation of what is written here in the second paragraph of section "Overview of Buffers, Queues, and Thresholds" it says

"In the Catalyst 6500 architecture, access into the switch fabric itself is almost never the bottleneck. Rather, on the transmit side, one or several ports are the likely destination for a majority of the packets entering the switch. As such, the receive-side port buffers on the Ethernet modules are relatively small compared to the transmit-side port buffers."

In your reply in point 1 you mention that Data contention will occur on the backplane and not on egress which seems to be contradictory to what is written in the document per my understanding. Please correct me if I have misunderstood.

With regards to the 6748 would it possibly help to make use of the buffers on the Janus Asic. Is that even possible? Per my understanding if its bursty traffic than this could help however am very hesitant as this might affect all the ports on the Rohini asic which I believe is the equivalent of the Coil asic

Thx for your help.

Edison Ortiz · ‎11-27-2009

Hi Edison,

Thx for replying. Could you please elaborate a bit on your reply in point 1. I was referring to this document "http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html" and per my interpretation of what is written here in the second paragraph of section "Overview of Buffers, Queues, and Thresholds" it says

"In the Catalyst 6500 architecture, access into the switch fabric itself is almost never the bottleneck. Rather, on the transmit side, one or several ports are the likely destination for a majority of the packets entering the switch. As such, the receive-side port buffers on the Ethernet modules are relatively small compared to the transmit-side port buffers."

You need to read between the lines. The 6148 does not have a connection to the switch fabric, it connects to the Supervisor via the bus (32Gbps shared connection). The 67xx modules have access to the switch fabric.

With regards to the 6748 would it possibly help to make use of the buffers on the Janus Asic. Is that even possible? Per my understanding if its bursty traffic than this could help however am very hesitant as this might affect all the ports on the Rohini asic which I believe is the equivalent of the Coil asic

Buffers add latency into the data flow. You don't want latency in a switched network so large buffers in a line card can be counterproductive.

While large buffers in the classic line card can be a marketing ploy for competitive reasons, classic line cards are often targeted for workstation connections. You don't want large buffers & latency while connecting to servers and interswitch links - you want the packet to have the same latency and speed entering and exiting a switch hardware. To mitigate the lack of buffers, it is often recommended to configure flow-control.

Regards

Edison

Please remember to rate helpful posts, thanks.

vdadlaney · ‎11-27-2009

Hi Edison,

ediortiz wrote:

You need to read between the lines. The 6148 does not have a connection to the switch fabric, it connects to the Supervisor via the bus (32Gbps shared connection). The 67xx modules have access to the switch fabric.

Thx for confirming that however for some reason I don't believe the author was referring to the switch fabric from a line card perspective since they do talk about all line cards in that document. I have however requested feedback on this and as is worded right now it does seem to be referring to cards that connect to the switch fabric so thx for pointing that out.

I am really sorry for being a pain here but I do want to understand this completely so can u pls elaborate on this part of your original reply

"On egress, you will be connected to a device that will perform line rate data thus buffering will have minimum use"

My understanding is that on egress means when traffic is leaving or going out of the switch so it would mean the Tx side, however buffers are always higher on the Tx side than the Rx side as is also mentioned in the document. From your above statement per my understanding it means that on egress buffering will have minimum use and if that is the case than why are the Tx buffers always higher than the Rx buffers.

Also as I have understood would it be true to say in the case of a 6748 that contention will occur on the Tx side and not on the fabric since it has a dedicated connection to the fabric however for the 6148 the contention will occur on the backplane since it dosen't connect to the fabric.

Wow now my head is spinning and I think i am overthinking this way too much. Sorry to bother. Appreciate if you could verify my understanding and correct if incorrect.Thx for all your help.

Edison Ortiz · ‎12-02-2009

My understanding is that on egress means when traffic is leaving or going out of the switch so it would mean the Tx side, however buffers are always higher on the Tx side than the Rx side as is also mentioned in the document. From your above statement per my understanding it means that on egress buffering will have minimum use and if that is the case than why are the Tx buffers always higher than the Rx buffers.

Also as I have understood would it be true to say in the case of a 6748 that contention will occur on the Tx side and not on the fabric since it has a dedicated connection to the fabric however for the 6148 the contention will occur on the backplane since it dosen't connect to the fabric.

Correct on both counts.

Regards

Edison

vdadlaney · ‎12-02-2009

ediortiz wrote:

From your above statement per my understanding it means that on egress buffering will have minimum use and if that is the case than why are the Tx buffers always higher than the Rx buffers.

Correct on both counts.

Regards

Edison

Hi Edison,

Thx for responding. In your reply to point 1 a couple of posts above you mentioned that buffering will have minimum use on egress. If that is true than why is it that the Tx buffers are usually higher than the Rx buffers on most line cards. Also the document seems to contradict by saying that "buffers are always higher on the Tx side than the Rx side as is also mentioned in the document"? Am i misunderstanding something over here? Thx for your help.

Edison Ortiz · ‎12-03-2009

In your reply to point 1 a couple of posts above you mentioned that buffering will have minimum use on egress.

You are taking my reply out of context.

If link partners have the same speed/duplex, you won't need to buffer the traffic on egress in either direction unless the receiving partner can't keep up with the sending partner. If the receiving partner can't keep up with the sending partner, we recommend implementing flow-control.

Also the document seems to contradict by saying that "buffers are always higher on the Tx side than the Rx side as is also mentioned in the document"?

The document contradicts what?

The sending device should be the one holding the packets in a temporary storage if the receiving device can't keep up.

If makes no sense for the buffers to be high on the receiving device (at ingress) if it can't process the traffic efficiently.

vdadlaney · ‎12-22-2009

ediortiz wrote:

In your reply to point 1 a couple of posts above you mentioned that buffering will have minimum use on egress.

You are taking my reply out of context.

If link partners have the same speed/duplex, you won't need to buffer the traffic on egress in either direction unless the receiving partner can't keep up with the sending partner. If the receiving partner can't keep up with the sending partner, we recommend implementing flow-control.

Also the document seems to contradict by saying that "buffers are always higher on the Tx side than the Rx side as is also mentioned in the document"?

The document contradicts what?

Hi Edison,

Sorry for not responding earlier. I completely missed this reply.

Quoting one of your earlier replies

"On egress, you will be connected to a device that will perform line rate data thus buffering will have minimum use."

What I understood from this was that egress (tx) buffering will be of minimum use so I was asking why if it is of minimum use do we even need high tx buffers if they are not going to be used.

The document contradicts my understanding of your reply as above and since you clarify in your last post that

"The sending device should be the one holding the packets in a temporary storage if the receiving device can't keep up.

If makes no sense for the buffers to be high on the receiving device (at ingress) if it can't process the traffic efficiently."

which means I have not interpreted your reply correctly and am still confused on"On egress, you will be connected to a device that will perform line rate data thus buffering will have minimum use."

Appreciate if you could simplify any further if possible. Thx for your help.