Can we check which EP is sending more traffic on the interface ?

[Network_Sarovani]

Hello Team , 

We have many alerts for high transmitted utilization on below mentioned interface , Do we have any option in APIC to see which server is sending/receiving more traffic on these interface ? In this way we can identify if there are any scheduled backups which runs at a particular time and causing high bandwidth utilization on the interface .

Interface Ethernet1/11 for node ACI-DC1-LF101 has a transmitted utilization of 78 % which is greater than the threshold of 75%.

Interface Ethernet1/11 for node ACI-DC1-LF101 has a transmitted utilization of 78 which is greater than the threshold of 75%.

[Network_Sarovani]

Can anyone answer this question please ?

[RedNectar]

Hi @Network_Sarovani ,

How about you set up a SPAN to capture traffic that the servers on that Eth1/11 interface are sending?

• Fabric > Access Policies > Troubleshooting > SPAN > ...
  • SPAN Destination Group >+ Create SPAN Destination Group
    • Then specify the destination IP where your Wirshark PC lives. Make sure you specify the Source IP/Prefix with a mask, like 1.0.0.0/16 so you can later determine which leaf sent what.
  • SPAN Source Group >+ Create SPAN Source Group
    • Then specify the Destination Group you just created
    • Add the Source EPG where the (possibly) offending Server lives
    • Repeat the previous step until all suspects are added
  • SPAN Filter Group .+ Create SPAN Filter Group
    • Keep adding the sources IPs of the suspect server until all are added

Of course, you can skip the filters and filter in Wireshark later if you have the bandwidth capacity on the link to the WireShark host.

From here on - it's Wireshark training. That costs extra

[Robert Burns]

I had started a reply last night, but my page/browser closed and I lost my draft.  In terms of what can be done natively with APIC capabilities, try this:

Fabric > Inventory > PodX > Leaf101 > Interfaces > Physical Interfaces > eth1/11 > Deployed EPGs tab.  This would at least show you which EPGs are utilizing the interface (unless this is an L3out interface).  This wont tell you the top utilizing EP, but at least it will narrow down the search and might prompt you to look at specific endpoints in the related EPGs.    It's also important to look for any patterns.  Like you mentioned, could Backup tasks be the culprit - definitely.  If this was the case you should notice the Threshold alarm time stamp should behave fairly consistent.  Ex. You may notice the alarm "last transition" timestamp regularly shows between 2300hrs - 0300hrs.  Similarly you could always ask your server team what their backup schedule looks like.  

Short of this I'm afraid Chris (RedNectar) hit the nail on the head by suggesting to setup a SPAN on the interfaces, grab all the traffic on a sniffer VM during a high utilization period, and then parse the capture for top packet senders/receivers.   Though the APIC is a very capable controller, its geared towards policy and security management, and not so much a full-fledge traffic analysis tool.  This would be something that ACI's Netflow and SPAN capabilities would serve well when integrated with a traffic analysis solution like Nagios, WhatsUp Gold, Solarwinds, Zenoss etc.

Can you tell us what is connected to eth1/11 on Leaf101 -  What type of device/host etc.

Robert

[Network_Sarovani]

Hpe storeonce 5500 model server , as guessed this is the Backup server and I could see bandwidth utilization is observed only during the off bossiness hours .

[Doug Byrd]

Thinking outside the box a bit...

You could set up Netflow on that interface and point it to a dummy address, then after a while you could login to the switch and look at the netflow cache, sort by Byte Count.  I'd suggest piping to a .csv file and use your favorite editor.  

LEAF# show flow cache ipv4
IPV4 Entries
SIP              DIP              BD ID    S-Port   D-Port   Protocol  Byte Count        Packet Count      TCP FLAGS  if_id       flowStart          flowEnd
10.xxx.xxx.xxx    10.xxx.xxx.xxx     4715     51935    38881    6         5196              13                0x1a       0x1a02c0ef  4083614963 4083617910

You could probably also pull it from the API as well. Quick look at API Docs shows analytics* classes to be hopeful.  I haven't had time to bang around in postman, though.