CBS350, Some routing is only works with ARP table is inserted manually

ping-er · ‎10-02-2023

Hi. I have a strange problem with inter-vlan routing and ARP.

As briefly,

I have to re-input a "Static" ARP Table entry after every re-booting of CBS350, to make the inter-vlan routing to work normally.

I can found this workaround after long trial-and-error.

This sounds strange but here below the full story. I want the problem is fixed in the future.

There are `host A`, `host B`, a `Router`, a `CBS350`. All settings are properly done to communicate like below flow.

Doing a ping from A to B.

host A - 192.168.52.121  -->  192.168.52.1 - Router - 192.168.10.1  -->  192.168.10.254 - CBS350 - 10.10.0.254  -->  10.10.0.29 - host B

# Let's say `host A` has the ethernet interface of 192.168.52.121,
# `Router` has the ethernet interfaces of 192.168.52.1 and 192.168.10.1,
# `CBS350` has 192.168.10.254  and 10.10.0.254, `host B` has 10.10.0.29

Above communication works normally when my workaround of adding the static ARP table entry is done.

- like `arp 10.10.0.29 aa:bb:cc:7a:37:ca vlan1010` in configure. (actually input is done by the Web)

Without this, CBS350 does not fill up the ARP table for the host B, for the 10.10.0.29 IP and then the communication fails ex) the ping is not replied.

Shortly, the simple topology of inter-vlan routing works normally as expected, like `host A --> CBS350 --> host B` but my case above is not that simple.

I already tried this simple path, and this success to make a ping. The ARP table in CBS350 for 10.10.0.29 is filled up immediately.

- A `Dynamic` entry is added, if I manually delete this by experimental then it added again.

host A - 192.168.52.121  -->  192.168.52.254 - CBS350 - 10.10.0.254  -->  10.10.0.29 - host B

But there is a security reason to make path to go through the original one.

So what is the reason of mal-functioning like the strange way?

- the host A is NOT directly connected subnet to CBS350?

- the other complicated configurations? so I attached the whole running-configuration below. (some info not related are erased)

Funny things can be listed.

- if CBS350 has no ARP Table entry for 10.10.0.29 then repeated ping from host A to B fails every time.

- if ARP Table is filled by another way, then the ping A to B starting to success.

- CBS350 Web - Administration - Ping to 10.10.0.29, this can make a `Dynamic` entry of ARP Table.

- the problem is that the Dynamic entry Age Out after given time (default 60000 sec), and by experiment to fix this to 10 sec, the entry delete by aging, and the ping A to B failing again after 10 sec.

- So the solution is to add a `Static` entry.

- Most foolish thing is that the Static entry is saved in the startup-config but this is not working at the re-booting.

- after re-booting CBS350, the Static entry appears the ARP Table, but!!! the ping A to B fails.

- delete it and re-enter the Static entry make the ping A to B work again. This kidding me.

- if the Static entry from the startup-config is kept in ARP Table, making own ping from CBS350 Web to host B is success but this not change the ping A to B fail. only after deleting the Static entry and re-entering Dynamic one or Static one make the ping A to B success.

To be sure,

- the arp function of host B is normal, since the other communications in his subnet is working all normally.

- even the CBS350 can ping to the host B. (this add the dynamic ARP item, after this, the routing suddenly works, but this item expires and then routing stop working.)

- Another static routing item is always success like `ping or udp traffic etc. to 10.11.0.5` from host A (including other many hosts).

I don't sure about what makes difference.

- YES directly to YES directly : no problem (meaning simple inter-vlan routing)

- NOT directly to YES directly : has problem?!? (my problem case)

- NOT directly to NOT directly : no problem (by 10.11.0.5 example)

#show startup-config
config-file-header
main-cisco-CBS350
v3.0.0.69 / RCBS3.0_930_770_008
CLI v1.0
file SSD indicator encrypted
@
ssd-control-start
ssd config
ssd file passphrase control unrestricted
no ssd file integrity control
ssd-control-end <something erased>
!
!
unit-type-control-start
unit-type unit 1 network gi uplink none
unit-type-control-end
!
vlan database
vlan 10,51-54,201-203,1010
exit
voice vlan oui-table add 0001e3 Siemens_AG_phone
voice vlan oui-table add 00036b Cisco_phone
voice vlan oui-table add 00096e Avaya
voice vlan oui-table add 000fe2 H3C_Aolynk
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone
voice vlan oui-table add 00e075 Polycom/Veritel_phone
voice vlan oui-table add 00e0bb 3Com_phone
port-channel load-balance src-dst-mac-ip
arp timeout 60000
no bonjour enable
bonjour interface range vlan 1
hostname main-cisco-CBS350
line console
exec-timeout 30
exit
line ssh
exec-timeout 30
exit
line telnet
exec-timeout 30
exit
passwords aging <something erased>
username <something erased> privilege 15
ip ssh server
ip http timeout-policy <something erased>
ip http timeout-policy <something erased>
clock timezone <something erased>
ip name-server  <something erased>
no pnp enable
!
interface vlan 10
 name office_lan_10
 ip address 192.168.10.254 255.255.255.0
!
interface vlan 51
 name office_lan_51
 ip address dhcp
 shutdown
!
interface vlan 52
 name office_lan_52
 ip address 192.168.52.254 255.255.255.0
 shutdown
!
interface vlan 53
 name office_lan_53
 ip address dhcp
 shutdown
!
interface vlan 54
 name office_lan_54
 ip address dhcp
 shutdown
!
interface vlan 201
 name dmz_lan_201
 ip address dhcp
 shutdown
!
interface vlan 202
 name dmz_lan_202
 ip address dhcp
 shutdown
!
interface vlan 203
 name dmz_lan_203
 ip address dhcp
 shutdown
!
interface vlan 1010
 name server_lan_1010
 ip address 10.10.0.254 255.255.255.0
!
interface GigabitEthernet1
 switchport mode trunk
 switchport trunk native vlan 54
 switchport trunk allowed vlan 10,54,201-202
!
interface GigabitEthernet2
 switchport access vlan 10
!
interface GigabitEthernet3
 switchport access vlan 51
!
interface GigabitEthernet4
 switchport access vlan 51
!
interface GigabitEthernet5
 switchport access vlan 52
!
interface GigabitEthernet6
 switchport access vlan 1010
!
interface GigabitEthernet7
 channel-group 2 mode on
 switchport access vlan 1010
!
interface GigabitEthernet8
 channel-group 2 mode on
 switchport access vlan 1010
!
interface GigabitEthernet9
 channel-group 2 mode on
 switchport access vlan 1010
!
interface GigabitEthernet10
 channel-group 2 mode on
 switchport access vlan 1010
!
interface GigabitEthernet11
 channel-group 1 mode auto
 switchport mode trunk
 switchport trunk native vlan 1010
 switchport trunk allowed vlan 54,201-203,1010
!
interface GigabitEthernet12
 channel-group 1 mode auto
 switchport mode trunk
 switchport trunk native vlan 1010
 switchport trunk allowed vlan 54,201-203,1010
!
interface GigabitEthernet13
 switchport mode trunk
 switchport access vlan 53
 switchport trunk native vlan 53
 switchport trunk allowed vlan 51-53
!
interface GigabitEthernet14
 switchport access vlan 53
!
interface GigabitEthernet15
 switchport access vlan 53
!
interface GigabitEthernet16
 switchport access vlan 53
!
interface GigabitEthernet17
 switchport access vlan 53
!
interface GigabitEthernet18
 switchport access vlan 53
!
interface GigabitEthernet19
 switchport access vlan 53
!
interface GigabitEthernet20
 switchport access vlan 53
!
interface GigabitEthernet21
 switchport access vlan 53
!
interface GigabitEthernet22
 switchport access vlan 53
!
interface GigabitEthernet23
 switchport access vlan 53
!
interface GigabitEthernet24
 switchport access vlan 53
!
interface GigabitEthernet25
 switchport access vlan 54
!
interface GigabitEthernet26
 switchport access vlan 54
!
interface GigabitEthernet27
 switchport access vlan 54
!
interface GigabitEthernet28
 switchport access vlan 54
!
interface GigabitEthernet29
 switchport access vlan 54
!
interface GigabitEthernet30
 switchport access vlan 54
!
interface GigabitEthernet31
 switchport access vlan 54
!
interface GigabitEthernet32
 switchport access vlan 54
!
interface GigabitEthernet33
 switchport access vlan 54
!
interface GigabitEthernet34
 switchport access vlan 54
!
interface GigabitEthernet35
 switchport access vlan 54
!
interface GigabitEthernet36
 switchport access vlan 54
!
interface GigabitEthernet37
 switchport access vlan 54
!
interface GigabitEthernet38
 switchport access vlan 54
!
interface GigabitEthernet39
 switchport access vlan 54
!
interface GigabitEthernet40
 switchport access vlan 54
!
interface GigabitEthernet41
 switchport access vlan 54
!
interface GigabitEthernet42
 switchport access vlan 54
!
interface GigabitEthernet43
 switchport access vlan 54
!
interface GigabitEthernet44
 switchport access vlan 54
!
interface GigabitEthernet45
 switchport access vlan 54
!
interface GigabitEthernet46
 switchport access vlan 54
!
interface GigabitEthernet47
 switchport access vlan 54
!
interface GigabitEthernet48
 switchport access vlan 54
!
interface Port-Channel1
 description link-to-server-room-cisco
 switchport mode trunk
 switchport trunk native vlan 1010
 switchport trunk allowed vlan 54,201-203,1010
!
interface Port-Channel2
 description link-from-gateway
 switchport access vlan 1010
!
exit
macro auto disabled
arp 10.10.0.29 aa:bb:cc:7a:37:ca  vlan1010
ip default-gateway 192.168.10.1
ip route 10.11.0.5 /32 10.10.0.1

(some info is erased, arp mac address is modified.)

M02@rt37 · ‎10-02-2023

Hello @ping-er,

Conduct packet captures on the CBS350 to observe ARP traffic during the boot process and communication attempts between host A and host B. And analyze ARP requests and responses for any inconsistencies.

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

ping-er · ‎10-04-2023

Hello M02@rt37,

I bring a laptop with the wireshark app is installed to capture the ARP packets.

Since the CBS350 itself has no packet capture ability right?

Where I mentioned in the body of the post as "ping A to B"

host A - 192.168.52.121 --> 192.168.52.1 - Router - 192.168.10.1 --> 192.168.10.254 - CBS350 - 10.10.0.254 --> 10.10.0.4 - host B
(the example's host B IP is changed but it is essentially the same example.)

If CBS350 has no ARP Table entry for 10.10.0.4, then the ping A to B is failed. (not replied.)

From CBS350 Web (screen captured below), there is no ARP entry for 10.10.0.4
And Wireshark capturing in the same subnet shows no ARP packet is shown which is looking for 10.10.0.4

Now, I can do ping from CBS350 to 10.10.0.4 from the CBS350 Web.

Then the Dynamic ARP entry is added to the Table.

And at the same time, wireshark captures the ARP packet looking for 10.10.0.4 sent from the CBS350.

Then the ping A to B works!!!! the reply from B is suddenly coming after the CBS350 ARP entry is added for the 10.10.0.4

But since this entry expires after the given settings (e.g. 60000), after then the ping A to B not works again.

The question is... why the ARP packet is not sent for 10.10.0.4 in this time??
- I think the difference is that this is during inter-vlan routing.

So the solution is simple. Add it as Static.

But after the re-booting, even the Static entry still exists, the ping A to B is NOT working.

Delete it and do again the above "Dynamic" or "Static" way makes the ping A to B works again.

This is the most weird and a bug suggesting part.

By the way, "observe ARP traffic during the boot process" is pointless, since no ARP packet will be sent for it. Because it is "Static".

M02@rt37 wrote:
Hello @ping-er,
Conduct packet captures on the CBS350 to observe ARP traffic during the boot process and communication attempts between host A and host B. And analyze ARP requests and responses for any inconsistencies.

paul driver · ‎10-02-2023

Hello
I assume that switch is the L3 switch for the lan if so is it actually enabled for ip routing?
config t
Ip routing
End
wr men

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

ping-er · ‎10-04-2023

Hello @paul driver,

in CBS350 Web, I can re-check the routing is ON.

Also you can see "no ip routing" phrase is not exists in the config text I put in the body of the post.

Plus, another routing is ALWAYS working normally.

host A - 192.168.52.121 --> 192.168.52.1 - Router - 192.168.10.1 --> 192.168.10.254 - CBS350 - 10.10.0.254 --> 10.10.0.1 - Another Router - 10.11.0.5 (Some sort of a NAT gateway)
if I manually delete the ARP entry for this, 10.10.0.1, then is always filled up by CBS350.

Fore mentioned ARP problem is shown up only in this path.

host A - 192.168.52.121 --> 192.168.52.1 - Router - 192.168.10.1 --> 192.168.10.254 - CBS350 - 10.10.0.254 --> 10.10.0.4 - host B

I infinitely running the ping every 1 second from my PC to continuously checking the routing is operating normally.

ping to 10.11.0.5 (always success)
ping to 192.168.10.254 (always success)
ping to 10.10.0.4 (only success when the Static ARP is re-inputted in CBS350, after rebooting)

@paul driver wrote:
Hello
I assume that switch is the L3 switch for the lan if so is it actually enabled for ip routing?
config t
Ip routing
End
wr men

KJK99 · ‎10-04-2023

@ping-er

You have configured your CBS350 for routing and you have successfully tested it. That’s all good. You run into a routing issue once you introduce some router into the setup, but you do not provide any information about the router. Instead you add some static ARP entry to the picture. Using static ARP to resolve that issue looks very unorthodox to me. That should work fine without it if proper routing is in place. I don’t think anybody here can help you with it until you provide sufficient IP information of that router and the client devices, basically a detailed end-to-end IP configuration.

Kris K

ping-er · ‎10-06-2023

Hello @KJK99,

"proper routing is in place" is proven by the ping A to B is working fine.

Now, I think something is misleading the point of the topic.

> "Using static ARP to resolve that issue looks very unorthodox to me"

Right, this situation is also unorthodox to me. too.

> "That should work fine without it if proper routing is in place"

The routing setting in here is so simple enough and it already working fine. But only when the ARP problem is manually handled.

> "I don’t think anybody here can help you with it until you provide sufficient IP information of that router and the client devices, basically a detailed end-to-end IP configuration"

No one can help me even if I provide the sufficient IP information about the router or clients, because this problem is not resides in the routing settings mistake.

THE ONLY thing which can help me is the bug fixed firmware about the ARP mal-function.

Let's look into the ARP problem.

ARP is solely a simple function. But it is not working as expected. It must fill-up the ARP table with the peer IP's MAC address if it don't know it now. (No matter the packet currently about to send is routed one or sending by CBS350 itself.)

if I don't have the pear IP's MAC in my table then send a ARP request packet and receive the reply, and next sending my any desired packets.
- Doing ping from CBS350 Web is work fine, filled up ARP table and also ping is done successfully.
  - CBS350 - 10.10.0.254 --> 10.10.0.4 - host B
  - if I delete the ARP table manually, it immediately re-fill the ARP table.
- During the directly connected net to directly connected net routing is also fine.
  - host A - 192.168.52.121 --> 192.168.52.254 - CBS350 - 10.10.0.254 --> 10.10.0.4 - host B
  - CBS350 ARP table is fill-up for 10.10.0.4's MAC.
- BUT the NOT directly connected net to directly connected net routing is causing the ARP mal-functioning.
  - host A - 192.168.52.121 --> 192.168.52.1 - Router - 192.168.10.1 --> 192.168.10.254 - CBS350 - 10.10.0.254 --> 10.10.0.4 - host B
  - ping A to B is failed, CBS350 ARP table is still empty about 10.10.0.4's MAC.
    - if I manually input Static entry or fill it by another method, then suddenly the ping A to B begin to success.
      - And if I delete the ARP table manually, then it fails again. (while running infinite ping the same time.)

May be I should change the title of this post to "CBS350, ARP mal-functioning in some case"

Because the fundamental reason of the problem is not caused by the in-correct routing settings in any router or host PC.