C9500-48Y4C 16.12.03a is ip routing enabled by default?

[saba1418]

Hello, I am having issues here. Due to hardware failure, I recently had to hastily replace my core with equipment that was still being configured for installation, and was not quite ready. I was still testing certain features to implement on our site, so during the mad dash to get everything back up and working I just had to go with it. Now after installation, although vlan traffic is flowing to the router and users and systems are connected properly, I am not able to connect to the devices on the management vlan (10.200.255.0/25). I am wondering that even though the routing table shows all vlans, as well as the default route set to send all traffic to the router, if the command "ip routing" still needs to be executed. Is it enabled by default? is there a way to know if its already enabled other than in the show config output?

[balaji.bandi]

show running-config view full - should show the hidden config.

[saba1418]

okay, done. Thank you for looking at this...

[balaji.bandi]

May be attachment has some issue not able to view in the community due to maintenance (may be)

show running-config view full  - this command will help you to see what is in the config, some hidden config may not show run

but by default ip routing not enabled in switches.

[saba1418]

CORE#show config

version 16.12
no service pad
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service internal
service sequence-numbers
service call-home
service unsupported-transceiver
no platform punt-keepalive disable-kernel-core
!
hostname CORE
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging buffered 51200
no logging console
!
aaa new-model
!
!
aaa authentication login default group radius local
aaa authentication login console local
aaa authentication dot1x default group radius
aaa authorization config-commands
aaa authorization exec default group radius local none
aaa authorization network default group radius
aaa accounting dot1x default start-stop group radius
aaa accounting exec default start-stop group radius
aaa accounting connection default start-stop group radius
!
!
!
!
!
!
aaa session-id common
clock timezone CST -6 0
clock summer-time extended_CST recurring
switch 1 provision c9500-48y4c
switch 2 provision c9500-48y4c
boot system bootflash:packages.conf
boot system flash:packages.conf
stackwise-virtual
domain 1
!
ip name-server [DNS IP's]
ip domain name hidden
!
!
!
login on-success log
!
!
!
!
!
qos queue-softmax-multiplier 1200
!
!
vtp domain hidden
vtp mode transparent
no device-tracking logging theft
!
crypto pki trustpoint TP-self-signed-3661210606
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3661210606
revocation-check none
rsakeypair TP-self-signed-3661210606
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
!
crypto pki certificate chain TP-self-signed-3661210606
certificate self-signed 01 nvram:IOS-Self-Sig#1.cer
crypto pki certificate chain SLA-TrustPoint
certificate ca 01 nvram:CiscoLicensi#1CA.cer
!
!
license boot level network-advantage addon dna-advantage
!
!
diagnostic bootup level complete
!
spanning-tree mode pvst
spanning-tree extend system-id
memory free low-watermark processor 186785
file verify auto
!
!
mac access-list extended Protocol-Filter
deny any any appletalk
deny any any amber
deny any any aarp
deny any any dec-spanning
deny any any decnet-iv
deny any any diagnostic
deny any any dsm
deny any any lat
deny any any lavc-sca
deny any any mop-console
deny any any mop-dump
deny any any msdos
deny any any mumps
deny any any netbios
deny any any vines-echo
deny any any vines-ip
deny any any xns-idp
permit any any
no errdisable detect cause gbic-invalid
errdisable recovery cause security-violation
errdisable recovery cause link-flap
errdisable recovery cause psecure-violation
errdisable recovery cause storm-control
errdisable recovery interval 60
!
redundancy
mode sso
!
!
!
!
!
transceiver type all
monitoring
!
!
vlan 100
name OEC-MGMT
lldp run
!
!
class-map match-any system-cpp-police-ewlc-control
description EWLC Control
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data packets, LOGGING, Transit Traffic
class-map match-any system-cpp-default
description EWLC Data, Inter FED Traffic
class-map match-any system-cpp-police-sys-data
description Openflow, Exception, EGR Exception, NFL Sampled Data, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-high-rate-app
description High Rate Applications
class-map match-any system-cpp-police-multicast
description MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual OOB
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control and Low Latency
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
class-map match-any system-cpp-police-dhcp-snooping
description DHCP snooping
class-map match-any system-cpp-police-ios-routing
description L2 control, Topology control, Routing control, Low Latency
class-map match-any system-cpp-police-system-critical
description System Critical and Gold Pkt
class-map match-any system-cpp-police-ios-feature
description ICMPGEN,BROADCAST,ICMP,L2LVXCntrl,ProtoSnoop,PuntWebauth,MCASTData,Transit,DOT1XAuth,Swfwd,LOGGING,L2LVXData,ForusTraffic,ForusARP,McastEndStn,Openflow,Exception,EGRExcption,NflSampled,RpfFailed
!
policy-map system-cpp-policy
!
!
!
!
!
!
!
!
!
!
interface Vlan100
description MGMT
ip address 10.200.255.20 255.255.255.128
no ip redirects
!
ip forward-protocol nd
no ip http server
ip http authentication local
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 [Router IP]
ip ssh version 2
!
!
!
!
!
!
!
control-plane
service-policy input system-cpp-policy
!
end

[saba1418]

Will executing the "ip routing" command cause any issues on an in production switch?

[balaji.bandi]

Command not have any issue, you are enabling to routing that's all, as long as you add correct route where to go after enabling will have an effect, but intervlan and   local routing no issue.

[saba1418]

Okay, thanks. I already have the default route to our router in the config. I assume enabling ip routing just allows the switch to route traffic between the vlan interfaces, but wont have any effect on traffic from vlans that aren't configured with a vlan interface, that traffic will still just route to the main router. I just wanted to make sure enabling the routing didn't require a reboot, or affect traffic/routes already in place.

[Richard Burts]

Running the command ip routing should not have any impact. Here are a couple comments and suggestions:

- you have posted only parts of the config so we can not be sure if the part we do not see would make any difference.

- you have configured a default route and we do not see ip default-gateway which certainly suggests that ip routing is enabled. default-gateway is what a device uses when ip routing is not enabled and ip route 0.0.0.0 is what is used when ip routing is enabled.

- the output of show ip protocol and of show ip route might help determine if ip routing is enabled.

- the posted config has just one vlan interface and we do not know what is assigned in vlan 100. But it certainly suggests that vlan 100 is for management traffic and that data is carried in other vlans. In that case ip routing and the default route is only for management traffic on this device. All data traffic would be forwarded at layer 2 to other devices which would do the routing for data traffic.

- you tell us that "I am not able to connect to the devices on the management vlan". It is not clear whether you are attempting to connect from this device of from other devices. Can you clarify? 

- are there devices connected in vlan 100 on this device?

- the default route indicates that the router is the next hop. Can this device ping the router address? Can the router ping this device address?

- are there any policies on the router that impact use of the subnet for vlan 100?

[saba1418]

Thank you for looking at this issue. My responses to your inquiries are below...

- the output of show ip protocol and of show ip route might help determine if ip routing is enabled.

CORE#show ip protocols
*** IP Routing is NSF aware ***

CORE#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 10.38.10.1 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 10.38.10.1
10.0.0.0/8 is variably subnetted, 8 subnets, 4 masks
C 10.38.8.0/21 is directly connected, Vlan1
L 10.38.10.208/32 is directly connected, Vlan1
C 10.39.1.0/24 is directly connected, Vlan25
L 10.39.1.1/32 is directly connected, Vlan25
C 10.39.2.0/24 is directly connected, Vlan35
L 10.39.2.1/32 is directly connected, Vlan35
C 10.200.255.0/25 is directly connected, Vlan100
L 10.200.255.20/32 is directly connected, Vlan100
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.8.0/21 is directly connected, Vlan30
L 172.16.8.1/32 is directly connected, Vlan30

- you tell us that "I am not able to connect to the devices on the management vlan". It is not clear whether you are attempting to connect from this device of from other devices. Can you clarify?

from our vpn, I am able to SSH to any and all of our edge switches through our radius server. I am not able to connect/SSH to any of our edge switches from within the campus network

- are there devices connected in vlan 100 on this device?

yes, all other edge switches and UPS's are connected on vlan 100, and they are pingable from this device as well

- the default route indicates that the router is the next hop. Can this device ping the router address? Can the router ping this device address?

yes/yes

- are there any policies on the router that impact use of the subnet for vlan 100?

I am not totally sure, as I am do not have access to the router CLI, but for the last year since I have joined this organization we have used V100 to manage all switches and it has worked flawlessly. Since replacement of the former core with the new core due to hardware failure, management/SSH has not worked correctly. The replacement was unplanned and we were rushing to complete the task, so we werent able to finish testing before going live, I/we are trying to get it right while the core is in production.

[Richard Burts]

Thank you for the update. The output of show ip protocol and of show ip route do confirm that ip routing is enabled. That was an important part of your original post and we now have a pretty definitive answer to that part of the question: ip routing is enabled.

There is still an issue about access to the switch. In the original post it seemed to be in general there was a problem with access to the switch. In this recent response it seems to be more specific that there is a problem with SSH access. Is that correct? If so a good next step would be to post the output of the command show ip ssh on the switch. And to eliminate the possibility that it might be a basic connectivity issue would you test whether you are able to ping to the switch?

In this recent response you tell us "I am not able to connect/SSH to any of our edge switches from within the campus network". I am not sure whether that might be another aspect of replacing the core switch or might be a different problem. To investigate that can you see if you can ping to an edge switch from a device within the campus network? If ping is not successful then would you do a traceroute and post its output.

[saba1418]

My responses are included below, and again - thank you for checking this out...

There is still an issue about access to the switch. In the original post it seemed to be in general there was a problem with access to the switch. In this recent response it seems to be more specific that there is a problem with SSH access. Is that correct? If so a good next step would be to post the output of the command show ip ssh on the switch. And to eliminate the possibility that it might be a basic connectivity issue would you test whether you are able to ping to the switch?

CORE#show ip ssh
SSH Enabled - version 2.0
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr
MAC Algorithms:hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
KEX Algorithms:diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 2048 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): TP-self-signed-3661210606
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCGiDWHs4J/mPCPtO/KGWufTlvJzFo+BWBomWOplBz8
o3bNabAkW6CcVwdmdvsPkGI216yCLBfJNyHbU/Efjt9gmDjDbNhq5pSnvFLGx4ArhMugheBmJjN5VvrW
A3grdF92ZNaWHdkWwyvXnjYoCets/9DS533oNG2eNkto2r1DiRjin0zD7QnSFTPkPowvUjIqdU8HEchi
WgN4C2HCwJO3ZtPGyLor6JAElFJlVfZsE2OoG1MphBUt21oxYeYvdjI0omSfkM41zlt9PPOW4sHERbET
xdORxSxX+tuuz9W2Z6vMXgQtXC/flmR27Mkxm6e1vnBJvkIm8A7lweGmeGtj

I can ping the switch having issues from other devices within our campus network that live on different vlans other than V100, as well as all devices that live on V100.

In this recent response you tell us "I am not able to connect/SSH to any of our edge switches from within the campus network". I am not sure whether that might be another aspect of replacing the core switch or might be a different problem. To investigate that can you see if you can ping to an edge switch from a device within the campus network? If ping is not successful then would you do a traceroute and post its output.

I can indeed ping all edge switches from devices within our campus network that live on different vlans other than V100.

[Richard Burts]

Thank you for the additional information. The output does demonstrate that SSH is enabled, and specifies that it must be version 2. I wonder if there is any possibility that your attempts to SSH might be specifying version 1?

It is interesting that you are able to ping the switch. So at this point we know these things:

- there is ip connectivity from the device attempting SSH to the core switch.

- the core switch has enabled SSH.

- It would seem that SSH should work but the device is not able to successfully SSH to the switch.

So why is SSH not working? Perhaps if you enable debug for SSH on the switch, attempt SSH access, and post any debug output?

[saba1418]

Sorry, I realized there is an important piece of information that I have inadvertently left out. Before the core replacement, the only way I could get the switch to communicate and respond to SSH attempts while connected to vlan 100 was to include an IP route of 0.0.0.0 0.0.0.0 10.200.255.1. Once it became the actual core, I had to change the IP route to 0.0.0.0 0.0.0.0 10.38.10.1, breaking the access henceforth. Also of important note, the vlan 1 interface ip will respond to SSH attempts, although not authenticated through radius. I am truly stumped at this point.

[Richard Burts]

Thanks for the update. Your discussion about having to change the default route is interesting, and I would like to think it might be the issue, suggesting that there was some problem about routing or something like that. Unfortunately if it is true that you are able to ping to the new core switch from devices in the campus network then there is no issue about routing/ip connectivity and the problem must be something else.

When you attempt to SSH to the new core switch what happens? Do you get any prompt, or any other type of response, or does it just hang?

I am not clear about a couple of things when you tell us "the vlan 1 interface ip will respond to SSH attempts, although not authenticated through radius."  

- am I correct in assuming that your normal SSH access uses the vlan 100 IP address?

- if the vlan 1 interface responds to SSH attempts why would it not be authenticated through radius?

It also occurs to me to wonder when you replaced the old core switch with the new switch, would the radius server recognize that this is a different device? (would the device name perhaps be different? or is there any other parameter that radius might be using that is different?)