Cisco Management interface on a L3 switch (routable or not) ?

[SJ K]

Hi all,

I have a L3 switch (3850) with a Fa0 Ethernet management port configured.

int fa0 - ip address 192.168.10.6 255.255.255.0

The L3 switch has ip routing turn on and hence in show ip route we can see

     192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.10.0/24 is directly connected, FastEthernet0
L        192.168.10.8/32 is directly connected, FastEthernet0
      192.168.7.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.7.0/24 is directly connected, Vlan4
L        192.168.7.250/32 is directly connected, Vlan4

S  192.168.0.0/24 via 192.168.7.251,  Vlan 4

This particular interface (192.168.10.8) is the connected to a L2 switch which is then connected to our servers' management interface.

Hence 192.168.10.8 is use as the gateway for our all devices/servers's management network.

============

Operators will connect in via the 192.168.0.0 network  to the 192.168.10.0/24 management network via the L3 switch and we thought the L3 switch will know how to route back.

However, from the 192.168.0.0 network, we are only able to connect to the L3 switch's management port (192.168.10.8) and no further then that .

q1)  Can management interface be use as a transit/ to route traffic to the remaining management network ?

Regards,

Noob

[Mark Malone]

If your using a management port on a 3850 for mgmt. traffic you should be using a vrf for extra security , is it an OOB network your building , as an example we have all our cisco equipment running on a parallel network through the mgmt. ports the are all under a specific ip range which then connects back to layer 2 stacks which directly connect to our internal firewalls , all our traffic locally for mgmt. ntp/syslog/ssh/netflow etc is sourced from the mgmt. interface by vrf  , the default gateway or default route depending if pure layer 2 or 3 is the firewall which process all mgmt. traffic

That way all our prod traffic is separated from mgmt. by mgmt. ports and vrfs isolate it from prod routing tables , is that what your trying to do ?

******************************

Placing the Gigabit Ethernet management interface in its own VRF has the following effects on the management ethernet interface:

• Requires configuring multiple features. Because Cisco IOS CLI may be different for certain management ethernet functions compared to other routers. You are required to configure or use many features inside the VRF.
• Prevents transit traffic from traversing the router. Because all module interfaces and the management ethernet interface are automatically in different VRFs, no transit traffic can enter the management ethernet interface and leave a module interface, or vice versa.
• Improves security of the interface. Because the Mgmt-intf VRF has its own routing table as a result of being in its own VRF, routes can only be added to the routing table of the management ethernet interface if you explicitly enter them.

The management ethernet interface VRF supports both IPv4 and IPv6 address families.

[SJ K]

Hi Mark!,

1st of all,  thank you so much for sharing your thoughts and advices on the thread !

I have never tried VRF before, but on 1st look, it seems like having a virtual router inside the physical router.  Can VRF share the same interface as the Physical router ?

But to the original topic -> My main concern is actually the functionality of the Cisco management port/interface -> whether it can be use for routing traffic (since it is appearing in the routing table).
Below is my setup (please ignore the fa0/3 and treat it real management port)

I am at my office network (192.168.0.0/24) and was trying to access the management network at 192.168.10.0/24.  Assume that the routes in the coreswitches for outgoing and return traffic are setup properly -> Below is what i have encounter

a) from 192.168.0.0 network, I am able to access  (ssh) 192.168.10.1 at the L3 coreswitch1, however i am not able to reach any other management devices below connected to management interface (via the Layer2 switch).

b) plug out the cable from the management interface (and no ip) and plug it to any other switch port + configure a vlan interface (svi) with the previous ip (192.168.10.1) == and i am able to reach all the other management devices.

It seems like the management interface is not able to route traffic and be use for transiting traffic.

Is the above behavior intended ? are we able to use management interface just like any routing port ?

Regards,
Noob

[Mark Malone]

Hi

The whole point of vrf is it isolates the routing table from production traffic , you don't want mgmt. traffic mixed in with production whenever you can avoid it , the mgmt. port is designed for OOB traffic not routing as you have found , it some cases like nx-os its a separate full device inside the chassis with its own ram cpu etc so if the device goes into a storm or flood no matter what remotely you can get on it which can be important if your devices are in a 3rd party data centre or very far away  

do not use mgmt. port as a routed port specifically designed for out of band , on a 3850 you cant even remove the vrf anyway from mgmt. ports if I remember right , its now on by default same for nx-os

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750x_3560x/software/release/12-2_53_se/configuration/guide/3750xscg/swint.html#wp1730622

Cisco

Even though the Ethernet management port does not support routing, you might need to enable routing protocols on the port (see Figure 13-4). For example, in Figure 13-4, you must enable routing protocols on the Ethernet management port when the PC is multiple hops away from the switch and the packets must pass through multiple Layer 3 devices to reach

Supported Features on the Ethernet Management Port

The Ethernet management port supports these features:

•Express Setup (only in switch stacks)

•Network Assistant

•Telnet with passwords

•TFTP

•Secure Shell (SSH)

•DHCP-based autoconfiguration

•SMNP (only the ENTITY-MIB and the IF-MIB)

•IP ping

•Interface features

–Speed—10 Mb/s, 100 Mb/s, and autonegotiation

–Duplex mode—Full, half, and autonegotiation

–Loopback detection

•Cisco Discovery Protocol (CDP)

•DHCP relay agent

•IPv4 and IPv6 access control lists (ACLs)

•Routing protocols

[SJ K]

Hi Mark,

Your like a walking documentation.

From the documentation,

By default, the Ethernet management port is enabled. The switch cannot route packets from the Ethernet management port to a network port, and the reverse.

q1) It seems like in that case, the management port can only receive traffic destined to itself and not another network - right ?

q2) In the earlier, you mentioned your setup for management network,  is it something like below (treat the 2911 as a firewall)

q3) for the Layer2 switch,  what should we do with its management interface ?

Regards,
Noob

[Mark Malone]

Hey

yes like yourself i found a lot of this stuff out from labbing it up best way to do I think anyway see yourself how it works  

1 Yes that's correct pure OOB mgmt port

2 Basically we run a parallel network off the mgmt. ports , so all our devices have a separate physical link through a stack of layer 2 switches which connects directly to our internal firewalls that process all our mgmt. traffic  

Then we also have our production network which links to our VSS and DC networks which also links to other firewalls for processing standard traffic

Then we also have SCS serial console switches for lights out setup were bit paranoid about outages and not being able to get to our kit  :)  even some of our SCS devices will eventually have 3/4g access for dial in on remote sites

All traffic is then sourced using the mgmt. port and traffic directed at the firewall

3 If the port has got a mgmt. interface it can be given an ip even on layer 2 switch and again give the switch a default gateway and source it from the mgmt. port , now like us some of our cahsis have these 3x20 cards that have no mgmt. port so all we did was justr source mgmt. traffic from a dedicated vlan SVI we setup just for switches without mgmt. ports , its really the best you can do to isolate the traffic at pure layer 2

As we were talking about 38s and vrfs this is an example of one setup in terms of mgmt. may be useful if you go that way , below isolates all mgmt. traffic only into a vrf and sends it to our firewall to be processed , I can add in a layer 2 example as well if required but its just the same with source of SVI and then no vrf used in syntax

aaa group server tacacs+ xtacacs
 server-private 172.x.x.x key 7 151F4E36366F237D2A64637F404632483002187F7D
 server-private 172.x.x.x key 7 141A57313E412272267F65687152235D3255177E76
 ip vrf forwarding Mgmt-vrf
 ip tacacs source-interface GigabitEthernet0/0
!
aaa authentication login default group xtacacs local enable
aaa authentication enable default group xtacacs enable
aaa authorization exec default group xtacacs local
aaa accounting exec default start-stop group xtacacs
aaa accounting commands 0 default start-stop group xtacacs
aaa accounting commands 1 default start-stop group xtacacs
aaa accounting commands 15 default start-stop group xtacacs
aaa accounting network default start-stop group xtacacs
aaa accounting connection default start-stop group xtacacs
aaa accounting system default start-stop group xtacacs

interface GigabitEthernet0/0
 description ** Network Managment Interface **
 vrf forwarding Mgmt-vrf
 ip address 172.x.x.x 255.255.254.0
 negotiation auto

ip route vrf Mgmt-vrf 0.0.0.0 0.0.0.0 172.FIREWALL
logging host 172.x.x.x vrf Mgmt-vrf

line vty 0 4
 access-class X in vrf-also

ntp source GigabitEthernet0/0
ntp server vrf Mgmt-vrf X.X.X.X

snmp-server source-interface traps g0/0
ip ssh source-interface GigabitEthernet0/0

flow exporter NetQos
 description export Netflow traffic to HQ
 destination x.x.x.x
 source GigabitEthernet0/0

[SJ K]

Hi Mark,

Thanks for the wonderful reply!

But on 3)

If the port has got a mgmt. interface it can be given an ip even on layer 2 switch and again give the switch a default gateway and source it from the mgmt. port

In my setup diagram above, you can see that the L2 switch is use to connect to the management interfaces of the rest of the devices; and the L2 switch is connected to the FW via a normal switch-port interface.

The L2 switch itself is access via its default VLAN interface through the FW and has it default gateway pointing back to the FW.

We know we cannot access the rest of the devices if we were to just setup a connection to the from the FW to the L2 switch management interface; and hence we need a connection from the FW to the L2 switch's switchport.

As such, what should be actually done to the L2 switch's management interface ?
If we are to assign an IP to it; does that means we will have 1 more connection to the FW ?

Regards,
Noob

[Mark Malone]

hope I understood that right , if you use the mgmt. port as the port for your mgmt. traffic you just give it an ip address stick it in the vrf connect it to your layer 2 mgmt. switch(where all your switches mgmt. ports are connecting before firewall) then give that port a layer 2 vlan on the mgmt. switch that's part of the ip range

If it does not have an mgmt. port use a separate vlan interface on the layer 2 switch then assign a port locally to it and then on layer 2 mgmt. switch again assign a local port to that vlan as well , this isn't as secure as using the actual mgmt. port but it still isolates it in another vlan , from the firewall don't allow intervlan communication from that vlan to other prod vlans

so either method is going to have the same physical link but it will either just come of the mgmt. port itself direct to mgmt. switch or just use a vlan and connect direct to the mgmt. switch by standard access port that then connects to the firewall

The way we have it done here is the only connections to the firewall are from the management stack , no switches directly connect to the firewall whether they have an mgmt. port or just purely on a separate vlan (everything is in same range /23 though) they all go through the mgmt. stack which has a connection to each firewall in the cluster

[SJ K]

Hi Mark,

Thanks but i am not entirely sure if i am reading you .. (sorry about that)...

Let's assume I have just 1 x L2 switch (named Switch2) and it, itself has a management port also.

Switch2's switchports are connected to all other devices' management ports.

Switch2 also has a switchport (not the management port) that is connected to the FW's port1.

FW's port1 and all the other management ports of the devices belong to the same subnet.

Management Network <<--- Switch2 --->> FW <<-->> Admin Network

On the other side of the FW, it is our Administrator network.

================================

Now, we can managed Switch2 (e.g. SSH) to it, by creating a VLAN interface on Switch2 and give it a IP (setting the default gateway to the FW's port1 ip).

We can also managed the rest of the devices in the management network via Switch2 (same vlan/subnet) through the FW (from the administrator's network)

q1) The problem is, Switch2 itself has a management port also. Do we use it at all ?
Do we still connect the management port of Switch 2 to a normal interface/switchport on Switch2 itself as well ?

================================

Or to summarize, if my Mgmt Switch has a management port, do i manage the MGMT switch via its management port, or via an SVI on its normal switchports ?  (if it is the prior, how should we connect the mgmt port of the mgmt switch ?)

Hope to hear from you soon.

Regards,
Noob

[Mark Malone]

Hey

don't worry struggling here myself trying to explain this :)

Or to summarize, if my Mgmt Switch has a management port, do i manage the MGMT switch via its management port, or via an SVI on its normal switchports ?  (if it is the prior, how should we connect the mgmt port of the mgmt switch ?)

So we don't manage our managed switch stack where all our switches mgmt. ports connect back to, the reason being we also have serial console switches in place and all our switches separately(including mgmt. stack) are connected back to them also as another form of redundant access,  our mgmt. stack of switches just has the svi vlan interface /23 and then an access port assigned on each switch on the stack back to each FW in cluster, we use opengear serial switches for console access, we just made an internal decision that its not production switch its only for management so the HUB stack where everything connects back could do with just serial access remotely and vlan mgmt. access , we did however use stacked switches that were dual power and separate feeds etc though to lessen the chance of an outage on it

so in short we don't manage the management switch like we manage our production switches apart from serial console  , you could though use the mgmt. port and connect it back to the firewall direct we had no spare ports though from what I remember as well and the SVI will be on your management stack switch already with just access ports back to the FW so its being managed just not by the actual mgmt. port

so it still can be managed through the svi way and mgmt. port could still be used but do you want to waste a firewall port on it , is it critical enough, its personal choice I suppose based on your network requirements and what else is in place for redundancy, if I had been given a spare port I probably would have connected it but just wasn't an option at the time of rollout of the out of band setup and the switch uses the same vlan interface and has access ports connected direct back to FW anyway on same subnet as everything else so its still managed but just like a switch that didn't have the option of a mgmt port.

[SJ K]

Hi Mark,

Thanks for the valuable insight and sorry for the late reply.

As per you said, I wouldn't be wasting a firewall port just to connect to the management port of the management switch since i am able to connect to the mgmt switch itself via its SVI interface/IP.

But at the same time, this means that all my devices are managed OOB using their management port, except for the mgmt switch itself (which is managed via the vlanIP).

Wouldn't this somehow fail the "OOB" concept ? (or it is inevitable ?)

Regards,
Noob