Strange output of "show version" on ISR4331/K9

[NetworkingGeek1]

Hello Community,

Can you please explain below output of

show version command

? I'm confused with it, why it indicates there are two operating systems IOS & IOS-XR on the same router? And plus both operating systems have different versions. How is it possible that different operating systems are running on the same router? Output is from Router - ISR4331/K9. Thank you.

Router# show version
Cisco IOS XE Software, Version 03.16.04b.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S4b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Mon 17-Oct-16 20:23 by mcpre 

[NetworkingGeek1]

@Ramblin Tech  Thank you for your reply. But I'm still confused. I'll put my question in other way:

Router# show version
Cisco IOS XE Software, Version 03.16.04b.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S4b, RELEASE SOFTWARE (fc1)

Does 

Cisco IOS XE Software Version 03.16.04b.S 

and 

Cisco IOS Software Version 15.5(3)S4b 

are indicates the same software here, but with just old and new numbering? Or it indicates two different software? And if it indicates different software, which one indicates IOS component? 

@Ramblin Tech  & @Joseph W. Doherty  Could you please clarify this? 

[Leo Laohoo]

---

@NetworkingGeek1 wrote:
Does 

Cisco IOS XE Software Version 03.16.04b.S 

and 

Cisco IOS Software Version 15.5(3)S4b 

are indicates the same software here, but with just old and new numbering? 

---

They are the same.  

[Joseph W. Doherty]

I believe two versions numbers are shown, whether different or the same, because there are two different logically and functional sets of code, and both these sets of code are needed to provide a working XE router.

Having a set of code with it's own versioning isn't uncommon.  Revealing this to end users, by/to end users, might not be supported, but if it's supported, the system might "exclaim" (display) as a message during boot up (BTW might be required by using someone else's software), or might require some

secret/hidden

command; and a whole range in between.  A good combination might use a matched set of version numbers or require something like @Ramblin Tech's secret decoder ring.

One example that comes to mind is Cisco's NBAR modules, each having its own version number.  A particular set comes with an IOS version, but they can be individually updated.

When discussing XE, there are two major sets of code.  The "new" OS Linux part, and the "adapted" classical IOS part.  Each having a version number, which can be the same number, or different numbers.  Something very important to keep in mind, only particular code sets will work, correctly, together.

The forgoing is somewhat like MACs and IPs.  Does a particular combination represent the same NIC?  To determine this you can use your secret decoder ring, MAC and ARP tables or use some MAC (via LAA) and IP numbering scheme to identify the same NIC.

The point of the prior is as a NIC has two major functions, L2 and L3, to support networking, each identified by a number, MAC and IP, which may somehow "match", ditto for XE.

To your question which is which component, I suspect, IOSd is the adapted classic IOS, v15, while v3 applies to Linux code set.

Again, to have an XE router, you need an acceptable combination of both, and using the same version number, for both, implies that.

[Ramblin Tech]

Hey Joe,

I vaguely remember someone saying about a thousand years ago that the first digit of the mcpdev/XE numbering referred to the underlying linux kernel numbering (or maybe that's just a false memory). That's a plausible explanation as, IIRC, the ASR1K was introduced in 2008 (-ish) with an

mcpdev 2.1

release, which fits with the availability of

2.x linux kernels

in that timeframe.

Linux 3.x kernels

started becoming available in 2011, which may be consistent (I haven't looked it up) with the availability of

3.x mcpdev

releases. But, AFAIK, there was always a one-to-one mapping of the mcpdev number to the IOS release that provided the IOSd PI features. But then again, mcpdev numbering lost any special meaning when that code train started being used as the baseline to build monolithic IOS images for SP products (MWR 2941, ASR901, 7600) that had classic Cisco kernels and not linux. 

In any event... I still believe Cisco exposing the mcpdev numbering to the public was a mistake, even if the intentions were good (publicizing the underlying linux kernel for market differentiation), as there has been nothing but confusion ever since. In hindsight, it might have been better to just add some kind of "XE" or "L" suffix to the regular IOS numbering to identify the presence of a linux kernel, like the old "S" trains did to indicate an emphasis on SP features, or the like "T" trains did to indicate feature-rich albeit extra-buggy code.

[Joseph W. Doherty]

". . . there was always a one-to-one mapping of the mcpdev number to the IOS release that provided the IOSd PI features."

That's not surprising, otherwise devs need to use their own secret decoding ring too.  Doing that, though, might mean the only change to some of the code was changing the version/release/patch number.  ; )

[NetworkingGeek1]

@Joseph W. Doherty  & @Ramblin Tech  So, can we say that in ISR 4331 which I bring as an example:

Cisco IOS XE Software, Version 03.16.04b.S - Extended Support Release

-> It's Linux Kernel code version.

Cisco IOS Software Version 15.5(3)S4b 

-> Version of IOS which runs as the component on top of that Linux Kernel.
And those versions should always match each other, even though they indicate different software, only in such version combination they can work with each other. 

Am I correct with the statement above? I made this conclusion from @Joseph W. Doherty statement.

[Joseph W. Doherty · Accepted solution]

"Am I correct with the statement above?"

Personally, I think that's correct, more-or-less.

Why more-or-less?

When XE first came out, just going by Cisco's publicly released information, I was under the impression, the Linux portion was mostly performing in a role much like a product like VMWare, allowing "classic" IOS to run under it.  Also much like OS|2 allowed DoS or Windows to run under it, or something like GNS3 running actual "classic" IOS binaries under it.

From the information that @Ramblin Tech provided, it sounds like "classic" IOS was, even initially, bound much tighter to Linux, than I understood.

Regardless, though, definitely there are two major components to XE, the Linux portion and the modified version of classic IOS to run under it.

For whatever reason (Jim's mentioned possible why this was done - as a "sales" feature [laugh - perhaps Cisco's version of "new and improved" or "lemon freshened"]), Cisco chose to make it clear there were these two major component parts of XE.

As to version numbers, software developers can do whatever they want, and whether different component numbers need match, or only "mate" with a specific corresponding number, again, up to the software developer's.

(As an aside, on one major software project I worked on decades ago, I proposed version numbers as part of the information we passed between programs exchanging data.  At first, the other developers say, huh?  I said, how else can we easily check we're actually usually the same "template"?  Interestingly, decades later, Cisco Catalyst 3750 appeared to have a version/release number embedded in their StackWise code, so that stack members running different IOS versions could determine if their StackWise code was compatible.  [I recall, if StackWise version number matched, but release number differed, stack would throw a warning, but if the StackWise version number differed, StackWise would throw an error message and the whole stack would not function as a single unit.])

[Ramblin Tech]

There is some good stuff on XE architecture here starting on slide 70:

https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2023/pdf/BRKARC-2092.pdf

What I have been referring to as "mcpdev" this preso calls out in one place as "BINOS" (BCN Internetworking OS). An important thing to note is that IOSd is not a VM hosted by linux kvm (or some other HV). IOSd (and related components) are linux processes. HTH.

[Joseph W. Doherty]

@Ramblin Tech, very nice!

However, look at slide 74, for

XE 3.7.x

"Monolitic IOSd", "Linux daemons hosting capability", etc., seems to much correspond to my recollection of my impression of XE as it initially (BinOS slide 72 [hmm, I remember a BCN - same? - telecommunications?]) was (in a prior post, you'll even find my mention of "daemons").

In

XE 16.x

and

17.x (and NG XE)

, shows major evolution!  Possibly some significant changes in

XE 3.x

NOVA variant too vs. the initial BinOS variant.

"An important thing to note is that IOSd is not a VM hosted by linux kvm (or some other HV)."

Never said it was exactly the same as VM hosting, just my "impression", from the information released at that time (2007?), XE was "mostly" "like".  I try (although don't always succeed - laugh) to use my words with care.  (BTW, and even for a VM OS, how do you think it might "manage" client virtual machines?  They might be hosted, principally, as a process under the VM OS.  Or, do you think, if using a Linux kernel, especially if customized, might not also support a VM environment?)

Unless you've into the subject of VM and emulations, there's a lot (like the devil) in the details in how it's done, especially when trying to avoid the software you're hosting, running like a pig.

For a while, OS|2 "claimed" Windows ran better under it than natively.  Possibly true, and such a situation not unique to OS|2.

I will say, some of the things you need to do to accomplish like, or better, performance, than running something natively, are usually very much tied to a very specific variant of the software you're hosting.  When Microsoft came out with one of their point releases for (don't recall which) DoS or Windows, it no longer ran under OS|2.  The "rumor" was the only reason for that release, was to break the software under OS|2.

Remember, your point about the XE component and IOSd component, being specific version matches?  Un-huh, see that for most Linux software?

Again, does what I initially thought (back in 2007), mean, parts of IOSd, weren't, even then, tied more directly into Linux?  Not at all, and would expect more and more of that as the XE evolved.  From your referenced presentation, there appears to have been much evolution.

I think my prior post, still pretty accurately sums XE up, although on a question I didn't address in that prior post, is IOSd, an OS in its own right?  As it appears even the first variant might have been unable to run without XE, i.e. natively on any platform, I would say the answer is no.  However, IOSd, then, and possibly still now, might do much that's normally managed by the OS.

Is the Linux component an OS?  Likely yes, although by itself, it doesn't do anything "useful"; which is true of the core of any OS.  They manage hardware, but additional OS programs (not the kernel portion), set up things to be useful.  For example, the kernel likely manages the disk controller, but a file management subsystem provides file management for on-line (not just disk) storage.

[edit Addendum]

To be clear, my statements are based on XE Cisco public information I recall seeing almost a decade ago.  Likely, only an XE dev, who really had internal access, and truly understands what was done to make XE, can accurately answer what XE was, and this is assuming all such information is NOT proprietary, and I suspect some was/is.

For those of us who know how to reverse-engineer, given enough (i.e. lots and lots and lots of) time, we likely could be fully accurate about XE, but generally, reverse-engineering is illegal for proprietary software.

We've certainly have come a long way from a very basic question - i.e. about XE version numbers!

Personally I've found this very interesting, and would like to thank @Ramblin Tech (Jim) for all the additional information provided.

Comment on @Leo Laohoo comments, definitely one might think of XE as a single entity, but the way it was/is structured is different from "classic" IOS and the NXOS and XR with their new from the ground-up approaches.

From the reference Jim provided, in a way, it's interesting XE hasn't been directly replaced by something like NXOS or XR, but perhaps, it's almost morphing into those kinds of OSs.

To put things into perspective, IBM's OS 360 took 5,000 man years to develop, and it too, has evolved much in its latest variants.  So, development of OSs, aren't trivial.  (BTW, developing IOS, might be compared with developing something like OS|2 but also including a whole office suite, web server and database server too.)

Linux, Linux, Linux.  I recall Linus Torvalds and I studied from the same text book, how to create a lite version of Unix (MINIX)

(forgoing confirmed by Wiki's "He bought computer science professor Andrew Tanenbaum's book Operating Systems: Design and Implementation, in which Tanenbaum describes MINIX, an educational stripped-down version of Unix.")

We differed in our reaction to that book.  I obtained a copy of MINIX to run (play with) on my PC (I also had at least one other variant of Unix too), but Linus ran with developing his own Unix variant, which, I believe, would not be where it's today if not for all the others that joined in its further development.  (Don't misunderstand, in no way is the forgoing meant to denigrate, in any way, what Linus accomplished, just, again, lots and lots and lots of man hours have gone into all the Linux variants, that Linus would likely never have been able to accomplish all that alone.)

[NetworkingGeek1]

@Ramblin Tech  So, can we simply say that in the example I bring on Router ISR 4331:

Cisco IOS XE Software, Version 03.16.04b.S - Extended Support Release 

-> BINOS version

Cisco IOS Software Version 15.5(3)S4b 

-> IOSd version which runs as linux processes.
And IOSd runs on top of BINOS. So, they're two different pieces of software, but IOSd runs on top of BINOS.
Am I correct with the statements above? 

[Ramblin Tech · Accepted solution]

@NetworkingGeek1 Yes, the infrastructure component (let's call that "BINOS") and control plane component (IOSd) that runs on top of the infra are two separate pieces of s/w bundled up and shipped together. I tend to think of the 3.16.4 release numbering as encompassing all components in the bundle: infra, IOSd control plane, data plane (code to drive the NPU and interfaces), etc. But, I think it is quibbling of me to say any other view on that is wrong (unless that other view is coming from the XE engineering org, in which case I'll just finally shut up).

Thankfully, as you have seen with your upgrade to 17.6.5, there is now only one release number system in use with XE. Whew!

[Joseph W. Doherty]

Ah, I remember modular IOS.  I recall it also quickly developed a reputation for having many issues not seen in the similar version for the non-modular IOS 

[NetworkingGeek1]

@Ramblin Tech  @Joseph W. Doherty  @Leo Laohoo   Thank you all for your answers.

[Joseph W. Doherty]

Just stumbled across: https://community.cisco.com/t5/networking-knowledge-base/cisco-ios-and-ios-xe-naming-convention-for-routing-platforms/ta-p/4520161

If you search for IOSd, you'll find 3 references, in the following:

On this ASR 1000 series router image there are two new elements: the IOSd version identifier and the release type.

We can see that the version for IOSd is 15.5 The “d” in IOSd stands for daemon. A daemon is a Linux program that executes in the background ready to perform an operation when it’s required. This is software that incorporates the routing protocol functionality of Cisco IOS software and is essentially Cisco IOS software code that runs as a separate process on a device.

And this:

The above, is a very concise (and, IMO, correct) description of what I was trying to explain.

In another recent posting, Peter Paluch, investigating high CPU, wanted to see the CPU stats for CPU processes.  I.e. what IOS is doing?

He wrote:

I've recently had an opportunity to help with another Cat4500 case, and I learned that here, at least the

show processes cpu

command (don't recall about

show memory

) produces a list of processes from the underlying Linux operating system. However, you won't see the processes inside the IOSd blob which are the ones that we both are interested in - such as IP Input or OSPF; we will only see iosd as a single Linux process. That is why I asked for the

show processes cpu

detailed process iosd sorted" output in my post that drills down into IOSd processes the usual way we were used to in old IOS.

Further confirms, IOSd, basically is a modified classic IOS, as it still does its own internal process management.

None of the above, contradicts what anyone has said on this subject, but also possibly shows, how

unique

(in networking) XE is, one OS using another OS (IOS), but unlike a "VM".  Further, Cisco, chose to

display

version information about these two major components.

BTW, for me, XE, conceptionally, isn't that unique.  In my IBM mainframe days, we had major subsystems, like a TP monitor (e.g. CICS) or a database system (e.g. IDMS), which, in many ways, do things that are only usually seen in the OS, alone.  In more modern systems, on a Linux OS, software "programs" like a database server or web server, also often mimic some functions (e.g. process and memory management) often unique to the OS.

Perhaps, one way to think of XE is to compare it to something else somewhat like it, like our human brain.. Which has a right and a left half, which work together, yet each half is known to do somethings better than the other half.

I whole heartily agree, with @Ramblin Tech, that, for whatever reason, when Cisco decided to showcase XE's both "halves", rather than the one unified combination, caused, and causes, much confusion.