topic Re: Massive Data Integrity Issues - Why the API is Worthless in Services Discussions

Massive Data Integrity Issues - Why the API is Worthless

anthony.t.nelson — Fri, 09 Sep 2016 18:27:04 GMT

I don't mean to be too inflammatory, but the data behind the API is all but worthless in its current state. There is little to no consistency with how Product ID's, or thereby the software versions, are written. Take the following OS: IOS-XE 3.10S.1. There are four variations on how the advisories reference this one piece of software.

Cisco IOS XE Software 3.10S 3.10.01S
Cisco IOS XE Software 3.10S 3.10.1S
Cisco IOS XE Software 3.10S.01
Cisco IOS XE Software 3.10S.1

And here is a list of those advisories and how they are "linked".

{'id': u'cisco-sa-20140326-ipv6', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20140326-sip', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'Cisco-SA-20140428-CVE-2014-2183', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'Cisco-SA-20140709-CVE-2014-3309', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20140924-metadata', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20140924-sip', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20140926-bash', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20141015-poodle', 'software': u'Cisco IOS XE Software 3.10S.1'}

{'id': u'Cisco-SA-20150113-CVE-2015-0204', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150310-ssl', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150325-ani', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150325-ikev2', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150325-iosxe', 'software': u'Cisco IOS XE Software 3.10S.1'}

{'id': u'cisco-sa-20150325-mdns', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150325-tcpleak', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150408-ntpd', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'Cisco-SA-20150729-CVE-2015-4293', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150923-fhs', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150923-fhs', 'software': u'Cisco IOS XE Software 3.10S 3.10.01S'}

{'id': u'cisco-sa-20150923-iosxe', 'software': u'Cisco IOS XE Software 3.10S.1'}

{'id': u'cisco-sa-20150923-iosxe', 'software': u'Cisco IOS XE Software 3.10S.01'}

{'id': u'cisco-sa-20150923-sshpk', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20150923-sshpk', 'software': u'Cisco IOS XE Software 3.10S 3.10.01S'}

{'id': u'cisco-sa-20151021-ntp', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20151130-iosxe3s', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20151130-iosxe3s', 'software': u'Cisco IOS XE Software 3.10S 3.10.01S'}

{'id': u'cisco-sa-20160127-ntpd', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20160323-dhcpv6', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20160323-ios-ikev2', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20160323-sip', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20160419-ios', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20160419-ios', 'software': u'Cisco IOS XE Software 3.10S 3.10.01S'}

{'id': u'cisco-sa-20160525-ipv6', 'software': u'Cisco IOS XE Software 3.10S 3.10.1S'}

{'id': u'cisco-sa-20160525-ipv6', 'software': u'Cisco IOS XE Software 3.10S 3.10.01S'}

Further, the SNMP value comes out as "03.10.01S". Some IOS-XE are "03.06.03.E" or "03.04.06.SG". Not to mention IOS-XE can be represented as 3.6.0S or 15.2(2)S.

Regular IOS has both:

Cisco IOS Software 12.4(7)
Cisco IOS 12.4(7)

Here are the advisories and softwares for this one software.

{'id': u'cisco-sa-20070110-dlsw',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20070131-sip',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20070509-iosftp',         'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20070522-crypto',         'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20070522-SSL',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20070808-IOS-voice',      'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20080326-dlsw',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20080708-dns',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20080924-iosips',         'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20080924-multicast',      'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20090325-ip',             'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20090325-mobileip',       'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20090325-webvpn',         'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20090826-cucm',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20090908-tcp24',          'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20090923-tunnels',        'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20100324-cucme',          'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20100324-ldp',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20100922-sip',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'Cisco-SA-20110610-CVE-2011-1631',  'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20110928-nat',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20120328-msdp',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20120328-pai',            'software': u'Cisco IOS Software 12.4(7)'}
{'id': u'cisco-sa-20120926-cucm',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20120926-sip',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'Cisco-SA-20130327-CVE-2013-1142',  'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20130327-nat',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20130801-lsaospf',        'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20130925-ipv6vfr',        'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20150325-tcpleak',        'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20150408-ntpd',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20151021-ntp',            'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20151120-ns',             'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20160127-ntpd',           'software': u'Cisco IOS 12.4(7)'}
{'id': u'cisco-sa-20160525-ipv6',           'software': u'Cisco IOS 12.4(7)'}

The question: Is Cisco going to normalize this data so I don't have to account for every variation through regex magic?

Re: Massive Data Integrity Issues - Why the API is Worthless

Omar Santos — Mon, 19 Sep 2016 22:19:56 GMT

Hi Anthony,

Thank you for your feedback. Is this data from the CVRF xml's or from the OVAL definitions? We are aware of issues with the CVRF data around version information. For IOS and IOS XE the best way to retrieve information about the versions is by either using the OVAL definitions via the API or the IOS Software Checker tool.

Re: Massive Data Integrity Issues - Why the API is Worthless

anthony.t.nelson — Thu, 29 Sep 2016 18:54:19 GMT

Omar,

I appreciate the response. However, I'm finding even further discrepancies going that route. Examine Advisory cisco-sa-20160928-dns, which has an OVAL and CFRV definition.

The OVAL definition only lists testing against the running config "config ip dns server", but does not have any IOS versions listed. Further, the OVAL definition does not include any of the advisory information that would be helpful for "more information", just a ref_url.

The CVRF includes all of the information you would want about the advisory and all the product names associated (IOS releases). But as you stated, the CVRF OS information does not have strong data integrity.

Finally, in order to piece all of this information together, I have to bounce the API twice as I only receive the XML URL from either OVAL or CVRF. And the /oval/Advisory/{advisory_id} doesn't seem to work all of the time from what I've seen.

Re: Massive Data Integrity Issues - Why the API is Worthless

Omar Santos — Tue, 04 Oct 2016 01:29:29 GMT

Hi Anthony,

Thank you for the heads up on this! Can you please check the OVAL definitions and please let me know if you encounter any problems. It appears that there was a synchronization problem; however, I have just checked them all and they do have the version information.

Re: Massive Data Integrity Issues - Why the API is Worthless

anthony.t.nelson — Wed, 05 Oct 2016 14:48:21 GMT

Just looked and the OVAL XML was updated for the few I checked last week and now includes IOS versions. Thank you for the update.

I'm likely going to build something against OVAL, but I'll still need to account for ASA's and other Cisco gear elsewhere. I like that the OVAL definition does include actual tests both code and by service selection.

Re: Massive Data Integrity Issues - Why the API is Worthless

Andrei Batyrov — Mon, 31 Oct 2016 14:13:28 GMT

Hi Anthony,

Where did you get the 'software' key? I cannot see it in return results from https://api.cisco.com/security/advisories/cvrf/all

Re: Massive Data Integrity Issues - Why the API is Worthless

klohse — Thu, 03 Nov 2016 09:20:33 GMT

Hi Omar,

i also tried the API, works pretty well but i have the same problem, the detailed XML does not contain all affected IOS versions, for example cisco-sa-20160419-ios.xml ( CVE-2016-1384), the Bug tool shows much more.

Will CVRF show all someday or is it better to use OVAL at all ?

I started with CVRF because it looked more simple to decode to me.

Re: Massive Data Integrity Issues - Why the API is Worthless

Omar Santos — Thu, 03 Nov 2016 15:36:39 GMT

Hi!

There are limitations with CVRF and as a matter of fact, we are standing a new TC in OASIS to enhance CVRF (standard):

https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=csaf

which is why we are recommending OVAL for that type of evaluation. On the other hand, I have better news... we are expanding the API very soon (within the next two months) to support IOS software checker. You will be able to submit an IOS or IOS-XE version and get all advisories that affect such version, along with the first fixed version information.

Regards,

Omar Santos

PSIRT, Security Research and Operations

Cisco Systems, Inc.

Email: os@cisco.com<mailto:os@cisco.com>

Phone: 1 919 392 8635<tel:1%20919%20392%208635>

PGP Key: 0x3AF27EDC

Re: Massive Data Integrity Issues - Why the API is Worthless

Andrei Batyrov — Fri, 04 Nov 2016 01:44:24 GMT

Hi Omar,

Good news regarding the Software Checker. By the way, do you know if they have plans to start supporting other OSes as well, like IOS-XR and NX-OS?

Re: Massive Data Integrity Issues - Why the API is Worthless

Omar Santos — Fri, 04 Nov 2016 06:07:02 GMT

Hi Andrei,

Cisco is starting to scope this for NXOS, but I don't have an ETA yet.

Re: Massive Data Integrity Issues - Why the API is Worthless

Andrei Batyrov — Fri, 04 Nov 2016 06:10:00 GMT

Sounds great, thanks for the update!

Re: Massive Data Integrity Issues - Why the API is Worthless

klohse — Mon, 07 Nov 2016 09:07:00 GMT

Hi Omar,

hmmm, i am wondering why CVRF is the Problem here, only more lines with product IDs need to be added.

My script does not care if there are 10 or 30 XML tags with product IDs, i don't see in the Standard that there is a limitation.

Re: Massive Data Integrity Issues - Why the API is Worthless

klohse — Tue, 08 Nov 2016 10:25:05 GMT

Hi Omar,

just found another problem, the Version history is not up to date in the CVRF XML file, API said update on Nov 2 to V1.2 but when downloading the detail XML still Shows 1.1 as the latest release.

-<DocumentTracking>

-<Identification>

<ID>cisco-sa-20161026-linux</ID>

</Identification>

<Status>Interim</Status>

-<RevisionHistory>

-<Revision>

<Description>Initial public release.</Description>

</Revision>

-<Revision>

<Description>Updated the investigated products as Vulnerable or Not Vulnerable.</Description>

</Revision>

</RevisionHistory>

-<Generator>

</Generator>

</DocumentTracking>

Re: Massive Data Integrity Issues - Why the API is Worthless

Omar Santos — Tue, 08 Nov 2016 23:14:13 GMT

Hi klohse,

I checked this myself and also with our development team that support the API, as well as the team that supports the creation of the CVRF file. We cannot reproduce that issue (testing with the advisory you mention below and with several others).

Currently, the version of the advisory is 1.3:

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20161026-linux

CVRF file: http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20161026-linux/cvrf/cisco-sa-20161026-linu…

<ID>cisco-sa-20161026-linux</ID>

</Identification>

<Status>Interim</Status>

<Description>Initial public release.</Description>

</Revision>

<Description>Updated the investigated products as Vulnerable or Not Vulnerable.</Description>

</Revision>

<Description>Updated the investigated products as Vulnerable or Not Vulnerable.</Description>

</Revision>

<Description>Updated the investigated products as Vulnerable or Not Vulnerable</Description>

</Revision>

</RevisionHistory>

OUTPUT from openVuln API:

bash-3.2$ openVulnQuery --cvrf --advisory cisco-sa-20161026-linux --json test.json

bash-3.2$ cat test.json

[

{

"advisory_id": "cisco-sa-20161026-linux",

"cves": [

"CVE-2016-5195"

"cvrf_url": "https://tools.cisco.com/security/center/contentxml/CiscoSecurityAdvisory/cisco-sa-20161026-linux/cvrf/cisco-sa-20161026-linux_cvrf.xml",

"first_published": "2016-10-26T20:00:00+0000",

"last_updated": "2016-11-08T20:28:14+0000",

"sir": "Medium"

}

]

All three appear to be in sync.

Re: Massive Data Integrity Issues - Why the API is Worthless

klohse — Wed, 09 Nov 2016 05:58:12 GMT

Hi Omar,

yes, pretty strange, i just ran an update (Nov 9 6:48AM CET) and now it is correct.

I swear i ran the same Software on Nov 7 8:36AM CET and there was only V1.1 in the XML because API said there was no update, so my tool did not download the update.

The XML tag "last updated" said 2016-11-02T21:12:29+0000

However, i will now try OVAL to decide what we will use in the future...

Kind regards,

Kai Lohse