Zero touch provisioning from F/W 7.3.7 up to 7.6.1

[brantwinter2004]

Hi - I spent an entire day trying to get automate the zero touch provisioning of SPA504G phones from the lowest possible F/W 7.3.7 up to the current 7.6.1.

• I have DHCP option 66 set to the web server address (example only) "http://192.168.10.10/provisioning" (note it is http as the old phones I don't think have the root CA's compatible with currently issued Cisco CA certs).
• I factory reset handsets
• They have an initial profile_rule of /spa$PSN.cfg
• I place an initial config file, spa504G.cfg, in /provisioning on the web server and confirm the URL works via a browser.
• The phones boot, I see them get an IP and the DHCP 66 option but then in the NGINX server log I see them try and get spa504G from the root of the webserver, not, /provisioning.
• I then set up a TFTP server ( I don't want one of these on the network to be honest) and the phones do pick up the config file from provisioning.
• The next issue, the old firmwares don't seem to support the evaluative logic in upgrade rules, ie:
• ($SWVER lt 7.5.2b) ? http://192.168.10.10/provisioning/spa50x-30x-7-5-2b.bin | http://192.168.10.10/provisioning/spa50x-30x-7-6-1.bit

So I am a bit stuck, I need to ensure I can get phones from the lowest possible F/W revision up to 7.6.1.

I know I have to get them to at least 7.5.2b before 7.6.1.

I want the spa504G.cfg file to come via a http server, not TFTP, and I then want to replace the profile_rule with a https:// URL to the proper config file.

The initial spa504G.cfg file needs to get the phone up to F/W 7.5.2b and then I can do the 7.6.1 F/W upgrade via the final xml profile_rule URL.

I found that in the spa504G.cfg file I could put the downgrade revision limit to 7.6.1 ad then the would prevent a 7.6.1 phone being downgraded to 7.5.2b after a factory reset.

I also thought that if the DHCP 66 option could not set the path as well as the IP to the initial .cfg file I may need to set up a virtual host for the URL name and just put it in the root of the virtual server.

Any help would be appreciated but the requirement is simple to get an SPA504G witg F/W 7.3.7 up to 7.6.1 with my current XML config on it.

On a side note, the 7.6.1 SPC tool would not run under Linux - kept giving a 'segmentation fault'.

[brantwinter2004]

I downloaded and flashed 7.5.6-XU and this is what I see in the host headers. Looks like I will be able to handle this case with some more regex:

192.168.10.97 - - [16/Mar/2016:17:57:49 +1000] "GET /cisco1.py HTTP/1.1" 200 3658 "-" "Cisco/SPA504G-7.5.6(XU) (58BFEA1104E6)(CCQ16380FYH)"

[Dan Lukes]

Great!

My conclusion has been based on behavior of an old -XU version and I has never analyzed it again. So it is good news.

[brantwinter2004]

Well that was a waste of time.... the phone with the 7.5.6(XU) firmware failed the SSL client auth so it couldn't talk to the provisioning server.

[Dan Lukes]

It's the matter of -XU flavor. It's firmware with cryptography capabilities restricted. It's dedicated to those countries where citizens are not allowed to encrypt their data - like France or Russia.

But you may not consider it stopper. We have client configuration configured to be 'optional' at WWW server layer.

Our provisioning script is willing to upgrade any phone - even phone unknown to us or phone with no client certificate installed. It doesn't harm. In such case we provide just those few options required to fire upgrade.

Full configuration is server to authenticated devices only.

It's provisioning script decision to provide either full or "upgrade only" configuration to particular client.

[brantwinter2004]

I still don't get this... I flashed 7.5.6(XU) to a factory reset phone. It reboots and the DHCP 160 option told it to go to the https URL (my Python code). I see the entry in the Apache access logs. The phone then reboots after picking up settings for upgrade_rule only. I verify this in the phone gui. 

So I know the 7.5.6(XU) phone has passed the ssl client auth on the Apache web server to have been able to change its factory default settings. 

The upgrade rule tells the phone to get f/w 7.6.1 from a http URL (no ssl for firmware downloads). But instead of getting the firmware it just hits the provisioning URL every 10s as per the periodic time I set via the python code. 

I can't work out why it won't get the f/w. If I force it via the URL (http://x.x.x.x/admin/upgrade?http://x.x.x.x/provisioning/spa-7.6.1.bin) to go to the exact URL in the upgrade_rule it downloads fen and auto provisioning continues. 

Pat first I thought it was the client ssl auth I have set up but it gets the initial configure from the python provisioning code so i must have the correct root certa on the Apache server. 

[Dan Lukes]

The *name* of firmware file needs to be taken into consideration.

Phone remembers the filename of latest firmware downloaded. It will never (unless reset to factory default has occurred) download the file with the same name again.

If the content of firmware file changes, the name must be changed as well or the upgrade will not occur.

I'm unsure it may explain the issue you are facing as I know no names you has use.

The upgrade rule tells the phone to get f/w 7.6.1 from a http URL (no ssl for firmware downloads). But instead of getting the firmware it just hits the provisioning URL

Full syslog&debug may (or may not) help to analyze it. Especially if manual (URL) form is working correctly.

Note the 10s for periodic resync sounds very low to me, even for tests.  It may overload the phone and affect behavior. May be the resync have priority over upgrade here and there's no time for upgrade. I has never tried as low value here, so I have no experience.

[brantwinter2004]

Between each testing attempt I have performed a factory reset. I changed resync_periodic to 180s.

So what I am observing is as follows:

• factory reset on F/W 7.6.1
• immediately on reboot I perform a manual (URL initiated) downgrade of firmware to 7.5.6(XU)
• phone reboots
• phone gets python script from profile_rule
• phone makes a https connection to provisioning script
• ie. mutual ssl auth is working
• and it can make an SSL connection to the provisioning server
• phone gets (XU) specific instructions which is to turn on syslog
• phone then retrieves profile from SSL provisioning server every 180s
• phone never actions the upgrade_rule
• but... if I go in to the GUI and change any character in the URL string that points to the http server holding the .bin files and save the settings, the phone immediately does an f/w upgrade to the 7.6.12 as per the upgrade_rule URL.

Note: a factory reset has been does at beginning of each testing cycle therefore the firmware names shouldn't come into the picture.

I also noticed the follow ing the syslog which is baffling. 192.168.10.22 is a host that was running a tftp daemon a few days ago when I was testing. This phone has been reset many times since then but I see this in the log where the phone attempts to get a generic spa.bin f/w from this tftp server:

2016-03-17
08:52:10
Debug
SEP58BFEA1104E6.local
local3
 
try upgrade: tftp://192.168.10.22(0x160aa8c0):69//spa.bin
2016-03-17
08:52:10
Debug
SEP58BFEA1104E6.local
local3
HTTPD_DEBUG
parse url fin (true)?tftp://192.168.10.22:0//spa.bin
2016-03-17
08:52:10
Debug
SEP58BFEA1104E6.local
local3
HTTPD_DEBUG
upgrade url (true)?tftp://192.168.10.22/spa.bin check start

[brantwinter2004]

I decided to change code logic to take 7.5.6(XU) to 7.5.7 before going to 7.6.1. After I did this it auto provisioning fine. The first call below to 

GET /resources/spa50x-30x-7-5-6-XU.bin

was the manual URL call I made to downgrade firmware to 7.5.6(XU) to allow auto provisioning to commence.

192.168.10.97 - - [17/Mar/2016:04:06:47 +1000] "GET /resources/spa50x-30x-7-5-6-XU.bin HTTP/1.0" 200 4312178 "-" "Cisco/SPA504G-7.6.1 (58BFEA1104E6)(CCQ16380FYH)"
192.168.10.97 - - [17/Mar/2016:04:09:53 +1000] "GET /cisco1.py HTTP/1.1" 200 2832 "-" "Cisco/SPA504G-7.5.6(XU) (58BFEA1104E6)(CCQ16380FYH)"
192.168.10.97 - - [17/Mar/2016:04:09:55 +1000] "GET /resources/spa50x-30x-7-5-7.bin HTTP/1.0" 200 4447179 "-" "Cisco/SPA504G-7.5.6(XU) (58BFEA1104E6)(CCQ16380FYH)"
192.168.10.97 - - [17/Mar/2016:04:13:05 +1000] "GET /cisco1.py HTTP/1.1" 200 2842 "-" "Cisco/SPA504G-7.5.7 (58BFEA1104E6)(CCQ16380FYH)"
192.168.10.97 - - [17/Mar/2016:04:13:07 +1000] "GET /resources/spa50x-30x-7-6-1.bin HTTP/1.0" 200 4443721 "-" "Cisco/SPA504G-7.5.7 (58BFEA1104E6)(CCQ16380FYH)"
192.168.10.97 - - [17/Mar/2016:04:16:19 +1000] "GET /cisco1.py HTTP/1.1" 200 4314 "-" "Cisco/SPA504G-7.6.1 (58BFEA1104E6)(CCQ16380FYH)"

[Dan Lukes]

Well, there seems to be some issues during updates between XU and non-XU flavors of the firmware of the same version.  But upgrade to newest firmware of non-XU firmware seems to work regardless the flavor of older firmware.

It's something I can live with. It doesn't sound plausible the Cisco will ship devices with the latest firmware installed, thus brand new devices will be upgraded by our framework all the times.  As we never upgrade to XU, the bias will be eliminated during first upgrade which will occur. I can be sure I will get phone properly upgraded to required firmware.

If I will meet a phone with latest XU image, it's a device patched by a rogue user. Such device should not be successfully attached to our network, so I don't care so much it may not be upgraded properly.

[brantwinter2004]

Also, once you have echo'd the values back to the phone ie.

    echo "<Upgrade_Enable>Yes</Upgrade_Enable>";

Is there any 'end of file' type string that has to be sent to the phone ? Once I have sent all values back to the handset what happens next ?

[Dan Lukes]

Provisioning still needs to be XML-like, despite generated on the fly. So you need to follow the structure. The configuration still needs to be enclosed in <flat-profile>...</flat-profile>

Thus </flat-profile> is the 'end-of-file' margin.

Once I have sent all values back to the handset what happens next ?

I most cases the phone will reboot to apply changes. If no boot-triggering value has been changed, then phone will return to idle state with new configuration applied.

Also see all those Resync_* options.

[brantwinter2004]

OK cool - that makes sense.

[brantwinter2004]

Is there any security on your setup ? I mean, if I knew a MAC address of one of your devices could I not spoof a HTTP request and then have an XML config file delivered to me ?

[Dan Lukes]

No. We use mutual authentication. Not only HTTP server use certificate to authenticate self, we require the phone use embedded certificate to introduce self as well.

As long as Cisco will not issue client certificate to such rogue user (or he will not tear out it from a phones chip) no unauthorized device will receive a configuration from us.

[brantwinter2004]

Are you 100% certain it is DHCP option 150 and not 159 ? DHCP option 150 seems to be for a 'list' of TFTP server IP's. I had a feeling that the old firmwares did not support DHCP option 159 anyway.