No "UC320W custom firmware for SPA504G phone" exists, as far as I know. Yes, UC320W push firmware to phone, but it's standard firmware.
Unless you upgraded UC320W firmware, there's no reason do downgrade/upgrade SPA504G. Particular UC320W version works with particular SPA504G version only.
You mentioned neither current nor intended new version of SPA504G, also, reason for upgrade remain undisclosed. Sorry, so hard to advise something valuable ...