It looks like Removing the switch and then adding back in does remove the ERL assignment. I suppose one could export the switch port details, delete the switch, re-add the switch then import back all the ports but the one you were trying to remove. Wow what a pain.
Sort of annoying but the application isn't designed for deleting many things, it maintains it in the database usually just flagged as deleted or inactive. Removing an assigned ERL will drop it back to default if a phone ends up on one of the ports, there's essentially not much benefit to that if you have a better ERL to leave assigned to it in case something changes and your documentation hasn't yet been updated. In my opinion anyway.
For a while I had an "unassigned" ERL but as it was essentially default I just set the broadest one to the equipment, and be more specific when I know more.