Building Repeatable Engineering Methodologies

Ahmadalkayyali · ‎07-14-2026

Introduction

This document describes how I built a repeatable way of working as an enterprise voice and network engineer.

It covers the five things I use on every project: templates, checklists, risk models, validation processes, and rollback procedures.

The goal is simple. Anyone should be able to pick up my work and do it again without me in the room. A good result once is luck. A good result every time is a method.

Prerequisites

Requirements

There are no special requirements for this document. It helps if you have done at least one change on a live voice system, and if you have felt that quiet worry during a maintenance window when calls did not come back the way you expected. That feeling is the reason this article exists.

Components Used

The examples in this document can apply to CUCM, UCCE, CVP, CUBE, Finesse, CUIC, SIP trunks, and other enterprise voice platforms, as well as Cisco IOS, IOS-XE, and Nexus.

Supporting tools I use:

A ticket or change system
A place to store backups and pre-check output
RTMT and whatever monitoring the business already trusts
A simple spreadsheet

The information in this document was created from lab and test environments. If your system is live, plan your change window first.

Background Information

When I started, every change was a fresh start. I opened a blank page, copied steps from an old email, changed a few names, and hoped. Sometimes it worked. Sometimes I spent the night explaining to a business team why their agents could not log in.

The turning point was not a new tool. It was one honest question: if I did this same task next month, would I do it the same way? The answer was no. And that was the problem.

Voice makes this worse than most areas, because one small change can touch many systems. A single SIP trunk change can pull in CUCM, CUBE, the carrier, firewall rules, certificates, call recording, UCCE scripting, and reporting. If you treat that as a one-off, you will forget something. Not because you are careless, but because there is too much to hold in your head at 11 p.m.

So I stopped treating each task as a one-off, and started building a method.

The Five Building Blocks

1. Templates

A template is a plan with the moving parts pulled out.

I noticed that most of my change documents were about 80 percent the same. Only the system, the business group, and the test steps changed. So I saved the common part once and left blanks for the rest.

I keep templates for:

Change requests
Maintenance window plans
Pre-check and post-check notes
Cutover plans
Rollback steps
Test call results
Communication emails
Vendor and carrier coordination
Implementation summaries

A template does not need to be complicated. Mine answer simple questions:

What are we changing?
Why?
What systems are impacted?
Who needs to be involved?
What is the risk score?
What is the test plan?
What is the rollback plan?
How will we know it worked?

The same idea works for config. On CUCM I do not build a new device pool, partition, or route pattern from memory. I keep a naming standard and follow it, so device number 40 looks exactly like device number 1. When everything looks the same, "different" becomes a clue instead of normal life.

Templates also help new engineers. They can follow the structure instead of guessing what belongs in a change plan.

2. Checklists

A template tells me what to write. A checklist tells me what to do around it.

Pilots use checklists. Surgeons use checklists. Not because they forget how to fly or how to cut, but because tired people skip steps. I am a tired person quite often.

Before the change

Business approval and maintenance window confirmed
DRS backup taken and verified, not just scheduled
Database replication healthy on all nodes
Certificates checked for expiry
DNS and NTP correct
Services running, device and agent registration counts recorded
SIP trunk and route list status recorded
Pre-checks captured and saved to a file
Rollback written down and tested
Peer review done
Carrier or vendor on call if needed

During the change

Record the start time
Follow the approved steps, in order
Capture output or screenshots at each major step
Do not change the plan mid-window unless there is a clear reason
Stop and reassess if something unexpected appears

After the change

Post-checks captured and compared with pre-checks
Test calls placed, inbound and outbound
Agents able to log in to Finesse
Reports and CDR data still flowing
Recording working if in scope
Monitoring clean for a set period
Ticket updated and lessons learned written down

One habit worth calling out: I re-run replication and registration checks about 30 minutes before the window, not just the day before. Things move. A node that looked fine at 9 a.m. is not proof it is fine at 11 p.m.

A checklist does not replace engineering knowledge. It supports it.

3. Risk Models

Not every change is the same size. Adding a description to a device is not the same as renewing certificates on a cluster. Early on I treated them the same, which meant I was either too slow on small work or too fast on big work.

So I score every change before it starts.

Question 1 point 2 points 3 points

User impact — who breaks if this goes wrong?	One user or one device	One site, one queue, or one business group	Whole cluster, all agents, or all inbound calls
Business impact	Internal only	Customer calls affected	Sales, support, or emergency calling affected
Technical reach	One setting, one system	Two or three systems	Routing, certificates, database, or SIP path
Recovery — how fast can I undo it?	Minutes	Needs a restore or a version switch	Hard, needs a rebuild or a carrier change
Testing	Full test plan possible	Partial test only	Cannot test until it is live

Add the points and you get a score from 5 to 15.

5 to 7 (Low): I do it myself, note it in the ticket.
8 to 10 (Medium): Peer review, change window, rollback written down.
11 to 13 (High): Lab test first, rollback tested, second engineer on the call.
14 to 15 (Critical): Everything above, plus business sign-off, vendor or carrier on standby, and a longer window than you think you need.

This is not fancy math. It is a way to have the same conversation every time, instead of arguing about feelings. It also gives leadership a simple number they can understand.

4. Validation Processes

Validation means proving the system is healthy, with facts, not with hope.

A change is not done because the command finished or the server rebooted. It is done when calls are working.

The rule I follow: capture the same checks before and after, then compare.

For a voice change, that usually means:

Phone and device registration counts
SIP trunk and route list status
Route pattern and translation behavior
Test calls, inbound and outbound
Agent login and state changes in Finesse
CUIC reports showing live data
Call recording working
CDR and CMR data still flowing
RTMT alarms and any unexpected service failures

Save the "before" to a file. Run the same checks after. Compare them. If the only difference is the thing you meant to change, the change is good. If something else moved, stop and look. Do not guess.

Two more things I always test:

Test what should work. The 800 number reaches the queue. Agents get calls.
Test what should NOT work. The old route is really gone. The blocked pattern is still blocked. Toll fraud paths are still closed. People forget this half, and it is where real problems hide.

5. Rollback Procedures

A rollback is the plan for going back. It is written before the change, not during the outage.

The worst place to write a rollback plan is in the middle of a failed cutover, with a manager standing behind you. So I write it while I am calm.

My rollback rules:

Back up first, and verify the backup. On CUCM that means a DRS backup that actually completed, not one that was scheduled and assumed. A backup you have never restored is a wish, not a plan.
Know your platform's undo path before you start. This is important, because rollback is not the same everywhere:
- CUCM upgrades: the version switch back to the inactive partition, if the upgrade left it usable.
- Virtual machines: a snapshot, if your support policy allows it, and taken with the system in a supported state.
- Config changes: the exported config, the previous route pattern, the previous trunk setting, written down exactly as it was.
- IOS and IOS-XE routers such as CUBE: configure replace from a saved config, or a scheduled reload as a safety net (see the note below).
- Nexus: NX-OS uses checkpoint and rollback, which is a different mechanism. Take a checkpoint before the change and roll back to it if needed.
The point is simple: rollback commands depend on the platform. Do not assume the method you use on a CUBE will work on a Nexus switch or a CUCM node. Check the guide for that platform before the window, not during it.
Set a time limit. If the change is not validated within a set time, roll back. I do not "just try one more thing." That is how a 30-minute problem turns into a five-hour outage and an angry morning.
Test the rollback. If it has never been run, it is not a plan.

Note on reload in: On supported IOS and IOS-XE devices, a scheduled reload is a useful safety net for a remote change. If you lock yourself out, the device reloads into the last saved config and you get access back. You cancel it with reload cancel once the change is validated, and you save only after that. Use it only if you understand how it behaves on your platform and software version, because behavior and known bugs vary by release. Check the command reference for your version first. This also does not apply to CUCM, UCCE, or CVP servers, which have their own recovery paths.

Example: Running a SIP Trunk Change from Start to Finish

Here is what the method looks like on a real change: updating a SIP trunk or a voice routing path.

1. Define the change. Which trunk or route? Which business group uses it? Which calls can be affected?

2. Review dependencies. Do we need firewall changes? Does the carrier need to do anything on their side? Does the contact center team need to test? Is a certificate involved?

3. Score the risk. If it touches inbound customer calls, it is not a small change. If rollback needs the carrier, the score goes up. If we cannot test until it is live, the score goes up again.

4. Run the pre-checks. Trunk status, route list behavior, recent call failures, alarms, registration counts. Save the output.

5. Implement during the window. Follow the written steps. Capture output as you go. Do not improvise.

6. Validate. Test calls in both directions. Check agent behavior if the contact center is in scope. Check recording. Check reports. Compare against the pre-checks.

7. Roll back if needed. If validation fails and the time limit is reached, go back. Do not wait and hope.

8. Document. What worked, what failed, what should be improved next time. Then update the template, so the next engineer starts one step ahead of where you started.

Verify

Use this section to check that the method is working, not just the change.

Ask after each project:

Could someone else repeat this from my notes alone?
Did I use a template, or did I copy an old email?
Did the pre-check and post-check output get saved anywhere?
Did anything surprise me? If yes, that surprise belongs in the checklist now.

A healthy sign: the checklist keeps getting a little longer, and the maintenance windows keep getting a little shorter.

Troubleshoot

Problem: The template does not fit this system. Do not bend the work to fit the template, and do not throw the template away. Add a section, or make a second template for that platform. Templates are meant to grow.

Problem: Nobody uses the checklist. It is probably too long. Cut it down to the steps that have actually burned the team before. A ten-item checklist people use beats a fifty-item checklist people skip.

Problem: The risk score feels wrong. That is fine. The score starts the conversation, it does not end it. If two engineers disagree, that disagreement is useful. Write down the reason and move on.

Problem: Rollback did not work. Then it was never tested, or it was the wrong method for that platform. Both are fixable. Test it next time, and write the platform-specific steps into the plan.

Related Information

The point of all this is not paperwork. It is this: I can hand my work to someone else, walk away, and the calls keep flowing.

One good deployment proves I got lucky. A method that works on the tenth deployment, on a platform I have not touched in a year, even when I am not there, proves something real.

Start with one template and one checklist. Add to them each time something bites you. In a year you will have a system, and you will not remember building it.

Cisco references worth reading:

Cisco. "Configuration Management: Best Practices White Paper" (Doc ID 15111) https://www.cisco.com/c/en/us/support/docs/availability/high-availability/15111-configmgmt.html
Cisco. "Upgrade Tasks," Upgrade and Migration Guide for Cisco Unified Communications Manager and the IM and Presence Service, Release 15 and SUs https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/15/cucm_b_upgrade-and-migration-guide_15/cucm_m_upgrade-tasks.html
Cisco. "Pre-Upgrade Tasks (Manual Process)," Upgrade and Migration Guide for Cisco Unified Communications Manager, Release 15 and SUs https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/15/cucm_b_upgrade-and-migration-guide_15/cucm_m_pre-upgrade-tasks-15.html
Cisco. "Restore the System," Administration Guide for Cisco Unified Communications Manager, Release 15 and SUs (Disaster Recovery System) https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/admin/15/adminGd/cucm_b_administration-guide-15/cucm_m_restore-the-system.html
Cisco. "Configuration Rollback Confirmed Change," System Management Configuration Guide, Cisco IOS XE 17.x https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/syst-mgmt/b-system-management/m_cm-config-rollback-confirmed-change.html
Cisco. "Configuring Rollback," Cisco Nexus 9000 Series NX-OS System Management Configuration Guide, Release 10.5(x) (checkpoint and rollback) https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/105x/config-guides/sys-mgmt/cisco-nexus-9000-series-nx-os-system-management-configuration-guide-release-105x/m-configuring-rollback-10x.html