cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5739
Views
25
Helpful
2
Comments
Ayodeji Okanlawon
VIP Alumni
VIP Alumni

     

     

    Overview

    When IPT first came to planet earth, the gods told us that we must ensure that everything is deployed using IP. They told us that DNS is unreliable, it doesn’t work well with IPT solution etc.

    Fast forward a few years and now the gods are forced to eat their words, because for some reasons and due to the demands of aliens on our planet, IPT now needs DNS.

    So what if you listened to the gods a few years back and due to the demands of your IPT inhabitants, you need to migrate your CUCM servers from IP based to FQDN. How do you go about achieving this goal without disrupting your services.

    I found myself in this dilemma recently and due to the lack of documentation on the exact approach to use for this, I have decided to document the process and highlights the pitfalls you are likely to face along the way and how to mitigate against such risks.

    Enterprise collaboration is no longer phone/device centric, it has evolved into a user centric solution and hence application centric.

    The challenge with this is that applications such as Cisco Jabber now rely heavily on network infrastructure such as PKI (certficates) and obviously DNS.

    Today you will find your self in one of two positions. Your current infrastructure needs to be migrated to a DNS reliant one or you are deploying a new solution and must be DNS enabled.

    Dependencies

    Before you embark on a project like this, you need to consider under lying elements that will be impacted. You need to ask yourself what the dependencies are. In my environment there were two major ones

    • IP Phones

    Your IP Phones have been using IP address to communicate with CUCM. Now you are going to change this behavior, hence DNS is absolutely crucial. You need to ensure all your phones have DNS. For this to be successful you need to know the DHCP source for all your devices.

    Once you have identified the DHCP source, you need to verify that the DHCP scope has DNS configured. This is critical. If you change your CUCM host name to FQDN without DNS in your IP phone's DHCP scope, then you run the risk of loosing so many phones during this change.

    Some devices may have their DHCP scope on gateways while others have DHCP on windows DNS server. This is critical.

    Dont assume your IP phones all have common DHCP source. Do your due diligence and find out what the DHCP source is especially in a multi site deployment.

    If you need to add DNS servers to your DHCP scope on Cisco IOS router here is the command to do it

    conf t

    ip dhcp pool POOLNAME
    dns-server 192.168.1.100 192.168.1.101

     

    • MGCP gateways

    MGCP gateways: This was a hidden one and could cause a lot of grief. Check your mgcp gateways and ensure that they are enabled for DNS.
            
    The reason for this is that some mgcp gateways are configured with ccm-manager config, in which case these devices will automatically download their config from cucm. Once the system>server name has been changed, the mgcp gateway will try and connect to the FQDN of the CUCM and without a valid DNS and domain lookup configured it will fail.

    To configure DNS on your mgcp gateway do the following:

    ccm-manager fallback-mgcp
    ccm-manager redundant-host UK-MA-SCMS03.net.live.net
    ccm-manager mgcp
    ccm-manager music-on-hold
    ccm-manager config server 192.168.20.15 192.168.20.16
    ccm-manager config


    Example below:

    WRS1#sh ccm-manager
    MGCP Domain Name: WRSW1
    Priority Status Host
    ============================================================
    Primary Domain Name Unresolvable UK-HM-SCMS04.net.live.net
    First Backup Domain Name Unresolvable UK-HA-SCMS04.net.live.net
    Second Backup None

    Once DNS is enabled you should see the following:
    Priority Status Host
    ============================================================
    Primary Registered UK-HM-SCMS04.net.live.net (192.168.20.14)
    First Backup Down UK-HA-SCMS04.net.live.net (192.168.20.44)

    Actual Change

    You are not out of the woods yet. The fact that you have done all your due diligence and everything looks good doesn't mean nothing van go wrong. A lot can still go wrong. So don't just run off and make the change. Read on....

    • CUCM configuration>system > server

    Ensure that you enter the correct FQDN of the cucm servers as defined in DNS. This is crucial. This could make a long night longer for you. So pay attention here. Its critical that you enter the correct FQDN in the CUCM configuration page.

    In my change, CUCM host name was changed from IP to only the DNS name and not the FQDN as showed in the image below

     

    The implication of this was huge

    NB: the FQDN of the servers were already added through the CUCM cli platform page, so the TVS , tftp certs contain the FQDN.

    The most critical issue was that the phones could not connect to their TVS server to verify their new configuration file

    Here is the process in detail:

    Once the phones downloaded the new configuration, in essence the phones download a new ITL file, Once ITL file is successfully received, phone then request a signed config file.
        
    Next phone must verify ITL before it can trust the configuration file.
        
    If the phone already has ITL it must verify the downloaded ITL files match the signature in the ITL /CTL or  connect to the TVS server if the ITL signature doesn’t match.

    The trace below details the process.

    ####

    866: NOT 15:17:58.550372 TFTP: [8]:Requesting ITLSEP0CD9969170BB.tlv from 10.105.40.15
         867: NOT 15:17:58.776273 TFTP: [8]:Finished --> rcvd 13384 bytes
    ####


    Next thing is the phone must verify the ITL file since it already has ITL file installed..Here since the phone is unable to verify the signer credentials from the existing ITL, It will try to contact its TVS server

    ####
    872: ERR 15:17:58.789752 SECD: CTL_GetFileSigner: Error: Failed to find signer credentials
    873: ERR 15:17:58.790558 SECD: TL_ValidateSignedTL: Error: Unable to retrieve signer info from TL
    ####

     

    Since the phone cant verify the new ITL from the existing it tries to connect to the TVS server but it goes via the TVS proxy  to know which TVS server it should connect to..the TVS proxy is the tftp server

    ####

    874: NOT 15:17:58.792914 SECD: setupSocketToTvsProxy: TVS client sock fd 9 bound to </tmp/secClntTvs_92_5878>
         875: NOT 15:17:58.794048 SECD: setupSocketToTvsProxy: Connected to TVS proxy server
        --
        875: NOT 15:17:58.794048 SECD: setupSocketToTvsProxy: Connected to TVS proxy server
         876: NOT 15:17:58.795033 SECD: tvsReqQueryCertificate: Sent Request to TVS proxy, len: 2684
         877: NOT 15:17:58.795648 SECD: tvsReqQueryCertificate: Waiting for response from TVS Proxy
         878: NOT 15:17:58.797387 SECD: clpTvsInit: Client message received on TVS proxy socket
         879: NOT 15:17:58.798363 SECD: processTvsClntReq: Success reading the client TVS request, len : 2684
         880: NOT 15:17:58.799056 SECD: processTvsClntReq: TVS Query Certificate request
         881: NOT 15:17:58.799670 SECD: processTvsClntReq: signer : CN=LELHM-SCMT01.net.live.net;OU=I.T;O=LAB;L=labham;ST=London;C=GB
         882: NOT 15:17:58.801202 SECD: processTvsClntReq: issuer : CN=LELHM-SCMT01.net.live.net;OU=I.T;O=LAB;L=labham;ST=London;C=GB
        

    #####


    Next TVS proxy informs the phone which TVS it should connect to..it should be first server in its cucm group.
        

    88: NOT 15:17:58.808350 SECD: getTvsSrvrSock: Expecting ip address, got hostname: SCMS04
     

    #### Here we see that the phone is unable to resolve the TVS server hostname###
     

       896: ERR 15:17:58.910323 SECD: EROR:getTvsSrvrSock: Failed to resolve hostname: SCMS04
     

     There phone says I cant verify my configuration file..and hence no registration..


        981: ERR 15:18:00.179985 SECD: EROR:getTvsSrvrSock: secReq_getTvsServer succeeded but got an empty ip address for index : 2
         982: NOT 15:18:00.181489 SECD: sendErrRespToClient: Sending the failed response to all TVS client and cleaning up
         983: NOT 15:18:00.182991 SECD: tvsReqQueryCertificate: Received the response from TVS proxy, status: 1
         984: NOT 15:18:00.183579 SECD: tvsReqQueryCertificate: The cert len received is 0
         985: NOT 15:18:00.184958 SECD: Failed validation using TVS, 0============>
         986: ERR 15:18:00.185784 SECD: EROR:verifyFile: sgn verify file failed </usr/ram/SEP0CD9969170BB.cnf.xml>, errclass 8, errcode 19 (signer not in CTL)
         987: ERR 15:18:00.186425 SECD: EROR:verifyFile: verify FAILED, </usr/ram/SEP0CD9969170BB.cnf.xml


    It is important to note that this step is really crucial. Several phones could be affected especially if they are generation two phones that have SBD enabled. In some cases you might need to delete the ITL files on the phones to get them back, so be careful here

    Database Replication

    This change will impact database replication. In some cases you could end up with corrupt cdr server list and as such you may need to rebuild the cdr server lists for the whole cluster.

    If you are not very confident on resolving database replication issue, then I suggest you open a pro active TAC case before you begin this. This was the most challenging part of the whole job. Depending on the size of your cluster, this could take a long time. I was working on a huge cluster with 13 servers, so it took me more than 7 hours to get dbreplication going again. Be prepared to battle this out!!!

    Do not just attempt to reset dbreplication if things are broken, this could just prolong your pain. If you have a corrupt cdr server list, then reset dbreplication wont help you. So make sure you know what the issue is and then you can use the right dbreplication command to address it.

     

    .

    Cluster Reboot

    Finally there is no need to reboot any server after this change.

    Comments

    Great as usual my friend!!

    +5 all day long 

    Carlo

    devnetjmt
    Level 1
    Level 1

     Awesome document! Thanks for sharing!!

    +5

    Getting Started

    Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: