Rebuilding a Cisco Unity Connection Publisher
Rebuilding a Unity Connection Publisher
So, your Cisco Unity Connection publisher has crapped out. Now what do you do? Maybe you had some data corruption or some other issue. In my case the Cisco UCS had a bad drive causing data corruption. So now the Unity Connection publisher has failed. We are going to go through the process of rebuilding the publisher server and copy the data from the subscriber to the publisher to get it up and running again.
This rebuild will entail gathering all the data (such as hostname, ip, DNS, NTP), installing a new publisher with the same version ip, DNS, passwords, etc. Add the Unity Connection subscriber to the Publisher config. Finally copy the data from the subscriber to the publisher.
So let's get into it!
Gathering Cluster Data
First, we need to Gather the Cluster Data from the Subscriber here is the output that you will see from the commands "show network cluster" and "show version active"
admin:show network cluster
172.16.88.10 UCPUB.company.net UCPUB Publisher callmanager DBPub not authenticated - INITIATOR since Wed Dec 11 22:09:36 2019
172.16.88.11 UCSUB.company.net UCSUB Subscriber connection DBSub authenticated
admin:show version active
Active Master Version: 12.0.1.21900-10
Active Version Installed Software Options:
Stop Replication on Subscriber
Next, we need to stop replication on Subscriber. This is so that the subscriber will not try to connect to the new publisher until the data is copied.
admin:utils dbreplication stop
********************************************************************************************
This command will delete the marker file(s) so that automatic replication setup is stopped
It will also stop any replication setup currently executing
********************************************************************************************
Deleted the marker file, auto replication setup is stopped
Service Manager is running
Commanded Out of Service
A Cisco DB Replicator[NOTRUNNING]
Service Manager is running
A Cisco DB Replicator[STARTING]
A Cisco DB Replicator[STARTING]
A Cisco DB Replicator[STARTING]
A Cisco DB Replicator[STARTING]
A Cisco DB Replicator[STARTING]
A Cisco DB Replicator[STARTING]
A Cisco DB Replicator[STARTED]
Killed dblrpc process - 8678
Completed replication process cleanup
Please run the command 'utils dbreplication runtimestate' and make sure all nodes are
RPC reachable before a replication reset is executed
Install the new Unity Connection Publisher
Now we need to install the new CUC Publisher I have not included any screenshots or output for this as it is a very straight forward install. Just create a new VM from the OVA and use all of the same names, ip, DNS, passwords, etc. You must install the same exact version as the subscriber. You need the bootable ISO for that version.
Once the publisher is installed, we now need to login and Update the Processnode Value on Publisher. Meaning we need to add the subscriber hostname or ip address as if we are installing a new subscriber. In the Cisco Unity Connection Administration go to System Settings --> Cluster. Click Add New and put in the hostname or ip address of the subscriber server.
You can run this command to verify that it is there. Run sql select name,description,nodeid from processnode <- on publisher
Now reboot the publisher. <-- This is very important. If you do not reboot the data will not be correct and the copy cluster renegotiate will fail.
Verify Cluster Authentication
Next Verify the Cluster Authentication
admin:show network cluster
172.16.88.10 UCPUB.company.net UCPUB Publisher callmanager DBPub authenticated
172.16.88.10 UCSUB.company.net UCSUB Subscriber connection DBSub authenticated using TCP since Thu Dec 12 02:04:33 2019
Server Table (processnode) Entries
----------------------------------
UCPUB
UCSUB
Run the CUC Cluster Renegotiate
Here is where the real magic happens! In previous version we would have to rely on backups and restoring backups. No longer. This is a much cleaner approach. We will connect the Subscriber to the New Connection Cluster and replicate the data. Here are the commands and the output. This is run on the SUBSCRIBER server!
Utils cuc cluster renegotiate
admin:utils cuc cluster renegotiate
Publisher server: UCPUB
Subscriber server: UCPUB
Run this command only if you are replacing a
disabled publisher server.
WARNING: This operation will overwrite all data and messages on
the publisher server UCPUB with data from this subscriber server
and will establish clustering with the new publisher server.
This command may take require several hours to complete. The duration
depends on the network bandwidth between servers and on the
total size of the data on this subscriber server.
Are you sure you want to continue? (y/n) : y
19/12/12 02:11:44 Disabling data replication...
19/12/12 02:11:49 Renegotiating ssh trusts...
19/12/12 02:11:54 Synchronizing platform and LDAP database...
19/12/12 02:15:49 Creating any missing messaging databases on the publisher...
19/12/12 02:15:53 Adding subscriber node to publisher...
19/12/12 02:16:02 Synchronizing Unity Connection databases...
19/12/12 02:22:30 Resetting sequences on publisher...
19/12/12 02:22:31 Synchronizing file systems...
19/12/12 02:22:45 Synchronizing message files for mail store UnityMbxDb1...
19/12/12 02:23:50 Copying cluster DSCP configuration to publisher node...
19/12/12 02:23:53 Copying cluster CCMEncryption Keys to publisher node...
19/12/12 02:23:59 Rebooting publisher node UCPUB...
19/12/12 02:25:11 Cluster renegotiation completed.
If the product is Registered with Cisco Smart Software Manager then Reregister the product from Publisher node
Process completed.
Database Replication
As you can see from the output the publisher will automatically restart. After the reboot you will want to do a "show cuc cluster status" on both servers to verify that all is looking good. Then you will want to run "utils dbreplication runtimestate" on the publisher several times until the replication has completed.
admin:show cuc cluster status
Server Name Member ID Server State Internal State Reason
------------- --------- ------------ -------------- ------
UCPUB 0 Primary Pri Active Normal
UCSUB 1 Secondary Sec Active Normal
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 0 Dec 12 02:28:06
SERVERS
Server Peer ID State Status Queue Connection Changed
---------------------------------------------------------------------------
g_ciscounity_pub g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 0 Dec 12 02:28:06
g_ciscounity_sub1 g_ciscounity_pub 100 Active Connected 0 Dec 12 02:28:06
g_ciscounity_sub1 101 Active Local 0
STATE
Source ER Capture Network Apply
State State State State
---------------------------------------------------------------------------
g_ciscounity_pub Active Running Running Running
g_ciscounity_sub1 Active Running Running Running
admin:show cuc cluster status
Server Name Member ID Server State Internal State Reason
------------- --------- ------------ -------------- ------
UCPUB 0 Primary Pri Active Normal
UCSUB 1 Secondary Sec Active Normal
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_ciscounity_pub 100 Active Connected 0 Dec 12 02:28:06
g_ciscounity_sub1 101 Active Local 0
SERVERS
Server Peer ID State Status Queue Connection Changed
---------------------------------------------------------------------------
g_ciscounity_pub g_ciscounity_pub 100 Active Local 0
g_ciscounity_sub1 101 Active Connected 428 Dec 12 02:28:06
g_ciscounity_sub1 g_ciscounity_pub 100 Active Connected 0 Dec 12 02:28:06
g_ciscounity_sub1 101 Active Local 0
STATE
Source ER Capture Network Apply
State State State State
---------------------------------------------------------------------------
g_ciscounity_pub Active Running Running Running
g_ciscounity_sub1 Active Running Running Running
admin:utils dbreplication runtimestate
Server Time: Thu Dec 12 02:52:52 PST 2019
Cluster Replication State: N/A
DB Version: ccm12_0_1_21900_10
Repltimeout set to: 300s
PROCESS option set to: 1
Cluster Detailed View from UCPUB (2 Servers):
PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
UCPUB 172.16.88.10 0.019 Y/Y/Y 0 (g_2) (2) Setup Completed
UCSUB 172.16.88.11 0.311 Y/Y/Y 0 (g_3) (3) Out Of Sync
admin:utils dbreplication runtimestate
Server Time: Thu Dec 12 02:53:32 PST 2019
Cluster Replication State: Only available on the PUB
DB Version: ccm12_0_1_21900_10
Repltimeout: Only available on the PUB
PROCESS option: Only available on the PUB
Cluster Detailed View from UCSUB (2 Servers):
PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & DB Status
----------- ---------- ------ ------- ----- ----------- ------------------
UCPUB 172.16.88.10 0.480 Y/Y/Y 0 (g_2) (2) Setup Completed
UCSUB 172.16.88.11 0.022 Y/Y/Y 0 (g_3) (3) Setup Completed
admin:utils dbreplication runtimestate
Server Time: Thu Dec 12 03:00:55 PST 2019
Cluster Replication State: Errors or data mismatches were found in utils dbreplication status at: 2019-12-12-03-00
Use CLI to see detail: 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2019_12_12_02_59_19.out'
DB Version: ccm12_0_1_21900_10
Repltimeout set to: 300s
PROCESS option set to: 1
Cluster Detailed View from UCSUB (2 Servers):
PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
UCPUB 172.16.88.10 0.021 Y/Y/Y 0 (g_2) (2) Setup Completed
UCSUB 172.16.88.11 0.307 Y/Y/Y 0 (g_3) (3) Out Of Sync
I ran into a little issue that the database didn't come back into sync. So, I ran "utils dbreplication rebuild all" to fix it.
admin:utils dbreplication rebuild all
******************************************************************************************************
WARNING !!!
This command can only be run from the publisher.
This command will run a combination of the following CLI commands on specified nodes:
utils dbreplication stop
utils dbreplication dropadmindb or dropadmindbforce
and utils dbreplication reset
This command may also affect other nodes including the publisher.
This command can take a considerable amount of time.
******************************************************************************************************
Do you want to continue? (y/n):y
Start rebuilding replication...
The process can be terminated by "delete process 15321" on a separate cli session
The detail status can be checked by "file view activelog cm/trace/dbl/sdi/rebuild_2019_12_12_03_02_11.out"
Starting rebuild command...
Number of nodes to be rebuilt: 1
Stopping replication setup...
Removing server(s) from replication network...
Signaling subscriber(s) for replication setup...
Rebuild command completed successfully...
-->UCSUB
Rebuild Completed: 1 Failed: 0
Duration: 2.00 minute
admin:utils dbreplication runtimestate
Server Time: Thu Dec 12 03:12:25 PST 2019
Cluster Replication State: BROADCAST SYNC ended at: 2019-12-12-03-09
Sync Result: SYNC COMPLETED on 718 tables out of 718
Sync Status: All Tables are in sync
Use CLI to see detail: 'file view activelog cm/trace/dbl/20191212_030800_dbl_repl_output_Broadcast.log'
DB Version: ccm12_0_1_21900_10
Repltimeout set to: 300s
PROCESS option set to: 1
Cluster Detailed View from UCPUB (2 Servers):
PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
UCPUB 172.16.88.10 0.020 Y/Y/Y 0 (g_2) (2) Setup Completed
UCSUB 172.16.88.11 0.304 Y/Y/Y 0 (g_3) (2) Setup Completed
After replication is complete it would be a great idea to reboot the cluster to make sure all is back in order. I hope this little guide is helpful. Thanks for reading.
Check out my YouTube Channel
https://www.youtube.com/channel/UCa2exyciw-M0BugrzCivodg
Do we have to register again using the license?
ReplyDelete