Prime Infrastructure: initHealthMonitor(): can not start DB

I recently had problem with Prime Infrastucture.

I found following output when do ncs start
ari-net-ncs/ncsadmin# ncs status
Health Monitor is running, with an error.
initHealthMonitor(): can not start DB
Ftp Server is Stopped
Database server is stopped
Tftp Server is Stopped
Matlab Server is Stopped
NMS Server is stopped.
CNS Gateway with port 11011 is down
CNS Gateway SSL with port 11012 is down
CNS Gateway with port 11013 is down
CNS Gateway SSL with port 11014 is down
Plug and Play Gateway Broker with port 61617 is down
Plug and Play Gateway config, image and resource are down on https
Plug and Play Gateway config, image and resource are down on http
Plug and Play Gateway is stopped.
SAM Daemon is not running ...
DA Daemon is not running ...
Syslog Daemon is not running ...
ari-net-ncs/ncsadmin# ncs stop verbose

Stopping Network Control System...

Stopping Tomcat...
Tomcat Stopped.

Network Control System successfully shutdown.


Plug and Play Gateway is being shut down..... Please wait!!!

Stop of Plug and Play Gateway Completed!!
SAM daemon process id does not exist
DA daemon process id does not exist
DA syslog daemon process id does not exist

ari-net-ncs/ncsadmin# ncs start verbose

Starting Network Control System...

Reporting Server Heap size = 3072m
XMP Server Heap size = 4096m
Starting Health Monitor
Starting Health Monitor as a primary
Checking for Port 8082 availability... OK
truststore used is /opt/CSCOlumos/conf/truststore
truststore used is /opt/CSCOlumos/conf/truststore
CERT MATCHED :
Updating web server configuration file ...
Starting Health Montior Web Server...
Health Monitor Web Server Started.
Starting Health Monitor Server...
Health Monitor Server Started.

Failure during Network Control System startup.  Check launchout.log for details.

ari-net-ncs/ncsadmin# ncs status
Health Monitor is running, with an error.
initHealthMonitor(): can not start DB
Ftp Server is Stopped
Database server is stopped
Tftp Server is Stopped
Matlab Server is Stopped
NMS Server is stopped.
CNS Gateway with port 11011 is down
CNS Gateway SSL with port 11012 is down
CNS Gateway with port 11013 is down
CNS Gateway SSL with port 11014 is down
Plug and Play Gateway Broker with port 61617 is down
Plug and Play Gateway config, image and resource are down on https
Plug and Play Gateway config, image and resource are down on http
Plug and Play Gateway is stopped.
SAM Daemon is not running ...
DA Daemon is not running ...
Syslog Daemon is not running ...
ari-net-ncs/ncsadmin#

I found following solution

1-I have checked the issue and observed that all the services of the primary are down .

2-Restarted the services many times without success .

3-Checked alert_wcs.log  under the following , and found that you have a full memory 100% :

/opt/oracle/base/diag/rdbms/wcs/wcs/trace/

4-Deleted Oracle archive , but I observed the following error :

"RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process"

So delete those archivelogs using "force" option as well.

RMAN > delete noprompt force archivelog all;

Prime Infrastucture How to add disk space

https://supportforums.cisco.com/docs/DOC-35173 

At times, it may be necessary to add more resources to the virtual machine Prime Infrastructure runs in.  Following are steps for adding a second virtual hard disk to Prime Infrastructure.

NOTE:  Making the existing virtual hard disk larger is not supported;  only adding an additional drive (or drives) is supported.

NOTE:  Always start any maintenance activity by knowing where the most recent backup is, or creating one. 

Step 1.  Shutdown Prime Infrastructure

     SSH to the Prime Infrastructure command line interface as the admin or other administrative user

     primeinfrastructure/admin# ncs stop

Step 2.  Shutdown the Virtual Appliance

     At the Prime Infrastructure command line interface

     primeinfrastructure/admin# halt

Step 3.  Add new virtual hard drive

     a.  Right click the virtual appliance in the VSphere Client







      b.  Select "Edit Settings..."

     c.  Select "Add..."

   
d.  Select Hard Disk, then Next

     e.  Select "Create a new virtual disk," then Next
  
 f.  Enter desired size and location for new virtual disk, then Next

     g.  Select Next

     h.  Select Finish

Step 4.  Power on the virtual appliance

     a. Right click the virtual appliance in the VSphere Client, select Power, then Power On.

 

 

 

 

 

 

 

 

 

 

 

 

Step 5. Login to Prime Infrastructure and confirm the additional disk space

     In the Prime Infrastructure GUI, go to Administration > Appliance to see the additional space

Procedure for upgrading HWIC firmware

Below is the procedure for upgrading firmware.

1. Download the file and rename the file "adsl_alc_XXXXX.bin".
2. Copy the firmware file to the flash:
3. For the new firmware to take effect, reboot the device.
4. The "show dsl interface atm 0" output gives the details of the firmware file in use

Note:

- The firmware file should not be renamed when being copied to the flash , Since the router looks for the presence of file named "adsl_alc_20190.bin" to initiate the flash upgrade process .

- If IOS is now able to find a valid firmware in the flash then it would always use that rather than the firmware embedded in IOS .If users want to use the embedded firmware, they have to ensure that there is no valid firmware file in flash.

EEM script for High CPU

This EEM won’t cause any risk on the switch , it will be automatically detected that HIGH CPU and it will storage some information that will help me out to analyze the cause of the

This is the EEM script:

 

event manager applet High_CPU

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt

entry-val 80 exit-op lt exit-val 75 poll-interval 1

action 0.0 syslog msg "High CPU DETECTED."

action 0.1 cli command "enable"

action 0.2 cli command "show process cpu sorted | append

NVRAM:high_cpu1.txt"

action 0.3 cli command " Show processes memory sorted | append

NVRAM:high_cpu2.txt"

action 0.4 cli command " Show process cpu history | append

NVRAM:high_cpu3.txt"

Cisco Switch High CPU

Here is some information which you will find very useful while troubleshooting Cisco switch high cpu.


show process cpu sorted

 http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html