Thursday 30 January 2014

Prime Infrastructure: initHealthMonitor(): can not start DB

I recently had problem with Prime Infrastucture.

I found following output when do ncs start
ari-net-ncs/ncsadmin# ncs status
Health Monitor is running, with an error.
initHealthMonitor(): can not start DB
Ftp Server is Stopped
Database server is stopped
Tftp Server is Stopped
Matlab Server is Stopped
NMS Server is stopped.
CNS Gateway with port 11011 is down
CNS Gateway SSL with port 11012 is down
CNS Gateway with port 11013 is down
CNS Gateway SSL with port 11014 is down
Plug and Play Gateway Broker with port 61617 is down
Plug and Play Gateway config, image and resource are down on https
Plug and Play Gateway config, image and resource are down on http
Plug and Play Gateway is stopped.
SAM Daemon is not running ...
DA Daemon is not running ...
Syslog Daemon is not running ...
ari-net-ncs/ncsadmin# ncs stop verbose

Stopping Network Control System...

Stopping Tomcat...
Tomcat Stopped.

Network Control System successfully shutdown.


Plug and Play Gateway is being shut down..... Please wait!!!

Stop of Plug and Play Gateway Completed!!
SAM daemon process id does not exist
DA daemon process id does not exist
DA syslog daemon process id does not exist

ari-net-ncs/ncsadmin# ncs start verbose

Starting Network Control System...

Reporting Server Heap size = 3072m
XMP Server Heap size = 4096m
Starting Health Monitor
Starting Health Monitor as a primary
Checking for Port 8082 availability... OK
truststore used is /opt/CSCOlumos/conf/truststore
truststore used is /opt/CSCOlumos/conf/truststore
CERT MATCHED :
Updating web server configuration file ...
Starting Health Montior Web Server...
Health Monitor Web Server Started.
Starting Health Monitor Server...
Health Monitor Server Started.

Failure during Network Control System startup.  Check launchout.log for details.

ari-net-ncs/ncsadmin# ncs status
Health Monitor is running, with an error.
initHealthMonitor(): can not start DB
Ftp Server is Stopped
Database server is stopped
Tftp Server is Stopped
Matlab Server is Stopped
NMS Server is stopped.
CNS Gateway with port 11011 is down
CNS Gateway SSL with port 11012 is down
CNS Gateway with port 11013 is down
CNS Gateway SSL with port 11014 is down
Plug and Play Gateway Broker with port 61617 is down
Plug and Play Gateway config, image and resource are down on https
Plug and Play Gateway config, image and resource are down on http
Plug and Play Gateway is stopped.
SAM Daemon is not running ...
DA Daemon is not running ...
Syslog Daemon is not running ...
ari-net-ncs/ncsadmin#

I found following solution



1-I have checked the issue and observed that all the services of the primary are down .

2-Restarted the services many times without success .

3-Checked alert_wcs.log  under the following , and found that you have a full memory 100% :

/opt/oracle/base/diag/rdbms/wcs/wcs/trace/

4-Deleted Oracle archive , but I observed the following error :

"RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process"

So delete those archivelogs using "force" option as well.

RMAN > delete noprompt force archivelog all;

2 comments:

  1. Hello

    Did this solved your problem ?

    Same here

    Thanks in advance

    ReplyDelete
  2. RMAN> delete noprompt force archivelog all;

    using target database control file instead of recovery catalog
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of delete command at 12/15/2015 20:55:21
    RMAN-06171: not connected to target database

    RMAN> connect target /

    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    ORA-09817: Write to audit file failed.
    Linux-x86_64 Error: 28: No space left on device
    Additional information: 12
    ORA-09945: Unable to initialize the audit trail file
    Linux-x86_64 Error: 28: No space left on device


    Any suggestions? Can't connect to database to delete the files?


    ReplyDelete