2010
02.10

Failed to shutdown DBConsole Gracefully

Category: IT / Tag: , , ,  / Add Comment
Unlike

Pretty average Oracle RAC Cluster (10g, two nodes). All of a sudden, trying to stop dbconsole results in an error:

[oracle@racnode1 log]$ emctl stop dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 10g Database Control ...
--- Failed to shutdown DBConsole Gracefully ---
 failed.

Similar behaviour when attempting to start it:

[oracle@racnode1 log]$ emctl start dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Agent Version     : 10.1.0.6.0
OMS Version       : 10.1.0.6.0
Protocol Version  : 10.1.0.2.0
Agent Home        : /opt/oracle/product/10.2.0/db_1/racnode1_DBSID1
Agent binaries    : /opt/oracle/product/10.2.0/db_1
Agent Process ID  : 24756
Parent Process ID : 24753
Agent URL         : https://racnode1:3938/emd/main
Started at        : 2010-02-09 13:48:34
Started by user   : oracle
Last Reload       : 2010-02-09 13:48:34
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :     3971
Size of XML files pending upload(MB)         :    50.11
Available disk space on upload filesystem    :    59.30%
Agent is already started. Will restart the agent
Stopping agent ... stopped.
Starting Oracle Enterprise Manager 10g Database Control ............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /opt/oracle/product/10.2.0/db_1/racnode1_DBSID1/sysman/log

Peeking into the emdctl.trc logfile, I found something that definitely smelled of expired certificates:

2010-02-09 13:54:14 Thread-4134193952 ERROR http: 6: Unable to initialize ssl connection with server, aborting connection attempt
2010-02-09 13:54:16 Thread-4133477152 WARN  http: snmehl_connect: connect failed to (racnode1:3938): Connection refused (error = 111)
2010-02-09 13:54:41 Thread-4134140704 ERROR ssl: nzos_Handshake failed, ret=29024

This thread on Oracle forums seems to confirm my suspicion.

And here’s what you do to fix the issue:

Ready your environment ($ORACLE_SID, $ORACLE_HOME, …). I “source” a script for that, each instance has its own.

[oracle@racnode1 ~]$ cat envDBSID.sh
export ORACLE_SID=DBSID1
export ORACLE_HOME=/opt/oracle/product/10.2.0/db_1
export PATH=/usr/local/bin:/bin:/usr/bin:/home/oracle/bin:"$ORACLE_HOME"/bin

[oracle@racnode1 ~]$ . envDBSID.sh

As I said, if the certificate is expired, dbconsole won’t shutdown cleanly. Fetch its pid and kill it manually.

[oracle@racnode1 ~]$ cat /opt/oracle/product/10.2.0/db_1/racnode1_DBSID1/emctl.pid  25608

[oracle@racnode1 ~]$ ps axo pid,command | grep 25608
25608 /opt/oracle/product/10.2.0/db_1/jdk/bin/java -server -Xmx256M -XX [..]

[oracle@racnode1 ~]$ kill 25608

[oracle@racnode1 ~]$ ps axo pid,command | grep 25608
[oracle@racnode1 ~]$

Run emctl secure dbconsole, it will generate the new certificates. Provide Oracle SYSMAN’s password and the hostname you’ll use (without domain name, in my case).
The URL displayed (the port number) will also tell you if you’re on the right instance and got the intended environment.

[oracle@racnode1 ~]$ emctl secure dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:5500/em/console/aboutApplication
Enter Enterprise Manager Root password :
Enter a Hostname for this OMS : racnode1

DBCONSOLE already stopped...   Done.
Agent is already stopped...   Done.
Securing dbconsole...   Started.
Checking Repository...   Done.
Checking Em Key...   Done.
Checking Repository for an existing Enterprise Manager Root Key...   Done.
Fetching Root Certificate from the Repository...   Done.
Updating HTTPS port in emoms.properties file...   Done.
Generating Java Keystore...   Done.
Securing OMS ...   Done.
Generating Oracle Wallet Password for Agent....   Done.
Generating wallet for Agent ...    Done.
Copying the wallet for agent use...    Done.
Storing agent key in repository...   Done.
Storing agent key for agent ...   Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode...   Done.
EMD_URL set in /opt/oracle/product/10.2.0/db_1/racnode1_RDS1/sysman/config/emd.properties
   Done.
Configuring Key store..   Done.
Securing dbconsole...   Sucessful.

Try and start dbconsole.

[oracle@racnode1 ~]$ emctl start dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:5500/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control ............................. started.
------------------------------------------------------------------
Logs are generated in directory /opt/oracle/product/10.2.0/db_1/racnode1_RDS1/sysman/log

[oracle@racnode1 ~]$ emctl status dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:5500/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /opt/oracle/product/10.2.0/db_1/racnode1_RDS1/sysman/log

Be prepared to handle the same situation in a few months. To see the new certificate expiry date, open any HTTPS URL served by dbconsole (i.e.: https://racnode1:5500/em/console/aboutApplication) and click on the lock icon your web browser should show somewhere.

3 comments so far

Add Your Comment
  1. Reply
    Pravs said: 2010.04.22 23:41

    Thanks for your blog..it helped me get my dbconsole working again…..I had applied a latest patch to my test database, but for some reason, when i tried to stop my dbconsole service, it did not shut down properly..didnt think too much at that time and applied the patches successfully..but was unable to restart the dbconsole and saw the trace file and stumbled onto your blog….!!!!

  2. Reply
    J Doherty said: 2010.04.25 20:12

    Thank you ! I had just spent 4 hours with Oracle Metalink.. I had EM backup jobs failing.. stuck waiting for output and assorted other problems… all linked back to this expiring certificate. Who ever chose 6 months for expiry date? That’s crazy. Thanks again. Your recipe fixed my problem completely.

  3. Reply
    Giuliano said: 2010.04.26 10:14

    Luckily this time, the trace file provided meaningful messages. That’s not always the case. :)
    Glad the post helped!

    ciao,
    –
    Giuliano

Your Comment