WebLogic Feature Pack: Troubleshooting

Intended for use with WebLogic Feature Pack V2.3.

The following material outlines problems you may encounter when running the WebLogic Feature Pack in your Cassatt Active Response environment, along with the steps to solve those problems.

Many of these procedures require that you know how to use BEA WebLogic 8.1 and that you understand how your WebLogic domains are configured, including settings like log file locations, paths, and ports. For questions about BEA WebLogic, and how to find configuration information using the Administration Console, refer to the BEA product documentation.

WebLogic Feature Pack cannot contact the node manager on a Managed Server

Description

Cassatt Active Response issues an alert listing the Managed Server IP address, listen port, and WebLogic Feature Pack service name.

WebLogic Feature Pack contacts node managers to ensure the Managed Servers are available for service deployments.

Resolution

1. Log into the Managed Server node as the user configured for the WebLogic Feature Pack service.
2. Determine whether the node manager is running:
   
   service nodemgr status
3. If the node manager is not running, start the node manager:
   
   service nodemgr start
   
4. If the service nodemgr status command indicates the node manager is running, check the status in the WebLogic Administration Console for a domain that uses that node manager:
   1. Select a machine that is online and click the monitoring tab.
   2. Look for the STATE attribute.
5. If the Administration Console indicates that the node manager is running, try your operation again; you may have encountered a timing issue.
6. Check the node manager logs for errors, which might indicate the source of the problem. Refer to the BEA product documentation for information about log errors.
7. If the problem persists, contact support@cassatt.com.

WebLogic Feature Pack cannot contact the WebLogic Administration Server

Description

Cassatt Active Response issues an alert listing the Administration Server IP address, listen port, and WebLogic Feature Pack service name.

Resolution

1. Open a browser and launch the WebLogic console by going to http:/IP_address:port/console, using the IP address and the listen port of the Administration Server node. If the console connects successfully, skip to step 3.
2. Log into the Administration Server node as the user configured for the service. Check to see whether the Administration Server for the WebLogic Feature Pack service is running using your favorite command, for example:
   
   ps -ef | grep startWebLogic.sh
   
   Search through the output for the directory name associated with the WebLogic Feature Pack service.
   

   If the service is not listed, skip to step 5.

3. If the service is running as indicated by step 1 or step 2, do one or more of the following:
   
   1. Check for network problems.
   2. Stop the Administration Server using the Administration Console.
   3. If the domain has a stopWeblogic.sh in the domain directory:
      
      Run the stopWeblogic.sh
      
      Make sure the Administration Server for the domain is no longer running:
      
      ps -ef | grep startWebLogic.sh
      
      If it has stopped, continue with step 3.  
   4. If there is no stopWebLogic.sh, or if running the script does not stop the Administration Server for the domain, use this command:

      ps -ef | grep startWebLogic.sh

      Identify the process ID associated with the startWeblogic.sh process for the domain and run this command:

      ps -ef | grep <process_id>

      This command returns the startWeblogic.sh process and a java process that was started with that script. Using those two process IDs, run this command to end the processes:

      kill <process_id for startWeblogic.sh> <process_id for the java process>

      Check to make sure that both processes were stopped, as follows:

      ps -ef | grep <process_id for startWeblogic.sh>
      ps -ef | grep <process_id for the java process>

4. Restart the Administration Server using this command, substituting your domain path if it differs from this default:
   
   nohup /home/bea/user_projects/domains/your_domain/startWebLogic.sh > /home/bea/user-projects/domains/your_domain/beaadmsvr-start.log 2>&1&

Cassatt Active Response cannot locate nodes that meet the requirements for the service

Description

Cassatt Active Response issues a resource constrained alert. Generally this means that WebLogic Feature Pack cannot start a service on any node in the service tier. Resources may be constrained because the tier is at its maximum nodes—or no additional nodes are available to be allocated to the tier—and all nodes in the tier are already running services; consequently no node meets the deployment constraints for the service.

Resolution

Resources may become available as the load on other services decreases and those services are undeployed. If constrained resources continue or occur at an unacceptable frequency, do one or more of the following to make additional resources available, depending on your situation:

If you received the alert in conjunction with a monitoring alert, see WebLogic Feature Pack is unable to collect monitoring data.

WebLogic Feature Pack is unable to collect monitoring data

Description

Cassatt Active Response issues an alert listing the IP address of the node that cannot be monitored.

Resolution

Verify that SNMP (or your monitoring source) is properly configured and reports the values configured for this service. Refer to the applicable Cassatt Active Response operating system installation instructions.

Insufficient NFS daemons

Description

Some node managers fail to start when a WebLogic Feature Pack service is initially brought up on the Managed Servers.

Resolution

If you are using SAN or dual-ported disk for Cassatt Active Response data storage, make sure you have enough NFS daemons running on the control node. Insufficient NFS daemons are likely the problem if you see text like the following in /tmp/nodemgr815811out (or the file that represents your service pack):

...
<Jan 10, 2006 10:28:55 AM PST> <Error> <NodeManager> <Failed to re-write the file nodemanager.properties because of exception
weblogic.security.internal.FileUtilsException: [Security:090195] Couldn't rename /home/bea/bea811/weblogic81/common/nodemanager/ nodemanager.properties41631.new
to /home/bea/bea811/weblogic81/common/nodemanager/nodemanager. properties>
+ set +x

See "Modifying NFS Daemons" in Control Node: Installing Red Hat ELAS .

WebLogic Feature Pack services become confused; WebLogic Feature Pack is unable to stop a service

Description

WebLogic Feature Pack minimum services is set to 0, but the WebLogic Feature Pack UI shows services are still running, or you are removing a service (using the ccserviceremove command; see Removing Services) and the removal hangs with the message "Waiting for service_name to stop," which repeats every 10 seconds indefinitely.

Resolution

Try one or more of the following in order from least intrusive to most intrusive.

Start the WebLogic Administration Server and use it to stop the managed servers as follows:

1. Check to see whether the Administration Server is running:
   
   ps -ef | grep startWebLogic.sh
   
2. If it is not running, start it as described in WebLogic Feature Pack cannot contact the WebLogic Administration Server.
   
   Tail the beaadmsver.log to make sure the Administration Server starts. If it doesn't, adjust any erroneous paths or ports.
3. Launch the WebLogic Administration Console and stop any deployed servers by selecting each managed server and choosing Control > Start/Stop > Stop this server...

Terminate processes, as follows:

1. Log in to the Administration Server node as the user configured for the service and terminate all of the processes related to the domain with the problem. Wait several minutes to see if the service has stopped.
2. Log in to each Managed Server node that is currently up as the user configured for the service, and terminate all processes related to the Managed Server being used by the domain. Note that halting Managed Server processes affects all other domains using that same Managed Server. Wait several minutes to see if the service has stopped.

Edit the service matrix, as follows:

1. Log into the control node(s) as a user with administrative privileges and stop the Cassatt Active Response service using the appropriate command(s):
   
   If using a single control node:
   /sbin/service collage-core stop

If using dual control nodes:

Use the clustat command to determine which control node is the active one: /usr/sbin/clustat

On the standby control node, run this command:
/sbin/service clumanager stop

On the active control node, run this command:
/sbin/service clumanager stop

2. Edit /cassatt/dde/servicematrix.xml as follows:

   Find the entry for the domain, and change any nodes marked as "STARTED" to "STOPPED."
   
   Change the minimum service count to 0.

3. Restart the Cassatt Active Response service using the appropriate command:
   
   If using dual control nodes:

start the clumanager on the active control node:
/sbin/service clumanager start

After clumanager starts on the active control node, use the same command on the second control node.

If using a single control node:
/sbin/service collage-core start

4. Log in to the Administration Server and Managed Server nodes as the user configured for the service and make sure that all processes related to the domain have been stopped (see above).

Once the services are stopped, you can continue by setting the service minimum to greater than 0 to restart the service from scratch (see WebLogic Feature Pack: Image and Service Management), or remove the service.

Invalid listen address for Administration Server

Description

During service configuration, the ccserviceconfigure command outputs a validation error similar to the following:

Admin address 127.0.0.1 does not match IP_address

Where IP_address is the IP address of the node in the Administration Server tier.

Resolution

This error happens when the Administration Server listen address in the WebLogic domain configuration is null, and is erroneously replaced with the loopback address in the error message. Add a listen address to the Administration Server in the WebLogic domain configuration.

JMS server fails to start

Description

Application failures are accompanied by a JMS server failing to start.

Resolution

Check the log file on the Managed Server for errors that are similar to the following:

####<Aug 17, 2006 8:32:18 PM MDT> <Error> <JMS> <collage0029.cassattmsp> <MedRec Server_3> <main> <<WLS Kernel>> <> <BEA-040368> <The following exception has occurred:

weblogic.jms.common.JMSException: Error binding destination to JNDI (jndi name = jms/XML_UPLOAD_MDB_QUEUE) at weblogic.jms.backend.BEDestination.advertiseDestinationImpl (BEDestination.java:903)

<more info>

Caused by: javax.naming.NamingException:
java.security.PrivilegedActionException
: javax.naming.NameAlreadyBoundException: XML_UPLOAD_MDB_QUEUE is already bound; remaining name 'jms'at weblogic.jms.common.JMSServerUtilities.bindAsSU(JMSServerUtilities.java:80) at weblogic.jms.backend.BEDestination.advertiseDestinationImpl (BEDestination.java:894)

JNDI binding errors like these could indicate a conflict between domains that use the same cluster multicast listen address and listen port. Check the domain's multicast values; multicast values must be unique to each domain.

Related Articles

WebLogic Feature Pack: Setup
WebLogic Feature Pack: Image and Service Management

top

Was this article useful? Tell us what you think.
Email infocentral@cassatt.com.