Received a call regarding a huge problem at one of my customers. Their Portal is behaving strange, performance is poor and after a reboot it is not starting at all. They tried several scenario’s themselves but nothing helped.
I brought the entire portal down and removed all logs en temp folder-content from WebSphere wp_profile in order to get a clean start, clean WP cache and clean logs. After doing that, i restarted the DB2 server and ran the stats and reorg scripts on the Portal databases just to be sure. Then started the Portal services.
No luck at all. Instead of being in-stable and running in a low perfomance mode :-), it didn’t start at all without any SystemOut.log lines indicating that there is something wrong. What the #$%##!@ is happening?
Deployment manager started perfectly, nodeagent started perfectly. No errors at all so far. Started Portal…..just log entries came up showing that the Portal will bind to several apps. And after that…… JVM crashed without any error, trace or debug showing what is wrong. Even starting Portal with TRACE on, nothing shows up in the logs.
Finally found this one at IBM KB’s as a possible cause of failing Portal to start:
When the java class cache is corrupted
It is possible that the Java cache is corrupted. A class cache is an area of shared memory of a fixed size that persists beyond the lifetime of any JVM that is using it. A JVM does not own the cache and there is no master and slave JVM concept; instead, any number of JVMs can read and write to the cache concurrently. A cache is deleted either when it is explicitly destroyed using a JVM utility or when the operating system restarts. A cache cannot persist beyond an operating system restart. Its purpose is to reduce the virtual memory footprint and improve JVM start up time. By default, this option is enabled starting with SDK 1.5 on all IBM platforms.
Run the WAS_HOME\profiles\profile_name\bin>clearClassCache.bat/sh command to clear up the Java cache of this WebSphere Application Server node from the common location of Java cache on the system level. Alternatively, you can also delete the content of the following directories:
- On UNIX-based platforms: /tmp/javasharedresources
- On Windows: C:\Documents and Settings\<userid>\Local Settings\Application Data\ javasharedresources
Tested it, but this file wasn’t removed after a reboot.
And YESS !!! When i removed this file, and restarted the server….everything was up and running again.
Now…..what’s causing this corruption of this file? Looking at the last modified timestamp of the corrupted file, it was exactly the same time as the latest Windows Update. 🙁 We don’t know for sure, but it’s too coincidental to let is pass by. We will keep an eye on it at the next Windows Update.
Visits: 255