Tech Shorts – WebLogic Server 12.2.1.1 – EM and Console Slow Response

We recently implemented a complete Oracle Fusion Middleware 12c (12.2.1.1) Stack for a client based on Oracle SOA Enterprise Deployment Guide (EDG). The design focus was to implement a highly available and scalable clustered environment which contained OSB, SOA, MFT, OWSM, ESS, BAM managed servers, and a highly available Admin Server. Each of the managed servers had their own dedicated VM with an active-passive Admin Server cluster. We performed extensive tuning and load testing to make sure the system can function under demand.

However, as we migrated to higher environments, the deployments screen would take a long time to render, even though all the deployed applications functioned as expected without any issue. It sometimes resulted in timeouts and would cause application deployments to fail. The overall performance of both the WebLogic Console and Enterprise Manager was sluggish, particularly affecting the deployment screen, with wait times of over 30 mins to render a single click action! If some of the servers were shutdown, the load time would improve slightly, but it was not a viable option to keep the server shut down. From the log files, everything appeared to be normal and there were no error messages.

Working together with Oracle Support, we noticed that in some of the thread dumps and Java Flight Recordings, a few of the managed beans took too long to respond and this was later identified as a known defect. In order to confirm that this was the issue, we disabled the new thread self-tuning functionality added in this release of WebLogic Server. In order to disable the new thread self-tuning functionality, we added the following JVM start-up parameter to all the WebLogic Servers and restarted.

-Dweblogic.UseEnhancedIncrementAdvisor=false   

After the restart, the WebLogic Console was significantly improved and the deployment screen would load in a few seconds, a great improvement from the 30 minutes before the thread tuning setting was applied.

This verified that their performance issue was due to the newly added functionality, and the temporary workaround was to disable it. At the time of writing this post, Oracle has released a patch that fixes this issue. The high-level steps are as follows:

1. Remove the parameter -Dweblogic.UseEnhancedIncrementAdvisor=false from all WebLogic Server startup
2. Apply Patch 23762529 for WLS 12.2.1.1.0 to all the servers in the domain
3. Restart all the servers and test by logging in to console and clicking on the Deployments tab
4. If it works, then apply Patch 24901211 for WLS 12.2.1 to all servers in the domain
5. Restart all the servers and test

This should resolve the performance issues caused by the new thread self-tuning functionality.