IBM InfoSphere Information Server SystemOut.log file is rapidly filled with PooledConnection errors
Problem(Abstract)
In Information Server, when the XMETA database goes down, the SystemOut log file is rapidly filled with WebSphere Application Server error messages. In a High Availability environment, this could happen when database fail over takes place.
Symptom
The SystemOut log file is rapidly filled with WebSphere Application Server ResourceAllocationException error messages similar to the following:
PoolManager E J2CA0046E: Method executeTestConnectionTask caught an exception during creation of the ManagedConnection for resource jdbc/ASBDataSource, throwing ResourceAllocationException. Original exception: <=================================>Exception Message -> DSRA8100E: Unable to get a PooledConnection from the DataSource jdbc/ASBDataSource.
Cause
The problem occurs when the database is down because of the settings for two properties in the WebSphere Application server configuration for
Data sources > <Information Server Data source name> > WebSphere Application Server data source
1. "Validate existing pooled connections" is checked
2. "Retry interval" is set to 0.
It does not appear that this was the WebSphere Application server behavior in earlier releases.
Diagnosing the problem
If the SystemOut log file is growing rapidly, check whether it contains error messages as mentioned above, and whether the XMETA database is up and running.
Resolving the problem
1. Ensure that the XMETA database is up and running, and free of issues.
2. Change the value of the "Retry interval" to a non-zero value. The value selected depends on your environment; how quickly the database is recycled/failed over. A lower value will result in more of the failure messages being logged while the database is down. A higher value could imply that users may have to wait for the retry interval to be completed before Information Server is usable, even if the database came online before that delay.
Try a value of 30 seconds, check the system behavior and then decide whether a lower or higher value is more appropriate. This value needs to be set for every WebSphere data source where the "Validate existing pooled connections" is checked.