XMeta

-
All platform users

 This document describes problem scenarios that can occur when using XMeta. For each scenario, the typical characteristics or symptoms of the scenario are described and a list of Must Gather information is provided.
Scenario: Database Deadlock
A database deadlock occurs when two (or more) database transactions each hold a resource that the other is attempting to acquire. Database deadlocks are automatically detected by the database and at least one of the transactions involved in the deadlock is terminated. The JDBC driver returns an exception in response to the termination. XMeta detects this exception and returns a DatabaseDeadlockException to the application.
It is important to recognize that in any non-trivial, multi-user database-based system, deadlocks will occur. While XMeta makes every effort to reduce the likelihood of deadlocks, there are certain scenarios where deadlocks cannot be avoided.
Characteristics
The most obvious symptom of a database deadlock is that XMeta will throw a DatabaseDeadlockException. This exception is logged in SystemOut.log or SystemErr.log. A message from DB2 is similar to:
Caused by: com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -911, SQLSTATE: 40001, SQLERRMC: 2
The database deadlock is recognized by the SQLCode: -911 and SQLERRMC: 2. If the SQLERRMC is a value other than 2, then this is not a true deadlock. It could be a lock timeout (SQLERRMC: 68 - see the following section).
In addition to the messages in the log file, the client application will report a failing operation. Some clients will display the error to the user (for example, DataStage Designer), some clients will have automatic retry logic (for example, IA), and some clients will show the same exception to their clients (for example, ISF services).
Files and Data to collect
  1. All WebSphere Application Server logs
  2. Database configuration settings
  3. DB2 diagnostic log
  4. Database deadlock detail
  5. Database statement snapshot
  6. Client logs for any client who sees a deadlock error.

Scenario: Database Lock Timeout
A database lock timeout occurs when a transaction is waiting to acquire a lock on a resource and another transaction is holding the lock. If the waiting transaction waits for too long, the database will abort the transaction. Some databases (for example, DB2) report this situation using the same error codes as a deadlock. As a result, XMeta interprets this as a deadlock and returns a DatabaseDeadlockException.
Characteristics
The database lock timeout symptoms are similar to the deadlock symptoms. The only significant difference from the deadlock scenario is that the reason code SQLERRMC is 68 (instead of 2):
Caused by: com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -911, SQLSTATE: 40001, SQLERRMC: 68
Since XMeta throws a DatabaseDeadlockException, the client responses to a lock timeout are the same as a deadlock (see the previous section).
Files and Data to collect
  1. All WebSphere Application Server logs
  2. Database configuration settings
  3. DB2 diagnostic log
  4. Database statement snapshot
  5. If the lock timeouts are a consistent problem, also capture the following information:
    1. A series of database lock snapshots taken at semi-regular intervals when the lock timeouts occur.
    2. A database application snapshot for any applications that are shown in the UOW Waiting state in the lock snapshots.
  6. Client logs for any client who sees a deadlock error.

Scenario: Lock -up
A lock-up is a situation where client applications freeze-up and the users are unable to do anything. This can be one client application or it can be every client application using the server.
Characteristics
The primary characteristic of this problem is a client application that stops responding. The application may show an hourglass. The application either never resumes or resumes only after a long period of time (for example, 1 - 2 hours).
Files and Data to collect
  1. All WebSphere Application Server logs
  2. Database configuration settings
  3. DB2 diagnostic log
  4. Database statement snapshot
  5. A series of database lock snapshots taken at semi-regular intervals when the lock timeouts occur.
  6. A database application snapshot for any applications that are shown in the UOW Waiting state in the lock snapshots.
  7. Client logs for any client that was active at the time of the lock up.

Collecting XMeta specific information
Files and Data to collect
Description
ASBServer\bin\imam_upgrade.logXMeta log file
ASBServer\bin\xmetaAdmin*.logXMeta log file
ASBServer\conf\database.propertiesXMeta properties file
ASBServer\conf\imam_staging_repository.propertiesXMeta properties file
DB2 Configuration and logs
Description
In a DB2 prompt, run the commands:
  1. get dbm cfg show detail
  2. get db cfg for <db-name> show detail
DB2 Configuration Settings
[DB2_HOME]\<DatabaseInstance>\db2diag.logDB2 diagnostic log: for example C:\IBM\SQLLIB\DB2\db2diag.log
Deadlock details
DB2 automatically creates a deadlock monitor at database creation time. If this monitor is still active, then the deadlock detail files can be found in this location:<DFTDBPATH>/<DatabaseInstance>/NODE0000 /SQL00001/db2event/db2detaildeadlockIf this monitor was dropped or deactivated, then create a new monitor in a DB2 command-line window as follows:create event monitor deadlock_mon for connections, deadlocks with details write to file '<output-directory>' buffersize 8 blocked maxfiles 1 maxfilesize none

This monitor can be turned on and off as follows:// Turn the monitor ON
set event monitor deadlock_mon state 1 // Turn the monitor OFF
set event monitor deadlock_mon state 0

After the deadlock occurs, extract the event details in text format as follows:
     db2evmon -path <event-output-directory> 
 Pipe the output of this command to a file to save the results in text format.
Monitor switches
The monitor logs are most useful if certain DB2 switches are enabled. These switches can be enabled or disabled in a DB2 command line prompt as follows:
DB2 UPDATE DBM CFG USING DFT_MON_UOW ON IMMEDIATE
DB2 UPDATE DBM CFG USING DFT_MON_STMT ON IMMEDIATE
DB2 UPDATE DBM CFG USING DFT_MON_TABLE ON IMMEDIATE
DB2 UPDATE DBM CFG USING DFT_MON_BUFPOOL ON IMMEDIATE
DB2 UPDATE DBM CFG USING DFT_MON_LOCK ON IMMEDIATE
DB2 UPDATE DBM CFG USING DFT_MON_SORT ON IMMEDIATE
DB2 UPDATE DBM CFG USING DFT_MON_TIMESTAMP ON IMMEDIATE
DB2 UPDATE DBM CFG USING HEALTH_MON ON IMMEDIATE
Restart DB2 for these to take effect:
    db2stop force
  db2start
Lock Snapshot
Run the following commands in a DB2 command line prompt:db2 connect to [db-name]
db2 get snapshot for locks on [db-name] > [output-file-name]
If necessary, reset the statistics, re-run the test code, and capture another snapshot. You can reset the statistics as follows:db2 reset monitor for database [db-name]
In some cases, it may be useful to run more test code and capture another snapshot without resetting the statistics.
Statement Snapshot
Run the following commands in a DB2 command line prompt:db2 connect to [db-name]
db2 get snapshot for dynamic sql on [db-name] > [output-file-name]
It is useful to reset statistics and/or capture multiple snapshots.
Application Snapshot
In some cases, you need to capture details about a particular connection to the database. DB2 refers to a connection as an application. First, identify the application ID. For effective debugging, in most cases where you need an application snapshot, you also need a lock snapshot. Take the lock snapshot first, identify which applications are in the UOW Waiting state, then take the application snapshots for those applications. In the lock snapshot, the application ID is referred as “Application Handle”:
Application handle                         = 175Application ID                             = *LOCAL.db2ieimd.081008181114
Sequence number                            = 00001
... 
Run the following commands in a DB2 command line prompt:db2 get snapshot for application applid [appl-id]
Statement Monitor
A statement monitor captures all of the individual statements that are executed against the database, in the order in which they were executed. The steps are similar to the deadlock monitor.
Run the following DB2 commands and procedures:
  1. Create the monitor:db2 create event monitor statement_mon for statements write to file '[output-directory]' buffersize 8 blocked maxfiles 1 maxfilesize none
     
  2. Turn on the monitor:db2 set event monitor statement_mon state 1
     
  3. Run the test code
     
  4. Turn off the monitor. This should create a (binary) .evt filedb2 set event monitor statement_mon state 0
     
  5. Convert the file to textdb2evmon -path [event-directory] > statements.txt

Popular posts from this blog

Shrink you container size up to 95%.

-
BLAFS is a bloat-aware filesystem for container debloating. The design principles of BLAFS are effective, efficient, and easy to use. It detects the files used by the container, and then debloats the container by removing the unused files. The debloated containers are still functional and can run the same workload as the original containers, but with a much smaller size and faster deployment. https://github.com/negativa-ai/BLAFS?tab=readme-ov-file
Read more

alma linux: dnf Module yaml error: Unexpected key in data

-
While applying recent updates to a server running Rocky Linux 8, an error presented itself. Module yaml error: Unexpected key in data: static_context [line 9 col 3] Module yaml error: Unexpected key in data: static_context [line 9 col 3] The solution turned out to be a simple one. Update the libmodulemd first to correct the problem then perform the dnf updates as usual. dnf update libmodulemd For a detailed explanation as to the cause of this problem.
Read more