VMware vCenter Server Appliance fails with the error: out of memory (2112303)

 Symptoms

• VMware vCenter Server Appliance fails and reports out of memory
• In the /var/log/vmware/vpx/vpxd.log file, you see entries similar to:

<YYYY-MM-DD>T<time> [7F1AEAF5E700 error 'Default' opID=HB-host-3012@125105-5f25df6b] [Vdb::IsRecoverableErrorCode] Unable to recover from 53200:7</time>

<YYYY-MM-DD>T<time></time> [7F1AEAF5E700 error 'Default' opID=HB-host-3012@125105-5f25df6b] [VdbStatement]SQLError was thrown: "ODBC error: (53200) - ERROR: out of memory

--> Failed on request of size 208.;

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_DUAL (DUMMY) SELECT 1 FROM VPX_PROPERTY_BULLETIN WHERE 1 = 0 AND OBJECT_TYPE=? AND OBJECT_MOID=? AND

PROPERTY_PATH=?; UPDATE VPX_PROPERTY_BULLETIN set GEN_NUMBER = ?, MOD_TIME = ?, OPERATION_TYPE = ? WHERE OBJECT_TYPE=? AND OBJECT_MOID = ? AND PROPERTY_PATH = ?; INSERT INTO VPX_PROPERTY_BULLETIN (

GEN_NUMBER, OBJECT_TYPE, OBJECT_MOID, PROPERTY_PATH, OPERATION_TYPE, MOD_TIME) WITH in_data (GEN_NUMBER , OBJECT_TYPE, OBJECT_MOID, PROPERTY_PATH, OPERATION_TYPE, MOD_TIME) AS (SELECT ?::numeric(22

<YYYY-MM-DD>T<time></time> [7F1AEAF5E700 error 'Default' opID=HB-host-3012@125105-5f25df6b] Alert:false@ /build/mts/release/bora-2183111/bora/vim/lib/vdb/vdb.cpp:3737

--> Backtrace:

--> backtrace[00] rip 00007f1b25574ff4 Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)

--> backtrace[01] rip 00007f1b2545ea22 Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

 

<YYYY-MM-DD>T<time></time> [7FA8D5D44700 error 'Default'] Unable to allocate memory

• The /var/log/Procstate.log file contains messages indicating that vpxd memory usage grew before the crash:

========== Start of system state dump at  <YYYY-MM-DD>T<time></time> ==========

procstate-20150313:root 4842 18.9 20.9 12566384 6918124 ? Ssl Mar06 1943:47 /usr/lib/vmware-vpx/vpxd

 

========== Start of system state dump at  <YYYY-MM-DD>T<time></time> ==========

procstate-20150313:root 4842 19.6 48.9 22775640 16173312 ? Ssl Mar06 2013:11 /usr/lib/vmware-vpx/vpxd 
 

Notes:

• In this example, 20.9 and 48.9 are the total system memory usage consumed by vpxd.
• The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

 Cause

This issue occurs when the changes in the resolution alter the way the ODBC driver utilizes its memory buffer space on select queries.

 

 Resolution

This is a known issue affecting the VMware vCenter Server Appliance 5.5.

 

This issue is resolved in VMware vCenter Server 5.5 Update 3, available at  VMware Downloads.

For more information, see the VMware vCenter Server 5.5 Update 3 Release Notes.

 

To work around this issue, edit the odbc.ini file.

Warning: You must upgrade to vCenter Server Appliance Update 2 (2001466 or higher) before implementing this resolution as it requires a vPostgres ODBC driver update to function properly. It may have adverse effects if applied on an earlier release.

To edit the odbc.ini file:

1. Log in to the vCenter Server Appliance console as root.
2. Run this command to backup the original odbc.ini:
   
   cp /etc/odbc.ini /etc/odbc.ini.orig
    
3. Open the /etc/odbc.ini file by running this command: 
   
   vi /etc/odbc.ini
    
4. Remove this line: 
   
   FetchBufferSize = 64000
    
5. Add these lines:
   
   useDeclareFetch = 1
   Fetch = 1024 
    
6. Save the changes and exit the text editor.
7. Stop the vpxd service by running this command:
   
   service vmware-vpxd stop
    
8. Start the vpxd server by running this command: 
   
   service vmware-vpxd start

 

 Related Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box. 

Vmware recommends you to implement the steps given in knowledge base article After upgrading to vCenter Server Appliance 5.5 Update 2, pg_log file reports this error: WARNING: there is already a transaction in progress (2092127) after upgrading to the vCenter Appliance 5.5 Update 2.