HANA Memory RCA Roadmap

.
 HANA memory RCA roadmap

OOM or
memory constraint Analysis
Delta Merge
Sizing is done
Properly?
Check Locks
Rev, Hardware, Crash Dump, Yes
Known issue?
Known issue
In temporary
calculation Review Sizing
No
Memory in
Parameter is No Data Store
Parameters
correct/optimized

Yes

Yes Memory in Generic

Issue only in one node Memory Issue
One Node

No
Memory Single query Investigate
Monitoring Memory Issue Single query

Analyze
Memory leak
Memory Leak
© SAP 2007 / Page 2
 Memory parameters(1)
global_allocation_limit
 90% of Physical RAM size. ( other 10% is reserved for Linux).
 Some situation it could be less than 90%, for example, due to license limitation.
 Even for large node with 1TB RAM, it’s not recommended to increase
global_allocation_limit.

 On most HANA systems, preprocessor is not needed(at least for now, might
change in the future), in some cases even the XSEngine and webdispatcher, you
can switch them off. (see SAP Note 1697613 to remove xsengine).

 Nameserver and Statisticsserver must never be switched off, and don’t try to limit
memory for nameserver.

 [In Development] HANA DB memory management may "transfer" memory that is

not used (in the sense of "Memory Used") but allocated by a HANA DB process to
another HANA DB process without performance penalty.

© SAP 2007 / Page 3

 Memory parameters(2)
Other memory parameters:
Global: Nameserver:
Global_allocation_limit: 0
reservedmemory: 128
Statement_memory_limit:
Preprocessor:
Indexserver:
maxmemory: 200
max_delta_memsize: 1000 (MB)

Memory_merge_decision_func:

Use_shared_memory: true
Statisticserver:
[joins] plan_cache_size: (Byte)
allocationlimit: 5%, 10%
[joins]udiv_cache_size: 0
minallocationlimit: 5120 (MB)
[sql]plan_cache_enabled: True
[sql]plan_cahe_statistics_enabled: false
[sql]plan_cache_size: (Byte)
[sql]query_cache_size (Byte)
[sql]plan_cahe_statistics_enabled:true

[cache]resultcache_enabled: no

[cache]resultcache_maximum_value_size_in_bytes

[ [joins] translator_cache_size 2000

© SAP 2007 / Page 4

 Memory parameters(3)
Identify non-default parameters

Run following SQL statement to identify non-default parameters:

select a.file_name, b.layer_name, b.tenant_name, b.host, b.section, b.key, a.value as

defaultvalue, b.currentvalue from sys.m_inifile_contents a
join ( select file_name, layer_name, tenant_name, host, section, key, value as currentvalue
from sys.m_inifile_contents b where layer_name <> 'DEFAULT' ) b
on a.file_name = b.file_name and a.section = b.section and a.key = b.key and a.value <>
b.currentvalue

© SAP 2007 / Page 5

 Memory issue with one specific node(3)

Which HANA process on which host is using most of the memory?

SQL Statement:
SELECT TOP 3 HOST, PORT, SERVICE_NAME, TOTAL_MEMORY_USED_SIZE
FROM M_SERVICE_MEMORY
ORDER BY TOTAL_MEMORY_USED_SIZE DESC

HANA Studio:

© SAP 2007 / Page 6

 Memory issue with one specific node(4)

 If high memory usage is because of row tables or happening on master node,

consider changing from Row Store to Column Store.

 Additional analysis needs to be done (what kind of queries run on the table,
DML/Selects and how the table is accessed from application, single columns or
select * ).

 Column store is preferable as it provides more compression and also they are
unloadable. In case of memory shortage, less frequently used tables would be
unloaded reducing the chance of OOM .

© SAP 2007 / Page 7

 Memory Monitoring (1)

• General information about memory (index & statistic server)

SELECT * FROM M_MEMORY
• General information about memory (all servers)
SELECT * FROM M_SERVICE_MEMORY
• Memory consumption of each allocator (index & statistic server)
SELECT * FROM M_HEAP_MEMORY
For detailed information about the first two views see:
http://hdb-doxygen.wdf.sap.corp:1080/dev/user/html/_m__m_e_m_o_r_y.html
https://wiki.wdf.sap.corp/wiki/display/ngdb/Memory+Values

© SAP 2007 / Page 8

 Memory issue with one specific query (1)

1. Analyze the modeling for the query, check the number of records, see if it's
selecting too many records, need optimization or not, can also find out memory
information from performance trace.

2. Use mm commands to analyze the memory used for the query.

> hdbcons
>> mm resetusage -r Pool
>> mm flag Pool -r -s astrace,dstrace
Execute the query.
>> mm callgraph -r -o /tmp/cg.dot Pool
>> mm flag Pool -r -d astrace,dstrace
>> mm resetusage -r Pool
dot -Tpdf -o /tmp/cg.pdf /tmp/cg.dot
dot -Tgif -o /tmp/cg.gif /tmp/cg.dot

3. To identify the amount of memory for the specific query:

Run command ‘resman shrink’ in the management console
management console: ‘mm gc –f’
HDB studio: ‘ALTER SYSTEM RESET MONITORING VIEW SYS.M_HEAP_MEMORY_RESET’.
SELECT CATEGORY, INCLUSIVE_PEAK_ALLOCATION_SIZE from SYS.M_HEAP_MEMORY_RESET
where port = 3xx03
Run the test query.
SELECT CATEGORY, INCLUSIVE_PEAK_ALLOCATION_SIZE from SYS.M_HEAP_MEMORY_RESET
where port = 3xx03

© SAP 2007 / Page 9

 Analysis Delta Merge (1)

Additional memory needed for delta merge process:

Memory usage
during delta merge

The minimum memory requirement for delta merge includes:

current main size + future main size + current delta + some
additional memory.

Even if a column store table is unloaded or partly loaded, the whole

table will be loaded into memory in order to perform the delta merge.
© SAP 2007 / Page 10
 Analysis Delta Merge (2)

The delta merge operation can be expensive for the following main reasons:
 The complete main storages of all columns of the table are re-written in memory. This
consumes some CPU resources and at least temporarily duplicates the memory needed for
the main storages (while Main 1 and Main 2 exist in parallel).

 The complete main storages are persisted to disk, even if only a relatively small number of
records were changed. This creates disk I/O load.

The negative impact on performance can be mitigated by the following

strategies:
 Executing memory-only merges. A memory-only merge affects only the in-memory
structures and does not persist any data.

 Splitting tables. The performance of the delta merge depends on the size of the main
storage. This size can be reduced by splitting the table into multiple partitions, each with its
own main and delta storages. The delta merge operation is performed at partition level and
only for partitions that actually require it. This means that less data needs to merged and
persisted. (there are disadvantages to partitioning tables that should also be considered.)

© SAP 2007 / Page 11

 Analysis Delta Merge (3)
stats and trace

Delta Merge statistics:

1) Look for table smart merges in storageserver merge statistics.

select * from M_DELTA_MERGE_STATISTICS where table_name = 'xyz' and motivation =
'SMART'

2) Look for application merge hints in storageserver merge statistics:

select * from M_DELTA_MERGE_STATISTICS where type = 'HINT' and table_name = 'xyz'

3) Visit the long term statisticsserver table delta merge history:

select * from _SYS_STATISTICS.HOST_DELTA_MERGE_STATISTICS where table_name =
'xyz'

Tracing for Delta Merge:

You can activate the logging of merge-related information in the
database trace for the indexserver component. The relevant trace
components are mergemonitor and mergedog. We recommend
the trace level INFO.

© SAP 2007 / Page 12

 Analysis Delta Merge (4)
HANA Alert

More information from delta merge, refer to How to Guide: “How To Delta
Merge for SAP HANA and SAP NetWeaver BW”
http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/10681c09-
6276-2f10-78b8-986c68efb9c8?overridelayout=true

© SAP 2007 / Page 13

 locks in HANA (1)
identify lock when it’s happening

 Long lock on uncommitted transaction could hold the memory from being
used by other transactions until the lock is released.

 It’s possible to find out the lock when it’s happening:

select * from SYS.M_RECORD_LOCKS
select * from SYS.M_TABLE_LOCKS
select * from SYS.M_BLOCKED_TRANSACTIONS

 To kill an existing lock:

ALTER SYSTEM DISCONNECT SESSION '<connection id>‘

How to check dead lock:

hdbcons 'context list -s' > ‘filename’

Monitor lock from HANA studio:

© SAP 2007 / Page 14

 locks in HANA (2)
Analyze history locks in HANA

When lock happened in HANA, it would be recorded in

indexserver_alert_<hostname>.trc

Example Trace file:

[23237]{246622}[6407795] 2013-04-07 02:14:54.176663 e Lock translock.cc(00110) : Lock timeout occurs while waiting
RECORD_LOCK of mode EXCLUSIVE:

waiting tx CONNECTION_ID=46622, CLIENT_HOST=<xxx>, CLIENT_IP=10.58.77.213, CLIENT_PIP=3796,

TRANSACTION_ID=56, TRANSACTION_ADDRESS=0x7f26ea2bb000, TRANSACTION_TYPE=USER,
TRANSACTION_CACHING_FLAG=0, TRANSACTION_ISOLATION_LEVEL=RC, THREAD_ID=4,
UPDATE_TRANSACTION_ID=6407795, QUERY_BUFFER_SIZE=15568, NUM_OPENED_CURSORS=1,
SQL_STATEMENT="Select * from T1 Where ID = 1 For UPdate"

CONTAINER_ID=1269, TABLE_ID=9378514, TABLE_NAME=SYSTEM:T1, RECORD_ID=[RS:SID=0x0000000a,

OFFSET=0x20581b0]

blocked by CONNECTION_ID=46624, CLIENT_HOST=<xxx>, CLIENT_IP=10.58.77.213, CLIENT_PIP=3796,

TRANSACTION_ID=38, TRANSACTION_ADDRESS=0x7f26b9168c00, TRANSACTION_TYPE=USER,
TRANSACTION_CACHING_FLAG=0, TRANSACTION_ISOLATION_LEVEL=RC, THREAD_ID=14,
UPDATE_TRANSACTION_ID=6406799

© SAP 2007 / Page 15

 Memory issue with temporary calculation

1. Check the history how memory behaving, if the memory keep climbing, most
possibly caused by keeping loading data into HANA. If there is no data loading
happening but memory keeps climbing, there is potential memory leak.

2. From memory history, if there is sudden drop in the memory used, most posisbly
caused by GC or memory consuming transaction cancelled.

3. Memory consumption by queries could be analyzed with view

M_HEAP_MEMORY ( field inclusive_in_use) under category Pool. More details
please refer to previous session “HANA Memory Analysis”.

4. Memory expensive queries may not be captured from “expensive statement

trace”, which is more from runtime point of view.

© SAP 2007 / Page 16

 Memory issues in data store(1)

 Identify the top 10 column store tables in size:

select top 10 MEMORY_SIZE_IN_TOTAL, HOST, PORT, SCHEMA_NAME, TABLE_NAME,
MEMORY_SIZE_IN_MAIN, MEMORY_SIZE_IN_DELTA
FROM "SYS"."M_CS_TABLES"
ORDER BY MEMORY_SIZE_IN_TOTAL DESC

 Identify the top 10 row store tables in size:

select top 10 (ALLOCATED_FIXED_PART_SIZE + ALLOCATED_VARIABLE_PART_SIZE ) AS
TOTAL_SIZE, HOST, PORT, SCHEMA_NAME, TABLE_NAME
FROM "SYS"."M_RS_TABLES"
ORDER BY TOTAL_SIZE DESC

© SAP 2007 / Page 17

 Memory issues in data store(2)
translation table

 Translation table is used for cached translation between two columns, used for
joining in the JoinEngine, the C++ allocator for translation table is
Pool/JoinEvaluator/TranslationTable.

 In certain scenarios when many temporary tables are joined, the cache grows
with "useless" translation tables.

 Limit the size of the cache with. ( or HANA studio indexserver.ini, joins):
ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') SET
('joins', 'translator_cache_size') = '500' with reconfigure

 Default value is 2000, but much lower values are probably enough for most
scenarios. However, if the cache is too small it will impact the performance.

 Use SYS.M_DEV_JOINENGINE_ to calculate the average memory usage of one

TT. Use this number to calculate a cache size with an acceptable memory usage.
select * from SYS.M_DEV_JOINENGINE_

 Shows detailed information for all available Translation Tables:

select * from SYS.M_DEV_JOIN_TRANSLATION_TABLES_

© SAP 2007 / Page 18
 Memory issues in data store(3)
Persistent layer

 Many different versions of tables will take too much memory, how to check:

select * from m_mvcc_tables

If there are too many versions (e.g. more than 1,000,000 or keeps growing), find out
who is blocking the garbage collection - possibility: (1) long-running or unclosed
cursors, (2) long-running Serializable/RepeatableRead-isolation-mode
transactions, (3) hanging threads.

 Make sure house keeping job running to bring the row store table size under
control, house keeping job only delete the data, not releasing memory, need to
run de-fragmentation to release memory.

 If a row store table fragmentation reaches 30% or 10GB, re-organization is

needed. Identify tables need reorganization: ( see note 1813245)
SELECT HOST, PORT, CASE WHEN (((SUM(FREE_SIZE) / SUM(ALLOCATED_SIZE)) > 0.30) AND
SUM(ALLOCATED_SIZE) > TO_DECIMAL(10)*1024*1024*1024) THEN 'TRUE' ELSE 'FALSE' END "Row store
Reorganization Recommended", TO_DECIMAL( SUM(FREE_SIZE)*100 / SUM(ALLOCATED_SIZE), 10,2) "Free Space
Ratio in %",TO_DECIMAL( SUM(ALLOCATED_SIZE)/1048576, 10, 2) "Allocated Size in MB",TO_DECIMAL(
SUM(FREE_SIZE)/1048576, 10, 2) "Free Size in MB" FROM M_SHARED_MEMORY WHERE ( CATEGORY = 'TABLE'
OR CATEGORY = 'CATALOG' ) AND PORT LIKE '%03' GROUP BY HOST, PORT;

© SAP 2007 / Page 19

 Memory issues in data store(4)
LOB column

 Lob columns could take a lot of memory, for less frequently used LOB columns,
put on harddisk instead of in memory.

 Check LOBs on disk:

SELECT * FROM SYS.M_TABLE_LOB_FILES

 Configuration: indexserver.ini in section [SQL]

 default_lob_storage_type = hybrid
 lob_memory_threshold = 1000

 default_lob_storage_type is applied for all new columns not explicitly qualified.

 lob_memory_threshold can be adjusted in bytes (only for hybrid lobs):

 >0: lob data with length <=lob_memory_threshold is stored in-memory,
>lob_memory_threshold goes on-disk
 0: all lob data is stored on-disk (a VirtualFile is created per LOB)
 -1: all lob data is stored in-memory (max supported length ~1GB)

© SAP 2007 / Page 20

 Memory issues in data store(6)
Convert row store to column store

Some row store tables could be converted to column store to:

1. Take advantage of compression. Column store table has much higher

compression rate than row store tables, and as a result, take much less memory
space for the same amount of records.

2. Analyze application SQL statement, for example, select (one column) is more
frequently used than select *.

3. For row store tables not frequently needed, make it unloadable. ( The statement
itself takes a lot of memory, so only do this for big tables.)

4. Not all of the row store tables could be converted to column store tables, for
example, some system tables, statistics tables.

Note 1634681 Database migration: Report to find large row store tables

© SAP 2007 / Page 21

 Analyze Memory leak

Management Console provides an easy way to analyze where memory was

allocated. Activate allocation stack traces for your allocator of interest, e.g. mm
flag Pool -r -s astrace, execute your test and use mm callgraph -r -o <file-
name> Pool to write a text file (with name <file-name>) which can be visualized
with dotty.

If too many edges are in the output file it might be good to reduce the edges to
those which have a high memory usage. Edges like

n0x00007fc5d9643e73 -> n0x00007fc5d955db3c

[label="608.0B\n19",fontname=arial];

can be erased.

You can also use mm top -r Pool to list the callstacks where memory was allocated.

Unfortunately using allocation stack traces takes some time, especially on Windows.
If you know the allocator which consumes too much memory you should restrict
stack traces to this allocator. You can see which allocator uses how much
memory by using the command mm list -s with Management Console.
© SAP 2007 / Page 22
 HANA sizing
SAP HANA sizing notes

SAP Notes for HANA sizing:

1793345 Sizing for Suite on HANA

1637145 SAP BW on HANA: Sizing SAP In-Memory Database

1855041 Sizing Recommendation for Master Node in BW-on-HANA

1514966 SAP HANA 1.0: Sizing SAP In-Memory Database

1609322 SAP HANA Platform Sizing

1799449 Sizing report ERP on SAP HANA database

1736976 Sizing Report for BW on HANA

© SAP 2007 / Page 23

Appendix
1. Memory Analysis details
 Analyze memory issue (1)
1. CLEAR SQL PLAN CACHE

Execute the following SQL commands:

a) select depth, host, port, category, inclusive_size_in_use from

m_heap_memory where depth < 3 and inclusive_size_in_use > 1073741823

order by inclusive_size_in_use desc;

b) alter system clear sql plan cache;

c) select depth, host, port, category, inclusive_size_in_use from

m_heap_memory where depth < 3 and inclusive_size_in_use > 1073741823

order by inclusive_size_in_use desc;

© SAP 2007 / Page 26

 Analyze memory issue (2)
2. collect stack traces

a) Perform this test only in a copy of production because the stack tracing will have
serious impact on the system performance. Make sure that you have a few
hundred gigabytes of disk free space. Restart the system before the execution.

b) at the OS console, logon as user <sid>adm, and execute the following commands:

> hdbcons

>> mm resetusage -r Pool

>> mm flag Pool -r -s astrace,dstrace

© SAP 2007 / Page 27

 Analyze memory issue (3)
3. collect stack trace

c) Execute the long running query. (In HANA Studio or Front-End tools)

d) After the query:

>> mm callgraph -r -o /tmp/cg.dot Pool

(To stop the traces )

>> mm flag Pool -r -d astrace,dstrace

>> mm resetusage -r Pool

© SAP 2007 / Page 28

 Analyze memory issue (4)
4. collect stack trace

e) Run command exit from dbcons, Back to Linux and convert the stats into PDF or
GIF.

Go to the directory
where you export
the DOT file

dot -Tpdf -o /tmp/cg.pdf /tmp/cg.dot

dot -Tgif -o /tmp/cg.gif /tmp/cg.dot

© SAP 2007 / Page 29

 Analyze memory issue (5)
5. Example callgraph

Different colors denote the ratio in

memory usage, RED ones mean more
Memory consumption, lighter color
means not important in memory
consumption.

Example PDF
attached
© SAP 2007 / Page 30
 To execute mm command from HANA Studio

In cases you don’t have the Linux level access to the HANA box, it’s possible to
execute the management console command from HANA studio with stored
procedure:

call SYS.MANAGEMENT_CONSOLE_PROC('mm list -f', 'hostname:30003', ?)

You can copy

out the return
result to a text
editor.

© SAP 2007 / Page 31

Thank you

Contact information:

F name MI. L name

Title
Address
Phone number