ASM Inherently Performs Asynchronous I/O Regardless

of lesystemio_options Parameter
A complete review of I/O behavior in Oracle database in ASM
and OS con guration
plus recommendations

The purpose of this document is to explain the behaviour of I/O calls depending database
parameters FILESYSTEMIO_OPTIONS and DISK_ASYNCH_IO when the Oracle RDBMS is
located on ASM diskgroups.

FILESYSTEMIO_OPTIONS controls the behaviour of I/O options given by operating system but it
has no a ect when ASM is used.

FILESYSTEMIO_OPTIONS enables asynchronous I/O (AIO) when the value is set to SETALL/
ASYNC.

📍 However, ASM bypasses the lesystem layer in this case because ASM I/O is entirely
controlled by DISK_ASYNCH_IO parameter.

AIO needs to be enabled/disabled by setting disk_asynch_io to parameter values TRUE/FALSE.

In order to test this behaviour database writer process (DBW0) is traced with strace during a
tablespace creation.

Use strace commnad in linux :

An example strace of dbw0 process when AIO is used shows io_submit/io_getevents calls.

gettimeofday({1227275539, 15789}, NULL) = 0

times(NULL) = 532384532
io_submit(13086720, 2, {...}) = 2
gettimeofday({1227275539, 16181}, NULL) = 0
io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532384532
io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1

and the strace of dbw0 process with AIO disabled shows pwrite64:

gettimeofday({1227274991, 462322}, NULL) = 0

times(NULL) = 532329776
pwrite64(22, "\36\242\0\0\3\0@\2\302\33\v\0\0\0\1\4VB\0\0\t\0\0\0\t\0\0\0\0\0\0\0"..., 8192,
6153068544) = 8192
times(NULL) = 532329776
times(NULL) = 532329776
pwrite64(24, "\36\242\0\0\3\0@\2\302\33\v\0\0\0\1\4VB\0\0\t\0\0\0\t\0\0\0\0\0\0\0"..., 8192,
6199205888) = 8192
times(NULL) = 532329776
fi
ff
fi
fi
 Test and Results:

100M tablespace is created and oracle dbw0 process of database instance is traced with strace
tool.

[root@ tmp]# ps -ef|grep dbw

oracle 10131 1 0 15:53 ? 00:00:00 ora_dbw0_DB2

...

$strace -af -p 10131 -o dbw.txt &

In a new sqlplus session (database instance), tablespace is created.

sqlplus / as sysdba

sql> create tablespace test1 data le '+DG1' size 100M;

Same test is repeated for each possible value to enable and disable asynchronous I/O with the
parameters and the following results are found:

Con g 1:
FILESYSTEMIO_OPTIONS=SETALL
DISK_ASYNCH_IO=TRUE

times(NULL) = 532400079
times(NULL) = 532400079
io_submit(5623808, 2, {...}) = 2
gettimeofday({1227275694, 483102}, NULL) = 0
io_getevents(5623808, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532400079
io_getevents(5623808, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532400079

When both parameters are activated to use asynchronous I/O, dbw0 process performs the
I/O asynchronously. The result is expected.

Con g 2:
FILESYSTEMIO_OPTIONS=NONE
DISK_ASYNCH_IO=TRUE
gettimeofday({1227275539, 15789}, NULL) = 0
times(NULL) = 532384532
io_submit(13086720, 2, {...}) = 2
gettimeofday({1227275539, 16181}, NULL) = 0
io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532384532
io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1

FILESYSTEMIO_OPTIONS is set to NONE.

This should normally disable the DISK_ASYNCH_IO when data les are located on a regular
lesystem (non-ASM).
fi
fi
fi
fi
fi
 However, since data les are located on ASM,
FILESYSTEMIO_OPTIONS has no a ect in this case. I/O is
asynchronous.

Con g 3:
FILESYSTEMIO_OPTIONS=NONE
DISK_ASYNCH_IO=FALSE

gettimeofday({1227274991, 462322}, NULL) = 0

times(NULL) = 532329776
pwrite64(22, "\36\242\0\0\3\0@\2\302\33\v\0\0\0\1\4VB\0\0\t\0\0\0\t\0\0\0\0\0\0\0"..., 8192,
6153068544) = 8192
times(NULL) = 532329776
times(NULL) = 532329776
pwrite64(24, "\36\242\0\0\3\0@\2\302\33\v\0\0\0\1\4VB\0\0\t\0\0\0\t\0\0\0\0\0\0\0"..., 8192,
6199205888) = 8192
times(NULL) = 532329776

Con g 4:
FILESYSTEMIO_OPTIONS=SETALL
DISK_ASYNCH_IO=FALSE
gettimeofday({1227275223, 67935}, NULL) = 0
times(NULL) = 532352937
pwrite64(21, "\36\242\0\0\3\0@\1\25k\v\0\0\0\2\4ZA\0\0\5\0\0\0\t\0\0\0\0\0\0\0"..., 8192,
5628780544) = 8192
times(NULL) = 532352937
times(NULL) = 532352937
pwrite64(22, "\36\242\0\0\3\0@\1\25k\v\0\0\0\2\4ZA\0\0\5\0\0\0\t\0\0\0\0\0\0\0"..., 8192,
5565865984) = 8192
times(NULL) = 532352937

In the last two cases, whatever the lesystemio_options is set, if DISK_ASYNCH_IO is false, IO is
done synchronously.

ASMLIB behaves similarly but with an exception in strace output.

🚩 When ASMLIB is used, ASMLIB performs AIO depending on

how the I/O is passed to it.
And It makes no decisions at all.
This is entirely up to ASM layer.
As the above tests also implies and ASM is entirely controlled by the disk_asynch_io parameter.

⛳ So, you can also control the ASMLIB by changing the value for disk_asynch_io parameter.

When using ASMlib, the result show read calls that look like this in the strace output:

read(16, "MSA\0\2\0\7\0P\0\0\0\0\0\0\0\30\245\215\16\0\0\0\0\0\360\347\267\0\0\0\0"..., 80) =

80
fi
fi
fi
fi
ff
 So this is asynchronous i/o.

DISK_ASYNCH_IO In 23c
DISK_ASYNCH_IO controls whether I/O to data les, control les, and log les is asynchronous
(that is, whether parallel server processes can overlap I/O requests with CPU processing during
table scans).

Property Description
Parameter type Boolean
Default value true
Modi able No
Modi able in a PDB No
Range of values true | false
Basic No
If your platform supports asynchronous I/O to disk, Oracle recommends that you leave this
parameter set to its default value.
However, if the asynchronous I/O implementation is not stable, you can set this parameter to false
to disable asynchronous I/O. If your platform does not support asynchronous I/O to disk, this
parameter has no e ect.
If you set DISK_ASYNCH_IO to false, then you can increase DB_WRITER_PROCESSES or use
DBWR_IO_SLAVES to simulate asynchronous I/O

In 19c By default, the DISK_ASYNCH_IO initialization parameter in the parameter le is set to

TRUE.

To enable asynchronous Input-Output on le system les:

1. Ensure that all Oracle Database les are located on le systems that support asynchronous
Input-Output.
2. Set the FILESYSTEMIO_OPTIONS initialization parameter in the parameter le to ASYNCH or
SETALL.

Note:If the le system les are managed through ODM library interface or Direct NFS Client,
asynchronous Input-Output is enabled by default.
There is no need to set FILESYSTEMIO_OPTIONS to enable asynchronous Input-Output in these
environments.

Asynchronous Input-Output Support

Note: On Linux, Automatic Storage Management uses asynchronous Input-Output by default.
Asynchronous Input-Output is not supported for database les stored on Network File Systems.
Oracle Database supports kernel asynchronous Input-Output. This feature is disabled by default.
By default, the DISK_ASYNCH_IO initialization parameter in the parameter le is set to TRUE to
enable asynchronous I/O on raw devices.
fi
fi
fi
ff
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
 Enabling Direct Input-Output Support
Direct Input-Output support is available and supported on Linux.
To enable direct Input-Output support:
• Set the FILESYSTEMIO_OPTIONS initialization parameter to DIRECTIO.
• Set the FILESYSTEMIO_OPTIONS initialization parameter in the parameter le to SETALL, which
will enable both asynchronous Input-Output and direct Input-Output.
DBWR_IO_SLAVES
DBWR_IO_SLAVES speci es the number of I/O server processes used by the DBW0 process.
Property Description
Parameter type Integer
Default value 0
Modi able No
Modi able in a PDB No
Range of values 0 to operating system-dependent
Basic No
DBWR_IO_SLAVES is relevant only on systems with only one database writer process (DBW0).
The DBW0 process and its server processes always write to disk.
By default, the value is 0 and I/O server processes are not used.
If you set DBWR_IO_SLAVES to a nonzero value, the number of I/O server processes used by the
ARCH and LGWR processes is set to 4. However, the number of I/O server processes used by
Recovery Manager is set to 4 only if asynchronous I/O is disabled (either your platform does not
support asynchronous I/O or disk_asynch_io is set to false).
Typically, I/O server processes are used to simulate asynchronous I/O on platforms that do not
support asynchronous I/O or that implement it ine ciently. However, you can use I/O server
processes even when asynchronous I/O is being used. In that case the I/O server processes will
use asynchronous I/O.
I/O server processes are also useful in database environments with very large I/O throughput,
even if asynchronous I/O is enabled.

Another method to checking i/o status:

Many times there is a requirement to check if Asynchronous I/O is working on Linux Platform, so
we can try to use it for our data les access inside database.

SOLUTION
The slabinfo maintains statistics about objects in memory. Some of the structs used by
Asynchronous I/O are treated as objects in the virtual memory, so we can look for those structs on
slabinfo.
The ones related to AIO are named kio*.

$ cat /proc/slabinfo | grep kio

Since Oracle Linux provides di erent kernels (including RedHat Compatiable Kernel, Oracle UEK),
the output could be slightly di erent.
fi
fi
fi
ff
ff
fi
ffi
fi
 If Async I/O is enabled:

$ cat /proc/slabinfo | grep kio

kioctx 270 270 128 9 9 1 : 252 126
kiocb 66080 66080 96 1652 1652 1 : 252 126
kiobuf 236 236 64 4 4 1 : 252 126

or
kioctx 518 670 384 10 1 : tunables 54 27 8 : slabdata 67 67 0
kiocb 595 720 256 15 1 : tunables 120 60 8 : slabdata 48 48 60

and if Async I/O is disabled or not in using:

$ cat /proc/slabinfo | grep kio

kioctx 0 0 128 0 0 1 : 252 126
kiocb 0 0 96 0 0 1 : 252 126
kiobuf 0 0 64 0 0 1 : 252 126

or
kioctx 0 0 384 10 1 : tunables 54 27 8 :
slabdata 0 0 0
kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0

In UEK4 the slabinfo entries for aio are no longer available. The caches used by the aio code are
merged with other caches:

$ ls -l /sys/kernel/slab | grep kio

lrwxrwxrwx. 1 root root 0 Dec 15 15:53 aio_kiocb -> :t-0000128
lrwxrwxrwx. 1 root root 0 Dec 15 15:53 kioctx -> :t-0000640

In the SLAB allocator there are three di erent caches involved.

The kioctx and kiocb are Async I/O data structures that are de ned in aio.h header le.

If it shows a non zero value that means async io is enabled.

If you have the source code loaded, you can review it at le aio.h.
This le is located under:
/usr/src/linux-<version>/include/linux/aio.h

These data structures are using to track the I/O requests, and are allocated as part of the
__init_aio_setup() call in aio.c.

Example strace of dbw0 process with AIO enabled (init.ora parameter lesystemio_options =
asynch) shows:
...
io_submit(3071864832, 1, {{0xb7302e34, 0, 1, 0, 21}}) = 1
gettimeofday({1176916625, 58882}, NULL) = 0
io_getevents(-1223102464, 1, 1024, {{0xb7302e34, 0xb7302e34, 8192, 0}}, {600, 0}) = 1
...

Example strace of dbw0 process with AIO disabled ( lesystemio_options = none):

...
pwrite64(21, "\6\242\0\0004\21\300\0\220B\243\0\0\0\1\6\207\357\0\0\1"..., 8192, 36077568) =
8192
times(NULL) = 1775653082
fi
ff
fi
fi
fi
fi
fi
 times(NULL) = 1775653082
pwrite64(21, "\6\242\0\0<\21\300\0\220B\243\0\0\0\1\6\254\0\0\0\2\0*"..., 8192, 36143104) =
8192
...

Also you can use OSWatcher tools to achieve more details and information.

Caveat for ASMLib

If Oracle ASMLib (see http://oss.oracle.com/projects/oracleasm ) is deployed, the kiocb structs
are not used.

ASMLib does not use the POSIX aio_*() functions.

You will never see any kioctx or kiocb structures from ASMLib.

It is far lower level than that.

In fact, ASMLib does AIO or SyncIO depending on how the I/O is passed to it, It makes no
decisions at all.

This is entirely up to kfk and the layers above it, kfk is entirely controlled by the disk_asynch_io
parameter.

So, we can check whether ASMLib is doing AIO by SQL*Plus command

"show param disk_asynch_io".

(You can disable AIO by setting disk_asynch_io=false)

With ASMLib, AIO is done via ioctl() calls (2.4 kernel), or read()
calls (2.6 kernel) on the ASM device.
Whether ASMLib uses aio depends on whether oracle is con gured to do aio, In oracle 10g, if
ASMLib is in use, the i/o is asynchronous, because oracle 10g enables aio by default.

The strace when using ASMlib will show read calls that look like this:
...
read(16, "MSA\0\2\0\10\0P\0\0\0\222\377\377\377@\313\373\5\0\0\0"..., 80) = 80
...

The rst 3 characters, byte-swapped, are ASM, indicating an ASMLib I/O command structure.

Note: Recommend "strace -c" which doesn't cause too much overhead.

I/O operations in UNIX and Linux systems typically go through the le system cache. Although
this doesn't represent a problem in itself, this extra processing does require resources. Bypassing
the le system cache reduces CPU requirements, and frees up the le system cache for other
non-database le operations. Operations against raw devices automatically bypass the le
system cache.
When a synchronous I/O request is submitted to the operating system, the writing process blocks
until the write is complete before continuing processing. With asynchronous I/O, processing
continues while the I/O request is submitted and processed. This allows asynchronous I/O to
bypass some of the performance bottlenecks associated with I/O operations.

Oracle can take advantage of direct I/O and asynchronous I/O on supported platforms using the
FILESYSTEMIO_OPTIONS parameter, whose possible values are listed below.
• ASYNCH - Enabled asynchronous I/O where possible.
• DIRECTIO- Enabled direct I/O where possible.
• SETALL- Enabled both direct I/O and asynchronous I/O where possible.
fi
fi
fi
fi
fi
fi
fi
 • NONE - Disabled both direct I/O and asynchronous I/O.

Most of the performance tuning issues can be related to I/O in any database. Oracle provides only
two main parameters to control I/O behaviour these are lesystemio_options and disk_asynch_io

Oracle recommends to set parameter

lesystemio_options to value 'setall' but it is not always good practise especially when SGA
is small.
setting it to setall lets your Oracle DB perform I/O operations without going to le system cache
and it saves overhead of double caching but if SGA is smaller and DB host machine has large free
memory then it is not good to set this parameter to value setall. In this case you should increase
DB_CACHE_SIZE and only then set lesystemio_options to setall.

As example if you setting up Oracle database to handle large I/O of new Dataware house project.
And set lesystemio_options to setall.
This maybe speeded up load operations aproximately by 5% whose caching in le system was
not useful as these were insert statements and not repeated.
At the same time increased SGA to quite high to cache I/O as much as possible so that repetitive
select queries can bene t from SGA as in this case when lesystemio_options was set to setall
then le system cache was not available.

So summary is be prudent when setting lesystemio_options to setall to enable direct read/write

and asynchronous operations.

Review dbwriter_processes a ect on i/o:

Other parameters to a ect write (as well as read) is dbwriter_processes.
When asynchronous I/O operations are slower in operating system in comparison to synchronous
I/O then turn o asynchronous I/O by setting disk_asynch_io to false and set multiple db writer
processes by increasing dbwriter_processes values from 1 to 2,3 or 4 suitable value to your
system.
Alternate is incrase dbwr_io_slaves from 0 to 2,3,4 suitable value.

You would be keen to disable asynchronous I/O when you see high average_wait on event
db_ le_parallel_wait.
Other reason for turning it o will be synchronous I/O is more reliable.

SQL> select event,average_wait from v$system_event where event like 'db le parallel write';
EVENT AVERAGE_WAIT
----------------- ------------
db le parallel write 28.2 [ centi seconds]

This is not a good ASYNCH I/O. Try Synchronous I/O and check.

Note 1:
Asynchronous I/O operations are more prone to block corruptions than synchronous operations
so many DBAs disable it and follow practice as mentioned in above paragraph.
So if you do not have standby databases and oracle 11g and newer then which autoamatically
recovers corrupted block on primary then you would not want asynchronous I/O.

Note 2: From 11g R2 for tuning purpose, the “db le async I/O submit” should be treated
as “db le parallel write” in previous releases.
fi
fi
fi
fi
fi
fi
ff
ff
fi
ff
fi
fi
fi
ff
fi
fi
fi
fi
fi
 db le parallel write
This event occurs in the DBWR. It indicates the time that DBWR spends waiting for I/O
completion.
If asynchronous I/O is available, then the db le asynch I/O submit wait event captures any time
spent in submitting I/Os to the underlying storage.
When asynchronous I/O is not available, db le parallel write captures the time spent during
submit and reap.
Wait Time: While there are outstanding I/Os, DBWR waits for some of the writes to complete.
DBWR does not wait for all of the outstanding I/Os to complete.

Some Oracle shops are plagued with slow I/O intensive databases, and this tip is for anyone
whose STATSPACK top-5 timed events shows disk I/O as a major event.

Example Statspack output:

Top 5 Timed Events

% Total
Event Waits Time (s) Ela Time
------------ ------------ ----------- -----------
db le sequential read 2,598 7,146 48.54
db le scattered read 25,519 3,246 22.04
library cache load lock 673 1,363 9.26
CPU time 2,154 934 7.83
log le parallel write 19,157 837 5.68

This tip is important to you if you have reads waits in your top-5 timed events. Remember, if disk
I/O is not your bottleneck then making it faster WILL NOT improve performance.
Also, this is a OS-level solution, and often I/O-bound Oracle databases can be xed by tuning the
SQL to reduce unnecessary large-table full-table scans.
Monitoring le I/O using the stats$ lestatxs view:

For optimal disk performance, Oracle should always use direct I/O to its data les, bypassing any
caching at the OS layer.

This must be enabled both in Oracle and in the operating system.

The debate:
Oracle supports multiple blocksized tablespaces in a single database including 2k, 4k, 8k, 16k
and 32k blocksizes.
The debate is about the bene t of di erent blocksizes (e.g. 8k, 32k) for reducing Oracle physical
disk I/O and about whether the SA is likely to con gure the Oracle server to use direct I/O for
Oracle.
Some claim that placing large-related things (i.e. indexes) in a large blocksize results in a single
physical fetch (less disk I/O), but others say this is NOT true because the OS blocksize (and the
JFS cache) result in multiple OS I/O.
According to this web page on direct I/O by Steve Adams:
http://www.ixora.com.au/q+a/io.htm
"It also avoids performance problems associated with using database block sizes that do not
match the le system bu er size exactly. (more).
Whether direct I/O is available, and the way in which it can be enabled, are dependent on the
operating system and le system type in use. Direct I/O can sometimes be enabled for an entire
le system, using a le system mount option. Application programs may also be able to request
direct I/O for a particular le by setting a ag when opening the le."
Adams goes on to note that direct I/O can increase the size of the RAM available for the working
set in the Oracle data bu ers:
"By switching to raw or direct I/O and by giving Oracle the memory that would otherwise be used
by the operating system to cache Oracle data, a much larger working set of data can be cached,
and a much higher cache hit rate can be sustained with obvious performance bene ts."
fi
fi
fi
fi
fi
fi
fi
fi
fi
ff
ff
fi
fi
fi
ff
fl
fi
fi
fi
fi
fi
fi
fi
 Oracle controls direct I/O with a parameter named lesystemio_options. According to this page by
Steve Adams the lesystemio_options parameter must be set in order for Oracle to read data
blocks directly from disk:
"Databases that use a combination of say raw log les and raw temp les with le system based
data les may wish to use kernelized asynchronous I/O against the raw les, but to avoid ine cient
threaded asynchronous I/O against the data les.
This can be done by allowing disk_asynch_io to default to TRUE, but setting lesystemio_options
to either none or directIO."

Checking your Server

Methods for con guring the OS will vary depending on the operating system and le system in
use. Here are some examples of quick checks that anyone can perform to ensure that you are
using direct I/O:
• Solaris - Look for a "forcedirectio" option. Oracle DBAs claim this option makes a huge
di erence in I/O speed for Sun servers. Here is the Sun documentation: http://docs.sun.com/
db/doc/816-0211/6m6nc6713?a=view
• AIX - Look for a "dio" option. Here is a great link for AIX direct I/O: http://www-106.ibm.com/
developerworks/eserver/articles/DirectIO.html

• Veritas VxFS - (including HP-UX, Solaris and AIX), look for "convosync=direct". It is also
possible to enable direct I/O on a per- le basis using Veritas QIO; refer to the "qiostat"
command and corresponding man page for hints. For HPUX, see Oracle on HP-UX - Best
Practices.

• Linux - Linux systems support direct I/O on a per- lehandle basis (which is much more exible),
and I believe Oracle enables this feature automatically. Someone should verify at what release
Oracle started to support this feature (it is called O_DIRECT). See Kernel Asynchronous I/O
(AIO) Support for Linux and this great OTN article: Talking Linux: OCFS Update.

I have found that the culprit in a few databases is tables that need purging or better indexing of
columns.
The memory/IO hogs can be seen with the following query. When I create the indexes or purge
rows, I nd my db block hit ratio jumping and the physical reads per hour dropping and the meg
of memory in the following query dropping:

SELECT COUNT(*)*8192/1024/1024 meg_in_memory,

o.OBJECT_NAME Object_in_Memory
FROM DBA_OBJECTS o, V$BH bh
WHERE o.DATA_OBJECT_ID = bh.OBJD
AND o.OWNER != 'SYS'
GROUP BY o.OBJECT_NAME
having count(*)>0
ORDER BY COUNT(*);

Regards,
Alireza Kamrani.
ff
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fl
ffi