Boosting Performance and Scalability With The InnoDB Upgrade for MySQL 5.

A MySQL White Paper July 2010

Copyright 2010, Sun Microsystems

Table of Contents 1 Executive Summary ..............................................................................................................3 2 Overview of the MySQL Database Server...........................................................................3 2.1 The MySQL 5.1 Storage Engine Architecture....................................................................3 3 MySQL 5.1 and the InnoDB Upgrade ...................................................................................5 3.1 Improved Performance, Scalability and Throughput..........................................................6 3.2 Performance Comparisons ................................................................................................6 3.3 Improved Manageability and Efficiency .............................................................................8 3.4 Improved Uptime and Reliability ........................................................................................9 4 Optimizing Performance, Security and Availability with MySQL Enterprise.................10 5 Conclusion ...........................................................................................................................10 6 Learn More ...........................................................................................................................11

Copyright 2010, Sun Microsystems

1 Executive Summary
MySQL used in conjunction with the InnoDB storage engine is a consistently high-performing and scalable transactional database. However, this server architecture was somewhat limited in its ability to scale by fully utilizing modern servers with more than four CPUs and advanced memory allocation capabilities. This meant that businesses couldnt make full use of their investment in multi-CPU machines, which is less than ideal when IT budgets are constrained. To meet this growing need, we have included a free and easy-to-install InnoDB Upgrade for MySQL 5.1 with MySQL versions 5.1.38 and higher. This Upgrade provides users with an immediate, seamless option for achieving higher performance and scalability, specifically on modern, multi-core, multi-CPU servers. This is a significant improvement that transparently provides InnoDB users with added performance and scalability gains without also requiring changes to their current MySQL schemas or applications. In this paper we provide a technical overview of the MySQL storage engine architecture used by the InnoDB Upgrade for MySQL 5.1. It then explores the features, performance and scalability gains InnoDB users can expect to see when enabling the InnoDB Upgrade in MySQL 5.1.38 or later.

2 Overview of the MySQL Database Server

The MySQL database server is the most popular and trusted open source database platform today, relied on by ISVs and OEMs to power their products, and by enterprises creating modern applications that serve todays Web-enabled applications. Whether its a Web 2.0 application that grows through the contributions of its active user base, a 24x7 online telecommunications application, or a hosted financial/CRM application that manages critical data used around-the-clock, both the creation of and demand for data are growing exponentially. Open source software is a key driver for many businesses, with a compelling combination of strong features and a small financial commitment that drives many to adopt full or partial open source stacks to pilot key systems. Data is often a businesss most valued and vital asset, making an applications database a truly mission-critical component. Naturally, smart businesses are discerning in their database selection and turn to proven leaders who offer technology thats reliable, high-performing, easy-to-use and most importantly, cost-effective. Reliable, high performing, easy to use, cost-effective: these are the attributes that describe the MySQL database server. Behind these attributes is an architecture that is unique in the database industry the MySQL storage engine architecture.

2.1 The MySQL 5.1 Storage Engine Architecture

A key differentiator between MySQL and other database platforms whether they are proprietary or open source is the MySQL storage engine architecture. The MySQL storage engine architecture allows application developers and database professionals to select a storage engine based on their application's needs while being completely shielded from having to manage specific application coding requirements. Graphically depicted, the MySQL storage engine architecture looks like Figure 1:

Copyright 2010, Sun Microsystems

Figure 1 - The MySQL Storage Engine Architecture The MySQL storage engine architecture provides a standard set of management and support services that is common to all the underlying storage engines. The storage engines themselves are the components of the database server that actually perform actions on the underlying data that is maintained at the physical server level. This efficient and modular architecture provides large performance and manageability benefits for those wishing to specifically target a particular application need such as transaction processing, data warehousing, high availability situations, etc. while enjoying the advantage of utilizing a set of interfaces and services that are independent of any one storage engine. From a technical perspective some of the key differences in storage engines include: Concurrency/Locking Some applications have more granular lock requirements (such as row-level locks) than others. Choosing the right locking strategy can reduce overhead and therefore help with overall performance. Concurrency/Locking also supports capabilities like multi-version concurrency control or snapshot read. Transaction Support Not every application needs transactions, and for those that dont, the added overhead can be avoided. But for those that do, there are a number of MySQL storage engines that support transactional features, such as ACID compliance. Physical Storage This involves things like overall page size for tables and indexes to the format used for storing data to physical disk. Different storage architectures can have an impact on the overall performance of both read and write operations based on application and data characteristics. Index Support Different application scenarios benefit from different index strategies. Each storage engine generally has its own indexing methods, although some indexing mechanisms (like B-tree indexes) are common to nearly all engines. Memory Caches MySQL offers storage engines whose data only resides in RAM, which results in very fast performance. Others use different variations of memory caches for transactions, transaction logs, tables, and indexes. Other areas Including things like multiple I/O threads for parallel operations, thread concurrency, database checkpointing, bulk insert handling, and more.

Copyright 2010, Sun Microsystems

Each set of the storage engine components is designed to offer a select set of benefits for a particular application. Conversely, avoiding a set of component features helps steer clear of unnecessary overhead. So, it stands to reason that understanding a particular applications set of requirements and selecting the proper MySQL storage engine can have a dramatic impact on overall system efficiency and performance. And a DBA or developer can certainly put multiple storage engines in play for the same application, giving the application extreme flexibility and high-performance. Beginning in MySQL 5.1, a new storage engine interface allows DBAs and developers to dynamically and easily install/uninstall storage engines within the MySQL Server. To add a new storage engine, the engines plugin shared library is loaded into MySQL using the INSTALL PLUGIN statement. For example, if the EXAMPLE engine plugin is named ha_example and the shared library is named ha_example.so, the engine would be loaded into MySQL with the following statement: INSTALL PLUGIN ha_example SONAME 'ha_example.so'; To unplug a storage engine from the MySQL Server, the UNINSTALL PLUGIN statement is used: UNINSTALL PLUGIN ha_example; MySQL 5.1.38 and higher distributions now ship with the InnoDB Upgrade shared library so users can dynamically INSTALL the InnoDB Upgrade and replace the built-in InnoDB within MySQL without compiling from source or re-linking MySQL. For current deployments and schemas, databases created with the built-in InnoDB are compatible with the InnoDB Upgrade and users can easily fall back to the built-in InnoDB configuration if desired. For specific instructions on downloading, version status, installing and enabling the new InnoDB Upgrade for MySQL 5.1 see: http://www.innodb.com/doc/innodb_plugin-1.0/index.html.

3 MySQL 5.1 and the InnoDB Upgrade

InnoDB is the mostly widely used storage engine for Web/Web 2.0, eCommerce, Financial Systems, Health Care, and Retail applications built on MySQL for good reason: it provides highly efficient ACIDcompliant transactional capabilities and includes unique architectural elements that assure high performance and scalability. Further, InnoDB is structurally designed to handle transactional applications that require crash recovery, referential integrity, high levels of user concurrency and SLA exceeding response times. MySQL 5.1 with the new InnoDB Upgrade provides immediate access to InnoDB improvements that: Scale MySQL performance to fully utilize the computing power of the most modern multicore architectures. Improve manageability, providing DBAs/Developers with better control/monitoring of key InnoDB performance and tuning parameters. Increase MySQL uptime and reliability.

The supporting InnoDB Upgrade features and enhancements are described below.

Copyright 2010, Sun Microsystems

3.1 Improved Performance, Scalability and Throughput

The new InnoDB Upgrade for MySQL 5.1 extends the current InnoDB storage engine built into MySQL by adding the following new features: Improved Default Thread Concurrency The InnoDB Upgrade defaults to allow an unlimited number of concurrently executing threads, leveraging the processing power of multi-threaded and multi-core systems. Users can override this default behavior by setting the value of innodb_thread_concurrency to the limit that best works for specific implementations. Control of Background I/O Threads Users now have two new configuration parameters for all platforms, innodb_read_io_threads and innodb_write_io_threads that allow for the setting of the number of background threads used for read and write requests. This helps users tune and scale their MySQL applications on high-end, multi-core systems. Control of Master Thread I/O Rate Users can now configure the overall I/O capacity and bandwidth available to InnoDB for running background tasks, via the new innodb_io_capacity in the my.cnf or my.ini files. Control of Using Operating System Memory Allocators Users can now control whether InnoDB uses it own memory allocator or leverages the more efficient allocators available in the current versions of the most commonly deployed operating systems. This is easily controlled by setting the value of the new system configuration parameter innodb_use_sys_malloc in the MySQL option file (my.cnf or my.ini). The default setting is 1, which instructs InnoDB to use the operating system resource. Control of Adaptive Hash Indexing Users can now disable the Adaptive Hash Indexing feature, which is enabled by default in the built-in InnoDB. This is useful when tuning applications or systems: too many threads blocked by RW-latch contention caused by the underlying Adaptive Hash Indexing process (as shown in the output from SHOW ENGINE INNODB STATUS) is a good indicator that this feature can be disabled to improve performance. Control of Insert Buffering Users can now disable the InnoDB insert buffer for operations involving working data sets or inserts that fit or almost fit into the InnoDB buffer pool. This keeps InnoDB from unnecessarily merging buffered insert changes into the InnoDB buffer pool retaining more buffer pool space and persistence of the most actively accessed data pages.. Improved scalability via Faster Locking Algorithm For most platforms (Unix, Linux, Windows), the InnoDB Upgrade now uses native atomic operations vs Pthreads to implement mutexes and read/write locks. This boosts InnoDB performance and scale, specifically on multicore systems. Restored Group Commit This feature allows InnoDB to issue a single write to the log file to trigger the commit action for multiple user transactions that commit at about the same time, significantly improving throughput.

A complete, detailed explanation of all of the new Innodb Upgrade performance and scalability features and how to enable and use them with MySQL 5.1 can be found here: http://www.innodb.com/doc/innodb_plugin-1.0/innodb-performance.html

3.2 Performance Comparisons

By enabling the InnoDB Upgrade for MySQL 5.1, users can expect an immediate and substantial performance and scalability gain. For quick reference, below are MySQL's internal DBT2 benchmarks comparing the performance of version 5.1 with the built-in InnoDB to version 5.1 with the InnoDB Upgrade. Figure 2 shows a significant boost from MySQL 5.1 with the InnoDB Upgrade related to completing transactions across a graduated number of active connections.

Copyright 2010, Sun Microsystems

Figure 2 5.1 with Built-in InnoDB vs. 5.1 with InnoDB Upgrade Performance Gains Similar results have been achieved when using MySQL's internal SYSBENCH Standard OLTP tests running SELECT/UPDATE/DELETE/INSERT queries against MySQL 5.1 with the same InnoDB enablement and configurations noted above are shown in Figure 3:

Figure 3 5.1 with Built-in InnoDB vs 5.1 with InnoDB Upgrade Performance Gains

Copyright 2010, Sun Microsystems

Understanding that results will vary for specific installations, the environmental specifics used for the internal MySQL internal DBT2 and SYSBENCH tests are provided here for those wishing to run similar benchmarks when evaluating MySQL 5.1 with the InnoDB Upgrade: Hardware: - dl360-g5-b box (Intel x86_64 (Dual Quad-Core Xeon 5345),RAM 16GB) OS: - SuSE 10 - kernel - 2.6.16 - fs - ext3 dbt2 test details: - version of dbt2 test - 0.37 - scale - 10 warehouses MySQL Server Configuration: - x86_64-glibc32 package InnoDB options: --innodb_status_file=0 --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_size=2G --innodb_additional_mem_pool_size=20M --innodb_log_file_size=650M --innodb_log_files_in_group=2 --innodb_log_buffer_size=16M --innodb_thread_concurrency=0 --innodb_flush_log_at_trx_commit=1 --innodb_flush_method=O_DIRECT

3.3 Improved Manageability and Efficiency

The InnoDB Upgrade for MySLQ 5.1 also provides better control over InnoDB internals so things like performance, scale, and storage can more easily be monitored, tuned and optimized for specific use cases and application loads. The key advances and features available in MySQL 5.1 with the InnoDB Upgrade are described below. Faster Index Creation Beginning in MySQL 5.1, MySQL allows a storage engine to add or drop indexes without copying the underlying data of the entire target table. The InnoDB Upgrade extends the built-in InnoDB capabilities by leveraging this MySQL feature to improve the efficiency and speed of creating and dropping indexes on InnoDB tables. Because the InnoDB Upgrade can be dynamically loaded, it is also now possible to use the Upgrades Fast Index Creation feature to quickly add or drop an index and then revert back to the built-in InnoDB for normal operations. Efficient Data Compression New compression options significantly reduce the storage requirements for the MySQL database and improve application throughput by reducing I/O workload, all with minimal overhead and operating expense. Users can specify new ROW_FORMAT and KEY_BLOCK_SIZE parameters in the CREATE TABLE and ALTER TABLE commands to store data pages in 1K, 2K, 4K, 8K or the standard 16K byte compression levels. The InnoDB Upgrade also provides new INFORMATION_SCHEMA tables around compression so users can monitor and tune their implementations for optimal efficiency.

Copyright 2010, Sun Microsystems

Efficient Storage Options for Large Objects and Variable-Length Columns Improves storage efficiency of large objects by allowing users to create tables using ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED. With these options long column values are stored fully off-page, and the associated clustered index record contains only a 20-byte pointer to the overflow page. New INFORMATION_SCHEMA tables Seven new tables provide information specific to InnoDB compression and transaction locking. o INNODB_CMP Contains status information on the efficiency and operations related to compressed tables. o INNODB_CMP_RESET Same as above, but reading from this table resets the reported statistics. o INNODB_CMPMEM Contains status information on the compressed pages that reside in the buffer pool. o INNODB_CMPMEM_RESET Same as above, but reading from this table resets the reported statistics. o INNODB_TRX - Contains information about every transaction currently executing inside InnoDB, including whether the transaction is waiting for a lock, when the transaction started, and the particular SQL statement the transaction is executing. o INNODB_LOCKS Contains a row for each blocked transaction that describes each lock the transaction has requested and what the transaction is waiting on. o INNODB_LOCK_WAITS Contains information about transactions that are waiting for a specific lock.

A complete, detailed rundown of all of the new InnoDB Upgrade for MySQL 5.1 Manageability and Efficiency related features, including important details on the new supporting File and Row Format features can be found here: http://www.innodb.com/doc/innodb_plugin-1.0/innodb-plugin-introduction.html#innodb-pluginintroduction-features

3.4 Improved Uptime and Reliability

The InnoDB Upgrade for MySQL 5.1 provides several new features that improve the uptime and reliability of MySQL and the overall usability of the InnoDB storage engine. The key new features and advancements include: Improved Uptime via Dynamic Control of System Configuration Variables These parameters can now be dynamically changed and applied without the need to re-start the MySQL server: o Innodb_file_per_table o Innodb_stats_on_metadata o Innodb_lock_wait_timeout o Innodb_adaptive_hash_index Improved Use of Disk Space TRUNCATE TABLE now releases disk space to the operating system. InnoDB Strict Mode An InnoDB Upgrade option matches MySQL 5.1 functionality (not available in built-in InnoDB) requiring complete syntax correctness of DDL/DML statements before database level activity is completed. Control of Optimizer Statistics Estimation Users can now control the number of sampled pages that are evaluated before InnoDB optimizer paths are re-calculated via the new global parm innodb_stats_sample_pages parameter. This parameter can be dynamically set at runtime and does not require the MySQL server to be re-started to take effect.

A complete, detailed rundown of all of the InnodDB Upgrades Uptime and Reliability related features and how to enable and use them with MySQL 5.1 can be found here:

Copyright 2010, Sun Microsystems

http://www.innodb.com/doc/innodb_plugin-1.0/innodb-other-changes.html.

4 Optimizing Performance, Security and Availability with MySQL Enterprise

For those who would like assistance with the InnoDB Upgrade for MySQL 5.1 or any combination of the MySQL storage engines and their specific performance tuning options, there is MySQL Enterprise. MySQL Enterprise is a subscription-based, cost-effective offering comprised of MySQL database and monitoring software plus Technical and Consultative support services for organizations delivering highly available, business-critical applications and services. MySQL Enterprise includes the following components:

MySQL Enterprise Server Is the most up to date and secure version of MySQL. Enterprise subscribers have access to the most current bug fixes to MySQL and the InnoDB Upgrade for MySQL 5.1, quarterly service packs, and hot fix builds of MySQL if lightning truly strikes. MySQL Enterprise Monitor and Advisors with Query Analyzer Provides global monitoring of performance, replication, security, and application availability across all MySQL servers. The Enterprise Monitor ships with a set of automated MySQL Best Practice Advisors that collect, monitor, and advise on specific metrics around performance tuning, tightening security, optimal replication set up and synchronization, and more. Users can customize monitoring schedules, severity thresholds, and network and email alerts to best fit their specific development and production environments and processes. The Enterprise Monitor also includes a visual Query Analyzer feature that helps DBAs and Developers quickly find and fix the most expensive queries running in their MySQL environments. There is also an advanced Replication Monitor that autodetects replication topologies and monitors master/slave synchronization status regardless of configuration. MySQL Technical and Consultative Support Services MySQL Enterprise includes 24x7x365 production to help ensure business-critical applications are continuously available and running at their peak. MySQL Production Support Services include: o Online Self-Help Support The MySQL Knowledge Base is a self-help tool that provides users with access to 2,000+ technical articles on MySQL specific topics that help quickly answer questions and solve problems. o Problem Resolution Support Allows users to work directly with the Oracle MySQL Support team via phone, email or an online interface for quick resolution of technical problems. o Consultative Support Allows users to work with Oracle MySQL Support team on the proper installation, configuration and deployment of MySQL products (and storage engines) and their advanced feature sets, and on best practices around the design and tuning of schemas, queries and application specific code. o Advanced Support for MySQL High Availability and Scalability Solutions Provides full production support for advanced MySQL features and third-party solutions that scale MySQL database and application performance and availability.

5 Conclusion
The MySQL database server continues to raise the bar for open source database performance, reliability and ease of use. With that in mind, MySQL versions 5.1.38 and higher now ship with the new InnoDB Upgrade for MySQL 5.1, providing users with immediate access to performance and scalability gains. MySQL and the InnoDB Upgrade for MySQL 5.1 combination provides current 5.1

Copyright 2010, Sun Microsystems

10

users with added ease of use, and higher performance and scalability, while sustaining the current production-ready and stable feature set of the MySQL 5.1 server. Given that InnoDB Upgrade for MySQL 5.1 now ships with MySQL 5.1.38 and can be quickly enabled, the move to the most current version of MySQL 5.1 is well worth the small effort.

6 Learn More
Technical and Consultative Support for MySQL and the InnoDB Upgrade for, MySQL 5.1 MySQL Enterprise Server, MySQL Enterprise Monitor and Query Analyzer: http://www.mysql.com/products/enterprise/ MySQL 5.1 Database Server Downloads Binary and Source: http://dev.mysql.com/downloads/mysql/5.1.html InnoDB Upgrade for MySQL 5.1 , Downloads and Documentation: http://www.innodb.com/products/innodb_plugin/ MySQL Performance Tuning and Optimization Resources: http://mysql.com/why-mysql/performance/index.html

Copyright 2010, Sun Microsystems

11