Linux and Mysql Tuning
Getting the most from your hardware
The Ubiquitous “About Me”
Ran datacenters for:
National Geographic
ExpertCity (now Citrix Online)
Fastclick (now Valueclick)
University of California (now broke);
Now CEO/founder of LogicMonitor.com
Probably just like you. (Not a kernel committer,
or poring over MySQL source code).
Your Mileage May Vary
Nothing I am going to say will have any impact
in your environment.
Unless it does.
So...test, measure, rinse, repeat.
What to measure?
Everything!
Database (innodb, MyISAM, replication), OS,
disk, application – all in as much detail as
possible.
Measure YOUR application
Get monitoring that can query db directly, and
slice/dice (registrations/sec; customer updates
received, etc).
Also instrument front ends (via JMX, WMI, web
status, etc) so you can measure effect on app
of db adjustments
Talk outline
Kernel IO scheduler tuning
InnoDB thread tuning
Stopping swapping
Swappiness
NUMA factors
Huge pages
Why?
Limits are always one of CPU, RAM, Disk,
Network.
But with better <CPU|RAM|Disk|network> the
limit will move to next object
Default configurations may be for:
Your server:
IO Scheduler Tuning
Kernel decides how to order disk operations, in
order to combine IO ops into one op, and also
minimize disk seeks, or not.
Defaults work. No benefit in other policies, only
harm. Don't bother testing.
But, it's trivial to test, so don't take my word for
it. With battery backed RAID cache, others
have reported good results with noop.
Kernel, hardware dependent. Test!
echo noop > /sys/block/sda/queue/scheduler
VM on ESX with no-cache RAID1
Switching from CFQ to Noop
Amount of non-merged writes overwhelmed
disk.
Switch to
Switch to NoOP
NoOP
VM on ESX no-cache RAID1
Switch from anticipatory to CFQ at 13:10
CFQ has slightly more reads merged, slightly
better performance time.
Switch to CFQ
Lots of IO where ordering doesn't
matter
4000 disk operations to SSD
Switching from CFQ to NoOP – slightly worse
IO completion time.
Switch to NoOP Switch to NoOP
But it saves CPU, right?
Not significantly, if at all.
Policy switched
Innodb Thread Concurrency
If database is slow, but CPU not busy...this may
be issue.
Defaults range from 4 (in some distro's) to
infinite, depending on version. 8 seems most
common default in real world.
“Infinite” changes between versions.
Default probably too low for “real” DB server.
Test! Benchmark first, then real life
Use a threaded benchmark.
sysbench --test=oltp --mysql-table-
engine=innodb --mysql-
socket=/var/lib/mysql/mysql.sock --max-
time=300 --db-driver=mysql --num-threads=10
--max-requests=300000 run
Use same benchmark, vary server config.
Results will vary with # users
How many user threads to test? Use real data.
My results, not yours
This is true on some hardware, with some
version of MySQL, with 10 client threads.
Your mileage WILL vary. Test.
Higher settings not always better.
More InnoDB threads – more
efficient use of CPU
16 Innodb ∞ Innodb
Threads Threads
8 Innodb threads
4 Innodb
Threads
Swapping
Use of swap not bad – swapping is bad.
OK (but not ideal):
Performance Killer:
Disable swap?
Erm … probably not.
“Because new pages are always being read in
and processes are always allocating new
memory, the OS will have to make a decision of
what pages to evict from physical memory. If a
page is dirty, it can only be evicted if there's
swap. So if you have dirty pages that are very
rarely used, swap allows you to keep more hot,
clean pages in memory.”
http://kerneltrap.org/node/3202
Easy things first
echo 0 > /proc/sys/vm/swappiness
echo “vm.swappiness = 0” >>
/etc/sysctl.conf; sysctl -p
This may be enough. If so, KISS.
NUMA factors
System tries to allocate memory local to the
CPU a process runs on.
In dual processor system, if one process tries to
grab > 50% of system's memory, it will use all
free memory local to one processor.
other process on that CPU requesting memory
may cause swap activity to get local memory –
even though there is free non-local memory.
http://jcole.us/blog/archives/2010/09/28/mysql-
swap-insanity-and-the-numa-architecture/
Numa, II
I couldn't reproduce:
Run mysql with affinity to a physical CPU, with buffer
pool that consumes all local memory.
Start Java process with affinity to same CPU. Mem
was allocated from other core. But.. no swap used.
But just in case...easy to address:
numactl --interleave all mysqld
If you are allocating more than 50% memory to
innodb (or any other single process), you are
guaranteed non-local memory access. So may
as well be deterministic, and possibly benefit
from this swap-fix.
HugePages
Currently huge pages are not swapped (may
change)
Will have some (probably very minor)
performance benefit (big in some cases)
But.. more complex (and thus risky.)
Need to enable in kernel, configure
kernel,ulimits, security groups, configure mysql.
See
http://dev.mysql.com/doc/refman/5.0/en/large-
page-support.html
HugePages
Set nr_hugepages too low – Mysql wont use,
and you just wasted memory. Ditto too high.
Pins Innodb buffer pool cache – but will
increase pressure on other memory if swapping
is triggered
Will reduce TLB cache lookups/misses. Does
that matter? Test!
Summary
My app/kernel/hardware/DB version is not
yours. YMMV.
Test IO scheduler, as its so easy.
Test InnoDB thread concurrency.
Address swapping in the simplest way you can.
Trend everything. All the time. Compare pre-
and post releases.
That's it!
Questions?
[email protected]
Www.logicmonitor.com
Blog.logicmonitor.com
