| Subcribe via RSS

The Dangers of Having status fields

August 27th, 2008 | No Comments | Posted in MySQL, MySQL Performance

Having a status column is very common in databases today. It can be used to denote a user status:

CREATE TABLE IF NOT EXISTS `user` (
`user_id` int(10) unsigned NOT NULL auto_increment,
`email` varchar(32) NOT NULL,
`pw_hash` char(40) NOT NULL COLLATE latin1_general_cs,
`status` enum('PENDING', 'ACTIVE', 'DISABLED') default 'PENDING',
`date_created` timestamp NOT NULL default CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`),
UNIQUE KEY `idx_email` (`email`)
);

or user-uploaded media status:

CREATE TABLE IF NOT EXISTS `media` (
`media_id` int unsigned NOT NULL auto_increment,
`owner_user_id` int(10) unsigned NOT NULL,
`title` varchar(32) NOT NULL,
`path` varchar(255) NOT NULL,
`description` varchar(128) NOT NULL,
`status` enum('PROCESSING', 'ACTIVE', 'FAILED', 'DISABLED') default 'PROCESSING',
`date_created` timestamp NOT NULL default CURRENT_TIMESTAMP,
PRIMARY KEY (`media_id`),
KEY `idx_owner` (`owner_user_id`)
);

Let’s say that we needed to retrieve a list of media associated with users. This can be easily accomplished with the following query:

SELECT SQL_NO_CACHE
`u`.`user_id`,
`m`.`media_id`
FROM `user` AS `u`
JOIN `media` AS `m` ON (`u`.`user_id` = `m`.`owner_user_id`);

If there are status columns involved, the query then becomes:

SELECT SQL_NO_CACHE
`u`.`user_id`,
`m`.`media_id`
FROM `user` AS `u`
JOIN `media` AS `m` ON (`u`.`user_id` = `m`.`owner_user_id`)
WHERE `u`.`status` = 'ACTIVE'
AND `m`.`status` = 'ACTIVE';

If non-active records are removed, the status columns dropped, and the first query run, there is a 15% increase in qps over the second query on the original tables in my test environment. The difference can becomes even more pronounced as the number of tables referenced in a JOIN increases!

If you must have status fields, choose the right data type. But if they can be avoided by archiving disabled records to “cold” databases and keeping “pending” records separate (say, until a user completes email verification), that is generally a better solution.

Tags: ,

Must we always escape values?

August 18th, 2008 | No Comments | Posted in MySQL, MySQL Performance, PHP

One of the cardinal rules of writing web applications is to escape user-generated input with functions like PHP’s real_escape_string. This is a great rule, but one that can have a negative impact on your application’s performance if used unnecessarily. For instance, when querying data with an integer parameter that is passed internally (not user-generated):

$query = "SELECT SQL_NO_CACHE * FROM `user` WHERE `user_id` = '" .
$mysqli->real_escape_string ( self::$user_id ) . "'";
$res = $mysqli->query ( $query );

The above code takes an average of 0.000922918319702 seconds to execute.

Whereas:

$query = "SELECT SQL_NO_CACHE * FROM `user` WHERE `user_id` = " . self::$user_id;
$res = $mysqli->query ( $query );

takes an average of only 0.000418901443481 seconds to execute.

Although the improvement is small (~0.0005 seconds), when your site runs millions (or tens-of-millions) of queries per day, the benefits begin to add up.

Tags:

Improving WordPress Performance with Basic MySQL Maintenance

August 17th, 2008 | No Comments | Posted in Random Tech

If your blog is anything like mine, the vast majority of comments are spam. Most blogs have at least a 50% ratio of spam-to-valid comments, and Pablowe has a 99.4% ratio (which is probably why there are so many Anti-Spam plugins for WordPress).

One of the most oft-executed queries (based on the MySQL general log statistics) is:

SELECT DISTINCT ID, post_title, post_password, comment_ID,
comment_post_ID, comment_author, comment_date_gmt, comment_approved,
comment_type,comment_author_url,
SUBSTRING(comment_content,1,30) AS com_excerpt
FROM comments
LEFT OUTER JOIN posts ON (comments.comment_post_ID = posts.ID)
WHERE comment_approved = '1' AND comment_type = '' AND
post_password = ''
ORDER BY comment_date_gmt DESC
LIMIT 10;

In order to keep this (and other) queries performing well, I put the following script in cron and schedule it to execute weekly:

<?php
// This should match your values in wp_config.php
$table_prefix = '';
$db_host = '';
$db_name = '';
$db_user = '';
$db_password = '';
// How long (in days) before comments are purged
$purge_age = 7; // DEFAULT: One Week
if ( $mysqli = new mysqli( $db_host,$db_user,$db_password,$db_name) ) {
$purge_query = 'DELETE FROM ' . $table_prefix .
'comments WHERE comment_date < DATE_SUB(NOW(), INTERVAL ' .
$purge_age . " DAY) AND comment_approved = 'spam'";
if ( ! $mysqli->query ( $purge_query ) ) {
die ( "Could Not Issue Query: Please check configuration\n" );
}
$mysqli->close ( );
} else {
die ( "Could Not Connect: Please check configuration\n" );
}
?>

Will this make your blog Digg Proof? Not by itself, but it can play an important part in the overall performance and scalability of your blog.

Tags:

OmniSQL 0.0.6 Released

August 13th, 2008 | No Comments | Posted in MySQL, OmniSQL

OmniSQL (a command line tool for DBAs needing to issue ad-hoc queries against sharded data) version 0.0.6 is officially released.

Instead of logging in separately to multiple databases to issue the same query, groups of databases can be specified in a configuration file and queries will be automatically issued against all targeted MySQL instances.

Let me know of any bugs found or features you would like to see in upcoming releases!

Download at http://code.google.com/p/omnisql/.

CHANGE LOG
- Fixed Bug #1: Multiple databases on the same host
- Fixed Bug #2: Multiple queries from STDIN
- Changed XML format of omnisql.cnf
- Added a README to show how to install and use

Tags: ,

The Query Performance Improvement Process

August 3rd, 2008 | No Comments | Posted in MySQL, MySQL Performance

The purpose of this post is to outline a general flow-chart for improving the performance of queryies in MySQL. Much has been written on using EXPLAIN to optimize queries, but there is a whole process that should be followed in order to maximize the effectiveness of query performance tuning. Following is a visual flow-chart of the process:

The Query Performance Improvement Process

Assuming that the problem query has been identified, the first question to be asked is:

1) Can the query be gotten rid of?

Surprisingly, the answer is often “yes”. As a result of the Rapid Application Development paradigm followed by many “Web 2.0″ companies, class interfaces are constantly changing and a data-set that used to be required, could no longer be. Equally as often, the query could be rolled up into another query with minimal effect on the other query’s performance. If the query can’t be gotten rid of or the data retrieved elsewhere,

2) Can the query be changed to select a smaller dataset with a more selective WHERE clause or eliminating columns that are returned but not used from the SELECT clause?

Some developers are trained to use SELECT * (link goes to an explanation of why “SELECT *” is evil) in all of their queries under the guise of “flexibility”, using perhaps only one or two columns. This can bog down the network if the unneeded columns are large, and eliminate the possibility of using covering indexes that could eliminate the need to read from disk when retrieving the result set. Additionally, a result can be made smaller by using a LIMIT clause if only a known few of the records will be used by the application. If neither of these approaches work,

3) Can the query be re-written?

If the query has a SUBQUERY, it might have more desireable performance characteristics if it were to be rewritten as a JOIN (tutorial here as well). Although the query re-writing process is outside the scope of this post, there are many good tutorials available elsewhere online.

4) Can indexes be added or changed?

Most people jump the gun and add/change indexes first, before trying to re-write the query. I do not suggest this because there are downsides to having too many indexes (slower INSERTs, poor use of memory, etc…). Some good rules-of-thumb for indexes are:

  • Index columns that are used in JOINs (typically Foreign Keys)
  • Use UNIQUE indexes when possible
  • Find places where you can use covering indexes
  • Ensure adequate selectivity on indexed columns
  • Use small datatypes (int vs. varchar)
  • Keep track of your data (selectivity can change over time)

5) Can the schema or code be changed?

Sometimes fixing a query is as simple as ensuring that joined columns have the same datatype or creating summary tables so that calculations are not performed each time the query is executed. More often, however, it can require that entire sections of code are re-written and data structures changed. This step is last in the list because it is usually the most time-consuming and error-prone.

Cache The Result

There are those who feel that caching a result (using memcached, for example) is a solution for badly-performing queries. I am not one of those people. If your cache servers die, for instance, wouldn’t you rather have the result be that the application performs more slowly, rather than becoming completely unresponsive? Only after your queries have achieved an acceptable level of performance should you explore the option of caching the results.

If you have gone through the entire process and been unable to fix the query, I would encourage you to repeat the process with a fresh set of eyes.

Tags: ,

Omni[My]SQL 0.0.5 Released

July 30th, 2008 | 1 Comment | Posted in MySQL, OmniMySQL, OmniSQL

The OmniMySQL project is now officially known as OmniSQL (for obvious potential legal reasons). I find myself using this tool more and more, so I thought it was about time to add some new features. The new project download site is now http://code.google.com/p/omnisql/. Let me know of any bugs and/or enhancements!

CHANGELOG:

0.0.5 - 30 July 2008

- Added the --list-groups command-line option
- Updated Documentation
- Added a version check on startup
- Changed the project name to OmniSQL
- Added support for piping commands through STDIN

Tags: , , ,

MySQL RSS Feeds

July 18th, 2008 | No Comments | Posted in FreeBSD, MySQL

Issue Five of MySQL Magazine just came out and it included the results from its Annual MySQL Usage Survey. Among a bunch of other cool information, it asked the question: “What is your favorite blog?” (Q26). The results were as follows:

MySQL Magazine Survey: What is your favorite blog?

The first interesting thing that I noticed was that the #1 result (with 100 votes) is not an actual blog, but rather a domain owned by a squatter (current asking price is $2,000USD). I assume that they meant http://www.mysqlperformanceblog.com/. So, in the spirit of sharing favorite blogs, I have compiled a list of all of the MySQL-related RSS feeds that I follow regularly. The list is maintained here.

http://dammit.lt/ – domas mituzas: vaporware, inc.
http://izoratti.blogspot.com/ – Ivan Zoratti’s Blog (on MySQL)
http://jpipes.com/ – Jay Pipes
http://mysql-dba.com/ – The MySQL DBA Feed Resource
http://www.planetmysql.org/ – Planet MySQL
http://www.mysqlperformanceblog.com/ – MySQL Performance Blog
http://feeds.feedburner.com/techrepublic/datacenter – Servers and Storage
http://ramonspage.blogspot.com/ – Ramon’s Page (MySQL Cluster)
http://mysql-ha.com/ – High Availability MySQL
http://blogs.sun.com/carriergrademysql/ – MySQL in Communications Blog
http://blog.plasticfish.info/ – Jon Stephens
http://mikaelronstrom.blogspot.com/ – Mikael Ronstrom
http://jonasoreland.blogspot.com/ – Jonas Blog (MySQL Cluster)
http://www.bcs.org/server.php?show=ConBlog.5 – Johnny’s Data Migration Blog
http://johanandersson.blogspot.com/ – Johan Andersson’s Blog (High Availability MySQL)
http://blog.kovyrin.net/ – Homo-Adminus Blog
http://highscalability.com/ – High Scalability
http://database-programmer.blogspot.com/ – The Database Programmer
http://crazydba.blogspot.com/ – Crazy DBA
http://www.xaprb.com/ – Baron Schwartz
http://datacharmer.blogspot.com/ – The Data Charmer
http://krow.livejournal.com/ – Brian Aker
http://www.jpipes.com/ – Jay Pipes
http://www.pythian.com/blogs/ – Pythian
http://jcole.us/blog/ – Jeremy Cole
http://www.pablowe.net/ – My Blog:)
http://www.honeysoftware.it/sleto/blog/ – Yabomsat! (Yet Another Blog On MySQL® Server Administration, monitoring and Tuning!)

MyMemCalc Version 0.0.2 Released

July 7th, 2008 | No Comments | Posted in Memcached, MyMemCalc

MyMemCalc, a command-line utility for basic reporting on Memcached Performance for groups of memcached servers, version 0.0.2 has been released.

ChangeLog:

* Added the --quiet command-line option to not display raw numbers
* Output now also includes Memcache Fill Ratio

Tags: ,

Memcached

July 6th, 2008 | 1 Comment | Posted in Memcached, MyMemCalc

The other day I was watching a Memcached Webinar (“Memcached for MySQL: Advanced Use Cases”) sponsored by MySQL, and it got me thinking about memcached from an operational perspective.

It seems to me that the implementation patterns for memcached are fairly well codified in the standard body of knowledge. What does NOT appear to be in the standard BoK is how to build a supporting infrastructure behind memcached to ensure that it is an effective and reliable component of the application architecture.

I often work with clients who have implemented memcached and seen great improvements in page load times and database load averages. However, most users have not taken the time to set up a supporting infrastructure like they would for webservers, database servers, or other traditional apps. So, I thought I would take a minute to talk about the MINIMUM actions one should take to support their memcache installations:

MONITOR

At a bare minimum, Memcached should be monitored to see if the process is running. There are many Nagios Plug-Ins available:

check_memcached.py from NagiosExchange
check_memcached from CPAN

NOTE: Different checks have different capabilities … evaluate and test for your specific needs

GRAPHING

Having visual graphs is very useful for trending over time and providing at-a-glance metrics. Cacti is a great tool for such, and there are readily-available templates for Memcached:

Templates from DealNews
MySQL Server Templates from faemalia.net

For those whose organization cannot or will not support Cacti or Nagios, one can use simple scripts such as mymemcalc to gather basic Memcached performance statistics.

REDUNDANCY/FAILOVER

The two most important questions in the Memcached FAQ are:

1) How is Memcached Redundant? (It isn’t)
2) How Does Memcached Handle Failover? (It doesn’t)

The redundancy problem cannot be fixed, except to ensure that a single (or multiple) memcache server failure will not render the application unusable.

The failover problem can be solved in a couple of ways:

1) Have a “hot spare” host available that can manually be turned on (with the same IP of the dead server) in the event of failure. This approach is very common and inexpensive to implement.
2) Have an active-passive failover with heartbeat, and automatically failover to the passive in the event of the primary server failing. This approach is more expensive and less used, but can result in less cache downtime in the long run.

WARMUP SCRIPTS

Some implementations may require memcache flushes on occasion. If the application has come to rely on memcache, it is possible that this can criple it until memcache is sufficiently populated with data as to alleviate the database load. The simplest way to warm up a memcache pool is to simply load the most popular pages of your site by using a crawler such as mycachewarmer. The loading process should automatically take care of the caching process. More advanced setups may require more custom scripts.

NAMESPACES

As mentioned before, pushing new features may require memcache invalidation for cached items affected by the code change. For example, if MySpace were to update Bulletins, they would probably want to invalidate all cached bulletin information without affecting any other features on the site. The easiest way to do this would be to invalidate all keys beginning with BULLETIN_*, but Memcached does not support namespaces. There are some common tricks to fake them, however:

1) Simulate Namespaces With Key Prefixes. This approach is simple, but can result in additional traffic to the memcache servers.
2) Have the application use separate cache pools for each feature. Continuing with the MySpace example, there would be a separate cache pool for bulletins, mail, comments, media, friends, etc… Although this approach has potentially more points of failure, it can make the feature-release process less taxing on the system as a whole.

CONCLUSION

Memcached can significantly reduce database load and increase application scaleability. With a little planning, it can also be among the most reliable and fault-tolerant tiers of an application.

Tags: ,

Changing The Exit Message in MySQL Client

July 2nd, 2008 | No Comments | Posted in MySQL

Sheeri Kritzer Cabral sent out a tweet yesterday wishing she could change the default MySQL exit text from “Bye” to “kthxbai!”. I figured that it was something that could be quickly done using a pager, but that turned out not to work because it is written after the pager would have been called. So I started digging around the source and found the following line in client/mysql.cc (line 1233):

put_info(sig ? "Aborted" : "Bye", INFO_RESULT);

and changed it to:

put_info(sig ? "Aborted" : "kthxbai!", INFO_RESULT);

So, Sheeri, this patch is for you:)

qubert:client rlowe$ bzr diff mysql.cc
=== modified file 'client/mysql.cc'
--- client/mysql.cc 2008-06-25 09:44:55 +0000
+++ client/mysql.cc 2008-07-02 02:28:19 +0000
@@ -1230,7 +1230,7 @@

#endif
if (sig >= 0)
- put_info(sig ? "Aborted" : "Bye", INFO_RESULT);
+ put_info(sig ? "Aborted" : "kthxbai!", INFO_RESULT);
glob_buffer.free();
old_buffer.free();
processed_prompt.free();

Tags: