The other day I was watching a Memcached Webinar (“Memcached for MySQL: Advanced Use Cases”) sponsored by MySQL, and it got me thinking about memcached from an operational perspective.
It seems to me that the implementation patterns for memcached are fairly well codified in the standard body of knowledge. What does NOT appear to be in the standard BoK is how to build a supporting infrastructure behind memcached to ensure that it is an effective and reliable component of the application architecture.
I often work with clients who have implemented memcached and seen great improvements in page load times and database load averages. However, most users have not taken the time to set up a supporting infrastructure like they would for webservers, database servers, or other traditional apps. So, I thought I would take a minute to talk about the MINIMUM actions one should take to support their memcache installations:
MONITOR
At a bare minimum, Memcached should be monitored to see if the process is running. There are many Nagios Plug-Ins available:
check_memcached.py from NagiosExchange
check_memcached from CPAN
NOTE: Different checks have different capabilities … evaluate and test for your specific needs
GRAPHING
Having visual graphs is very useful for trending over time and providing at-a-glance metrics. Cacti is a great tool for such, and there are readily-available templates for Memcached:
Templates from DealNews
MySQL Server Templates from faemalia.net
For those whose organization cannot or will not support Cacti or Nagios, one can use simple scripts such as mymemcalc to gather basic Memcached performance statistics.
REDUNDANCY/FAILOVER
The two most important questions in the Memcached FAQ are:
1) How is Memcached Redundant? (It isn’t)
2) How Does Memcached Handle Failover? (It doesn’t)
The redundancy problem cannot be fixed, except to ensure that a single (or multiple) memcache server failure will not render the application unusable.
The failover problem can be solved in a couple of ways:
1) Have a “hot spare” host available that can manually be turned on (with the same IP of the dead server) in the event of failure. This approach is very common and inexpensive to implement.
2) Have an active-passive failover with heartbeat, and automatically failover to the passive in the event of the primary server failing. This approach is more expensive and less used, but can result in less cache downtime in the long run.
WARMUP SCRIPTS
Some implementations may require memcache flushes on occasion. If the application has come to rely on memcache, it is possible that this can criple it until memcache is sufficiently populated with data as to alleviate the database load. The simplest way to warm up a memcache pool is to simply load the most popular pages of your site by using a crawler such as mycachewarmer. The loading process should automatically take care of the caching process. More advanced setups may require more custom scripts.
NAMESPACES
As mentioned before, pushing new features may require memcache invalidation for cached items affected by the code change. For example, if MySpace were to update Bulletins, they would probably want to invalidate all cached bulletin information without affecting any other features on the site. The easiest way to do this would be to invalidate all keys beginning with BULLETIN_*, but Memcached does not support namespaces. There are some common tricks to fake them, however:
1) Simulate Namespaces With Key Prefixes. This approach is simple, but can result in additional traffic to the memcache servers.
2) Have the application use separate cache pools for each feature. Continuing with the MySpace example, there would be a separate cache pool for bulletins, mail, comments, media, friends, etc… Although this approach has potentially more points of failure, it can make the feature-release process less taxing on the system as a whole.
CONCLUSION
Memcached can significantly reduce database load and increase application scaleability. With a little planning, it can also be among the most reliable and fault-tolerant tiers of an application.
Tags:
Memcached,
MyMemCalc 