Wednesday, January 23, 2013

Toward Memcached

This week at Flex we're going through release candidates for version 4.6.X which includes a number of big performance enhancements we recently made to improve speed for high throughput systems or companies that use large quotes.  It's taken longer than expected because we decided to be somewhat aggressive with the performance tuning and tuning of this kind always has ripple effects.  The last week and half or so has been consumed with fixing new issues raised by QA.  Yesterday the new version dropped to QA for regression testing.

This means the new version is feature complete, all known bugs have been fixed and the software is currently in regression resting.  This is the normally the last stage in our release process, but this time we're going to add a brief period of beta testing to test the new version's memory footprint in a production environment.  The reason for this is that one of the key strategies we used to improve performance was to expand the Hibernate Second Level cache.  This made the performance numbers much better, but a potential downside of this strategy is that the increased memory usage could - in theory - exceed available heap, increase garbage collection overhead, and increase swap usage on the servers and all the potential virtual memory paging that goes along with it.

A Distributed Cache

One of the unique conditions for this beta test is that we already know what we'll do if we run into memory problems on the production servers, so there's no need to sit around and wait for the issue to crop up.  We also know that the solution to this hypothetical memory issue is also something we'll have to do once Flex moves to a fault tolerant high availability architecture later this year - which is to use a distributed cache.

This means that the memory we need for caching is offloaded to another server or cluster of servers.  This solves the problem of running out of memory on a local server and it deals with the concurrency issues you'd run into on a multi-server architecture.

There are a number of options for a distributed cache in Java, notably Terracotta, but we decided to go with a much simpler option called memcached - which we'll use via Amazon's Elasticache service.

Memcached

Memcached is a dirt simple in memory cache server originally developed for LiveJournal and later extended and refined by Facebook.   To this day memcached plays a huge role in Facebook's architecture with over 800 memcached servers in production.  You can read more about Facebook's experience with memcached here:

Once the current release of Flex was feature complete, we decided to get a jump start on memcached integration while the release wound its way through QA.  The biggest stumbling block for integrating Flex with memcached turned out to be the lack of an existing memcached library for Hibernate 4 - we had to build it.

Flex Alto

We also had the problem of needing to support customers, particularly those who self host, who don't want or need to use memcached.   We needed an architecture that supported seamlessly switching between an on heap cache implementation and memcached or anything else that might come along.  To support this, we added a number of new features to an open source library we sponsor called alto.

(Feel free to explore the source code on github here.)

We needed three basic features to make memcached work with our system and to support interchangable caching strategies:

  • The ability to switch caching implementations via a JNDI injected parameter.
  • A simplified caching abstraction so we code to an abstraction instead of a proprietary caching architecture.
  • Glue between that caching abstraction and Hibernate 4.

Pluggable Cache Implementations

The first one was easy.  We created a new Spring factory bean that takes a bean id as a property that can easily be injected via JNDI and returns the Hibernate Region Factory (cache implementation) that corresponds with that id when needed.  Source here.

Simple Caching Abstractions

Next, we wanted a simplified abstraction for the generic idea of a cache, one that came without the complexity of JSR-107.  For that we created a simple interface called AltoCache:


public interface AltoCache {
public Object get(String region, String key);
public void put(String region, String key, Object value);
public boolean isCached(String region, String key);
public void remove(String region, String key);
public void clear(String region);
public AltoCacheStatistics getStatistics(String region);
public AltoCacheStatistics getStatistics();
public void get(String region, String key, Future<Object> callback);
public void put(String region, String key, Object value, boolean async);
public CacheKeyGenerator getKeyGenerator();
}
As you can see from the interface, it's a pretty simple key/value pair abstraction with some asynchronous get/put support.

We then created implementations of this interface for a simple in memory HashMap, EHCache, and memcached.

Hibernate 4 Integration

We then took the caching API defined by Hibernate 4 and developed an implementation targeting the AltoCache interface.  It's a complicated API, so this took quite a bit of time and testing to get up and running.

Perhaps the two biggest issues that came up during the implementation process relate to Hibernate's concept of regions and how read/write locks would work in a distributed architecture.

For the first issue, Hibernate likes to divide the cache into sections of related cache entries called regions and Hibernate relies on this region concept to evict entire cache regions that for some reason or another it believes are stale.  Trouble is, memcached doesn't support the idea of key namespaces or regions, so we had to use the memcached append method to maintain a list of keys that define each region, effectively creating virtual regions.  It feels a little ugly to me, but there's no other way to handle it until memcache introduces a region concept, which given the philosophy behind memcached, particularly how it uses key hashes to distribute keys across various servers, seems unlikely.

Local Locks, Distributed Locks

The final issue pertaining to mutex locks during cache reads and writes is a pretty easy problem to solve for now, since we only need local locks.  But we also know that we'll eventually need a more complicated locking mechanism that works on a multi-server architecture.

We first created a simple lock abstraction that can be plugged into our Hibernate 4 Cache implementation or used on its own...

public interface LockProvider {
public String lock(String lockId);
public boolean unlock(String unlock);
public boolean isLocked(String lock); }
Then we created a simple implementation using Java's reentrant locks for use when locks are local in scope - and that's good enough for now.

But anticipating what's coming down the pike with a multi-server architecture, we stubbed in a simple telnet based lock service we called lockd.  The idea is that something with same simple approach used for memcached adapted for mutex locks would be a wonderful thing.  We designed lockd to be easily embedded in an application via Spring for small clusters where the locks replicate to each node in the cluster.  Ultimately, though, we'll wait for another lock service to emerge or develop a C++ implementation that supports the protocol.  We may even fork memcached to start and modify it to be a lock server.

Pragmatically, we really shouldn't be in the distributed lock service business and we hope a propeller head created lock service comes along or our friends at AWS introduce a lock service.

In Conclusion

We know it's been a long wait for the new release.  We really are close now and it should be out of QA later this week or early next week.  We'll immediately beta test it on one of our servers (we're planning on testing it on one of our European Union servers) and get a feel for whether or not memcached will be required for general deployment.

In the meantime, we've gotten a head start on memcached integration and full regression testing of Flex backed by memcached will start as soon as the current release clears QA.

After this release gets out and is running stable, we'll get back to clearing out fast track issues and finishing up our new crew scheduling system.

No comments:

Post a Comment