Circuit Breaking

I'm doing a fun little Rust project today on my Home DNS server. I thought I would try and blog about doing this at the same time that I'm writing the code, might get a little weird.

The service caches lookup results and has the option to use a Memcache server, or by default a local Moka cache. Both of these implement the same trait so that the implementation is transparent to the callers. Pattern is some pretty basic OOP stuff. First we have the trait, I'll only include one method for brevity.

#[async_trait]
pub trait DNSCache {
   async fn get_lookup(&self, name: &Name, record_type: &RecordType) -> Option<Vec<Record>>;
}

Then we have a remote (Memcache) implementation and a local (Moka) one. The contract of both of these is the same. Moka supports TTLs on inserts in the same way that Memcache does.The main differences are in the instantiation of the underlying cache. The provider does a simple switch on the configuration to determine which version of the cache to hand to a caller.

#[derive(Clone)]
pub struct CacheProvider {
   cache: Arc<dyn DNSCache + Send + Sync>,  
}


impl CacheProvider {
   pub fn new(config: &Option<MemcacheConfig>) -> anyhow::Result<Self> {
       match config {
           None => Ok(Self {
               cache: Arc::new(LocalInMemoryCache::new()),
           }),
           Some(config) => Ok(Self {
                cache: Arc::new(RemoteDNSCache::new(config)?),
           }),
        }
   }


   pub fn get_cache(&self) -> Arc<dyn DNSCache + Send + Sync> {
       self.cache.clone()
   }
}

Having the provider in place gives me a good spot to put in the circuit breaking logic. The plan is, if the Memcache impl starts failing the code should hand out the Moka version. The circuit breaker pattern is a pretty well worn so I went shopping for a crate and picked the first the thing that looked good circuitbreaker_rs. After that I made some changes to the provider so that it held two version of the cache.

#[derive(Clone)]
pub struct CacheProvider {
   local_cache : Arc<LocalInMemoryCache>,
   remote_cache: Option<Arc<RemoteDNSCache>>,
}

Next I'm going to integrate circuitbreaker-rs into the RemoteDNSCache. So that if the call fails, the breaker is keeping track of the state. Turns out this was super easy to do. The client caller code goes from :

   async fn get_lookup(&self, name: &Name, record_type: &RecordType) -> Option<Vec<Record>> {
   ...
       let cached: Result<Option<Vec<u8>>, MemcacheError> = self.client.get(key.as_str());
   ...
}

We just need to wrap the breakable call with the circuit_breaker :

   async fn get_lookup(&self, name: &Name, record_type: &RecordType) -> Option<Vec<Record>> {
   ...
   let cached : BreakerResult<Option<Vec<u8>>, MemcacheError> = self.circuit_breaker.call(||
           self.client.get(&key)
   );
   ...
   }

This is where I ran into an issue with the design of my program. The naive way I was using this in the provider was to check the state of the circuit and return the cache impl that was appropriate. Like so

match remote.circuit_breaker_state() {
   State::Closed | State::HalfOpen => {
       remote.clone()
   }
   State::Open => {                       
       warn!("cache circuit is open, returning local");                                          self.local_cache.clone()                   
   }               
}

I tested this out by running the service and then stopping Memcached. This yielded a couple good findings.

First, I had my connection, read and write timeouts too high. This caused the circuit to fail too slowly. In practice these should all be a second so the circuit will flip much more quickly when there are issues.

Secondly the circuit would flip but wouldn't flip back when Memcached became available again. The issue should have been pretty obvious but without exercising the call() method of the circuit breaker it won't advance its internal state. A way forward is for the provider to become more of a facade, delegating to the appropriate cache impl. After doing that things started working as expected. What the new code looks like with the provider implementing the cache and handling the delegation of which underlying cache to use.

async fn get_lookup(&self, name: &Name, record_type: &RecordType) -> Result<Option<Vec<Record>>, anyhow::Error> {
       match self.remote_cache.as_ref() {
           None => {
               self.local_cache.get_lookup(name, record_type).await
           }
           Some(remote) => {
               let result = remote.get_lookup(name, record_type).await;
               match result {
                   Ok(value) => {
                       Ok(value)
                   }
                   Err(error) => {
                       error!(error = %error, "get lookup error");
                       self.local_cache.get_lookup(name, record_type).await
                   }
               }
           }
       }
   }

This works as you'd expect. The code needs to call the remote.get_lookup(...) to advance the circuit breaker but it will get an immediate error when the circuit is open.

This was an interesting exercise. Writing what I was doing while I was doing it didn't seem to slow me down that much. This post probably doesn't make a ton of sense to anyone but me, but luckily nobody really reads these posts but me :)