App Fabric cache Issue – Local Cache items are evicted using IIS

29 בJanuary 2014

Local Cache

One of the best features of app fabric cache is Local cache(near cache). The local cache is “in process” (in memory) cache of items that was recently used. The great thing is how items in this cache are invalidate. The local cache has two methods:

  1. Timeout base – trivial
  2. Notification base – meaning each time interval (default is 300 ms) the local cache will get a list of modification made on items from the distributed cache and invalidate only those who changed. (Note: the notification should be enable in the named cache side)

The notification method is pretty awesome because it means we have transparent use of cache that enables local cache that is sync in resolution of 300 ms which is pretty good.

Read more on local cache here.

The Problem

Recently one of my costumers encountered an issue regarding the local cache (Near cache) of app fabric on premise version (not Azure). He notice that the worker process (w3wp.exe)has high CPU usage and after debugging we discovered that most of the time the process was spending on deserialization of objects that was suppose to be in the local cache. I had two theories for it to happen:

  1. Items are invalidate in the local cache – maybe updated by other process.
  2. Items are evicted because we reached the max items in the cache (10000 items by default)

both of them were overruled:

  1. The specific items were pretty static and no other process is updating them
  2. number of items in the cache was ~5000 not even close to the default – 10000

So I had no choice but to enable tracing of app fabric and I got this strange message: “Memory pressure was detected in the client”. I checked the machine memory usage: there was around ~5G RAM free memory – so why we got this strange memory pressure message. In this phase I decided to dig deeper and looking in app fabric cache code using reflector tool – (I use Dot Peek from Jet brain which is great) and this is what I came up with (on the assembly Microsoft.ApplicationServer.Caching.Client.dll):

  • The class ClientMemoryPressureMonitor – which gets delegate and has internal timer and each time interval check memory level and in case it is high execute the delegate
  • The class EvicationManager is using the class above passing it the OnMemoryPressure method which evict items in the local cache
internal void OnMemoryPressure(ClientMemoryPressureLevel memPressure)
  {
    if (!this.TakeEvictionInProgressLatch())
      return;
    try
    {
      long num = this._localStore.ActualItemCount();
      this._localStore.CountLocalCacheItem = num;
      long countItemsToEvict = (long) ((double) num * ((double) this._evictionProp.PcntCleanup / 100.0));
      if (countItemsToEvict <= 0L)
        return;
      long lwmCountLocalCacheItem = num - countItemsToEvict;
      this.EvictExpireCompact(countItemsToEvict, lwmCountLocalCacheItem);
    }
    finally
    {
      this.ReleaseEvictionInProgressLatch();
    }
  }
  • The internal class MemoryPressureStatus is implementing the method GetMemoryPressure
public ClientMemoryPressureLevel GetMemoryPressure()
   {
     ulong num = (ulong) Process.GetCurrentProcess().VirtualMemorySize64;
     if ((int) (num * 100UL / this._memoryLimit) > this._highMemoryMark)
     {
       if (Provider.IsEnabled(TraceLevel.Warning))
         EventLogWriter.WriteWarning("DistributedCache.ClientMemoryMonitor", "Private Bytes {0}, Memory Limit Mark: {1}", (object) num, (object) this._memoryLimit);
       return ClientMemoryPressureLevel.High;
     }
     else
     {
       if (Provider.IsEnabled(TraceLevel.Info))
         EventLogWriter.WriteInfo("DistributedCache.ClientMemoryMonitor", "Private Bytes {0}, Memory Limit Mark: {1}", (object) num, (object) this._memoryLimit);
       return ClientMemoryPressureLevel.Low;
     }
   }
  • The memory measured is the Virtual Memory and not private bytes

This gave me the full picture, and by using performance monitor I notice that worker process of IIS was taking 19G of virtual memory (I don’t know why and if it’s bad) .This cause the items to get evicted from the local cache.

The Solution – Microsoft support Works Smile

Understanding the problem was very important stage. Then we contact Microsoft support for help with the specific details of the problem. I tried google it a bit without any luck but Microsoft support was familiar with the issue and quickly supplied  the solution: cumulative update package 4 – http://support.microsoft.com/kb/2800726

Issue 1: The local cache feature of AppFabric Caching may not work correctly
The local cache feature of AppFabric Caching does not work correctly in some scenarios. Objects from the local cache are evicted aggressively.
Cause
Memory pressure-related eviction is based incorrectly on virtual bytes.
Resolution

After you apply this hotfix, local cache memory pressure-related eviction is changed to be based on private bytes.”

So this was the solution GetMemoryPressure was fixed to be based on private bytes.

Notes

Pay attention the update required .NET 4.5:”To apply this cumulative update package, you must have AppFabric 1.1 for Windows Server installed. Additionally, you must have the Microsoft .NET Framework 4.5 installed.”

This my be problematic in some cases – so although it may be risky you can modify only consumer side app fabric assemblies without applying the update to App fabric Server.

I tested it and it worked Smile (still not sure if there are any side affects and we didn’t get some of the hot fixes the update contains)

Summery

I hope this post will help others – because Local cache not working can greatly effect the application performance. One more important thing is don’t take anything for granted – Even a product that is mature (in version 1.1 and pretty old) can have bugs – in our case because the use of local cache is transparent we can think we are using it without really using it.

Cheers Offir.

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*