NHibernate Access Performance

*1/5/2008 Update - Source code is now available for download if you would like to test these findings yourself. Download Here*

*11/19/2007 Update - It turns out my instincts were correct. Upon further review of the code the CodeDom field getter/setter was actually using the CodeDom property getter/setter. I had a very hard time understanding how the CodeDom reflection optimizer improved private field level access, now it turns out that it did not. An updated chart has been posted at the bottom of this article *

Recently I was involved in a discussion on the NHibernate forums regarding how to implement the null object pattern which later moved to a discussion regarding the performance impact of such a pattern. I have been told many things regarding the performance impact of reflection and more specifically the performance impact of reflection access of a property versus a field, but I have never actually researched these items myself. I finally took the time to closer examine the impact of NHibernate access mechanisms, and the results really surprised me!

I've always been told that accessing private members is significantly slower than accessing public members (due to Code Access Security checks). Because of this I used to prefer to access properties instead of private fields. However, after noticing that even with a very large application the reflection impact of field access versus property access wasn't noticeable I shifted gears to believe that all items which should not be settable from code should rely on NHibernate's nosetter access mechanism. Basically why expose a setter for your object's Id property when it is an identity column that only NHibernate should ever populate? This makes our code safer and helps ensure that someone who is new to NHibernate does not try to set that id value due to a misunderstanding of how this new O/RM paradigm works.

Test Overview

I figured it was about time to do some actual tests which showed the impact of using public property reflection versus private field reflection. I decided to write a small .NET console application which would include a simple type which contained a single private field and a single public property which wrapped that field. See the below class:

public class SampleObject
{
private int val;

public int Val
{
get
{
return val;
}
set
{
val = value;
}
}
}

I then chose the mechanism I would use to measure the relative performance. I decided on using threading to allow both property access and field access to run at exactly the same time. I figured that I did not want my test results to be impacted by unknown differences in system environments while the tests were running. I figured if both tests are running at exactly the same time both accessors would be subject to exactly the same system constraints. I determined that I would start two threads, and then immediately call Join on both threads from the main application thread which would allow me to evaluate the differences. Note that I calculate the processing time within each thread to ensure the time is as accurate as possible.

Now that I know how I will compare my field access versus property access I had to determine the metrics I wanted to measure. NHibernate 1.2 provides a few options regarding how it should access or set the values of fields and properties. The tool will always use reflection, but there are actually multiple ways NHibernate can use reflection.

1) Basic Getter/Setter
NHibernate's first and most basic mechanism for accessing/setting property values is via the NHibernate.Property.BasicGetter and NHibernate.Property.BasicSetter classes. Basically these classes wrap up a System.Reflection.PropertyInfo instance and then use that PropertyInfo's GetValue and SetValue methods to get or set the value via reflection. This is probably the mechanism most developers have used to get/set values via reflection (if they have used reflection).

2) Field Getter/Setter

NHibernate offers NHibernate.Property.FieldGetter and NHibernate.Property.FieldSetter to provide the basic getter/setter functionality to fields as well as properties. This works the same as above but uses the System.Reflection.FieldInfo class instead of the PropertyInfo class.

3) CodeDom Reflection Optimizer

Due to the performance impact of using reflection to get/set values NHibernate introduced the concept of "reflection optimizers". Basically NHibernate will build a custom class which it will use to access the field/property value of your object and then access the value via a delegate to this newly created accessor method to allow NHibernate to bypass reflection for each access.

The CodeDom mechanism of reflection optimization is used to support .NET 1.1. NHibernate writes it's own C# code in code which wraps up your property/field access and then runs this dynamically generated C# code through the built in System.CodeDom.Compiler.CodeDomProvider class or more specifically the Microsoft.CSharp.CSharpCodeProvider class which is used to generate a C# code compiler and then compile the dynamically written C# code. After compiling NHibernate uses reflection to dynamically create an instance of the new class and then proceeds to use that class for it's access.

4) Lightweight Reflection Optimizer

this technique of reflection optimization is very similar to the above mentioned CodeDom optimization except that it does not require a C# compiler. I believe it is called lightweight since it does not incur the overhead of the compiler. Instead this technique relies on the System.Reflection.Emit.ILGenerator class to dynamically build the accessor class. This basically skips the compiler and provides the output which the above compiler would provide.

Testing Code

For all tests I used the NHibernate types IGetter, ISetter and IReflectionOptimizer to ensure that my code followed the exact same code path that users of NHibernate can expect.
Now that I have my testing technique

For my testing loops I used the following code:

public void TestGet()
{
object value;
   DateTime begin = DateTime.Now;
   for (int i = 0; i < NUM_LOOPS; i++)
{
  value = getter.Get(obj);
}
Time = DateTime.Now - begin;
}

public void TestSet()
{
object value;
   DateTime begin = DateTime.Now;
   for (int i = 0; i < NUM_LOOPS; i++)
{
  setter.Set(i);
}
Time = DateTime.Now - begin;
}

And then to run the actual tests I needed the following code:

Thread propertyThread = new Thread(propertyContainer.TestGet);
Thread fieldThread = new Thread(fieldContainer.TestGet);

propertyThread.Start();
fieldThread.Start();

propertyThread.Join();
fieldThread.Join();

The prior loop is written for each case passing in the appropriate Getter/Setter.
After each run the results are output to the console window for me to analyze.

Test Results

Now for the good stuff! How did the results turn out? First for reference I ran this test on a slightly outdated computer (Athlon XP 3700+ 2GB Ram Vista Ultimate) with release code. The results I found were not at all what I expected. Note that for each scenario I ran the methods through 100,000 loops so the times shown are the time it takes (in milliseconds) to get/set a field/property value 100,000 times and with two simultaneous threads. Each bar graph pair was run simultaneously. See the graph:

	Property (msecs)	Field (msecs)
Basic Getter	524.38	391.58
Basic Setter	671.83	505.83
Lightweight Getter	13.67	27.34
Lightweight Setter	29.30	15.62
CodeDom Getter	15.62	25.39
CodeDom Setter	35.15	13.67

Now, the first thing that jumps out at you is that the Reflection Optimizer strategies are significantly faster than the non optimized techniques. Well duh, it has optimize in the name right? That was to be expected. I don't know that I thought it would make this much of an impact, I expected something more along the lines of 3 times faster, but not 38 times faster! This part was really encouraging.

The part that threw me for a loop was that when using basic reflection the private field access was faster than the public property access? I never would have guessed that. Haven't we all been hearing about how slow private field access is for a long time?

Given this I can't possibly see why someone would avoid using field level access. With this sort of performance it seems to provide the cleanest access mechanism to hydrate your objects and then to analyze and save the changes. From now there is no way I'll put a setter on a property where it does not make sense from a public API perspective. I feel like the results of these test lifted some chains off my shoulders. (OK, very light chains made from plastic, but chains none the less!)

Is anyone else surprised by these results? I find them encouraging, but if I was making predictions before running the tests, this is not what I would have expected.

If anyone would like the full source code (it's about 210 lines of code total) leave me a comment and I will e-mail it to you.

--John Chapman

*11/19/2007 Update - Below is an updated chart that includes the correct data. It turns out that a bug in the code caused the CodeDom field getter/setter to use the property getter/setter instead. The results now make a lot more sense. I'm sorry for any confusion.

Note that the tests had to be re-ran and hence the numbers came out slightly differently this time. Also note that the items had to be abbreviated (LW = LightWeight and CD = CodeDom). Take note that the CodeDom optimizer does nothing for field level access, which is what I originally expected (since after reviewing the code I saw that for non Basic Getter/Setters it just calls the provided IGetter/ISetter.

With this run I also added a direct property access getter and setter as a base line. This helps to show the true performance of using reflection (or the reflection optimizers of NHibernate). Note that no direct value is provided for the field since direct access of a private field is not possible.

*For Updated Results with 10,000,000 Loops see follow up post: NHibernate Access Performance Revisited.

--John Chapman