Why I Am Sick Of Hearing About Deferred Execution

Since the announcement of LINQ we've heard plenty about "deferred execution", this term that has appeared like its some sort of LINQ magic feature.  Personally, I think I need to come up with my own term and claim it's something awesome too.  I'm really tired from hearing about it.

On Wednesday, July 15th I went to a Great Lakes Area .NET Users Group talk by Bill Wagner where he was talking about Extension Methods and how to make proper use of them.  Now, don't get me wrong, I have a lot of respect for Bill.  I don't mean to criticizing Bill in any way.  So Bill, if you read this, I really don't mean any disrespect by this.  It was simply your use of the term that made me recall my feelings on this topic.

Bill was doing a demo where he showed various LINQ extension methods and showed that by making use of these extension methods we were able to harness the power of DFERRED EXECUTION! 

The first example Bill showed was Enumerable.Range(Int32, Int32) where it returns an IEnumerable<Int32>.  Bill then shows that when he calls the Take() extension method it only iterates through the first x of the items in the range, not the full list of items identified by the range.  Ok yes, this is true.  We didn't have to create a new list and populate it with a million items, just to pull the first 5 items.

Bill later went on to discuss how if you use a LINQ query with variables, you can change those variables after you have defined the query.  His code looked something like the following:

var range = Enumerable.Range(0, 1000000);

var maxValue = 40;

var items = from r in range
            where r < maxValue
            select r;

var takenItems = items.Take(30);

maxValue = 20;

foreach (var i in takenItems)
{
  Console.WriteLine(i);
}

Output:

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Now yes, you define your LINQ query, change your variable after the fact and then consume that class.  Yes, it takes into account the change in your variable.  Yes, this occurs after you defined your query, so deferred execution is a term that makes sense.

Ok, I'll give in a bit, I'm ok with the term, but not the way its talked about.  The magic isn't LINQ, and understanding what is going on is not just about understanding LINQ.  It's the fundamentals of how LINQ works which people should really understand.

I'm going to say this one more time before I move on "Deferred Execution is not a LINQ feature".  It's a closure feature/implementation pattern.

First let me try to explain the implementation pattern piece by creating my own "Deferred Execution" code which works exactly the same way as as the Range method Bill demonstrated. (Note that this is not necessarily built with production quality in mind).

public class MyRange : IEnumerable
{
    private class RangeEnumerator : IEnumerator
    {
        private int? _current;
        private bool _complete = false;
        private readonly int _minValue;
        private readonly int _maxValue;
        
        public void Dispose()
        {
            
        }

        public bool MoveNext()
        {
            if (_current == null)
            {
                _current = _minValue;
                return true;
            }
            
            if (_current < _maxValue)
            {
                _current += 1;
                return true;
            }
            else
            {
                _complete = true;
                return false;
            }
        }

        public void Reset()
        {
            _current = null;
            _complete = false;
        }

        public int Current
        {
            get
            {
                if (_current == null || _complete)
                {
                    throw new InvalidOperationException();
                }

                return _current.Value;
            }
        }

        object IEnumerator.Current
        {
            get
            {
                return Current;
            }
        }

        public RangeEnumerator(int minValue, int maxValue)
        {
            _minValue = minValue;
            _maxValue = maxValue;
        }
    }

    private readonly int _from;
    private readonly int _to;

    public IEnumerator GetEnumerator()
    {
        return new RangeEnumerator(_from, _to);
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    public MyRange(int from, int to)
    {
        _from = from;
        _to = to;
    }
}

That's actually really simple code isn't it?  There is nothing revolutionary in that code.  Any one of us could have implemented that in C# 1.0. 

Now, let's look at a case with closures.  LINQ internally is using closures (via lambda expressions) to perform its queries.  So lets say I write my own closure. 

var range = new MyRange(0, 1000000);

var maxValue = 40;

Func expression = i => i < maxValue;

maxValue = 20;

foreach (var i in range)
{
    if (!expression(i))
    {
        break;
    }
    else
    {
        Console.WriteLine(i);
    }
}

Output:

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Huh, wouldn't you guess it, it also shows this magical LINQ "deferred execution" behavior.

So what's the point of all this?  One, I'm probably too easily set off on topics like this.  Second, we shouldn't look at "deferred execution" as some sort of LINQ magic but rather a pattern that can provide us many benefits with our own code.  Deferred execution allows us to enhance performance and flexibility of our applications.  This is something we can all make use of in our algorithms, even if we aren't utilizing LINQ.

And in regards to the Extension method talk by Bill, I really enjoyed it.  It was simple enough for people to learn about new C# 3.0 features.  You talked about it well and gave good examples.  I'm just frustrated that people seem to write this stuff off as magic even though they are simple concepts.  Plus, this term seemingly just appeared with LINQ even though the concept has been around for a long time.

Subtle Bugs When Dealing With Threads

Pop quiz, what's wrong with the following code?

public void Unsubscribe()
{
	if (_request != null)
	{
		ThreadPool.Enqueue(() => _service.Unsubscribe(_request));
	}
}

public void Subscribe(string key)
{
	Unsubscribe();

	if (!String.IsNullOrEmpty(key))
	{
		_request = new Request(key, handler);
		ThreadPool.Enqueue(() => _service.Subscribe(_request));
	}
}

Does everyone see the issue? There is a critical bug in the above code which isn't always readily apparent.

Try to find it...

I actually wrote code like this today (same concept, different implementation) and immediately saw some serious defects. Honestly, I'm lucky the issues popped up right away, these sorts of things tend to not appear right away, but jump up to bite you at a later point.

In this case the issue is the use of closures. When using a closure it copies the fields from outside the lambda expression into the expression meaning that my use of _request from within the lambda is actually the same reference that exists outside the lambda expression. So in the above case the Unsubscribe lambda gets executed on a new thread (from the pool) but by the time it actually executes the _request has already been changed.

In this case, you're actually unsubscribing from a request that most likely hasn't even been subscribed yet. And to top it off you haven't unsubscribed from the old request yet either. Obviously the above is a race condition where the exact output isn't guaranteed. There is a chance it works perfectly (though doubtful with a true thread pool). There is a chance the new request is subscribed first and then immediately unsubscribed as well.

The simplest way to resolve this issue is by changing the variable which is captured from one which is shared between both closures to one that is unique to each closure. As shown here:

public void Unsubscribe()
{
	if (_request != null)
	{
		var localRequest = _request;
		ThreadPool.Enqueue(() => _service.Unsubscribe(localRequest));
	}
}

public void Subscribe(string key)
{
	Unsubscribe();

	if (!String.IsNullOrEmpty(key))
	{
		_request = new Request(key, handler);
		var localRequest =_request;
		ThreadPool.Enqueue(() => _service.Subscribe(localRequest));
	}
}

However, what I really want to solve this type of problem going forward is to develop something which is process aware, much like the Saga in NServiceBus. Of course my goal is not to be a long running, persistable process like the Saga in NServiceBus, but the process portion is what I'm looking at.

StackOverflow.com, Uh oh?

So I mentioned in my last post that I have began listening to podcasts.  I have a lot of respect for both Jeff Atwood (Coding Horror) and Joel Spoelsky.  So when I saw they were working together on a new project and publishing their conversations regarding their new product, I figured I had to listen.  Now, they've posted around 9 episodes now, but I've only had a chance to listen to the first couple so far.

Honestly, I'm a bit concerned by what I heard.  In their first episode I felt like they gave Microsoft technology based developers a really bad name.  Now it may well be largely true (which is probably part of my concern), but I wish there were more resources to correct this bad name, rather than encouraging it.

Basically, during the podcast you will hear something along the lines that Microsoft technology developers basically resort to the Google-Copy & Paste programming development.  Microsoft technology developers are called pragmatic in that they don't care what the right solution is, or how clean it is, or well it works so long as it does work. 

Now, I'm not saying that using google to find answers to interesting problems is a bad thing.  I'm not even saying that if you ever copy and paste code you're a bad developer, but ideally the developer is learning from the blog post instead of just finding something which seems to work and moves about their business.  Honestly, typically these samples you find in blog posts are not thorough enough for a true production deployment.  The point of these postings should be to educate people about new concepts, not try to do their job for them.

Stackoverflow.com from what I gathered wants to be the place to replace google as the first place where you search for doing your job.  Now they stated their goal is to be the first hit on google for all of your searches, but really I think they would be happier if you went straight there instead of google.

Now Stackoverflow is not Microsoft specific, it is meant to appeal to developers on all platforms.  However, they seem to be looking at the Microsoft centric market as their main target.  Honestly, I think these guys will be successful.  They both have large followings, and I think there is huge demand for systems that can essentially do their jobs for them.  I just wish it appeared to be a more helpful resource that helped developers grow, instead of just allowing them to get by.

This all being said, the podcast is worth checking out.  These are two extremely intelligent people, and by listening to these podcasts you essentially get a look inside their heads and how they think.  I don't have to agree with Stackoverflow.com, or the topic which they discuss, I'm still able to learn from it while I listen.  To both Joel and Jeff, thank you for posting your phone conversations as podcasts, it has been a great learning experience for me.

Deep Fried Goodness

So I realize I'm going to look like a bit of a sellout based on my procrastinating, but I really meant to write this earlier.  With my newly purchased iPhone and my increased amount of travel, I've recently started listening to Podcasts.  I honestly never saw the point before.  I rarely get an hour where I can really listen to a podcast.  I have always thought of reading as being a simpler and more effective mechanism for learning.  However, while traveling (especially on a plane) I find that a properly timed podcast can provide a lot of information that otherwise I wouldn't be able to consume.

I saw that Keith Elder (and Chris Woodruff) had a new podcast called Deep Fried Bytes, and I figured I may as well see what it is.  I'm actually one of those people that first met Keith because I recognized his picture from his blog.  Not knowing really what was good for pocasts (besides the obligatory Hanselminutes and DNR) I figured it was worth a shot.

After listening to their episode on interview war stories I was really impressed.  They had some really intelligent people talking about interviewing.  This was a topic, which I have to admit, was not something that immediately peaked my interest.  But what you find is that when many smart people sit down to have a talk, something good will result.  Now, after I picked myself up off the floor from hearing a C# MVP call the using keyword "Obsolete", I realized that they have a winning format.

Plus honestly, Keith Elder is the kind of guy where he doesn't need to have anything good to say.  The way he talks and presents himself can be entertaining almost regardless of the topic.  If you listen to podcasts I recommend you go try these guys out.  If you don't listen to podcasts, I recommend giving them a shot anyway.

As for my contribution to the topic at hand, I suppose I had a little bit of a war story.  From an interviewing side I do remember talking to one guy who's resume really looked great.  He had all sorts of great items written down from projects he had worked on in the past.  While inquiring about these items it became more and more clear that this person really didn't understand the concepts which he had written he had previously implemented.  After a few questions trying to get this candidate to talk about items on his blog he eventually answered that he had nothing to do with those tasks.  They were all completed by other people and he didn't understand how they worked.  He then apologized for writing misleading (or factually incorrect) items on his resume, and we ended the interview.

As an interviewee.  I just remember the Microsoft interview I had.  When I was graduating from Case Western Reserve University I had an on-campus interview with a representative from Microsoft.  I wanted to be a programmer since I was a small child (maybe 12 or 13 years old) and working for Microsoft was always a dream for me.  I had seen their campus (my family lived in Portland, OR at the time, and we saw their campus while visiting the Seattle area), and everything seemed like the perfect opportunity for a young geek in love with software.  From my interview, I really only remember a single technical question which I was asked.  Now keep in mind I wasn't claiming to be an expert at C or any other language at the time.  I had some professional experience working in VB.NET Beta, as well as some experience in developing relatively simple applications in C, C++, Java, PHP, Basic, Perl and the early versions of C#.

Anyways, he asked me "What is the fastest way to reverse a string in C?".  Ok, well I am familiar with C, and I'm familiar with how strings work in C.  I understand pointers, and pointer arithmetic, and immediately I think this must be pointer arithmetic.  Well, before I could even start talking about my response he says "Ohh, and it doesn't use pointer arithmetic.".  Uh ohh, at that point I pretty much froze.  I didn't know what to do.  I'm not a C expert.  I haven't written any C code in a while, let alone overly complex C code, and I need to know what the fastest way to reverse a string is in that language?  Well, lets just say the rest of that apparently didn't go over so well, and I wasn't asked any other technical questions.  I probably didn't handle the curve ball so well, but that was that.

I still remember how dumb I felt when I later learned just how many people from one of my classes landed jobs at Microsoft.  While in a class my senior year I remember the professor asking who was going to work for Microsoft, and there must have been at least 30 hands in the room that went up.  The said part of it to me too was I WAS the curve buster in that class.  I remember taking a test where the curve was so bad that a 68 became an A, yet I had scored a 98.  I was trying to figure out where I went wrong at that point.  Oh well, that's just how it goes.

Well, enough about me and my interviewing war stories, you need to go have a listen to Deep Fried Bytes.

--John Chapman

Been Gone

I just wanted to make a post so people knew I was still alive.  I've been extremely busy over the past two months.  I haven't forgotten about this blog, and I really want to get back to writing.  I really want to get further on the Sudoku series.

Lately I've been focusing on architecture of a Model-View-Presenter based WPF application.  The current complexities which are being analyzed is that users will have many duplicate views open at the same time, each working on a different item.  So picture 40 or more edit windows open on the same time, some for different items, some for the same items.  Some have similar child widgets, some have different child widgets.  Screens still need to talk to each other, but they have to talk to the appropriate screen.  Oh, and btw, performance is absolutely crucial on this app. 

The standard Model-View-Presenter samples you see really don't address these issues.  Typically there is one main form which acts as a shell, and there is one region where a given view can live.  Not here, that view can be in any number of places, and there can be any number of them.  It makes the problem a bit more complex.  We have some solutions, but I can't really discuss them here.

But after looking at the standard Model-View-Presenter samples, it make me feel more strongly that I really need to find the time to get back to the sudoku series, and hopefully I can produce a well documented series about why each choice is made.

Sorry for the delay.  I will be back.

--John Chapman 

New Adventures

I was originally going to write this on Tuesday, but I realized that with it being April 1st, people may have thought I was joking.  I have decided to leave Ryder.  Tomorrow, Friday April 4th will be my last day.

I have found an opportunity to work as an independent contractor building an application dealing with a domain that I am very passionate about.  I can honestly say that the application I'll be working on is one that I would love to use myself.  How often do we get to say things like that as developers?  I figured this was an opportunity I had to take.

Ryder has served me very well.  I first started at Ryder at the end of 2004.  I've had the chance to work with some great people.  I've also had the chance to interact with people of all levels in the company.  There were people at Ryder that took good care of me, helping obtain 24" monitors (to replace 17" CRTs for most people) to increase productivity as well as state of the art development machines.  I'll miss working with them.

I worked on the LMS team at Ryder, which is probably the largest externally facing application at Ryder (don't quote me on that).  I do know that they are backfilling my position, so if you are looking for a new adventure yourself, send them your resume.

--John Chapman

Sudoku Part 5: A Look At The UI Architecture

Part 5 has been a long time coming.  Originally, the plan was to present an entire UI in this post, which I've come to realize is simply not feasible. 

I have been attempting to learn WPF as part of this exercise.  Learning an entirely new UI framework while putting together this piece has proven a little bit difficult, and as a result the progress of the application has slowed significantly.  So for the rest of these pieces, I'll try to break down the UI further to show the choices I make while it's being developed and why.

Today, we're going to take a look at the architecture I plan to use for the WPF sudoku game and why.  We're going to take a look at my simple implementation of MVP (Model-View-Presenter).  We're going to take a look at the advantages of a MVP architecture and how that fits with our goal of utilizing Behavior Driven Development to build our application.

Architecture Description

In Part 1 (Defining The Solver Behavior) I mentioned that we would be using Castle Windsor in this project.  One of the great things about Windsor, and really any good inversion of control framework, is that you can minimize the invasion of container logic in your code via nested dependency injection.  What I mean by this, is that you can use the container to resolve a single service and it will automatically populate all dependencies for all children.  From our perspective, this will mean that the container logic can be restricted to just the UI, resulting in one less dependency throughout the application.

From previous parts we never made explicit use of Windsor, but rather we defined our inner dependencies via our available constructors.  For example, our RecursiveGenerator that we built in Part 4 took an ISolver in the constructor which will instruct Windsor to automatically populate the appropriate implementation for our generator.  This process is known as Constructor Injection.

I mentioned that we'll be using a Model-View-Presenter based architecture for our Sudoku game.  The Model-View-Presenter obviously breaks down to three pieces.  There are many variations of the MVP pattern (See Jeremy Miller's Build your own CAB series for some awesome overviews of various techniques), but the way we will use it for our purposes can be described as follows:

1. View - The portion of the application that actually displays data to the user as well as listens for user input.  Such as watching for button presses or mouse movement.  For this application we will be using WPF for the view.
2. Presenter - Handles the UI behavior logic.  For example the view will show the data to the user, but the presenter will determine what data should be shown to the user and how it should be formatted.  Also, while the view listens to see if a mouse moves, or a button is clicked, how that action is handled belongs to the presenter.  So it would be possible for our game to have many "New Game" buttons on the view, but it would defer the appropriate actions to the presenter to determine what should happen when any of those buttons are pressed.
3. Model - Handles the data which the presenter operates on.  This would typically represent the description of the business domain which is being solved by our application.  In our case this will represent the data needed to track the game which is being played, such as the puzzle which is being worked on, the current status of the solution which the user is building as well as supporting information such as the game clock or at least the time when the game was started.

So, why go through all of this?  Especially for an application which seems as simple as a sudoku game?  Well, first of all, we are doing this as an exercise, not necessarily the exact application.  Secondly, and far more importantly, this separation of concerns allows us to keep logic more closely aligned to a purpose, and allows us to have fully testable behavior.  By moving our presentation behavior from the XAML "code-behind" to a separate presenter object, we can test our presentation without the need for an actual UI.  This is a major advantage over the old fashioned applications.  We can now have a lot more faith that our application behaves as expected via our automated testing.

Additionally, MVP does not force us to be tied to the WPF view.  It would be possible for us to re-use the same model and presenter in a WinForms application (or potentially silverlight and other web-based frameworks) with no change, only a different view.  This will be considered out of scope for the time being. 

WPF Implementation

First off, I want to make a disclaimer. I am not a WPF expert.  If there is a better way to implement this in WPF, I would love to hear about it.  I'm learning WPF as I go with this sample application.

My first idea was to create a base class which would initialize the appropriate presenter for that view.  I quickly ran into a bit of a problem with WPF.  It's valid to create your own base class instead of Window, but it adds a lot of complexity to the XAML, and Visual Studio didn't seem to appreciate it too much. 

Since I only needed the base class to resolve the appropriate presenter via my container, I figured it would be ok to just have the convention of every view calling a static Initialize method.  This keeps the XAML simple, and only adds one extra line of code to all views.

Application Implementation

I chose to initialize Windsor in the Application class, which is where the static Initialize also lives.  The Application class looks like the following:

public partial class App : Application, IContainerAccessor
{
 public static IWindsorContainer Container { get; private set; }
 
 IWindsorContainer IContainerAccessor.Container
 {
  get
  {
   return Container;
  }
 }
 

 protected override void OnStartup(StartupEventArgs e)
 {
  InitializeContainer();
  base.OnStartup(e);
 }

 private void InitializeContainer()
 {
  Container = new WindsorContainer(new XmlInterpreter());
 }

 public static void InitializePresenter<T>(T view)
 {
  Presenter<T> presenter = Container.Resolve<Presenter<T>>();
  presenter.Wireup(view);
 }
}

Note that this is as far as our container needs to go into our sudoku application.  When the application first starts we need to initialize our container based on the application configuration file.  The InitializePresenter method is what we expect every view to call when the view is initialized.  The type paramter (T) is the interface type which the view itself implements.  That is the view that the appropriate presenter will use in order to interact with the view.  The Presenter<T> will server as the abstract base class for all presenters.  So in this case we're asking Windsor to locate and instantiate the appropriate presentation logic for our particular view.  Once the presenter is located we call the base class method Wireup which tells the presenter to begin observing any events exposed by the view.  This would also be the time that the presenter could perform any initialization logic.  The Presenter base class looks like the following:

public abstract class Presenter<T>
{
 public T View { get; private set; }

 public void Wireup(T view)
 {
  View = view;
  Initialize();
 }

 protected abstract void Initialize();
}

The View implementation would then contain the following:

protected override void OnInitialized(EventArgs e)
{
 base.OnInitialized(e);
 App.InitializePresenter<IBoardView>(this);
}

Where obviously in this case the view implements the IBoardView interface.  Our appropriate presenter implementation would be defined as:

public class BoardPresenter : Presenter<IBoardView>

To wrap up our initial implementation we would need to configure our Windsor Container in our configuration file as follows:

<configuration>
  <configSections>
    <section name="castle"
        type="Castle.Windsor.Configuration.AppDomain.CastleSectionHandler, Castle.Windsor" />
  </configSections>

  <castle>
    <components>
      <component 
        id="generator" 
        service="Sudoku.Bll.Generator.IGenerator, Sudoku.Bll" 
        type="Sudoku.Bll.Generator.RecursiveGenerator, Sudoku.Bll" />
      <component 
        id="solver" 
        service="Sudoku.Bll.Solver.ISolver, Sudoku.Bll" 
        type="Sudoku.Bll.Solver.RecursiveSolver, Sudoku.Bll" />
      <component 
        id="boardpresenter"
        lifestyle="transient" 
        service="Sudoku.UI.Presenter`1[[Sudoku.UI.Views.IBoardView, Sudoku.UI]], Sudoku.UI" 
        type="Sudoku.UI.Presenters.BoardPresenter, Sudoku.UI" />
    </components>
  </castle>
</configuration>

In this case our presenter would have an IGenerator parameter in its constructor which would tell Windsor to construct the defined generator, which we've already seen requires a solver implementation.  Therefore these three specific components are necessary to fully configure our Windsor container for this example.  One item to note is that we have specified a lifestyle of transient for our presenter.  This is to signify that every presenter should have its own containing class (since the state of the associated model will matter), whereas our generator and solver services have no defined state and can safely be use the default lifestyle of singleton.

At this point we have the basics defined for a Model-View-Presenter application in WPF.  We are using Castle Windsor to manage our dependencies.  Our View has no knowledge of our presenter implementation, and the presenter has no knowledge of our view implementation.  All aspects are configurable and easily changeable.  Plus, since our presenter does not know our view implementation it has opened the door to potential porting to an alternative UI framework.

In the next iteration I hope to examine the needed behavior of our presenter and potentially the implementations which satisfy that behavior.

--John Chapman