Failed to build gem native extension on Windows

So there I was lovely new computer, I installed Ruby 193, old tests which needed it, don’t ask, and of cause the ruby dev kit, then a quick bundle install and I will be ready to go, but instead I get a cryptic message from the install saying something like.

C:\devkit>gem install rdiscount --platform=ruby
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing rdiscount:
       ERROR: Failed to build gem native extension.

C:/ruby/bin/ruby.exe extconf.rb
C:/ruby/bin/ruby.exe: No such file or directory -- extconf.rb
(LoadError)

after a bit of googling with bing, I turned up the Windows Rubyinstaller Troubleshooting instructions.

Now even though I had not installed ruby using the installer, but using Chocolatey the trouble shooter did have a nice section displaying the text above which was close to what I had on my screen, apparently there are some registry settings that effect the dev kit.

Reading down it shows 4 registry keys

REG QUERY "HKCU\Software\Microsoft\Command Processor" /s
REG QUERY "HKLM\Software\Microsoft\Command Processor" /s
REG QUERY "HKCU\Software\Wow6432Node\Microsoft\Command Processor" /s
REG QUERY "HKLM\Software\Wow6432Node\Microsoft\Command Processor" /s

that need to be checked, and the expected results, all but the third one worked fine, the third one, simply did not exist in my registry!

Another quick google with bing and I discovered that happily I was not the only one with the issue, but I did not find any real solutions.

The following is what I did to solve the issue, it “worked on my machine”, but editing the registry comes with risks and your results may be different.

I had a look at the registry entries for the ones that worked, both the HKCM keys

HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor
    CompletionChar    REG_DWORD    0x40
    DefaultColor    REG_DWORD    0x0
    EnableExtensions    REG_DWORD    0x1
    PathCompletionChar    REG_DWORD    0x40

HKEY_LOCAL_MACHINE\Software\Wow6432Node\Microsoft\Command Processor
    CompletionChar    REG_DWORD    0x40
    DefaultColor    REG_DWORD    0x0
    EnableExtensions    REG_DWORD    0x1
    PathCompletionChar    REG_DWORD    0x40

and both entries were the same, one for the 32bit environment and one for the 64 bit.  So taking a quick leap in the dark and looking at the working entry for HKCU I fired up RegEdit and added an entry for the missing key making it exactly the same as the existing entry.

HKEY_CURRENT_USER\Software\Wow6432Node\Microsoft\Command Processor
    CompletionChar    REG_DWORD    0x9
    DefaultColor    REG_DWORD    0x0
    EnableExtensions    REG_DWORD    0x1
    PathCompletionChar    REG_DWORD    0x9

and for reasons I do not understand it all worked, native gems now build correctly on my machine and I can complete my bundle install.

Posted in Uncategorized | Leave a comment

There Is No State Only Events

I had got to the next feature, “Get an existing basket”, this was going to be a simple save the basket on creation, load it by id, return 404 if not found type of blog post, I even had the code written to store the basket state in a file using the id as the file name!

One of the joys of blogging and writing code for fun, is that you can change your mind, and learn new things along the way, the book club where I work has just finished the book

DDD BookWhich got me thinking about my little basket API and how it could be used to investigate the whole idea of state and events.  At work we have just started to use events (we call them messages) in Amazon Web Services to trigger off tasks like sending welcome emails. 

The book takes domain aggregates and the event store to the next level instead of saving object state and losing the meaning of the action, events are saved and the object (Aggregate) is rebuilt by playing these events back.  This has the advantage of being very saleable and because each event is saved the object can be rebuilt to any state in its history which can be a very useful tool in debugging applications.  It is also thought that because the event store is append only the speed of an application using this technology can be faster and support a greater number of transactions than a traditional application using state storage.

So lets have a quick look at the context the basket Aggregate, and what I am currently expecting it to interact with.

BasketBoundedContext

The basket bounded context contains our basket aggregate and a product cache that is maintained using events from the product aggregate.  The basket will raise events that are consumed by the order service to maintain its own cache.

So what events will the basket be raising, currently I can only think of three

  • Basket Created: contains the basket id
  • Product added to basket: contains the product id, product description and product price
  • Product removed from basket: contains the product id

All the events raised will be stored in an events store when we want to get the baskets state we can reload the events replay them and the resulting aggregate will be complete.

I am expecting an order service to be listening for basket messages so that when an order is placed it can continue its processing form its own basket cache, in a similar way to the basket context listens to all product messages to maintain its own product cache.  That’s the theory anyway.

Ok great so that’s event sourcing in a nutshell, but what does it mean for our basket code, well to keep it simple I am not going to do the publish event bit yet, so I will simply save events to an event store. and use these events to re-create the objects state. so first we need to add to the basket create method, so that it will store a basket created event.

Notice we don’t need any new tests for this, our tests covers the creation of the basket from the outside, how we do that is of no interest to the outside world, so we can simply add code here and we know its covered.

public static Basket CreateWithId()
{
  string id = GenerateBasketId();

  // create event
  var basketEvent = new CreateBasketEvent {Id = id};

  // store event in basket event store
  var filename = string.Format("./EventStore/Basket/{0}", id);
  var sterilizer = new XmlSerializer(typeof (List<CreateBasketEvent>));
  var writer = new StreamWriter(filename);
  sterilizer.Serialize(writer,new List<CreateBasketEvent>{basketEvent});
  writer.Close();

  // return new basket
  return new Basket { Id = id };
}

The even class itself is simple a dumb dto with a property id

namespace CheckoutBasket.API.Events
{
    public class CreateBasketEvent
    {
        public string Id { get; set; } 
    }
}

In the get method all we will have to do now is load the events and replay them through the basket to re-create its state.

But first we need a failing test for getting the basket.

public class WhenGettingAnExistingBasket : BddBase
{
   private Browser _browser;
   private Basket _basket;
   private string _basketId;

   protected override void Given()
   {
       var bootstrapper = new DefaultNancyBootstrapper();
       _browser = new Browser(bootstrapper);

       var response = _browser.Get("/basket");

       var basket = response.Body.DeserializeJson<Basket>();
       _basketId = basket.Id;
   }

   protected override void When()
   {
       var response = _browser.Get(string.Format("/basket/{0}", _basketId));

       _basket = response.Body.DeserializeJson<Basket>();
   }

   [Test]
   public void The_returned_basket_id_is_the_expected_basket_id()
   {
       _basket.Id.ShouldBe(_basketId);
   }
}

This test simply creates a basket then attempts to get that basket from the application, of cause at the moment it fails because our router does not know what to do with a basket request with an id attached, so lets fix that.

public class BasketHander : NancyModule
{
   public BasketHander()
   {
       Get["/basket/{id?}"] = _ =>
       {
           var basket = Basket.CreateWithId();

           return Response.AsJson(basket);
       };
   }
}

Here I have simply added the “{id?}” to the end of the root for get, this means the id is an optional part of the route, now our tests fails because we are getting a new basket each time with a different id to the one supplied, as our service only knows how to create baskets.

    public class BasketHander : NancyModule
    {
        public BasketHander()
        {
            Get["/basket/{id?}"] = _ =>
            {
                Basket basket;
                
                if (_.id == null)
                    basket = Basket.CreateWithId();
                else
                    basket = Basket.GetWithId(_.id);
                
                return Response.AsJson(basket);
            };
        }
    }

Here we are checking for the existence of the id, and either creating a new basket if the id is null or attempting to get the basket if the id is not null, we are still not doing any error checking but this is still the first pass and early stages, lets have a look at the GetWithId method.

public static Basket GetWithId(string basketId)
{
  var filename = string.Format("./EventStore/Basket/{0}", basketId);
  var sterilizer = new XmlSerializer(typeof (List<CreateBasketEvent>));
  var reader = new StreamReader(filename);

  var events = (List<CreateBasketEvent>) sterilizer.Deserialize(reader);
  reader.Close();

  return new Basket(){Id = events[0].Id};
}

Here again I have gone for the simplest thing that can possibly work, we are only working with one event at the moment so its a case of getting the file, desterilising the list of events and creating a new basket object and giving it the correct id.

We now have passing tests again.  The code is very simple but we need to be able to handle more than one type of event, we currently know of at lest three so lets refactor the code a bit to move us in that direction.

First up lets give the CreateBasketEvent class a base class which all future events can be derived from called ApplicationEvent our serializer can then be standardised to use this base class, it turns out that in order for the serialization to work we need to decorate the base class with an attribute so it knows what has inherited from it.

[XmlInclude(typeof(CreateBasketEvent))]
public class ApplicationEvent
{
}

public class CreateBasketEvent : ApplicationEvent
{
    public string Id { get; set; } 
}

Now we can generalize the serialization and deserialization so that it can in theory cope with any sort of event based on the ApplicationEvent.

public static Basket CreateWithId()
{
  string id = GenerateBasketId();

  var basketEvent = new CreateBasketEvent {Id = id};

  var filename = string.Format("./EventStore/Basket/{0}", id);
  var sterilizer = new XmlSerializer(typeof (List<ApplicationEvent>));
  var writer = new StreamWriter(filename);
  sterilizer.Serialize(writer,new List<ApplicationEvent>{basketEvent});
  writer.Close();

  return new Basket { Id = id };
}

public static Basket GetWithId(string basketId)
{
  var filename = string.Format("./EventStore/Basket/{0}", basketId);
  var sterilizer = new XmlSerializer(typeof (List<ApplicationEvent>));
  var reader = new StreamReader(filename);
  var events = (List<ApplicationEvent>) sterilizer.Deserialize(reader);
  reader.Close();

  return new Basket {Id = ((CreateBasketEvent)events[0]).Id};
}

Now we need to extract all the serialization into some sort of event store class this will have two methods. The first will be to save an event to the store, the second will be to get a list of events from the store, the code is refactored from the basket create and get methods.

public interface IEventStore
{
   void Store(ApplicationEvent applicationEvent, string path);

   IList<ApplicationEvent> Retrieve(string path);
}

public class EventStore : IEventStore
{
   public void Store(ApplicationEvent applicationEvent, string path)
   {
       var sterilizer = new XmlSerializer(typeof(List<ApplicationEvent>));
       var writer = new StreamWriter(path);
       sterilizer.Serialize(writer, new List<ApplicationEvent> { applicationEvent });
       writer.Close();

   }

   public IList<ApplicationEvent> Retrieve(string path)
   {
       var sterilizer = new XmlSerializer(typeof(List<ApplicationEvent>));
       var reader = new StreamReader(path);
       var events = (List<ApplicationEvent>)sterilizer.Deserialize(reader);
       reader.Close();

       return events;
   }
}

Notice we haven’t added any tests, we are simply at the moment moving code around, the use of tests written with the behaviour in mind and reducing the use of mocks allow us to refactor code, this is still a very naïve implementation, there is no concurrency we are not posting events and there is still a lot of work to do, but by not using mocks until the tests become too slow and only then mocking the boundaries of the application we will be able to implement refactoring’s changing the structure of the application as required.

But before we refactor more lets get the next test written, that is for a basket that does not exist.

public class WhenGettingANonExistingBasket : BddBase
{
   private string _basketId;
   private Browser _browser;
   private BrowserResponse _response;

   protected override void Given()
   {
       var bootstrapper = new DefaultNancyBootstrapper();
       _browser = new Browser(bootstrapper);

       _basketId = Guid.NewGuid().ToString().Replace("-", ""); 
   }

   protected override void When()
   {
       _response = _browser.Get(string.Format("/basket/{0}", _basketId));
   }

   [Test]
   public void then_the_response_should_be_404_not_found()
   {
       _response.StatusCode.ShouldBe(HttpStatusCode.NotFound);
   }
}

This test of cause fails nicely with an internal server error, because the stream reader in the event store cannot find the file with the events, so we need to handle this first, easiest solution is to check for the file, and return null if not from the event store.

public IList<ApplicationEvent> Retrieve(string path)
{
  if (! File.Exists(path))
      return null;

  var sterilizer = new XmlSerializer(typeof(List<ApplicationEvent>));
  var reader = new StreamReader(path);
  var events = (List<ApplicationEvent>)sterilizer.Deserialize(reader);
  reader.Close();

  return events;
}

Now we still get an internal server error but its in the basket because we are trying to access the first event which at this point we know should be a create, but we have no events.  This we can handle by checking that we have events and returning null if we do not.

public static Basket GetWithId(string basketId)
{
  var filename = string.Format("./EventStore/Basket/{0}", basketId);
  var eventstore = new EventStore();
  var events = eventstore.Retrieve(filename);

  if (events == null)
      return null;

  return new Basket {Id = ((CreateBasketEvent)events[0]).Id};
}

So now we are getting an http.ok status back from the method, this is because our handler does not check what it gets back from the basket call, checking this for null and returning the correct response is the last step to make the test pass.

public class BasketHander : NancyModule
{
   public BasketHander()
   {
       Get["/basket/{id?}"] = _ =>
       {
           Basket basket;
           
           if (_.id == null)
               basket = Basket.CreateWithId();
           else
               basket = Basket.GetWithId(_.id);

           if (basket == null)
               return HttpStatusCode.NotFound;

           return Response.AsJson(basket);
       };
   }
}

And now the test passes.  Although the code at this point is not great, there is still a lot we could do to improve the basket and event store,  we do have the basics in place next time we can look at adding a product to the basket, which will force us to look at the event store, think about multiple event types and how we rebuild the aggregate from events.

Posted in Programming | Tagged , , | Leave a comment

Adding An Id!

Ready to start again with Red?

Looking back at our requirements next I see that the basket should have a unique id, at the moment I have no clue about where I am going to store the baskets so there is nothing to help me generate ids, and I don’t want an incremental number appearing in the URI of our API calls as this could be a security risk.  To this end I think I will use a GUID, with the “-“ removed will make a nice unique id for the basket.

First up lets write the test, we already have a fixture calling our create method so we can simply add a new test method that checks for the existence of the id.  Starting again the same as before creating a test method.

        [Test]
public void The_returned_basket_has_an_id()
{

}

then getting the baskets id property

        [Test]
public void The_returned_basket_has_an_id()
{
_basket.Id
}

Which of cause does not exist on the basket, so we need to create it

    public class Basket
{
private IList<object> _items;

public Basket()
{
_items = new List<object>();
}

public bool IsEmpty { get { return ! _items.Any(); } }
public string Id { get; set; }
}

and now simply to assert that the id is not empty.

        [Test]
public void The_returned_basket_has_an_id()
{
_basket.Id.ShouldNotBeEmpty();
}
 
This test runs red because we are never setting the id.
 
The next phase.  Getting to Green as quickly as possible.
 
We want to set the id to a new GUID without the “-“ between each section of the string which we can achieve in the handler.
 
    public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
return Response.AsJson(new Basket(){Id = Guid.NewGuid().ToString().Replace("-","")});
};
}
}
And our code goes green again. Its time to take a breath and review our code.
 
The Refactoring phase. 
 
Do we want to be doing all that id generation within the handler? Probably not, first lets use the introduce variable refactoring so we have a basket variable that is filled with a created basket.
 
    public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
var basket = new Basket(){Id = Guid.NewGuid().ToString().Replace("-","")};

return Response.AsJson(basket);
};
}
}

This will allow us to move the id creation code to a method on the basket, call it from the handler and remain green.

    public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
var basket = new Basket();
basket.Id = basket.CreateBasketId();

return Response.AsJson(basket);
};
}
}

This is better, we have moved the id generation to the basket, but its still not great.
To create a basket we now need to not only call new, but we also have to remember to set the id, what would be nice is to be able to call something like  “Basket.CreateWithId()” and be returned a new basket with the id set. 
 
This is a factory method, so lets implement that and see how this effects our code.
 
    public class Basket
{
private IList<object> _items;

private Basket()
{
_items = new List<object>();
}

public bool IsEmpty { get { return ! _items.Any(); } }
public string Id { get; set; }

public static Basket CreateWithId()
{
return new Basket { Id = GenerateBasketId() };
}

private static string GenerateBasketId()
{
return Guid.NewGuid().ToString().Replace("-", "");
}
}
 
Quite a bit has gone on here,
We have a new static method called CreateWithId this returns a basket, and also calls the GenerateBasketId method  to set the id, because this method is now only used within the class I have made it private so we cannot generate ids outside of the basket class.  I also made the constructor of the basket private so the only way now to create baskets is through the factory method, which means we now have encapsulated the idea of basket id generation within the basket and all baskets created will have unique ids.
 
Back to the handler which can call this new factory method to get the basket.
 
    public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
var basket = Basket.CreateWithId();

return Response.AsJson(basket);
};
}
}

Our handler is back to looking clean and simple, and the basket code looks ok, I am not sure about the chaining of the “Guid.NewGuid().ToString().Replace()” there is an awful lot of dots in there, and I usually like to keep them to a minimum to avoid the whole train wreck problem, but in this case its within a method, and I am not sure splitting it out would give much benefit at the moment.
 
So I think we are done with basket creation for the minute.
 
Looking at the requirements, the next task has to be getting a basket with an id, this implies we have baskets stored somewhere so at some point soon we are going to have to deal with storage.  We will also have to cover not finding the basket, and how we are going to return errors, lots to think about and decisions to be made.
 
All the code for this post is in the branch create basket in the git hub repository
Posted in Uncategorized | Leave a comment

The First Test

We are at the point of writing our first test, but first lets review the whole test first pattern, that is the hopefully well known

Red
Green
Refactor

although it seams to be well known I think there has grown up a lot of myth and folk law around the process that gets in the way of actually carrying out what is on the surface a simple process.  But a bit like chess this simplicity hides a complex thought process that has to be practiced and when performed well leads to code that is well designed, extensible and easy to maintain. 

Lets take this one stage at a time and work through the first test and see how the process translates into code.

Red
Within this phase we write our test, as will be seen we will also be writing some production code, in order to keep what we create compiling and syntaxially correct, we will not worry about duplication or design at this point, our aim is to write a purposefully failing test.

Our first test is about the creation of a basket, as discussed in the previous post, our first task is to create a test class in the  Checkout basket specifications project this class will inherit from the BddBase class.  All the following code was created in the “createbasket” branch of the repository.

    public class When_calling_the_basket_api_without_an_id : BddBase
{
protected override void Given()
{

}

protected override void When()
{

}
}

Our code at this point still compiles. 
 
The “Given” method is where we will setup our system to run the specific set of tests, in the “When” method we will call the actual public method under test, ideally this method should be a single line where we get a result from our call that our test methods can assert on, but as we will see this is not always the case. 
 
Where we start writing code now depends on the situation, if we have classes and methods that we can call already then we could start with creating the setup and calling the method.  But in our case we have nothing, so we must start with the test method.
 
Naming our test can be complex, I always suggest that the best idea is to get something down and change it as you start to write the actual code, the name of a test like everything else is refactorable, but in order to get to the refactor stage we need something in code.  In our case we are saying that the call will result in a new empty basket, so lets say that.
 
        [Test]
public void Then_a_new_empty_basket_is_returned()
{

}

Notice our code still compiles, I am not sure how we are going to test for newness of our basket, but checking for empty should be easy, if our basket has a collection of items, then we can check for there being no items in the basket.   Lets code that, we will need a field variable to hold our basket as this will be returned inside the “When” method,
 
        [Test]
public void Then_a_new_empty_basket_is_returned()
{
_basket
}

Now our code will not compile, because we do not have a declaration for the “_basket” field, lets add that.

        private Basket _basket;

adding this declaration will require the creation of a “Basket” class, in order that we continue progressing at this point I would suggest simply creating the class in the same file as we are currently working in.

    internal class Basket
{
}

I created the class using a Resharper shortcut, and the tooling has created it internal, but at this point I don’t care, we want to make progress so lets leave it for the time being.  Back to our test we can now assume our field variable contains a basket, so the next bit is to get the items in the basket.
 
        [Test]
public void Then_a_new_empty_basket_is_returned()
{
_basket.Items
}
 
Our code won’t compile because “Items” is not defined on the “Basket” class so lets add that now
 
    internal class Basket
{
public IList<Object> Items { get; set; }
}

Back to our test and finally we can see if there are any items in the basket, and assert on the result of the check.
 
        [Test]
public void Then_a_new_empty_basket_is_returned()
{
_basket.Items.Any().ShouldBe(false);
}
 
And our code is back to compiling again. 
 
But the test will fail because _basket is not set anywhere!  so lets fill out the given and when steps.  A quick lookup at the Nancy documentation shows that we can create a bootstrapper that is handed to a special test browser, that we can use to make our API call on, this is almost a copy from the Nancy documentation.
 
        protected override void Given()
{
var bootstrapper = new DefaultNancyBootstrapper();
_browser = new Browser(bootstrapper);
}

protected override void When()
{
var response = _browser.Get("/basket");
}

Now we have a response, which we need to turn back into our basket so our test will work.  As it turns out the Nancy Response object has a method for converting JSON back into an Object, so we use that to get our basket from the JSON.
 
        protected override void When()
{
var response = _browser.Get("/basket");

_basket = response.Body.DeserializeJson<Basket>();
}

We are still compiling but our test fails! Why?
 
At each stage we are writing our code we always need to be aware of why the test is failing, as this can help guide us on what code we need to write next, and it ensures that we actually understand the code we are writing!  In this case the response is getting a 404 not found error.  Which is expected, we have not written any code to handle the “Get Basket” call
 
This completes the red phase of our process, notice how we always tried to keep the code compiling even during the writing of the test, we only created just enough to get the test to fail even when failing was simply not compiling.  We are now ready to make the test pass.  
 
Our test file looks like this.
 
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using Nancy;
using Nancy.Testing;
using NUnit.Framework;
using Shouldly;

namespace CheckoutBasket.Specifications
{
public class When_calling_the_basket_api_without_an_id : BddBase
{
private Basket _basket;
private Browser _browser;

protected override void Given()
{
var bootstrapper = new DefaultNancyBootstrapper();
_browser = new Browser(bootstrapper);
}

protected override void When()
{
var response = _browser.Get("/basket");

_basket = response.Body.DeserializeJson<Basket>();
}

[Test]
public void Then_a_new_empty_basket_is_returned()
{
_basket.Items.Any().ShouldBe(false);
}
}

internal class Basket
{
public IList<Object> Items { get; set; }
}
}

 
Green 
It is during this phase that a lot of us being to make our mistakes, we start to worry about design, and clean code and all the rest of the very good practices that we will be worrying about in the future, but to worry about them now, and attempt to implement them is a mistake.
 
Our task at this point is to make the test pass as quickly as possible. 
 
One of the very good practices, that can help to keep us moving fowrward is to say that if you have not made the test pass within one pomodoro (25 minutes) then you should back out of your code changes remove the test and write a test that does a smaller chunk of work.   I would add to this, that if you haven’t changed the error message within 5 minutes of starting then you should back out, because what you are attempting to do is too big and should be broken down into smaller tasks.
 
Do not be afraid to back out of a test and rethink the problem, remember our tests are there to support the design and implementation of our code, so if we have bitten off more than we can comfortably chew then its always best to back out, redesign the test and plunge in again.   The future code will always be better, because your understanding of the problem and its domain is going to be enhanced as you explore we each new test.
 

I cannot over emphasise this enough, we need to get the test passing as quickly as possible,

At this point in the process we are going to break every rule of coding, cutting and pasting code, copying from the internet, stealing from other projects do what ever you have to, make the test pass, and do it quickly. 

Ward Cunningham put is best when he said do the simplest thing that can possibly work, simplicity is the shortest path to a solution, notice he didn’t say best, or well engineered, its the shortest which at this point is what we want, we need our code to work, for the test to be green and for us to be able to move on as quickly as possible. You have permission to do your worst break the rules, and make the test pass.

For us we need a handler for the basket get method, so lets quickly put this together, add a new class to the “CheckoutBasket.API” project.

Nancy uses the constructor of the class to setup routing for a method.

using Nancy;

namespace CheckoutBasket.API
{
public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
return null;
};
}
}
}
 
Our code compiles, but the test still fails, but the message has changed to object not set to an instance of an object.
 
Because we are returning null from our method! And therefore the basket in our test is set to null, the error is caused by trying to access the items collection on the null basket. 
 
So we need to move the basket class we created earlier into the “CheckoutBasket.API” project (that’s a simple f6 in Resharper) make it public so our code compiles, Then return a new basket with a new collection of items from the basket handler as JSON.
 
using System.Collections.Generic;
using Nancy;

namespace CheckoutBasket.API
{
public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
return Response.AsJson(new Basket() {Items = new List<object>()});
};
}
}
}

And our test now passes, well done, commit to save all of our changes and its onto the one stage that is the most important and the most neglected.
 
Refactor
 
When we wrote the code to make the test pass we ignored all the rules for clean code and good design, because we needed to get to green quickly.  Developers almost naturally try to design when they code at they are writing it, which means that the refactor stage is often forgotten or is delayed until some point later when there is “more” code to look at. 
 
What I want to show now is that we need to think about it after each passing green, even if there is only a singe line of code!
 
So lets take a look at our code starting with the test we have written.  Looks fairly reasonable, and in truth it is not bad, but we can improve it. 
 
The test states in its title that “Then a new empty basket is returned”, well we know its new because the handler creates a new basket, but the test doesn’t know that, but it does check for the basket being empty.  I think we can rename this test a bit to reflect what the code is actually saying, the code ensures the basket is empty, so lets rename it to “Then an empty basket is returned”.
 
The actual code in the test is a bit more complex, we are actually reaching into the object grabbing its items collection and then calling an extension method to check to see if it is empty, not exactly intuitive.
 
How about having an IsEmpty property on the basket this would stop us from reaching into the object and will tidy up our tests.
 
In the basket class create a read only property IsEmpty and copy the code from our test into it so that it returns true if “Any()” returns false;
 
    public class Basket
{
public IList<object> Items { get; set; }

public bool IsEmpty { get { return ! Items.Any(); } }
}
 
And change the test to call our new property
 
        [Test]
public void Then_an_empty_basket_is_returned()
{
_basket.IsEmpty.ShouldBe(true);
}

 

Now looking at the handler we are returning a new basket which is correct but should be also be doing the initialization of the items collection in the handler as well, lets move it to the constructor of the basket, stopping the handler knowing about the internals of the basket.

    public class BasketHander : NancyModule
{
public BasketHander()
{
Get["/basket"] = _ =>
{
return Response.AsJson(new Basket());
};
}
}

We can now also make the items property private as no one else accesses it.

    public class Basket
{
public Basket()
{
Items = new List<object>();
}

private IList<object> Items { get; set; }

public bool IsEmpty { get { return ! Items.Any(); } }
}

Actually we can make the items list a field, because no one needs access to it currently, so lets do that as it simplifies our code even more.
 
    public class Basket
{
private IList<object> _items;

public Basket()
{
_items = new List<object>();
}

public bool IsEmpty { get { return ! _items.Any(); } }
}

The last bit of refactoring I want to do is within the solution it self moving the handler and basket classes to their own folders, I want to start to add a bit of structure to the project to ease navigation in the future.
 

create basket first test

With that I think we are just about done, the end of the initial test first development phase, notice how we took time over the refactoring,.

We should have taken as much time over the naming of our tests as this helps with designing the API, the only phase we rushed to get through was the getting to green, where we broke all the rules and “Just made it work”.

Once green we went back and applied all the laws of good coding practice, refactoring the code to make it maintainable and easy to read.

All through the process we aimed to keep the code compiling, and the tests running, even while building the very first test, then once failing we got to green as quickly as possible.  again keeping the code compiling, only at this last stage do we apply the rules of software design, and because our tests are on the outside looking in, we are able to refactor the internals of our application without breaking tests.  Next up will be the id and some thoughts on storage.

Posted in Programming | Tagged , | Leave a comment

A Bit of Up Front Planning

They do say that in Agile there is no up front planning, well I like to do a little bit, just so I know the direction I should set off, and the bit of space it gives allows me to think about the problem, for me a problem always sounds difficult if not insurmountable, before a bit of thought.

In the last blog post I spoke about how I approach test first development in a way that improves the ability to refractor code, reducing the requirement for mocking between layers of an application by only mocking at the application boundaries.  I thought it would be good to build an application in this way, and show how the tests allow for refactoring and changes over time.

I need a project to work on that can start very simply and grow but does not have large visual requirements, so I chose the typical basket component as my application.  Think of the checkout basket API as a single service of a larger shopping system, each part communicating over http using a JSON format.  In there complete system there would be a web front end or mobile application, other services that provide search, and product listing and our checkout basket, which will provide the all the basket, order calculations ready for the customer to checkout through a payment system. 

It may be our basket system will call other related systems within the system a product service comes to mind as a possibility, but this will come out in requirements later in development.

To this end I have created a git repository where all the source will be located, the master will contain the release version, and a branch will exist for each stage of development.

So now some hammock driven development, we are going to build the basket using a service in C#, so other than C# what else will we use, I am going to need some form of Web API framework, I thought about using the build in MVC, or maybe the Web API but for what I want it seams a bit heavy and I want a test framework that is easy to use and fits in with the behaviour style tests I am going to write, I am thinking of using NancyFx for the basic API, it supports tests and has little setup requirements and will support the JSON format easily.  As for the test framework, that will probably be NUnit or something based on it as this is the simplest test framework and its well known within the C# community.

The basket service is going to have some functionality so that the basket can be created, have items added to it, items removed from it and to be able to be returned to a caller with all the items that it contains listed.  There is also a probably that there will be some totalling of items and application of discounts, and other requirements I haven’t even thought about yet.

Now we have the problem and where it fits into the application, some thoughts about the sort of frameworks and tooling we want to use, nothing concrete at this point but some things to ponder before diving into the computer.

I have setup a basic project, the idea is that each stage of the development will be a feature branch that is merged back into the trunk, so each stage of development can be seen, along with all the mistakes and failures that typically haunt a project along with all the successes!

I think a good starting point will be the basket creation,  in a REST ish style thinking about the basket as a resource, I think our caller will make a call to the checkout basket API to “get” a basket without an id will be the return a new basket, and with an id will return a existing basket.

/basket
or
/basket/{basket id}

The first tests I will write then are going to be to do with creating a basket, calling basket without an id, this will return the JSON representation of a basket, something like

{ “Id” : “a unique basket id” }

This will be expanded as time goes on, but currently we just need a unique id, enough thinking lets write our first failing test.

So getting to the code I have created a branch called “createbasket”, I have two tests in mind, the first will be that we have a id and the second is that the basket is empty, I am planning on returning the basket as JSON from the call. at this point the project looks like this.create basket solution start We now need to add a couple of NuGet packages which will assist in our development, to the “CreateBasketAPI” project I added the Nancy package and to the “CheckoutBasket.Specifications” project I added the NUnit package, the Nancy.Testing package and the Shouldly package. With these installed we are ready to start.

I like to use the Given – When syntax for my tests, to this end I create a small base class which my tests can inherit from to provide simple hooks for this syntax.

namespace CheckoutBasket.Specifications
{
[TestFixture]
public abstract class BddBase
{
[SetUp]
public void Setup()
{
Given();

When();
}

protected abstract void Given();

protected abstract void When();
}
}

With this in place I am ready to write the first test of the project.

Posted in Programming | 1 Comment

Test First Development

Last time I wrote about how we are increasing the numbers of tests and how they are damaging our code.  Another problem we have that is stopping us getting the full benefits of refactoring code is the proliferation of mocks within our tests, this almost forces the code to mimic the production code epically when using some of the modern behaviour test frameworks that helpfully mock all the dependencies of a class. 

Why is this bad? before answering that lets first think about why we mock, in fact to forestall any arguments lets call this mocking or stubbing out of things fakes. 

Why do we fake things? When test driven development was introduced by Kent Beck some 16 odd years ago, computers were a very different think, I was writing code in visual basic 6, on Windows Server NT, for the dot net world testing started with the release of NUnit for the dot net framework, this was around 2002, computers were a lot slower, so in order to keep tests fast we faked the slow bits.

One of the core tenants of test driven development was and is the tests must be quick, we are running all of them all the time, so they have to be very quick, this was the primary reason to fake some components.  Move on to today and we are faking more and more of the systems, not just the boundaries but between layers and I think it is this that has made our tests fragile and reduced the chances of refactoring.

I think in order to progress and improve the development experience and gain the benefit of refactoring we need to go back to the origins of the practice, back to Kent Becks original idea.

1. Write the failing test, along with enough production code to make the test fail, what do I mean, we want to write the test so it compiles, runs and fails, this will involve writing some production code, creating a new method etc, the test will be red and it will be obvious why the test will not pass.

2. Now we need to get to green as quickly as possible, this is where we cheat, lie steal and plagiarise code, copy pasting where necessary, my policy is we should stick it all in the one place the method we are testing, don’t worry about design or style get it working make the test pass, get to green as quickly as possible.

3. Once the test is green, we hit the stop button and review the mess we have just made, what is wrong, how many things is our method doing, can we see any duplication, and we then refractor.  Notice we do not create code at this point, we are extracting classes methods etc from the code we have written, the test will fail if we break something and even if you add three levels of dependencies during the refactoring pushing concepts down the stack, it all started in that one place, so by definition it must be tested!   Also notice I have said nothing about faking anything, I would write to go right through the stack at this point calling the database or other services as required, only when I have finished refactoring would I stop again and run my tests to check for speed and then and only then would I introduce a fake object right out on the boundary where the test slows down.

Doing this means that the next test I write will go though the same process, and because we are only mocking where we actually have to, in order to keep the tests fast we can refractor the hell out of the internals of our application all the time knowing because we will have 100% code coverage that we have not broken anything.

If we build the whole application like this our tests will look very different to the implementation, in fact we may have fewer tests than using the modern practice and because we are only using fakes where necessary to improve the speed of the tests the internals of our application are open to complete refactoring without danger of breaking tests.  In fact the tests are now an asset changes can be introduced using the same technique and because each test is covering a specific behaviour the number of tests failing because of change, not because of broken code should be greatly reduced

Posted in Programming | Tagged , | Leave a comment

Too Many Tests

So here we are, there are now so many layers of tests that can be performed on a unit of code that even developers asking questions on stack overflow can’t keep them in line, is it a unit, iteration, functional, black box, white box or what ever test, and I think it has began to harm what we are trying to do.  Lets face it there are actually only two types of tests:

1.  There are the tests that are written by a developer during development that lead to the creation of production code, what I am going to call for this argument  “developer tests” because they are written to aid development

2. and there are tests that may be written by a developer, but it can equally be some other third party, that test production code but do not lead to the creation of production code, and for this argument I will call it “functional tests”, because we are testing already written  functionality.

And that’s it, all the various layers of testing can be slotted into one of these two categories. 

So why is this important?

It goes back to the whole test first pattern of working and a stage that gets forgotten in a lot of the process and because its forgotten its not practiced enough and our code is suffering because of it.

What am I talking about? Refactoring

Remember the test first work pattern

Write a failing test “RED”
Make the test pass “Green”
Then REFACTOR the code!

When you make a change to the internals of your code, how many tests break?

Do you spend more time “fixing” your tests then writing production code?

I think we have all been there at some point, and felt that the tests just knew too much about how the code worked, they had become tightly coupled, did it feel wrong?

When Dan North introduced “Behaviour Driven Development” before all the frameworks were invented to “support” it he talked about how it was a way of helping people discover what tests to write. Even highly skilled developers have trouble getting started when doing test first development, the give the first test a silly name “monkey” until we discover what its doing, just to make headway is not unfamiliar.

I think we have with the need to mock or stub stuff pasted ourselves into a corner which makes it hard to start,  Lets talk about behaviour, what is it.  how about if we change the question,

What happens if there for the given input there is no output result, or the output result is a fail. if we are working on some update or create feature.

Remember we are working at the boundary here, either at the controller or  handler of an API or MVC website application, the execute method of a program etc.  What is our code going to do?

Well its either going to return false, nothing  an error result or throw and exception.

there is our first test :)  Working on the negative case first can help, in getting started, even if we change our minds later and decide that our negative result should do something else, we have a starting point, we now can write a test that checks for our desired result and in the sprit of doing the simplest thing, I tend to plump for returning false, unless we already have a pattern in the application for negative cases.

so at this point we have a test and a handler that has a single method that returns false.  Notice no mocks, no tiers in application no service calls.  This is one of the most difficult things for developers to do, but if we start doing the simplest thing that can work, we will get working code yes it will all start off in the controller and that’s where the important step of refactoring comes in.

When we have the code working in the controller we can refactor out into services and repositories or what ever pattern the code needs, I would even continue to call these in the test, create real things not mocks right up to the far boundary of the application be that an API client call or the database or file store. Why? I want the tests to run fast, and crossing process boundary’s is the slow bit of any application, so if I can’t get something “in memory” then that’s the point to mock, but the code I am refactoring is going to be fast, it all started in the handler remember.

At this yes I know people will say but that’s not a unit test, but I am not writing a unit test, its a developer test, and its testing a single unit of work isn’t it?

The code all started from a single method call, we refactored it into services to make the code readable, maintainable and hopefully reusable, but it originated in one method, its a single unit of work, I am mocking the boundary to keep the test quick.  Remember the tests were green before we started refactoring so the code worked, as we never refractor in red.

we can change the internals of our application without changing any tests the tests are working at the boundary testing behaviour and have no interest in how something works, only in the behaviour it exhibits.

We will actually have fewer tests, and greater freedom because we can now refactor in two directions either pushing code down the application, or conversely bringing code up the application stack or we can refactor across as we do now creating private methods within the application.

Build out the behaviour at the boundary, refactor the code when you have enough code to make it worth while, think clean code, is the method (usually the handler) doing too much? more than one thing? then its time to refactor and push the functionality down your application stack.

It takes practice, and yes your developer tests will cover calls across multiple classes, but remember it is a single unit of work, the code originally started within the one method its doing a single thing, it maybe complex, but it is still a single thing, methods should only ever do a single thing.

Another side effect of working in this way is the only way to create private methods and other tiers of the application is to refactor to them, they are extracted from the boundary code, which also solves the question often asked, how do I test private methods? Answer: you don’t! they are tested by being called by the method from which they were extracted.

Posted in Programming | Tagged , , , | Leave a comment