Specifying properties using Lambda expression trees instead of Reflection

For my TranslationTester project I need users to be able to specify properties on classes so that they can specify that one property maps to another, and to indicate which property they are mapping (or even excluding).

Up until now I’ve only been able to achieve this using strings that match the property names. This is bad because a) it’s difficult to use, and b) it doesn’t support refactoring very well – (although the tests give helpful exceptions when a property has been renamed it would still need to be manually corrected)

Whilst attending DDD7 I saw a demo on LINQ by Jon Skeet and saw a type of Lambda expression I’d not really seen before that was being used to specify a Key on  class. This got me to thinking that this is basically what I want to do. Looking into it I found a few other resources explaining how to do this.

Here’s what I came up with:

target.AddMapping(f=>f.Property1,t=>t.Property1);

which is achieved by:

public SimpleMapping<TFrom, TTo> AddMapping<TProp>(
  Expression<Func<TFrom, TProp>> fromProp,
  Expression<Func<TTo, TProp>> toProp)
{
  var fromName = fromProp.ToPropertyName();
  var toName = toProp.ToPropertyName();
  return AddMapping(fromName, toName);
}  

...

public static class ExpressionExtensions
{
    public static string ToPropertyName<TFrom, TProp>(this Expression<Func<TFrom, TProp>> propertyExpression)
    {
      var lambda = ToLambda(propertyExpression);
      var prop = lambda.Body as MemberExpression;
      if (prop != null)
      {
        var info = prop.Member as PropertyInfo;
        if (info != null)
        {
          return info.Name;
        }
      }
      throw new ArgumentException();
    }    

    private static LambdaExpression ToLambda(Expression expression)
    {
      var lambda = expression as LambdaExpression;
      if (lambda == null)
      {
        throw new ArgumentException();
      }
      var convertLambda = lambda.Body as UnaryExpression;
      if (convertLambda != null
          && convertLambda.NodeType == ExpressionType.Convert)
      {
        lambda = Expression.Lambda(convertLambda.Operand, lambda.Parameters.ToArray());
      }
      return lambda;
    }
}

Note: Some of this is borrowed/inspired very heavily from Moq, particularly the conversion of lambda convert expressions.

One of the nifty things about it is that the syntax of the two Expressions enforces a lot of the requirements on the property types. By using TProp the properties that are referenced in the expressions must be of the same (or common base) type.

I’ve been using the new way of referring to properties and I must say I’m really happy with it!

Casting reflected types – What I learnt

For my Translation Tester I originally wanted a user to be able to add a mapping between two properties of different types, so for example they could say that a property of type Int16 would be directly assigned to a property of type Int32 by the translator. This turned out to be real tricky to get working, and after much analysis I’m not sure it should even be done as part of a ’simple mapping’.

The problem is that the types of the parameters are found using reflection, so you then need ways to tell whether there are any commonly supported ways to convert one type to another, and then a way to actually do the conversion to verify that the mapping was fulfilled.

To expand the problem let’s list some of the possible scenarios:

• Primitive type that can be implicitly cast to another primitive type (e.g. Int16 assigned to Int32)
• Primitive type that can explicitly cast to another primitive type (e.g. Int32 to Int16)
• Primitive/Struct/Class to Primitive/Struct/Class where an implicit cast operator has been added
• Primitive/Struct/Class to Primitive/Struct/Class where an explicit cast operator has been added

This gets pretty confusing, particularly with all the permutations for the last two.

IConvertible/Convert.ChangeType

When dealing solely with the primitive types there is an interface IConvertible that they all support, that allows conversion between the primitive types. The Convert.ChangeType() method simply provides a nice wrapper around the IConvertible interface. One problem here is that for several of the primitive types they have the whole interface, but individual conversions will always fail at run-time (e.g. decimal to DateTime).

So this facility can be used for primitive types at comparison time, but to determine whether a simple mapping should be allowed you’d have to actually attempt a dummy conversion, using something like Activator.CreateInstance().

TypeDescriptor.getConverter

I posted a question on StackOverflow.com, one of the answers suggested I look into TypeConverters. From what I can see the built in TypeConverters are focussed on converting from and to strings. This facility could be used to provide Converters for primitive types and even allow users to add converters for custom types. But this all seems rather heavy.

Dynamic Cast

This post contains an interesting implementation for dynamic casting based on looking for implicit (and if necessary explicit) cast operator methods on the reflected types and using them to perform a cast. One problem is that the primitive types don’t use cast operators to cast between themselves, rather they use the IConvertible interface. So this wouldn’t work for primitive-to-primitive conversions.

What I’ve done

This all seemed to be getting really confusing, so what I’ve done at the moment is only allow simple mappings to be added where the types are identical. This makes the code a lot easier, and also removes any magic being hidden from the user; if they want to map a property to a property of another type they can specify this using a complex mapping (predicate).

What I might do

Having written all this down I think it might be possible to do what I originally intended. using the following steps to add a mapping (confirm that it’s possible):

• If both types are primitives – Consult a local dictionary of valid mappings
• else use the Dynamic Cast method to determine if there is a cast operator specified
• If no cast operator specified see if there is a custom converter (perhaps based on TypeConverter).

Then the following steps to verify the mapping

• if both types are primitives use Convert.ChangeType() to convert the from type to the to type.
• else use the cast operator or custom converter specified.

Another thought I’ve had is that you might want to limit the allowed conversions to just implicit conversions (where you wont lose precision/data), this could be achieved by splitting the primitive mappings into 2 lists, and separating the dynamic cast into 2 checks.

Translation Tester – Progress update

It’s been a little while since I blogged anything about my TranslationTester project; I have however been hard at work doing some coding for it. The source is all now on google code as are the user stories that I originally blogged along with some new ones. I’ve done a reasonable amount of work and now have something that’s semi-usable, although it is currently limited to simple mappings.

So far the user stories I’ve done are:

Exclude property

Simple mapping

Verify ‘from’ instance

I’m currently working on the facility to automatically add mappings based on identical property names.

Translation Tester, part 4 – High Level Design

In the fourth part of my Translation Tester series I’d like to give a brief description of the overall mechanism I’m planning to implement for testing translations. The main aim is to test a class that does a translation between two types; this breaks down into the following tests:

• Test that all properties have been mapped (or excluded from the test)
• Test that all the specified mappings are fulfilled by the translator
• Test that the types being translated have not had changes which invalidate the translator.

The Translation Tester will allow a developer to create a specification for the translation; the translation will be based on public properties of the types being translated (may extend this to public fields and methods in the future) and will be found using reflection when the specification is created.

A specification will then be built up by adding mappings between properties on the types, several types of mapping will be supported; the most basic being where one property is directly assigned to another property with the same type. Complex mappings such as where one property maps to several other properties after going through various modifications will be supported by allowing the developer to specify Predicate type delegates that should be called to test the mapping.

The test for whether there are any unmapped properties will be based solely on the ‘From’ type, as a translation will be treated as unidirectional, and I believe it only really makes sense that the ‘From’ type be fully specified in the translation.

My development stack

A couple of days ago I started developing my TranslationTester. To do this I needed to get my home PC up to speed in terms of a development stack. At work I have a pretty much MS stack (Rhino Mocks being about the only exception), but I wanted to see what the open source offerings would be. So currently I have:

• IDE – Sharp Develop, (3.0 Beta 2)
• .Net Framework – (3.5 SP1)
• Windows/.Net SDK – (6.1)
• Code/Style Analysis – FxCop (1.36), StyleCop (4.3)
• Testing – NUnit (2.4.8)
• Source Control Client – Tortoise SVN (1.5.3)
• Source Control Server – Google Code
• Issue Tracking – Google Code

So far I’m pretty impressed, particularly with #Develop. The integration between them all is top notch; One of my worries with Subversion was that to get integrated source control with Visual Studio required a commercial plugin (as far as I’m aware), with #Develop and TortoiseSVN there’s great integration and it tracks and indicates which files you’ve changed right in the IDE.

I’ve setup a Google Code project for the TranslationTester, I’ll start linking to it soon, once I’ve got a bit more up there.

Extending unit testing frameworks with the Translation Tester

While I’ve been working on my TranslationTester project, I’ve been wondering whether the tool should either be a stand-along tool, or whether it would make sense to contribute it to one of the many unit testing frameworks out there (NUnit, MbUnit, xUnit). I don’t think it’s ‘frameworkey’ enough to contribute to the main products, but perhaps it could be contributed as some sort of extension/plugin etc.

My current thinking is that I’ll create it framework-independent, and then maybe write simple wrapper classes that allow it to be used in each framework in a more integrated fashion, e.g. Define a base TestFixture class for NUnit.

Translation Tester, part 3 – Requirements

In the 3rd part of the Translation Tester series I intend to come up with some initial requirements for the product. This will then be used to create a product backlog and drive development via ‘Test Driven Development’. I’m going to try and express the requirements as ‘user stories‘.

The users

First a quick introduction to the users; I realise that I should probably have some real users providing my user stories, and in due time I hope to add to my initial user stories with real end-user stories.

The Traditional Developer is a developer who writes code and then writes some unit tests afterwards to test parts of their code.

The Test Driven Developer uses tests to drive the design and development of their code

The term Developer will be used to refer to all developers.

The Development Manager manages a team of potentially changing developers over a period of times, potentially on several projects.

The Build Engineer manages and monitors a product’s automated build.

The Stories

So without further ado lets start writing some stories, although they’ll probably go from simple to complex no prioritisation or ordering is intended at this stage.

As a Developer
I want to test my translator classes
So that I have confidence that they work

As a Developer
I want to specify that a property should not be translated
So that I can have minimal classes

As a Developer
I want to be able to clearly see properties that were excluded from the translation
So that as requirements change I can easily identify the changes to be made and bug fixing should be easier

As a Developer
I want to know when my code change has broken another area of the code
So that I can have confidence in making changes without the risk of regression

As a Developer
I want to reduce the amount of ‘boilerplate’ code for testing translators
So that I can work more efficiently

As a Developer
I want to be able to specify complex translations where necessary
So that as much as possible of the translation can be tested

As a Developer
I want the tests for my translator to specify the desired output, not how the output is achieved
So that my tests are more robust, and don’t just duplicate the production code

As a Developer
I want each test to test one aspect of the translator
So that a failed test clearly indicates the reason for the failure and other failures are not hidden

As a Developer
I want to exercise the translator with a wide range of inputs
So that I can test the translator handles a wide range of inputs correctly

As a Developer
I want to be able to use a mocking framework to test that a call was made as part of the translation
So that I can test parts of the translation that do not easily allow for state based testing

As a Developer
I do not want to be forced to use a specific mocking framework
So that I can work the way I (and my team) work.

As a Development Managers
I do not want to be tied to any other dependencies (such as on a specific unit testing framework)
So that I can choose what tools the team will use and change my mind in the future

As a Build Engineer
I want to automate running of the translation tests
So that I can be notified of a failure without any manual intervention

As a Test Driven Developer
I want to be able to start from a small skeleton translator and test class
So that I can get quick feedback on my development

As a Test Driven Developer
I want each small test to indicate the work that must be done in the translator to make the test pass
So that I can use the tests to drive my development.

Translation Tester, Part 2 – Approach

In part 1 of this series I described the problem I’m trying to solve. Before I delve into solving the problem I’d like to take a quick moment to reflect on the development approach I plan to take for this project.

This is the first time I’ve tried to do any development outside of work, so I’m going to acknowledge from the outset that I don’t have a very detailed approach in mind, and I’m more than likely to change it as I go along. This fits quite well with my experience of agile development methodologies, so I’m going to try and put as many of the agile techniques into practice that I can. I’m pretty much development centric, so I’d also like to have a look at some of the related skills relating to requirements analyis and project management.

So as I see it the stages that I’ll need to go through in approximate order are:

1. Initial requirements
2. Iteration 0 – Get a skeleton solution with Continuous Integration etc
3. High Level Design – I don’t want to get into too much detail at this stage, but just taking a moment to think of a broad design might help
4. Find a suitable requirement/use case and implement it using Test Driven Development
5. Analyse what comes out of the implementation and update requirements etc as needed. Lessons learned etc.
6. Repeat steps 4 and 5 until all the requirements have been met.

So that’s my very rough approach, I’m sure I’ve missed oodles, so please drop a comment to suggest any pit-falls etc.

Translation Tester, part 1 – The problem

Introduction

This post is the 1st in what may become a series of posts detailing the development of a test support tool for testing ‘Translators’. I’m planning on blogging the development of this tool for a few reasons. One reason is to try and gauge if there is any interest in the tool (may impact whether I bother to continue development and maybe even extending it to a releasable open source product). Another reason is so that if anyone out there thinks there is a tool out there that already does this they can tell me and save me a whole load of effort

Some useful information before I state the problem…

It may be worth me starting by explaining what I mean by a ‘Translator’; When you have a layered architecture in a software product/suite you often have de-coupled representations of concepts for each layer rather than having a common set of representations for all the layers. This means that whenever you cross a layer boundary you need to convert from one representaion of the data/concepts to another, the class that does this conversion I would call a Translator.

Entity Translator diagram

Modern software development seems to be more and more layered, and with some concepts like Domain Driven Design (DDD) and The Onion Architecture advocating decoupling different parts (or layers) of an application from each other, it seems to be more and more prevalent.

The problem

The problem I’ve found with Translator classes is that they can be pretty difficult to unit test reliably, flexibly and effectively.

1. Unit tests often seem to over-specify the behaviour and are pretty much just directly duplicating the translator code.
2. Writing unit tests for translators can be long and laborious with seemingly little benefit, it’s almost mindless coding. However a lot of bugs seem to be due to translators, so they obviously do need to be tested.
3. When a new property/field is added to one of the types being translated from/to the test will often continue to pass, and there will be no indication that the translator is not attempting to translate the field.
4. An untranslated property often doesn’t cause an immediate problem (compilation failure, unit test failure, immediate run-time failure). Rather the missing data will show up some time later as a hard to diagnose bug.
5. When a bug is identified it can often be traced back to a translator not mapping a field, when you have several layers each with each layer boundary having a translator there can be a lot of places to look!
6. Over time translator tests go out of date, and not even due to any changes of the class being tested changing (the translator), rather that one of the types being translated has changed.