Monday, December 30, 2013

DUnitX and my plans

I have been a big fan of using unit testing frameworks.   I use both DUnit and DUnitX for my Delphi tests.

I am right now contributing to DUnitX as I see it as the future framework of choice for Delphi Developers.

I just wanted to talk in public about some of work I have going on around DUnitX.   My goal is simple to get feedback from the Delphi community.

IDE Expert

It is really easy to setup a test project and test fixtures with DUnitX, but I thought it could be easier.

So I created a new Delphi Open Tools Expert that does the following:

  • File | New | Other | Delphi Projects | DUnitX Project
    • Creates a new DPR/DPROJ
    • DPR Source is modeled after the DUnitX example unit tests.
    • Base Project Search Path set to $(DUnitX)
    • Optionally creates a Test Unit
  • File | New | Other | Delphi Files | DUnitx Unit
    • Creates new Delphi unit
    • Adds   DUnitX.TestFramework to uses
    • Creates a new class with correct attributes, you get to specify class name
    • Optionally creates Setup and TearDown methods
    • Optionally creates Sample Test Methods.
    • Registers the TestFixture in the initialization section.
Basically it's not much, but it provides a framework to reduce your time to get to writing actual test code.    I am nearly done with this code, I wrote most of it during the Christmas Break.   Hopefully in the next week I can finish this.  It's going to take some time, as I have to build a machine with Delphi 2010 through XE5 on it to test this functionality as I only have XE and XE5 installed right now.

To see the current state of the code check out this expert branch
Update:  Code is now part of the master branch on the Main Project

Data Driven Test Cases


The potential to have data driven test cases is the main reason why I started looking at DUnitX.   
  [TestFixture]
  TMyTestObject = class(TObject)
  public
    [Setup]
    procedure Setup;
    [TearDown]
    procedure TearDown;
    // Sample Methods
    // Simple single Test
    [Test]
    procedure Test1;
    // Test with TestCase Atribute to supply parameters.
    [Test]
    [TestCase('TestA','1,2')]
    [TestCase('TestB','3,4')]
    procedure Test2(const AValue1 : Integer;const AValue2 : Integer);
  end;

The method Test2 above shows up as two different tests, the first time it's called with 1 and  and second time it's called with 3 and 4.

I recently made a change to underlying structure of the code. First I created a new record called TestCaseInfo followed by two new abstract attribute classes
 

 
   ///  
   ///    Internal Structure used for those implementing CustomTestCase or
   ///    CustomTestCaseSource descendants.
   ///  
   TestCaseInfo = record
 
     ///  
     ///    Name of the Test Case
     ///  
     Name : string;
 
     ///  
     ///    Values that will be passed to the method being tested.
     ///  
     Values : TValueArray;
   end;

  TestCaseInfoArray = array of TestCaseInfo;


  /// 
  ///   Base class for all Test Case Attributes.   
  /// 
  /// 
  ///   Class is abstract and should never be, used to annotate a class as a
  ///   attribute.   Instead use a descendant, that implements the GetCaseInfo
  ///   method.
  /// 
  CustomTestCaseAttribute = class abstract(TCustomAttribute)
  protected
    function GetCaseInfo : TestCaseInfo;  virtual; abstract;
  public
    property CaseInfo : TestCaseInfo read GetCaseInfo;
  end;

  /// 
  ///   Base class for all Test Case Source Attributes.   
  /// 
  /// 
  ///   
  ///     Class is abstract and should never be, used to annotate a class as a
  ///     attribute.   Instead use a descendant, that implements the
  ///     GetCaseInfoArray method.    
  ///   
  ///   
  ///     Note: If a method is annotated with a decendant of
  ///     TestCaseSourceAttribute and returns an empty TestCaseInfoArray, then
  ///     no test will be shown for the method.
  ///   
  /// 
  CustomTestCaseSourceAttribute = class abstract(TCustomAttribute)
  protected
    function GetCaseInfoArray : TestCaseInfoArray; virtual; abstract;
  public
    property CaseInfoArray : TestCaseInfoArray read GetCaseInfoArray;
  end;


With these two classes, I changed TestCaseAttribute to descend from CustomTestCaseAttribute.   
Then I changed the architecture to create a Test based on the TestCaseInfo record structure, that is obtained by either the CaseInfo or the CaseInfoArray properties of the abstract classes.

This little change provides for some really nice functionality, for example I have working sample that uses FireDAC to provide the values to my tests method

  FireDacTestCaseAttribute = class(CustomTestCaseSourceAttribute)
  protected
    FTestName : String;
    FConnectionName : String;
    FSelectStatement : String;
    function GetCaseInfoArray : TestCaseInfoArray; override;
  public
    constructor Create(const ATestName : String;const AConnectionName : String; const ASelectStatement : String);
  end;

  // Which then can be used like this:
 [FireDacTestCase('TestName','MyDBConn','select strVal, IntVal from Table');
 procedure MyTestMethod(AValue1 : String; AValue2 : Integer);

// Right now the test come out names 'TestName1', 'TestName2', 'TestName3' although that will 
// change before I commit my code to allow specifying values.  

//Most likely passing the TValuesArray with a Format call on testName like this:

 [FireDacTestCase('TestName%0:s%1:s','MyDBConn','select strVal, IntVal from Table');
 procedure MyTestMethod(AValue1 : String; AValue2 : Integer);

This attribute is not in the DUnitX.TestFramework.pas to avoiding creating dependencies on those that don't use this functionality.       This needs quite a bit of work before it's polished enough for general use, but it's what I am working on next after the expert is submitted as a pull request.  

Note: Since FireDac changed it's naming, I believe it might be XE4 or XE5 specific.


Repeat Attribute


DUnitX defines a RepeatAttribute that is currently not implemented.    I made an attempt at implementing it, and I don't like it.    If anyone has better idea, I would be happy to entertain it.

Otherwise, I have some small improvements I think I will make and will submit it again.

My implementation can be found in Pull Request #26  which won't auto merge due another change being implemented first,  but can be viewed in a working state in my RepeatAttribute branch.

Future Plans

Things I want see in a Unit Testing framework is vast, I not sure what I will start on next but here are some areas I am considering with no preference on order.  Note: this is not a road map as some may never be done (at least by me)

  • Test Categories similar to NUnit as I have 10k+ of DUnit tests currently and categories might make it easier to group run on the ones I am interested in.
  • VCL and Firemonkey  GUI Runner
    • Understands and can filter on categories
    • Quick Filter by name
    • Run all or specified tests
    • Simplify the DUnitX project source and make easier to select which runner is used.
      • VCL
      • Firemonkey
      • Console
  • Something similar to GUITesting.pas found in DUnit
  • Load and run tests stored in BPLs.
  • Data Driven Tests Enhancements
    • dbExpress
    • CSV Test Case Source
    • XML Test Case Source
    • TestCaseSource Attribute
      • IOC looks up the Test Case Source Builder Interface
      • Other Sources implement Test Case Builder Interface and Register it in IOC Container.
  • Remote Test Framework
    • Think mobile, tests are on the device the runner is on your development machine.
    • Think Test Farm, multiple machines, with different configurations all running tests.
  • Implement TestInOwnThreadAttribute 
  • Find better way to test new functionality in DUnitX.

Thank you

And last but not least I would like to thank Vincent Parrett and his team for the set of great tools he as produced for the Delphi community.

I use the following:
  • FinalBuilder - Build Automation Tool - Commercial - Well worth the price!!!!
  • ContinuaCI - Build Server - Commercial - Free single for a single build server/agent.
  • DUnitX - Unit Testing Framework - Open Source
  • Delphi-Mocks - Mocking Framework - Open Source
  • DUnit-XML - DUnit XML output in NUnit Style - Open Source
There is more Open Source he has produced all listed on GitHub.




12 comments:

  1. An IDE expert for DUnitX is about the only thing holding me back from using it. Looking forward to seeing it finished :-)

    ReplyDelete
  2. "I have to build a machine with Delphi 2010 through XE5"

    I already have this Robert, so if I can help (testing etc) then let me know. For now, I can spare the odd day here and there.

    Cheers

    Rob

    ReplyDelete
    Replies
    1. I have working XE and XE5 packages, but I need to be able to save the packages for each version of delphi then verify they load and operate correctly. Tonight I got Windows 8.1 loaded on a VM and downloaded the installs.

      Delete
  3. I really do not like this attribute abuse....
    How do you debug this?
    You now lack of compile time checking.
    How do you make your code maintainable?
    This is just not pascal any more.

    ReplyDelete
    Replies
    1. Attributes are metadata looked up at runtime. Declaration of attributes, and the attribute code is compile time checked.
      To debug problems with attributes you typically need to know where you are looking for and acting on the given metadata.
      I would agree this is not typical pascal... But then I don't want old style pascal, I want to see new language features appearing.

      Delete
    2. This is were I do not like it.
      See what Jeff said:
      "Don't use fancy OOP features just because you can. Use fancy OOP features because they have specific, demonstrable benefit to the problem you're trying to solve. You laugh, but like Rico, I see this all the time. Most programmers never met an object they didn't like. I think it should be the other way around: these techniques are guilty until proven innocent in the court of KISS."
      See http://www.codinghorror.com/blog/2004/10/kiss-and-yagni.html

      IMHO attributes should define only interface specification, not implementation.
      For instance, it may add some information to access an external API in an interface type definition (like is done with XE5 and Android's JNI). I'm fine with that, and find it pretty useful.
      But if attributes are just another way of putting code at the interface part - I do not like that, sorry. And you are stuck to TValue kind of parameters... real-world methods will probably need to supply objects or stubs/mocks...

      Such use of attributes is IMHO just a trick to test a feature, and has no benefit - only the drawback of making a confusion between interface and implementation part of unit.
      In Java and C#, there is no distinction between interface and implementation - so it even less confusing to put some logic within attributes.
      But even in Java or C# there is some backward model into plain POJOs/POCOs and code-based configuration instead of annotations/attributes, even for ORMs.

      Delete
    3. If you have a better idea on how to implement Data Driven tests I am listening.
      I have a key requirement is that each item being tested needs to have it's own test name.

      Delete
    4. Hi Rob, I don't know how to do it in DUnitX but in DUnit it is quite easy, I do it all the time. You need to descend a new TestSuit class, override the AddTests method and then use RegisterTest.

      Delete
    5. Actually what I don't like is the necessity to specify a name when using the testcase attribrute as you don't need that in nunit either.
      See how I did that for DUnit: http://stackoverflow.com/questions/8999945/can-i-write-parameterized-tests-in-dunit

      Another method to write data driven tests is using DelphiSpec which might be compatible (or integrated?) with DUnitX in the future.

      Delete
  4. I saw that you are using an abstract class "class abstract(TCustomAttribute)".
    I read somewhere, that the abtsrtact class keyword has no meaning in Delphi. It was just introduced for .Net compatibility and is still there for backward compatibility. Is that right?

    (http://docwiki.embarcadero.com/RADStudio/XE5/en/Classes_and_Objects)
    Note: Delphi allows instantiating a class declared abstract, for backward compatibility, but this feature should not be used anymore.

    ReplyDelete
    Replies
    1. abstract classes really do have no meaning other than helping to remind the developer not to create instances of them. I believe the docwiki is talking about instantiating an abstract class being allowed, but should not be used. I don't believe it's talking about the abstract keyword.

      Delete
  5. The class abstract is intended to prevent the class being instantiated, and was introduced in relation to .NET. In Delphi, you can actually instantiate a class marked as abstract, but you should not.

    ReplyDelete