Saturday, March 29, 2014

Why Delphi generics are annoying

First of all: I like generics, I love them. You can do all kinds of neat stuff with them... until they bite you. Then they crash your compiler which might make your IDE hang or actually generate wrong source code.

So today I will talk about about the smaller or bigger annoyances of Delphi generics.

They turn your binary into a fat bastard


This might not be a problem at first but you might end up in a situation where your application reaches 36MB just because you are heavily using generic types. And actually duplicated code that is exactly the same for every bit. This by the way is one of the reasons your empty default applications are growing constantly for the recent releases of Delphi: because now generic collections are used all over the place in the runtime. And when I use TList<TButton>, boom, another 10K added to your binary size (approx in XE5). Too bad if you are using more advanced and feature rich collection types like Spring4D. Then every use adds 65K (actually we got down to that number from close to 100K a few months ago).

How could this be solved? Either by the linker fixing this by figuring out identical methods and removing duplicates (the C++ linker has an option for that called COMDAT folding) or the compiler itself could be smart and generate code for equal types only once. Of course he should pay attention to any use of TypeInfo(T) or T.Create for example because that code is actually different. C# does something like this.

Even worse when you think you just use an abstract class (like TEnumerable<T>) in an interface and it suddenly compiles in TList<T> although you never ever created one! Why? Because in the implementation of TEnumerable<T>.ToArray a TList<T> gets created. And because that method is public and virtual (I guess) it won't be smartlinked out.

If you ever tried delegating a generic interface with the implements keyword you either suffered from the bug I mentioned earlier or you realized that this again adds to the binary size because the compiler creates stub methods that makes this possible. I tried to achieve this using interface delegation but that did not result in reducing the binary size to a number I like (actually it was 15K just for interface stubs! Yes the IList<T> interface has quite some methods)

type
  TObjectList<T: class> = class(TList<TObject>, IList<T>)
    // ...
  end;

Interface methods must not have parameterized methods


Also known as compiler error E2535. So you cannot write something neat like this:

type
  IEnumerable<T> = interface
    function Select<TResult>(
      const selector: TFunc<T, TResult>): IEnumerable<TResult>;
  end;

C# has them and hell even Java has them! As far as I know this is because of some implementation detail and the same reason why you cannot have parameterized virtual methods.

No helpers for generic types


How much easier would that make our lives because we actually could extract some code from the generic class to a helper which we might not be used for every instantiation we are using and thus help us with the binary size problem. If we also had helpers for interfaces we could realize something like the method above in a helper because it does not need to be implemented in some class by design (is just needs the iterator). Someone might scream that helpers are not meant to be used by design but I disagree. I actually would like to have them improved so we could use more than one at a time. Even more so now that we have all these helpers for intrinsic types sitting in the SysUtils unit that make it impossible to add own behavior without hiding (or copying, argh) the built-in one. It could also avoid making ugly design decisions to make the underlying array of a TList<T> public to everyone without caring about the fact that it might contain garbage.

The compiler just goes bonkers


If you ever really heavily used generics you might experienced all kinds of funky internal errors where it suddenly stops and the marker points to the line after the last one in a unit or the compile times just goes beyond minutes you know what I am talking about. Even more if you tried to find workarounds to the previously mentioned problems. Like you made a generic record with lots of methods and watch the compiler spinning in circles performing "shlemiel" lookups for the stuff it is compiling. Or it completely fails doing anything and instead running in circles allocating memory until everything is lost.

Addendum:
After I wrote this I noticed another problem. Imagine this class:

type
  TFoo<T: class> = class
  strict private
    procedure Bar(const item: TObject); overload;
  public
    procedure Bar(const item: T); overload;
  end;

What method do you expect to be called when having a TFoo<TButton> and calling Bar passing in a TButton? The second of course. First it matches the argument and second it's the only public method. Do you think anything should be different when you have a TFoo<TObject>? No? Well the compiler does not agree and happily calls the private one.

No support for conditional compiling


This could solve the problem we discussed earlier where I have a generic but want to use specific implementations depending on the type parameter. Currently something like this is not possible:

function TCollections.CreateList: IList<T>;
begin
{$IF TypeInfo(T).Kind = tkInterface}
  IList<IInterface>(Result) := TList<IInterface>.Create;
{$ELSEIF TypeInfo(T).Kind = tkClass}
  IList<TObject>(Result) := TObjectList<TObject>.Create;
{$ELSE}
  Result := TList<T>.Create;
{$IFEND}
end;

Finding solutions


The good news for those concerned about binary size (and with Spring4D that supports the mobile platforms being released soon you should or your customers might hate you for apps that are dozens of MB big): Spring4D will support something similar to how C# handles generics. That means with the SUPPORT_GENERIC_FOLDING switch on (it is disabled by default and not supported on Delphi 2010) it will just create TList<TObject> and TList<IInterface> when you are using the TCollections.CreateObjectList<T> and the new TCollections.CreateInterfaceList<T> methods. That means only 2.5k overhead for every list you are creating that holds objects or interfaces (implementing something for pointers is left as an exercise to the reader).

Sunday, March 23, 2014

Sorry Uncle Bob - but NO!

I just came across this article and was very shocked by the code and so I decided to travel back in time to read the original article by Robert C. Martin and I have to say:

I sincerely have to disagree with you, Uncle Bob!

Let me explain why


While he is trying to separate DI container code from his application logic - which is a good thing but not necessary if you design your architecture without a certain DI container in mind (as Mark Seemann writes in his comment to Martins post) - he does one ugly mistake. He introduces a static factory to his BillingService class.

This thing is nothing else than a service locator call in disguise!

While he shows how nicely you can test the BillingService class he does not show how hard it will be to test the class that contains the logic he commented with "Deep in the bowels of my system." earlier in his post. Because this is where things start to get really messy. In this code he reaches into the BillingService.factory and uses it to create a new BillingService instance which in his real application will call into his DI container and retrieve a fully functional BillingService instance. But how are you supposed to test this? The constructor of DeepInTheBowelsOfMySystem obviously does not tell you anything about BillingService or a factory. This a case when the API lies to you - and in this case really badly. Since you not only call into some dependency that you might pass into your SUT. No you call into some static member and we know that global states are bad, right? This means for a test to succeed we need to setup the BillingService.factory which we will encounter after running into a null pointer exception (or access violation).

This code is as bad as calling into a service locator at this place to get a BillingService instance.

How to solve this problem?


Either pass in a BillingService into your DeepInTheBowelsOfMySystem class or pass in a BillingService factory. Both are things that can be done with "Poor Man's DI" and with most Dependency Injection Containers. Like Spring4D supports factories out of the box. If you register something like this:

myContainer.RegisterType<IBillingService, TBillingService>;

Then the container will automatically resolve something like this:

constructor Create(const billingServiceFactory: TFunc<IBillingService>);

When you see this constructor you immediately know what dependency it expects and it does not take you couple of runs or reading through the code to find out what you need to setup for a test. The SUT should tell you what it needs and nothing more.

As you can see it is very easy to step into traps that lead to hard to test code and it's not always trivial to solve this - so even knowledgeable people like Robert C. Martin might step into these.

Wednesday, March 5, 2014

Spring4D roadmap

Ok, actually less of a roadmap but more of some news regarding the project - but I thought with the recently published Delphi roadmap for this year this might be a good title. ;)

What has been done?


Spring4D has been moved to BitBucket some while ago as most of you know - for those that do not this is a reminder as the project on Google Code will not be continued. If anyone knows a nice way of redirecting from there to the new page let me know. While we did this we also changed from svn to git which made it much easier to work than with svn - except a few times when git was a bit confusing if you never worked with it before. I will share a few things on that in another post.

In the past few months we made some nice changes and cleaned up the source. The collection types have been improved and I am certain we found a good balance between complexity and usability. I hope they will be used by other projects to have one accepted collection libary (apart from the RTL ones) in order to be compatible between these projects (think of an ORM library that uses the same collection types as the MVVM or binding framework for example).

Also Honza RameŇ° and Jeroen Pluimers have been busy on making Spring4D ready for OSX, iOS and Android. These changes are currently still in an extra branch but if you are interested in using the library on these platforms check them out.

The are also some nice additions to the dependency injection container which I will talk about in detail in another post.

We are now down to fixing bugs that we (or you) find - please feel free to report them in the issue tracker. During that time we will also improve source documentation and write guides on how to use Spring4D. Our plan is to finally (the project turns 5 this year, can you imagine) officially release a version 1.0 by the end of march.

What's next?


After the release some work on DSharp will be done removing the redundancies (like the collection types and the multicast events) and it will be then using Spring4D.Base.

Some things that we are planning to add to Spring4D this year are:

- adding more collection types
- improving the RTTI extensions
- adding some utility types (like Weak<T>)
- adding expression types (similar to those from DSharp)
- improving the dependency injection container

While we want to add features Spring4D will still focus on some core features and leave things like GUI or database things for other projects. If you have suggestions or want to contribute in any way feel free to contact us. You are also welcome to discuss or ask things in the Spring4D google group. We are also eager to hear from you if you are using Spring4D in your projects.

Monday, February 24, 2014

Packages and initialization

Packages are an approved concept and they have been working fine for over a decade now, right? Well yeah, they should be. However my coworkers and me have been bitten by them not only once. Especially since our software consists of lots of DLLs that are linked against the RTL/VCL, 3rd party and some of our own packages.

Everything alright so far you say? Actually not because of some simple fact: initialization parts of units inside of packages might not be executed when the package gets loaded but when a binary gets loaded that uses it. Not a problem? Not really unless your initialization and finalization parts are doing the correct thing.

Let's take a look at some unit that might be part of a package:

unit Foo;

interface

procedure Bar;

implementation

uses
  Contrns;

var
  list: TObjectList;

procedure Bar;
begin
  if not Assigned(list) then
    list := TObjectList.Create;
  // do something with list
end;

initialization

finalization
  list.Free;

end.

What do you think happens when you load a DLL that uses this unit given that no other module has used this unit yet? Yep, the initialization part gets executed. Since we are using lazy initialization here there is nothing to execute. Now we call Bar and the list variable gets initialized since as we know global variables are initialized with zero (or in our case nil). Then we unload the DLL and our finalization part gets called and cleans up the instance. What do you think happens when we load the DLL again and call Bar? Most likely an AV. Wait what? Yes, list has not been set to nil because remember this unit is part of a package and thus list has not been set to nil again!

There are several places in Delphi and 3rd party source I have seen that are coded like this. Our solution? Make sure every unit that causes trouble is referenced in the main application EXE to force initialization at startup and finalization at shutdown of the application.

P.S. Did anyone shout "Never use FreeAndNil" back there?!

Sunday, December 8, 2013

Dependency Injection best practices - lazy initialization

“You gotta know when to be lazy. Done correctly, it's an art form that benefits everyone.” Nicholas Sparks, The Choice

I admit this was taken out of context a bit - but we will see how this applies to dependency injection.

Identifying the problem


When dealing with dependency injection - and when I am saying dependency injection I am referring to the software design pattern - you are facing one problem pretty quickly. The first instance you create (or resolve using the container at the very root of your application) needs everything injected that you will ever need, either directly or something you inject takes that. This will result in tons of wire up code - and this is one reason why using a DI container might be a good idea for an application that you design with this pattern in mind - however a DI container might save you from writing tons of boilerplate code but it still does not solve the problem of instantiating the whole object graph at the start of the application on its own.

There are several ways on how not to do dependency injection - please read the linked article first as it will explain some important things that I felt would be redundant to repeat because they are very well explained (and the few C# code snippets are easy to understand).

...

Things to avoid


You are back? Good.

Let's talk about the service locator and the GlobalContainer - both things you might have seen in several Spring4D examples. If you are writing an application with DI in mind - don't use the service locator.

The service locator might be a nice way of introducing the DI container to an already existing application because you can plug it in at some place deep within the existing code and then kick off the container from there. But don't (ab)use it just to replace constructor calls with something else or to get around actually doing dependency injection. Using the service locator is not DI!

Another important thing to know about the use of the DI container - it does not serve to produce value objects - objects that only hold some values (think of TCustomer or TOrder) but for service objects - objects that actually do some work (like TCustomerValidator or TOrderProcessor). With that in mind you understand the possible problem of "I need to pass some data that is only known later in the application to the container" actually is not a problem. If you need to create customer or order objects then you either register a factory to the container that takes the necessary arguments and constructs an instance or you create that object in your code. No, this is not evil and there is no tight coupling you need to remove - if you follow some guidelines that Misko Hevery explains in one of his articles.

Now what about that GlobalContainer singleton we see in all the Spring4D examples? Actually if you want to make it 100% correct you should create your own TContainer instance - you remember you only need that at the start of your application to register all your types and then resolve the first instance and from there on it will never be seen again throughout your whole application.

If you ever have heard someone telling you that you should put your classes in your implementation part of the unit and then register them in the initialization part of that unit - never ever do that please!
First you are making these classes untestable (because not accessible from outside) without the DI container - always write your code in a way that it is testable without a container. Second you are tightly coupled to the GlobalContainer instance - what if you created your own one - you would be screwed.

Solving the problem


But now let's get back to our initial problem. Having things that might be needed later or even never throughout one run of the application. That is when we need lazy initialization.

Let's see how the example from the other article would look in Delphi:

container.RegisterType<IExampleService, TExampleService>('default').AsDefault;
container.RegisterType<IExampleService, TAnotherExampleService>('another');

container.RegisterInstance<TFunc<string, IExampleService>>(
  function(name: string): IExampleService
  begin
    Result := container.Resolve<IExampleService>(name);
  end);

container.Build;

Assert(container.Resolve<IExampleService> is TExampleService);
Assert(container.Resolve<TFunc<string, IExampleService>>
  .Invoke('another') is TAnotherExampleService);

So if we had another class that would take this factory as an argument on its constructor the container would inject this anonymous method there - keep in mind that we used RegisterInstance which returns the same anonymous method every time. In this example it is completely valid because the anonymous method has no state but pay attention when you use variable capturing.

You can imagine that this will be much code to write if you have many service types and you want to resolve many of them lazily. But just like other DI containers the Spring4D container has built-in support for that. Take a look at this code:

type
  THomeController = class
  private
    fService: IExampleService;
    fServiceFactory: TFunc<IExampleService>;
    function GetService: IExampleService;
  public
    constructor Create(const serviceFactory: TFunc<IExampleService>);
    property Service: IExampleService read GetService;
  end;

constructor THomeController.Create(const serviceFactory: TFunc<IExampleService>);
begin
  inherited Create;
  fServiceFactory := serviceFactory;
end;

function THomeController.GetService: IExampleService;
begin
  if not Assigned(fService) then
    fService := fServiceFactory();
  Result := fService;
end;

container.RegisterType<IExampleService, TExampleService>('default').AsDefault;
container.RegisterType<IExampleService, TAnotherExampleService>('another');
container.RegisterType<THomeController>;
container.Build;

Assert(container.Resolve<THomeController>.Service is TExampleService);

As you can see we did not register the factory method anywhere. The container took care of that and passed it to the constructor.

We now have the lazy initialization logic inside the getter because the container did just give us a factory method. Every time we would call it the container would run its logic and create a new service (unless we registered it as singleton of course). But Spring4D contains the lazy type - so we can reduce this code a bit and make use of that:

type
  THomeController = class
  private
    fService: Lazy<IExampleService>;
    function GetService: IExampleService;
  public
    constructor Create(const service: Lazy<IExampleService>);
    property Service: IExampleService read GetService;
  end;

constructor THomeController.Create(const service: Lazy<IExampleService>);
begin
  inherited Create;
  fservice := service;
end;

function THomeController.GetService: IExampleService;
begin
  Result := fService;
end;

The rest of the code is pretty much the same. The Spring4D container has built-in support to TFunc<T>, Lazy<T> and ILazy<T> where T needs to be a registered service type that can be resolved without a name - either by only having one implementor or having specified AsDefault on any.

So this was the first article on dependency injection best practices - more will follow. As always your feedback is very much appreciated.

By the way - in case you missed this - Spring4D is now hosted on Bitbucket.

Sunday, December 1, 2013

Future plans for Spring4D and DSharp

Some of you may have read the public letter about Spring4D project two weeks ago. So this is the promised post about the roadmap (well more of some plans that are not set in stone yet but in our minds) for the future.

First of all I am really happy when I hear that people are using Spring4D in their projects and that it is not just an experimental thing to bring some Java or .Net flavor into the Delphi world. And as always software only becomes better when it gets used. In the past there were many features added because you as the users needed them.

However communication and sharing experiences between you as users and us as the development team is still not as good as I wished. In the past there were many channels used to share experiences or ask for help. Getting them focused more in one place will help to share and build knowledge - so for everyone that is not following us please join us on groups.google.com/forum/#!forum/spring4d.

The project is now hosted on Bitbucket in a Git repository which will make the workflow easier than it was with Subversion. We will follow the Gitflow Workflow which might look confusing first but is really easy. For you as user it means, clone from the master if you want the stable and thoroughly tested source. For stuff currently in development either clone the develop branch or one of the possible feature branches.

Some changes have been made to the collection types recently which are important things I will talk about in a few. These are mostly non breaking changes but there were some small API changes which are easy to apply after you got the latest version.

The DI container got some really nice features in the past months which will be subject of an extra blog post that will follow soon.

Ok, but what about the future?

One thing that is often criticized about the project is the poor documentation - apart from some blog posts here and there - this is something we want to change.We will improve the source documentation (using DocumentInsight actually makes this fun!) and the general documentation about the different features of the framework as well as best practices for dependency injection.

Another goal will be to improve unit tests as some parts are not as well tested as they should be. This will also help adding new features in the future and/or refactor existing ones.

Some other projects will work more closely with Spring4D - one being about ORM - more about that when we have more details.

We want to make Spring4D a de facto standard for Delphi developers.

There are also some interesting things about Spring4D in Nicks upcoming book - so you might to take a look.


Another project that closely works with Spring4D is DSharp. Currently there are some redundant parts in both frameworks and if you are using both it sometimes is hard to decide which one to use. Also this might cause incompatibilities. We are working on reducing and finally removing these.

DSharp had a very simple but powerful MVVM implementation for some while. This was very basic and more of a proof of concept than something you can use for bigger applications but I am happy to tell you that we are bringing you a MVVM solution for Delphi that is based on Caliburn Micro. Some people might have seen some previews by Jeroen Pluimers on the ITDevCon or the BE Delphi this year. Currently we are aiming for a beta in next spring and more details on that project will follow.


Are you interested in helping in one of the projects, have questions or just want to share your experience?Please let us know!

Sunday, June 23, 2013

Implementing custom iterators with delayed execution

The spring4d users amongst you will be familar with methods like IEnumerable<T>.Where which just returns the elements of an enumerable (mostly a list) that match a certain condition passed as delegate.

Today we will take a look how that is achieved and what requirements should be met when implementing such a method. The method itself looks like this:

function Where(const predicate: TPredicate<T>): IEnumerable<T>;

You can see that it just returns an enumerable so in fact you could chain multiple Where calls and get an enumerable that only contains the elements that match every condition.

The execution of the filtering will be done delayed so as long as we don't use that enumerable in a for-in loop nothing will actually happen. In fact you can still modify the list you called Where on and it will consider these elements when starting the iteration. That means when we call Where another enumerable object will created that just saves the informations to perform the filtering: the source and the delegate.

Also the execution will be streamed. That means the elements in the original source will only iterated once and only when they are needed. So if you for example cancel the loop early it will not have iterated all elements in the source enumerable.

In the spring4d branch there is a new class called TIterator<T> that implements both IEnumerable<T> and IEnumerator<T>. Usually you have seperate classes for enumerable and enumerator but in our case the enumerable is just saving the state for the operation and passes these informations to the enumerator class because that one is doing the real work in the MoveNext method. So we can get rid of the redundant information and put them into the same class.

So we look at this:

type
  TIteratorBase<T> = class(TEnumerableBase<T>, IEnumerator)
  protected
    function GetCurrentNonGeneric: TValue; virtual; abstract;
    function IEnumerator.GetCurrent = GetCurrentNonGeneric;
  public
    function MoveNext: Boolean; virtual;
    procedure Reset; virtual;
  end;

  TIterator<T> = class(TIteratorBase<T>, IEnumerator<T>)
  private
    fThreadId: Cardinal;
  protected
    fState: Integer;
    fCurrent: T;
  protected
    function GetCurrent: T;
    function GetCurrentNonGeneric: TValue; override;
  public
    constructor Create; override;
    function Clone: TIterator<t>; virtual; abstract;
    function GetEnumerator: IEnumerator<T>; override;
  end;

Let's look at the GetEnumerator method:

function TIterator<T>.GetEnumerator: IEnumerator<T>;
var
  iterator: TIterator<T>;
begin
  if (fThreadId = TThread.CurrentThread.ThreadID) and (fState = 0) then
  begin
    fState := 1;
    Result := Self;
  end
  else
  begin
    iterator := Clone;
    iterator.fState := 1;
    Result := iterator;
  end;
end;

This ensures that you get a new instance as enumerator when necessary. In the case of the same thread and its first iteration it just returns itself (which is the most common case).

To implement the Where operation we basically just need to implement 3 methods: Create, Clone and MoveNext - let's take a look at the class for that:

type
  TWhereEnumerableIterator<T> = class(TIterator<T>)
  private
    fSource: IEnumerable<T>;
    fPredicate: TPredicate<T>;
    fEnumerator: IEnumerator<T>;
  public
    constructor Create(const source: IEnumerable<T>;
      const predicate: TPredicate<T>);
    function Clone: TIterator<T>; override;
    function MoveNext: Boolean; override;
  end;

As said above the real work is done in the MoveNext method - constructor and Clone method are a no brainer.

function TWhereEnumerableIterator<T>.MoveNext: Boolean;
var
  current: T;
begin
  Result := False;

  if fState = 1 then
  begin
    fEnumerator := fSource.GetEnumerator;
    fState := 2;
  end;

  if fState = 2 then
  begin
    while fEnumerator.MoveNext do
    begin
      current := fEnumerator.Current;
      if fPredicate(current) then
      begin
        fCurrent := current;
        Exit(True);
      end;
    end;
    fState := -1;
    fEnumerator := nil;
  end;
end;

Now this is the real interesting part. As in the requirements the source is not iterated as long as we do not iterate the filtered enumerable. When MoveNext is called for the first time we get the enumerator from the source. And it is only proceeded as much as needed. Since Delphi does not support some yield syntax and compiler generated iterator blocks like C# this source looks a bit more complicated as it would with that syntax support but I guess it is easy enough to understand and implementing these iterators is pretty easy since you only need to implement the MoveNext method (apart from a pretty much straight forward constructor and Clone method).

However just to tease you - wouldn't it be nice if we could write something like this?

function TEnumerable<T>.Where(const delegate: TPredicate<T>): IEnumerable<T>
var
  item: T;
begin
  for item in Self do
    if predicate(item) then
      Yield(item);
end;

If you are interested more about that topic I suggest reading Jon Skeets Reimplementing LINQ to Objects series.