Is Marshmallow and Spring.Persistence.Mapping.RttiExplorer Thread-Safe?

[Todd Flora]

We are using Marshmallow in a multithrreaded environment and seem to be seeing some thread unsafe issues.

For now we have determined that these issues seem directly related to RttiExplorer and the TOrmAttribute base attribute class.,

For instance we are creating some of our own attributes based on TORMAttribute and using TRttiExplorer to lookup fields and seed values in our PODO's based on the lookup attribute.

When we run with 10 threads we are periodically seeing exceptions related to the TORmAttribute.GetMember method and involving Unicode String. Here some example stack traces that we are seeing.

EXCEPTION ClassName: (EOutOfMemory) Message: Out of memory

Occured at Address: $0040B6F1 in Procedure "@UStrSetLength$qqrr20System.UnicodeStringi", Line 29157

FULL STACKTRACE:
(0000A6EC){PrismMQ.exe } [0040B6EC] System.@UStrSetLength$qqrr20System.UnicodeStringi (Line 29156, "System.pas" + 27) + $0
(00005CBE){PrismMQ.exe } [00406CBE] System.ErrorAt$qqrucpv (Line 5324, "System.pas" + 3) + $4
(00005D02){PrismMQ.exe } [00406D02] System.Error$qqr20System.TRuntimeError (Line 5335, "System.pas" + 1) + $7
(0000A6EC){PrismMQ.exe } [0040B6EC] System.@UStrSetLength$qqrr20System.UnicodeStringi (Line 29156, "System.pas" + 27) + $0
(0000A76D){PrismMQ.exe } [0040B76D] System.@UStrCat$qqrr20System.UnicodeStringx20System.UnicodeString (Line 29269, "System.pas" + 21) + $0
(01011E0E){PrismMQ.exe } [01412E0E] Prism.Model.LookupService.Model.Lookupservice.TLookupService.Execute$qqrrx18System.Rtti.TValue (Line 234, "Prism.Model.LookupService.pas" + 39) + $E
(0101221E){PrismMQ.exe } [0141321E] Prism.Model.LookupService.Model.Lookupservice.TLookupService.Execute$qqrrx18System.Rtti.TValue (Line 321, "Prism.Model.LookupService.pas" + 126) + $28
(010420EB){PrismMQ.exe } [014430EB] Prism.Model.Base.BusinessProcess.Model.Base.Businessprocess.TBaseBusinessProcess.Execute$qqrrx18System.Rtti.TValue (Line 51, "Prism.Model.Base.BusinessProcess.pas" + 15) + $B
(010422F3){PrismMQ.exe } [014432F3] Prism.Model.Base.BusinessProcess.Model.Base.Businessprocess.TBaseBusinessProcess.SetResourceName$qqr20System.UnicodeString (Line 66, "Prism.Model.Base.BusinessProcess.pas" + 1) + $9
(00F4532A){PrismMQ.exe } [0134632A] Prism.Service.InitializationConsumer.Service.Initializationconsumer.TInitializationConsumer.ProcessMessage$qqr51System.%DelphiInterface$24Btjmsinterfaces.IMessage% (Line 158, "Prism.Service.InitializationConsumer.pas" + 52) + $E
(00F44F98){PrismMQ.exe } [01345F98] Prism.Service.InitializationConsumer.Service.Initializationconsumer.TInitializationConsumer.Execute$qqrv (Line 87, "Prism.Service.InitializationConsumer.pas" + 8) + $6
(000D3031){PrismMQ.exe } [004D4031] System.Classes.ThreadProc$qqrxp22System.Classes.TThread (Line 14161, "System.Classes.pas" + 21) + $5
(00009400){PrismMQ.exe } [0040A400] System.ThreadWrapper$qqspv (Line 23677, "System.pas" + 45) + $0
(00012388){kernel32.dll} [75173388] BaseThreadInitThunk + $10
(00038F70){ntdll.dll   } [76F09F70] Unknown function at RtlInitializeExceptionChain + $61
(00038F40){ntdll.dll   } [76F09F40] Unknown function at RtlInitializeExceptionChain + $31


EXCEPTION ClassName: (EAccessViolation) Message: Access violation at address 00406DC3 in module 'PrismMQ.exe'. Read of address 1BE1C8D4 (OS Exception)

Occured at Address: $00406DC3 in Procedure "Move$qqrpxvpvi", Line 5698

FULL STACKTRACE:
(00005DC3){PrismMQ.exe } [00406DC3] System.Move$qqrpxvpvi (Line 5698, "System.pas" + 49) + $0
(0005A497){ntdll.dll   } [76F2B497] Unknown function at LdrRemoveLoadAsDataTable + $D4E
(0005A466){ntdll.dll   } [76F2B466] Unknown function at LdrRemoveLoadAsDataTable + $D1D
(0005A409){ntdll.dll   } [76F2B409] Unknown function at LdrRemoveLoadAsDataTable + $CC0
(0000F12E){ntdll.dll   } [76EE012E] KiUserExceptionDispatcher + $A
(000338B0){ntdll.dll   } [76F048B0] RtlInitializeCriticalSectionEx + $DA
(00007D55){PrismMQ.exe } [00408D55] System.@AfterConstruction$qqrxp14System.TObject (Line 16999, "System.pas" + 2) + $5
(0000BCCC){PrismMQ.exe } [0040CCCC] System.DynArraySetLength$qqrrpvpvipi (Line 33858, "System.pas" + 9) + $6
(0001EACC){ntdll.dll   } [76EEFACC] ZwAllocateVirtualMemory + $C
(0000DFA3){KERNELBASE.dll} [7690EFA3] VirtualAllocEx + $3E
(0000E015){KERNELBASE.dll} [7690F015] VirtualAlloc + $13
(000044D0){PrismMQ.exe } [004054D0] System.AllocateLargeBlock$qqrui (Line 1345, "GETMEM.INC" + 8) + $A
(000044A5){PrismMQ.exe } [004054A5] System.LockLargeBlocks$qqrv (Line 1316, "GETMEM.INC" + 2) + $9
(000044E6){PrismMQ.exe } [004054E6] System.AllocateLargeBlock$qqrui (Line 1354, "GETMEM.INC" + 17) + $0
(00004D10){PrismMQ.exe } [00405D10] System.SysReallocMem$qqrpvi (Line 3943, "GETMEM.INC" + 115) + $0
(00005BD9){PrismMQ.exe } [00406BD9] System.@ReallocMem$qqrrpvi (Line 4513, "System.pas" + 21) + $0
(0000A6EC){PrismMQ.exe } [0040B6EC] System.@UStrSetLength$qqrr20System.UnicodeStringi (Line 29156, "System.pas" + 27) + $0
(0000A77F){PrismMQ.exe } [0040B77F] System.@UStrCat$qqrr20System.UnicodeStringx20System.UnicodeString (Line 29278, "System.pas" + 30) + $0
(01011FF8){PrismMQ.exe } [01412FF8] Prism.Model.LookupService.Model.Lookupservice.TLookupService.Execute$qqrrx18System.Rtti.TValue (Line 279, "Prism.Model.LookupService.pas" + 84) + $2B
(0101221E){PrismMQ.exe } [0141321E] Prism.Model.LookupService.Model.Lookupservice.TLookupService.Execute$qqrrx18System.Rtti.TValue (Line 321, "Prism.Model.LookupService.pas" + 126) + $28
(010420EB){PrismMQ.exe } [014430EB] Prism.Model.Base.BusinessProcess.Model.Base.Businessprocess.TBaseBusinessProcess.Execute$qqrrx18System.Rtti.TValue (Line 51, "Prism.Model.Base.BusinessProcess.pas" + 15) + $B
(010422F3){PrismMQ.exe } [014432F3] Prism.Model.Base.BusinessProcess.Model.Base.Businessprocess.TBaseBusinessProcess.SetResourceName$qqr20System.UnicodeString (Line 66, "Prism.Model.Base.BusinessProcess.pas" + 1) + $9
(00F4532A){PrismMQ.exe } [0134632A] Prism.Service.InitializationConsumer.Service.Initializationconsumer.TInitializationConsumer.ProcessMessage$qqr51System.%DelphiInterface$24Btjmsinterfaces.IMessage% (Line 158, "Prism.Service.InitializationConsumer.pas" + 52) + $E
(00F44F98){PrismMQ.exe } [01345F98] Prism.Service.InitializationConsumer.Service.Initializationconsumer.TInitializationConsumer.Execute$qqrv (Line 87, "Prism.Service.InitializationConsumer.pas" + 8) + $6
(000D3031){PrismMQ.exe } [004D4031] System.Classes.ThreadProc$qqrxp22System.Classes.TThread (Line 14161, "System.Classes.pas" + 21) + $5
(00009400){PrismMQ.exe } [0040A400] System.ThreadWrapper$qqspv (Line 23677, "System.pas" + 45) + $0
(00012388){kernel32.dll} [75173388] BaseThreadInitThunk + $10
(00038F70){ntdll.dll   } [76F09F70] Unknown function at RtlInitializeExceptionChain + $61
(00038F40){ntdll.dll   } [76F09F40] Unknown function at RtlInitializeExceptionChain + $31

Is it possible that this is a thread safety issue with TRttiExplorer and TOrmAttribute or are we maybe doiing something wrong with our custom attribute. Here is our attribute code.

Our Lookup Service

unit Prism.Model.LookupService;

interface
uses
      System.Rtti

    , Spring.Collections
    , Spring.Container
    , Prism.Model.Base
    , Prism.Model.Interfaces
    , Prism.Model.Orm.Session
    , Prism.Model.Service.Interfaces
    ;

type

  TLookupService = class( TInterfacedObject, IService )
  private
    FORMSession: TORMSession;
    FResourceName : String;
    function VarIsNullOrEmpty(const value: Variant): Boolean;
  public
    constructor Create;
    destructor Destroy; override;
    procedure SetOrmSession( ASession: TORMSession );
    function Execute(AValue : TValue): TValue;
    procedure SetResourceName(AName : String);
  end;


implementation

uses
    System.Variants
  , System.SysUtils

  , Spring
  , Spring.Reflection
  , Spring.Persistence.Mapping.Attributes
  , Spring.Persistence.Core.Utils
  , Spring.Persistence.Mapping.RttiExplorer

  , Prism.Core.Common
  , Prism.Model.Attributes
  , Prism.Model.Orm.Interfaces
  , uDebugLogger
  ;



type
  TValuesCache = IDictionary<string,Variant>;

var
  Lock : TObject;
  DoNotUseDirectlyValuesCache: TValuesCache = nil;
{
  You see the above variable name right ? ? ? ?
  Why do think it is named that way?

  So don't use it directly or you will be shot.
  You must lock it and unlock it before using it with the following methods two methods.

  Get it? Got it? Good!
}


function LockValuesCache: TValuesCache;
begin
  TMonitor.Enter(Lock);
  Result := DoNotUseDirectlyValuesCache;
end;


procedure UnLockValuesCache;
begin
  TMonitor.Exit(Lock);
end;




procedure ClearField( AField: TRttiField; AModel: TModelBase );
var
  LValue: TValue;
  RValue: TValue;
begin

  { The Nullable<T> type in spring has has a readonly property called HasValue
    which returns True when the Nullable<T> has a value (I know, Shocker!)
    The issue here is this technique doesn't work with Prism properties because
    we have a setter on each property that doesn't take HasValue into consideration
    during the set property process.

    example:
      procedure TCustomer.SetStoreSid(const Value: Nullable<Int64>);
      var
        V:Variant;
      begin
        if (FStoreSid <> Value) then
        begin
          FStoreSid := Value;
          NotifyPropertyChanged('StoreSid');
        end;
      end;

    Under the covers, if FStoreSid has been previously set, its FHasValue field
    will have a '@' character in it and thus HasValue will evaluate to True and
    the implicit class method for Nullable<T> will assign Value to FStoreSid.
    FStoreSid is declared as Nullable<Int64> and its Default<T> value is 0. It
    can never be NULL. So we have to go after the FStoreSid field directly in
    this method: "ClearField"

    On the other hand, we could change all the setters to this format:

      procedure TCustomer.SetStoreSid(const Value: Nullable<Int64>);
      var
        V:Variant;
      begin
        if (FStoreSid <> Value) and ( Value.HasValue ) then
        begin
          FStoreSid := Value;
          NotifyPropertyChanged('StoreSid');
        end
        else
          FStoreSid.Create( V );
      end;

      And then setting the Property via rtti would work. We don't want to change
      all the setters so we'll use this method: "ClearField"
   }

  if IsNullable( AField.FieldType.Handle ) then
  begin
    RValue := AField.GetValue( AModel );
    RValue.SetNullableValue( LValue );
    AField.SetValue( AModel, RValue );
  end
  else
  begin
    RValue := AField.GetValue( AModel );
    AField.SetValue( AModel, RValue.Empty );
  end;
end;


{ TLookupService }

function TLookupService.VarIsNullOrEmpty(const value: Variant): Boolean;
begin
  Result := VarIsNull(value) or VarIsEmpty(value);
end;

procedure TLookupService.SetOrmSession(ASession: TORMSession);
begin
  FORMSession := ASession;
end;



procedure TLookupService.SetResourceName(AName: String);
begin
  FResourceName := AName;
end;

constructor TLookupService.Create;
begin
  FOrmSession := nil;
end;

destructor TLookupService.Destroy;
begin
  inherited;
end;

function TLookupService.Execute(AValue : TValue): TValue;
var
  ClearAttrs : IList<ClearValueAttribute>;
  ClearAttr : ClearValueAttribute;
  Lookups : IList<LookupAttribute>;
  Lookup : LookupAttribute;
  Prop : TRttiProperty;
  Value : Variant;
  Params : IList<IParam>;
  Param : IParam;
  LookupValue, ReturnValue : TValue;
  ParamCnt : Integer;
  Children : IList<TObject>;
  Child : TObject;
  Key : string;
  Val : string;
  AModel : TModelBase;
  ValuesCache : TValuesCache;

begin
  Logger.Log(llVerbose, ClassName + '.Execute', leEntry);
  try
    if not Assigned(FOrmSession) then
      raise exception.Create('Orm Session must be set before this service can be executed: ' + ClassName);
    AModel := AValue.AsObject as TModelBase;
    Result := nil;
    if Assigned( AModel ) then
    begin

      {
        If a field in the incoming model object is marked with the clear attribute then its value
        Needs to be cleared So find all the clearAttributes and call Clear field on each.
      }
      ClearAttrs := TRttiExplorer.GetClassMembers<ClearValueAttribute>( AModel.ClassType );
      for ClearAttr in ClearAttrs do
       ClearField( ClearAttr.RttiMember.AsField, AModel );

      {
        Now get all the lookup attributes to start performing the lookups
      }
      Lookups  := TRttiExplorer.GetClassMembers<LookupAttribute>( AModel.ClassType );
      for Lookup in Lookups do
      begin

        Value := null;

        {If a lookup attribute also has Clear marked to true then start by clearing the value}
        if ( Lookup.ClearSetProperty ) then
          ClearField( Lookup.RttiMember.AsField, AModel );

        {
          We will build the key as the data is available:
          Expected Key = "TABLENAME":"WHERE_PARAM_NAME"="WHERE_PARAM_VALUE"; Etc...

          For TenantSid and ControllerSid they don't have a table name because they are static lookups,
          So use the field name instead of tablename for key on this one.
        }
        if ( Lookup.Tablename.IsEmpty ) then
          Key := Lookup.MemberName+':'
        else
          Key := Lookup.Tablename+':';

        ParamCnt := 0;
        Params := Lookup.Params;
        for ParamCnt := 0 to Params.Count-1 do
        begin
          Param := Params.Items[ParamCnt];
          LookupValue := TRttiExplorer.GetMemberValue( AModel, Param.Value );
          {
            If we don't have a lookup Value for any of the Parameters in the model object
            then we cannot perform the lookup so just ignore this lookup by breaking out of the
            loop
          }
          if LookupValue.IsEmpty then break;

          {Check for Nullable type}
          if IsNullable( LookupValue.TypeInfo ) then
          begin
            {Get the value}
            LookupValue := TRttiExplorer.GetMemberValueDeep( AModel, Param.Value );
            {
              And again if the Lookup value is not available then we are done and cannot perform
              this lookup so just ignore this by breaking out of the loop
            }
            if LookupValue.IsEmpty then break;
              Param.Value := LookupValue.AsVariant;
          end
          else
            Param.Value := LookupValue.AsVariant;

          Val := VarToStr(Param.Value);
          {Start Building a key for the Cache so we can cache the value for future lookups of this value}
          Key := Key + Param.Name + '=' + Val + ';';
        end;

        {
          If we didn't break out of the above loop early then that means we found all the param values
          in the Model object so we can perform the lookup
        }
        if ( ParamCnt = Params.Count ) then
        begin

          if ( Lookup.Tablename.IsEmpty ) then
            Logger.Log(llVerbose, 'Perform Lookup: ' + Lookup.MemberName, Lookup.Sql)
          else
            Logger.Log(llVerbose, 'Perform Lookup: ' + Lookup.Tablename, Lookup.Sql);
          {First of all based on the Key we created above lets see if the value is cached}

          ValuesCache := LockValuesCache;
          try
            if not( ValuesCache.TryGetValue( Key, Value )) then
            begin
              {If not cached then get the value from the database and cache it}
              Value  := FORMSession.ExecuteScalar<Variant>( Lookup.SQL, Params);
              ValuesCache.Add(Key,Value);
            end;
          finally
            UnLockValuesCache;
          end;

          {Finally if we have a valid value let's set the lookup value into the associated field on the
           model object.}
          if not( VarIsNullOrEmpty( Value )) then
          begin
            {Get the property value we arer about to set with this value.}
            Prop := TRttiExplorer.RttiCache.GetProperty( AModel.ClassType, Lookup.SetProperty );
            {Convert the value from the database or cache to a TValue type}
            LookupValue := TValue.FromVariant( Value );
            {
              If we can cast this value to model Objects property type then set it.
              otherwise we will ignore this lookup. Maybe we should be throwing an exception here ? ? ?
            }
            if TUtils.TryConvert( LookupValue, Prop.PropertyType.Handle, AModel, ReturnValue ) then
              Prop.SetValue( AModel, ReturnValue );
          end;
        end;

      end;

      {Process any one to many children in the same manner.}
      Children := TRttiExplorer.GetRelationsOf( AModel, OneToManyAttribute );
      Logger.Log(llVerbose, 'Process ChildLists: ' + Children.Count.ToString);
      for Child in Children do
      begin
        if Assigned( Child ) then
          Execute( Child as TModelBase );
      end;

    end;
  finally
    Logger.Log(llVerbose, ClassName + '.Execute', leExit);
  end;
end;

initialization
  Lock := TObject.Create;
  DoNotUseDirectlyValuesCache := TCollections.CreateDictionary<string,Variant>;

finalization
  Lock.Free;
  DoNotUseDirectlyValuesCache := nil;


end.

Our Lookup Attribute

  LookupAttribute = class(TORMAttribute)
  private
    FTableName: string;
    FSelectCol: string;
    FWhereCol: string;
    FWhereProperty: string;
    FSetProperty: string;
    FClearSetProperty: Boolean;
    FSQL: string;
    function GetParams: IList<IParam>;
  protected
    function GetWhereClause: string;
  public
    constructor Create( const ATableName: string;
                              ASelectCol: string;
                              AWhereCol: string;
                              AWhereProperty: string;
                              ASetProperty: string;
                              AClearSetProperty: Boolean = True); overload;

    constructor Create( const ASQL: string;
                              ASetProperty: string;
                              AClearSetProperty: Boolean = True); overload;
    function SQL: string;virtual;
    property Params: IList<IParam> read GetParams;
    property Tablename: string read FTableName;
    property SelectCol: string read FSelectCol;
    property WhereCol: string read FWhereCol;
    property WhereProperty: string read FWhereProperty;
    property SetProperty: string read FSetProperty;
    property ClearSetProperty: Boolean read FClearSetProperty;
  end;

implementation

{ Lookup }

constructor LookupAttribute.Create( const ATableName: string;
                                          ASelectCol: string;
                                          AWhereCol: string;
                                          AWhereProperty: string;
                                          ASetProperty: string;
                                          AClearSetProperty: Boolean = True);
begin
  FSQL              := '';
  FTableName        := ATableName;
  FSelectCol        := ASelectCol;
  FWhereCol         := AWhereCol;
  FWhereProperty    := AWhereProperty;
  FSetProperty      := ASetProperty;
  FClearSetProperty := AClearSetProperty;
end;

constructor LookupAttribute.Create(const ASQL: string; ASetProperty: string; AClearSetProperty: Boolean);
begin
  FSQL              := ASQL;
  FSetProperty      := ASetProperty;
  FClearSetProperty := AClearSetProperty;
end;

function LookupAttribute.GetParams: IList<IParam>;
var
  Params: TArray<string>;
  Cols: TArray<string>;
  p: Integer;
  Cnt: Integer;
begin

  Result := TCollections.CreateList<IParam>;
  Params := WhereProperty.Split([',']);
  Cols := WhereCol.Split([',']);

  Cnt := Length(Params);
  for p := 0 to Cnt-1 do
    Result.Add( TParam.Create( Cols[p], Params[p] ));

end;

function LookupAttribute.GetWhereClause: string;
var
  Cols: TArray<string>;
  Col: string;
  p: Integer;
  Cnt: Integer;
begin

  p := 0;
  Cols := WhereCol.Split([',']);
  Cnt := Length(Cols);
  for p := 0 to Cnt-1 do
  begin
    if ( p = 0 ) then
      Result := 'where ';
    Col := Cols[p];
    Result := Result + Col + ' = :'+Col;
    if ( p < Cnt-1 ) then
      Result := Result + ' and ';
  end;

end;

function LookupAttribute.SQL: string;
const
  fsSQL = 'Select %s from %s %s';
begin
  if ( FSQL.IsEmpty ) then
    Result := Format( fsSQL, [SelectCol, Tablename, GetWhereClause])
  else
    Result := FSQL;
end;

[Stefan Glienke]

Put a lock into or around TRttiExplorer.GetClassMembers calls and then check again. As you can see for yourself that code is not thread-safe.

[Todd Flora]

Yes I can see that. I have added the lock and the failure still occurs.

It seems that it is returning the attributes as is, and unless I misunderstand how attributes are created, that being that there is only one attribute instance for each usage in a given class, then it would seem that possibly the GetCustomAttributes method of the TRttiHelper might also be unsafe as it is returning the single attribute instance and not a copy of the attribute?

Or maybe that is OK but then the GetClassMembers method should both Lock and then make a copy of each attribute before returning them in the IList.

[Stefan Glienke]

No, it should not create a copy of them but it should only initialize their properties once. Please try that and let me know if that changes anything.

[Todd Flora]

My Apologies, but I am not exactly sure what you are suggesting. Are you suggesting that I just remove the first For Loop from the GetClassMembers method as seen below?

class function TRttiExplorer.GetClassMembers<T>(classType: TClass): IList<T>;
var
  rttiType: TRttiType;
  rttiField: TRttiField;
  rttiProperty: TRttiProperty;
  attribute: TORMAttribute;
  attribute2: T;
begin // TODO: use inherited here for getting the attributes?!
  FLock.Acquire;
  try
    Result := TCollections.CreateList<T>;
    rttiType := TType.GetType(classType);

//    for attribute in rttiType.GetCustomAttributes<TORMAttribute> do
//    begin
//      attribute.EntityType := rttiType.Handle;
//      attribute.MemberKind := mkClass;
//      attribute.MemberName := rttiType.Name;
//    end;

    for rttiField in rttiType.GetFields do
    begin
      for attribute2 in rttiField.GetCustomAttributes<T> do
      begin
        attribute2.EntityType := rttiType.Handle;
        attribute2.MemberKind := mkField;
        attribute2.MemberName := rttiField.Name;
        attribute2.RttiMember := rttiField;
        Result.Add(attribute2);
      end;
    end;

    for rttiProperty in rttiType.GetProperties do
    begin
      for attribute2 in rttiProperty.GetCustomAttributes<T> do
      begin
        attribute2.EntityType := rttiType.Handle;
        attribute2.MemberKind := mkProperty;
        attribute2.MemberName := rttiProperty.Name;
        attribute2.RttiMember := rttiProperty;
        Result.Add(attribute2);
      end;
    end;
  finally
    FLock.Release;
  end;
end;

[Todd Flora]

Never mind I thought about it some more and understand now.

I am trying only one call t this at the class level and caching the result for use by the multiple threads. I will see how that works

[Todd Flora]

So I followed a similar pattern as to how Linus uses TRttiExplorer, Creating a Cache for the Entities Rtti data and only calling the GetClassMembers method once during class registration at startup, and then using the Cache during processing with multiple threads and this has resolved my issue.

Once again I have to praise you for your outstanding technical support. Who would have thought that you could get this level of technical support on an open source proejct without a paid support contract. Thank you immensely.

Regards,

Todd Flora

[Stefan Glienke]

You are welcome points to the paypal button on the overview page ;)