TValueHelper.FromVariant, TFmtBcdVariant return type also needs to handle Floats

Issue #135 wontfix
Todd Flora created an issue

A while back we made a suggestion to fix an issue with Delphi XE5 and earlier where Oracle was returning TFmtBcdVariant types, and they were losing precision because of a bug in XE5.

Please see issue #64 for the previous fix.

We have now run into an issue where the Bcd Type (which is really a decimal type) is also being used by Firedac to return floating point types and now the above fix fails as it tries to cast a floating point number to an int64

We have come up with a fix for this and wanted to present it to you, so that you might consider integrating it into the latest code.

From the Spring.pas unit see the following changed method.

class function TValueHelper.FromVariant(const value: Variant): TValue;
type
  TCustomVariantTypeInfo = record
    Name: string;
    VType: TVarType;
  end;
const
  CustomVariantTypes: array[0..2] of TCustomVariantTypeInfo = (
    (Name: 'SQLTimeStampVariantType'; VType: varDouble),
    (Name: 'SQLTimeStampOffsetVariantType'; VType: varDouble),
    (Name: 'FMTBcdVariantType'; VType: varUnknown)
  );
var
  typeName: string;
  i: Integer;
begin
  case TVarData(Value).VType of
    varEmpty, varNull: Exit(Empty);
    varBoolean: Result := TVarData(Value).VBoolean;
    varShortInt: Result := TVarData(Value).VShortInt;
    varSmallint: Result := TVarData(Value).VSmallInt;
    varInteger: Result := TVarData(Value).VInteger;
    varSingle: Result := TVarData(Value).VSingle;
    varDouble: Result := TVarData(Value).VDouble;
    varCurrency: Result := TVarData(Value).VCurrency;
    varDate: Result := From<TDateTime>(TVarData(Value).VDate);
    varOleStr: Result := string(TVarData(Value).VOleStr);
    varDispatch: Result := From<IDispatch>(IDispatch(TVarData(Value).VDispatch));
    varError: Result := From<HRESULT>(TVarData(Value).VError);
    varUnknown: Result := From<IInterface>(IInterface(TVarData(Value).VUnknown));
    varByte: Result := TVarData(Value).VByte;
    varWord: Result := TVarData(Value).VWord;
    varLongWord: Result := TVarData(Value).VLongWord;
    varInt64: Result := TVarData(Value).VInt64;
    varUInt64: Result := TVarData(Value).VUInt64;
{$IFNDEF NEXTGEN}
    varString: Result := string(AnsiString(TVarData(Value).VString));
{$ENDIF}
    varUString: Result := UnicodeString(TVarData(Value).VUString);
  else
    case TVarData(Value).VType and not varArray of
      varSmallint: Result := TValue.From<TArray<SmallInt>>(Value);
      varInteger: Result := TValue.From<TArray<Integer>>(Value);
      varSingle: Result := TValue.From<TArray<Single>>(Value);
      varDouble: Result := TValue.From<TArray<Double>>(Value);
      varCurrency: Result := TValue.From<TArray<Currency>>(Value);
      varDate: Result := TValue.From<TArray<TDateTime>>(Value);
      varOleStr: Result := TValue.From<TArray<string>>(Value);
      varDispatch: Result := TValue.From<TArray<IDispatch>>(Value);
      varError: Result := TValue.From<TArray<HRESULT>>(Value);
      varBoolean: Result := TValue.From<TArray<Boolean>>(Value);
      varVariant: Result := TValue.From<TArray<Variant>>(Value);
      varUnknown: Result := TValue.From<TArray<IInterface>>(Value);
      varShortInt: Result := TValue.From<TArray<ShortInt>>(Value);
      varByte: Result := TValue.From<TArray<Byte>>(Value);
      varWord: Result := TValue.From<TArray<Word>>(Value);
      varLongWord: Result := TValue.From<TArray<LongWord>>(Value);
      varInt64: Result := TValue.From<TArray<Int64>>(Value);
      varUInt64: Result := TValue.From<TArray<UInt64>>(Value);
      varUString: Result := TValue.From<TArray<string>>(Value);
    else
      typeName := VarTypeAsText(TVarData(Value).VType);
      for i := 0 to High(CustomVariantTypes) do
        if SameText(typeName, CustomVariantTypes[i].Name) then
        begin
          case CustomVariantTypes[i].VType of
            varDouble: Result := Double(Value);
            varUnknown: {FMTBcdVariantType}
            begin
              if BcdScale(VarToBcd(Value)) > 0 then
                Result := BcdToDouble(VarToBcd(Value))
              else
              begin
                Result := {$IFDEF DELPHIXE6_UP}Int64(Value);
                {$ELSE}StrToInt64(VarToStr(Value));{$ENDIF} // see QC#117696
              end;
            end
          else
            raise EVariantTypeCastError.CreateRes(@SInvalidVarCast);
          end;
          Exit;
        end;
      raise EVariantTypeCastError.CreateRes(@SInvalidVarCast);
    end;
  end;
end;

Also the dbrtl.dcp needs to be added to the Spring.base package, so that the compiler does not complain that Data.FmtBcd exists both in dbrtl.dcp and spring.base when compiling Spring.Persistence.

My apologies for not catching this earlier, and thanks for all your hard work on this project as always. Todd.

Comments (9)

  1. Stefan Glienke repo owner

    Thanks, we will look into it but I can tell you that unfortunately the presented solution can't be accepted because Spring.Base must not get a dependency to dbrtl.

  2. Todd Flora reporter

    Understood. Here is a version using Frac from sysutils

             case CustomVariantTypes[i].VType of
                varDouble: Result := Double(Value);
                varUnknown:
                begin
                  if (Frac(Extended(Value)) > 0) then
                    Result := Extended(Value)
                  else
                    Result := {$IFDEF DELPHIXE6_UP}Int64(Value);
                      {$ELSE}StrToInt64(VarToStr(Value));{$ENDIF} // see QC#117696
                end
    

    And it returns the higher precision extended type.

  3. Todd Flora reporter

    Converting a large FMTBcd (18 char or more I believe) with no Scale to a Double will cause a loss of precision as double is not large enough to hold it. That is what the VarToBcd method was doing and what was fixed by QC 117696 so converting to double would reintroduce the bug.

    Oracle does not know the difference between an Integer and a floating point number and Firedac returns them all as TFMTBcd variants. Soooo if a number has no scale then it really is an integer, Since int64 can hold up 19 character numbers it works well in this scenario. If you don't want to provide the int64 conversion when there is no scale, then I would suggest that you return them as extended as it has higher precision than double. The nice thing about what my second suggested fix does is that it alleviates the need for the calling process to convert to an integer when the result really is an integer anyway.

  4. Stefan Glienke repo owner
    1. Have you tried defining some data type mappings so FD does not return a FMTBcd but a Int64 for number(19) columns which would be the correct type so far?
    2. Your fix is not a fix but a hack because it only works on 32bit Intel platform (see http://docwiki.embarcadero.com/RADStudio/Seattle/en/Internal_Data_Formats#The_Extended_type)

    The only true solution to me is to cause FD not to return a FMTBcd but define a data type mapping for all the different column data types you are using because there is no way to find out what the Variant really holds without using the Data.FmtBcd unit. And if we have to do that it would require some callback mechanism for TValue.FromVariant because we cannot put the dependency on the dbrtl package in the Spring.Base package.

    So please try the FD data type mappings and then report back.

    Edit: I just tried this and I could properly let FD convert Number(19) into Int64 resulting in a TLargeIntField instead of TFMTBCDField and Number(19,4) into Double resulting in a TFloatField.

  5. Stefan Glienke repo owner

    If you don't want to use the data type mapping another solution could be to use a custom adapter and implement the GetFieldValue functions of its resultset adapter accordingly.

    Anyway I am closing this now.

  6. Todd Flora reporter

    Sorry I didn't get back to you sooner, been slammed.

    Requiring a developer to add several lines of code to each Firedac connection he creates seems like a less than optimal solution imo. The second solution I provided is only two lines of code and fixes the issue for every firedac connection created for use with Marshmallow. Additionally the use of Double instead of extended could be used for the Frac portion of the if statement but the int64 portion would definately loose precision if double were used.

    Here is the code that would have to be added to each connection. And this still does not guarantee that Firedac will not still return dtBCD sometimes if one of these map rules does not cover a situation.

     with FFDConnection.FormatOptions do
        begin
          OwnMapRules := True;
          with MapRules.Add do
          begin
            ScaleMin := 0;
            ScaleMax := 0;
            PrecMin := 0;
            PrecMax := 4;
            SourceDataType := dtBcd;
            TargetDataType := dtInt16;
          end;
          with MapRules.Add do
          begin
            ScaleMin := 0;
            ScaleMax := 0;
            PrecMin := 5;
            PrecMax := 8;
            SourceDataType := dtBcd;
            TargetDataType := dtInt32;
          end;
          with MapRules.Add do
          begin
            ScaleMin := 0;
            ScaleMax := 0;
            PrecMin := 9;
            PrecMax := 19;
            SourceDataType := dtBcd;
            TargetDataType := dtInt64;
          end;
          with MapRules.Add do
          begin
            ScaleMin := 1;
            ScaleMax := 20;
            PrecMin := 0;
            PrecMax := 15;
            SourceDataType := dtBcd;
            TargetDataType := dtDouble;
          end;
          with MapRules.Add do
          begin
            SourceDataType := dtDateTimeStamp;
            TargetDataType := dtDateTime;
          end;
        end;
    

    In a perfect world I think we would get Embarcadero to fix this, but in this world we live in if I had a vote I would say this issue should be fixed in the code we have control over for now at least. Thanks for your consideration. BTW the simple fix I provided has been in our copy of the code and working great ever since I submitted this.

  7. Log in to comment