off-by-1 normalization error in ImageUtility::NormalizeUShort

Issue #1237 resolved
Michael Smith created an issue

Description

below is current code for function NormalizeUShort on lines 85-92 of \NINA.Image\ImageAnalysis\ImageUtility.cs:

        /// <summary>
        /// Converts a value from range [0;65535] to [0;1]
        /// </summary>
        /// <param name="val"></param>
        /// <returns></returns>
        private static double NormalizeUShort(double val, int bitDepth) {
            return val / (double)(1 << bitDepth);
        }

UShort has range 0-65535 but this code is using normalization with 65536 as max value for normalization instead of 65535

I suggested changing denominator

(double)(1 << bitdepth)

to

(double)((1<<bitdepth)-1)

Steps to Reproduce

NormalizeUShort(65535, 16)

returns 65535/65536 = 0.9999847412109375

Expected behaviour

NormalizeUShort(65535, 16)

should return 65535/65535 = 1.0

Actual behaviour

NormalizeUShort(65535, 16)

returns 65535/65536 = 0.9999847412109375

Comments (2)

  1. Log in to comment