neural network to predict ISO/aperture/shutter speed

Issue #1031 on hold
Former user created an issue

Hi all,

I would like to contribute code to predict ISO, shutter speed, and aperture settings. One scenario I see for this is an established wedding photographer who has a style and doesn't want to miss a shot by spending 5-10 seconds adjusting things between the many scenes she moves between. Another scenario is giving a "second shooter" a camera trained with the main photographer's settings. The second shooter would then take pictures with a similar style to that of the main photographer. Another scenario is to be able to share styles with other photographers. Lastly, depending on the algorithm's efficiency, this could be applied to video.

The neural network would attempt to learn the three settings based on the metadata available in all of the previous good shoots. When on a photo shoot, the network would predict the correct settings based on the previous pictures taken. For those unfamiliar with neural networks, the bulk of the processing would take place on the photographer's computer, when the training phase occurs. The actual prediction code would be running on the camera and would not be a burden for the camera's ARM processor.

I have trained my first ISO prediction network on the metadata for about 20,000 images that are from "good" photo shoots. Right now I can get about 90% accuracy for base ISO prediction, and this can probably be improved. (Disclaimer: this number was from testing on my training set, but I want to verify I can get access to my needed inputs before refining things.) As inputs I am using:

  • Measured EV
  • Camera Temperature
  • Measured RGGB
  • WB RGGB Levels Measured
  • Color Temp Measured
  • Raw Measured RGGB
  • Lens
  • Blue Balance
  • Field Of View
  • Hyperfocal Distance
  • Light Value
  • Red Balance

This is all data that I collected from metadata. I have two big questions before I continue:

1) Are all of my inputs independent from ISO, shutter speed, and aperture? We need the input values to be (fairly) independent from those settings. 2) Is it possible to read these inputs on the fly with Magic Lantern?

Would anyone be interested in collaborating on this add-on? I am currently writing the code in Python/Octave, but I have the 550d build environment installed (I own a 550d, but eventually I need to get this working on the 5dm2), so I can also write the ARM C code for the neural network to predict the settings based on the input data. What would be very helpful is for someone to be able to provided the input values and add the GUI code and the proper hooks to the buttons - e.g. press the button half-way and the neural network predicts and then set the predicted settings. This all assumes that the list above is valid and readable, or that we can predict based on inputs that are valid and readable.

Best regards

Comments (3)

  1. Alex

    Regarding your parameters:

    • Measured EV - easy to read (AE_VALUE) but not accurate in my experience.
    • Camera Temperature - yes (why it's relevant?)
    • Measured RGGB - no direct access to raw data. In LiveView you can get RGB data after YUV conversion.
    • WB RGGB Levels Measured - not sure what this is.
    • Color Temp Measured - if you can compute it from LiveView RGB data, fine. I don't know how. Besides, why is color temperature relevant to auto ISO? (assuming you are shooting raw)
    • Raw Measured RGGB - not sure why it's different from measured RGGB.
    • Lens - lens name? why is this relevant?
    • Blue Balance - what's that?
    • Field Of View - from focal distance; why is relevant?
    • Hyperfocal Distance - same
    • Light Value - what's that?
    • Red Balance - what's that?

    For M mode auto ISO, I would suggest to compute ISO from the histogram - by exposing to the right without clipping and without underexposing (whenever possible). No need to take into account other factors - here you only want to minimize noise and avoid clipping.

    For Av mode auto ISO, you can also take into account the focal length, and IS status - to know whether motion blur is acceptable or not. You might identify the amount of motion in the scene in LiveView - when the scene looks stationary, choose a low ISO, when it moves, choose a higher one.

    For Tv mode, you probably want a good balance between noise and lens sharpness (that is, try to set the aperture at the sweet spot).

  2. Former user Account Deleted

    Hi Alex,

    Thanks for response. I just noticed it. By the way, I thought all of these got pushed to the mailing list. Is that not the case?

    Measured EV - easy to read (AE_VALUE) but not accurate in my experience.

    Good to know.

    Camera Temperature - yes (why it's relevant?)

    Probably doesn't contribute anything. Will remove next go around.

    Measured RGGB - no direct access to raw data. In LiveView you can get RGB data after YUV conversion.

    Does YUV conversion take a lot of processing? Is that available in M mode?

    Color Temp Measured - if you can compute it from LiveView RGB data, fine. I don't know how. Besides, why is color temperature relevant to auto ISO? (assuming you are shooting raw)

    I'm not sure why it is relevant - maybe it doesn't contribute anything. Some of these things may not make sense, but as a whole all these variables combined to make a very good prediction for ISO. Yes, RAW mode is an assumption at the moment.

    Lens - lens name? why is this relevant?

    Yes, lens as reported in metadata.

    Blue Balance - what's that?

    I don't know yet.

    Field Of View - from focal distance; why is relevant? Hyperfocal Distance - same

    I don't know yet.

    Light Value - what's that? Red Balance - what's that?

    Not sure yet..

    Again, regarding things that are relevant or not, it's hard for me to say at the moment. I basically threw all that data at the neural network and it worked. It would take more time to see what is contributing more or less. Of the variables I listed, what do you think contributed most to ISO? I'll train and test a new network based on your thoughts.

    For M mode auto ISO, I would suggest to compute ISO from the histogram - by exposing to the right without clipping and without underexposing (whenever possible). No need to take into account other factors - here you only want to minimize noise and avoid clipping.

    Initially M mode (with RAW format) is my first goal. Is histogram data available (or can it be derived) in my the metadata of my training set? That's the main constraint - if histogram is the best predictor of ISO. Otherwise I would have to have my friend install a version of ML that would record his histogram data into his RAW metadata and I would have to wait to retrain the neural network after he takes enough good photos. I would like to be able to train based on people's existing RAW photos, that way many more people could use this feature more quickly.

    Best regards, Sam

  3. Log in to comment