Summary: Sony’s ARW (v2.3.1) embedded lens-correction data

Here’s a summary of what I’ve found on regarding the lens-correction data that Sony embeds in their ARWs. This is tested with a Sony A7 (ILCE-7) and the Sony/Zeiss 35mm F/2.8 and 55mm F1/1.8 lenses. I would believe that this also holds for other cameras using v2.3.1 of the ARW format (e.g., A7r, A7s, A7m2) but haven’t tested it.

The A7 allows to separately set 3 possible corrections in the menu (‘Auto’ vs ‘off’).

• Vignetting correction
• Chromatic Aberration correction
• Distortion correction

Not every lens seems to support all modes, my 55mm has the distortion correction greyed out to ‘Auto’, so it can’t be switched off. However, Sony is embedding the data into the ARW in any case anyway, no matter the setting (at least for CA/dist, probably also Vignetting). It’s up to the software to decide if to use it.

Also interesting is Adobe’s take on this:

• Lightroom automatically applies CA correction and the DNG-converter writes CA-correction opcodes into the DNG-file.
• Distortion correction is not applied in Lightroom. However, the DNG-converter writes an non-standard XMP-tag into the DNG-file that saves the decoded correction data. It does not write distortion-correction opcodes (although they’re part of the DNG-spec), so other DNG-readers won’t apply lens-correction either (unless they interpret Sony’s MakerNotes on their own).
• As far as I can tell, Lightroom and the DNG-converter completely ignore any Vignetting correction information.
• The DNG-converter also inserts 2 colour calibration matrices depending on the lens (fix per lens type and in addition to the colour matrices specified in Adobe’s camera profile). I’m not quite sure why (different lenses should not influence hue shifts?) and where it gets the data for this.

I do not know where Sony stores the Vignetting correction information but the CA and Distortion corrections are here:

• Distortion correction: SR2SubIFD tag 0x7982 and mirrored in Exif SubIFD tag 0x7037. 17 signed integers (16 bit), first one is always 16 (denoting the number of numbers to follow), the remaining ones describe one radial transformation function.
• CA correction: SR2SubIFD tag 0x7980 and mirrored in Exif SubIFD tag 0x7035. 33 signed integers (16 bit), first one is always 32, followed by 2 times 16 numbers, describing 2 radial transformation functions (for colour planes red and blue, denoting their shift vs green).

The logic and format behind the 3 different types of transformation function encodings (dist, CA-red, CA-green) seems to be the same although the precision is much different. CA-correction numbers are always multiples of 128, while distortion are multiples of 1. Plotting the tag values of a set of 116 photos makes for some pretty graphs:

CA-red:

CA-green:

Distortion:

The exact format of these 16 numbers is not known but based on how the DNG-converter interprets them, they are scaling factors to 16 base transformation functions (polygonal of 6th degree), whose sum gives the final radial transformation function (see my previous post). It’s probably a bit more complicated than that, I’m not sure where the base functions come from – they are likely computed themselves based on some fixed parameter.

Decoding the base functions from Adobe’s DNG-conversion is ok for CA-correction (see previous post) but difficult (i.e., imprecise) for the distortion correction, most likely due to the much higher range of the possible values. By running a least square fitting algorithm, I have deducted a set of functions that (at least for my test set of photos) should have a maximum deviation from Adobe’s results of 3-4 pixels in the picture corner at the strongest distortion in my set. For most pictures the deviation should be less than 1 pixel shift, which is good enough for me.

More things left to figure out (I will stop here though…):

• What’s the exact meaning of the set of 16 numbers, i.e., what are the original base functions and how are they derived?
• Why does Adobe’s DNG-converter add the camera calibration matrices and where does the data come from?
• Where is the vignetting information stored in the ARW?
• What do the tags 0x0064/0x189c denote? It’s another 16 value transformation function, seems to be 1:1 correlated to the distortion correction function and has very similar values (just a slight delta). Adobe does not seem to interpret them for DNG-creation.
• And what do the other ~380 currently undocumented tags in the Sony-data mean…? 🙂

btw: Huge thanks to the developers of Exiftool. Amazingly useful, couldn’t even have started on looking into this without it!