Beam tilt estimation - group micrographs?

[Oliver Clarke]

Hi Sjors,

The option of refining beam tilt seems like it could be very helpful for those of us collecting using moderate image shift!

In such cases (e.g. see attached), we have decent (?) prior values for beam tilt on a per micrograph basis, and it would be great to be able to refine these per micrograph or by dividing the dataset into beam tilt groups (e.g. 9 groups in the attached example).

Currently we are doing this ad hoc by splitting the dataset into nine separate datasets, but being able to use all the data at once while refining beam tilt values seems like it would give better results.

Cheers Oli

[Sjors Scheres]

Hi Oli, This is a useful suggestion. It will take a bit of time to do this... thanks, Sjors

[Sjors Scheres]

Meanwhile, just proceed like you are doing, and use joinStar jobtype to combine all nine again later. A bit cumbersome, but doing exactly this is what got Wim's data from 2.9 to 2.2A, like he showed at the GRC. We are now writing this up. S

[Oliver Clarke]

Great, thanks Sjors!

Oli

[Takanori Nakane]

Hi Oli,

Now you can define rlnBeamTiltGroupName in your particle STAR file and each group will be refined separetely.

Takanori & Jasenko

[Oliver Clarke]

That's great, thanks so much Takanori and Jasenko! Look forward to trying it!

[Oliver Clarke]

Hi Takanori, Sjors, Jasenko,

Anchi Cheng has a suggestion re this feature, see attached - she suggests that supporting refinement of the center of the beam tilt distribution (when the input star file already contains per micrograph beam tilt values estimated from image shift) would be a useful addition. I guess this would mean refining one beam tilt group, but using the overall beam tilt to adjust each estimated per micrograph value present in the input star.

Cheers Oli

[Ricardo Righetto]

Hi,

We implemented rlnBeamTiltGroupName support in FOCUS where we give a label (an integer number) to each movie corresponding to the beam tilt group it belongs to, in the output micrographs_ctf.star file from FOCUS. However it seems that RELION-3 throws away this column upon particle extraction. Is it a bug or are we doing something wrong?

Thanks, Ricardo

[Takanori Nakane]

Hi Ricardo,

The commit 7c7a874 will copy rlnBeamTiltGroupName from the micrograph STAR file when it is absent in the particle STAR file.

[Ricardo Righetto]

Thanks Takanori, will test it on our micrograph STAR files!

[Ricardo Righetto]

Hello again,

It doesn't seem to work as expected - when we extract particles from a micrograph STAR file with the rlnBeamTiltGroupName column, the first particle from each micrograph seems to take the correct value, but the remainder of the particles for that micrograph receives an 'empty' value in this column. Grouped BeamTilt refinement later then doesn't work as expected. If it helps you I could send you the corresponding STAR files via e-mail.

Thanks once again!

[Takanori Nakane]

Confirmed. Sorry, I wasn't careful enough. Fixing now...

[Takanori Nakane]

Hi Ricardo,

The commit 99d96a3 should fix the problem.

[Ricardo Righetto]

Hello again Takanori,

Now the particle STAR files get the rlnBeamTiltGroupName values correctly from the micrograph STAR file, thanks a lot for correcting this. Now the problem is with the beam tilt estimation itself: RELION still refines the whole dataset as a single beam tilt group. Do we have to pass a special flag or something to relion_ctf_refine_mpi in order to enable grouped beam tilt refinement?

Thanks a lot!

[Takanori Nakane]

I am sorry but we realised that we had two redundant columns rlnBeamTiltGroupName and rlnBeamTiltClass. The first is string and the latter is integer. We decided to remove the first (commit 6a1a78a9). So please use rlnBeamTiltClass instead.

[Ricardo Righetto]

Hi Takanori,

Sorry, I forgot to confirm earlier that the grouped beam tilt refinement is now working nice! The output has quite some verbose on the beam tilt classes - I guess some kind of sanity check - but that's fine. For example see: https://pastebin.com/BkFfLZQt

Thanks a lot for solving this issue! Merry Christmas to you and the RELION team.

[Ricardo Righetto]

Hi Takanori and RELION team,

I'm afraid the issue that commit 7c7a874 was supposed to fix is back: if an imported micrographs.star contains the rlnBeamTiltClass column, it is again throwing it away upon particle extraction...

Thanks in advance!

[Takanori Nakane]

When did the problem come back? Are you extracting from AutoPick, or re-extracting from particles.star?

[Ricardo Righetto]

I'm using the latest version of the source code from this repo and noticed the problem for the first time today. I am extracting from AutoPick.

[Takanori Nakane]

When was it working before? Were you also extracting from AutoPick at that time?

[Ricardo Righetto]

On 13-12-2018 I confirmed that the extraction was working. That was the last time I tried to do this before today for a dataset with multiple rlnBeamTiltClass values. At that time it was from a re-extraction from a particles.star file and we were still using the rlnBeamTiltGroupName label, which was later deprecated in favor of rlnBeamTiltClass, which I manually changed in the .star file. I just tested and at the moment it's not working either for AutoPick nor re-extraction with my .star files.

[Takanori Nakane]

Hi, it works just fine here. My input micrograph STAR file contains the following columns:

_rlnMicrographName #1 
_rlnCtfImage #2 
_rlnDefocusU #3 
_rlnDefocusV #4 
_rlnCtfAstigmatism #5 
_rlnDefocusAngle #6 
_rlnVoltage #7 
_rlnSphericalAberration #8 
_rlnAmplitudeContrast #9 
_rlnMagnification #10 
_rlnDetectorPixelSize #11 
_rlnCtfFigureOfMerit #12 
_rlnCtfMaxResolution #13 
_rlnBeamTiltClass #14

[Ricardo Righetto]

Strange. My input micrographs.star file contains the following columns:

_rlnMicrographName #1 _rlnCtfImage #2 _rlnDefocusU #3 _rlnDefocusV #4 _rlnDefocusAngle #5 _rlnVoltage #6 _rlnSphericalAberration #7 _rlnAmplitudeContrast #8 _rlnMagnification #9 _rlnDetectorPixelSize #10 _rlnCtfFigureOfMerit #11 _rlnCtfMaxResolution #12 _rlnPhaseShift #13 _rlnBeamTiltClass #14

But the extracted particles.star contains:

_rlnCoordinateX #1 _rlnCoordinateY #2 _rlnAutopickFigureOfMerit #3 _rlnClassNumber #4 _rlnAnglePsi #5 _rlnImageName #6 _rlnMicrographName #7 _rlnMagnification #8 _rlnDetectorPixelSize #9 _rlnCtfMaxResolution #10 _rlnCtfFigureOfMerit #11 _rlnVoltage #12 _rlnDefocusU #13 _rlnDefocusV #14 _rlnDefocusAngle #15 _rlnSphericalAberration #16 _rlnCtfBfactor #17 _rlnCtfScalefactor #18 _rlnPhaseShift #19 _rlnAmplitudeContrast #20 _rlnOriginX #21 _rlnOriginY #22

I even did git pull (already up-to-date) and re-compiled to make sure I am running the latest version.

[Takanori Nakane]

Are you sure your PATH points to the right directory? Did you also try make clean before make?

[Ricardo Righetto]

You are right: the problem was with my PATH in my cluster submission script for CPU jobs that was pointing to a wrong, outdated version. All other submission scripts were fine, that's why I didn't notice it before. It works indeed. Sorry for the confusion!