Quick Start with Experimental Data

This guide walks through processing real experimental single-particle imaging (SPI) data. We recommend completing at least one simulated reconstruction first using the Quick Start with Simulations guide before proceeding.

Setting Things Up

As in the simulation case, create a reconstruction directory to keep things organized:

./dragonfly_init -t spi

This creates a folder named spi and compiles the necessary executables.

Data Source

This tutorial uses data from the AMO end-station of the Linac Coherent Light Source (LCLS), published as Reddy, et al. Scientific Data 4, 170079 (2017).

Download single hits from CXIDB. Follow the HDF5 link to download the data files. Each file contains the dataset photonConverter/pnccdBack/photonCount with photon converted data from the 4x4 down-sampled pnCCD detector.

Experimental Geometry

Add the [parameters] section to your config.ini:

[parameters]
detd = 586
lambda = 7.75
detsize = 260 257
pixsize = 0.3
stoprad = 40
ewald_rad = 650.
polarization = x

For units and other details, see Configuration Guide. The ewald_rad parameter is the radius of curvature of the Ewald sphere in voxels. The 3D grid size is determined by the highest resolution detector pixel distance. A value of 650 gives a 125x125x125 voxel volume.

Detector File

Add the [make_detector] section to create the detector geometry file:

[make_detector]
in_mask_file = aux/mask_pnccd_back_260_257.byt
out_detector_file = data/det_pnccd_back.h5

Generate the detector file by running:

dragonfly.utils.make_detector

Data Conversion

Convert HDF5 data to EMC format for each file:

dragonfly.utils.convert.h5toemc -d photonConverter/pnccdBack/photonCount <HDF5_file>

This creates an EMC file in the data folder. For other options, run:

dragonfly.utils.convert.h5toemc -h

The EMC file header contains the total pixel count from the configuration file, so the [parameters] section must be complete before converting data.

View your data using the frame viewer:

dragonfly.frameviewer

Click ‘Random’ a few times to inspect the patterns.

EMC Parameters

Add the [emc] section to config.ini:

[emc]
in_photons_list = experiment_files.txt
in_detector_file = make_detector:::out_detector_file
output_folder = data/
log_file = EMC.log
num_div = 10
need_scaling = 1
beta = 0.001
beta_schedule = 1.41421356 10

Create the photon list file (e.g., experiment_files.txt) with paths to all EMC files:

data/run_001.emc
data/run_002.emc
...

The beta parameter controls the sharpness of the orientation probability distribution. The beta_schedule specifies the deterministic annealing schedule where beta is multiplied by sqrt(2) every 10 iterations. These aid convergence with high signal data.

Running and Monitoring

Serial execution:

dragonfly.emc 100

MPI parallel execution:

mpirun -np 8 dragonfly.emc 100

On a cluster, expect around 800 seconds per iteration at the start, decreasing to ~240 seconds by iteration 100.

Monitor reconstruction progress with the autoplot GUI:

dragonfly.autoplot

This displays likelihood values, best volume, and correlation coefficients.

Check Your Work

After running, verify your reconstruction by viewing the output volumes. Compare with known results from similar experiments to ensure quality.