qFitLatent seeks to learn a latent vector that is capable of producing the qFit multiconformers with fidelity to the side chain ensembles. Fundamentally, by treating the multiconformers as the ground truth, a distribution of side chain conformations arises out of the atomic positions, occupancies, and B-factors. These experimental representations lend themselves naturally to a Gaussian mixture of states, where each side chain rotamer is treated as a single state with probability and variance. This Gaussian mixture can then be mapped directly to the input multiconformers, and the two distributions can be compared.

To learn a representation capable of recapitulating the side chain ensembles, we adopt an invariant point attention backbone to operate on sequence embeddings, such that the model has knowledge of the amino acid identities and the relevant structural biology driving local dynamics. We are experimenting with multiple formulations of the model, including a generative version conditioned on sequence and backbone only, as well as a variational autoencoder that masks residues/regions and reconstructs them from the latent vectors. The goal of this project is to craft a latent manifold that practically models the multiconformers created by unused density. An accurate representation of local dynamics through the lens of multiconformers unlocks an untapped but complementary dynamics representation: another piece of the puzzle.

Figure 1. qFitLatent reconstruction of side chain multiconformers in a validation protein not seen during training. The rotamer of the ground truth (qFit, purple) central valine is correctly predicted by the model (qFitLatent, salmon).

Check out the code here.

WaterFlow in one pass: Flow ODE integration carries a sampled prior onto candidate water sites; the confidence model scores each candidate, and clustering leaves a ranked, discrete set of kept waters.

Predicting accurate coordinates for ordered water remains a challenging, unsolved problem. Static structure prediction is close to experimental accuracy, and protein-ligand cofolding is improving quickly, but neither models solvent, so the models return a dry protein. Most water placement is instead physics-based, grounded in molecular dynamics and solvation thermodynamics (Nguyen et al., 2012), which is slow and only as accurate as the force field and solvent model underneath. Deep learning and data-driven methods such as GalaxyWater-CNN (Park & Seok, 2022), HydraProt (Zamanos et al., 2024), and most recently SuperWater (Kuang et al., 2025) showed a learned model can beat physics-based coverage, but precision and recall trade off steeply: SuperWater recovers only around 27% of deposited waters at high precision, as claimed in their paper.

Our new preprint introduces WaterFlow, which outperforms SuperWater at every operating point. We also analyze how much of this gain, and the gap to ideal prediction, lies in the quality of the model itself versus the quality of the data used to train these models.

What is WaterFlow?

WaterFlow takes a protein structure and returns the same structure with ordered waters added. A generator, trained with flow matching (Lipman et al., 2022), learns a velocity field that transports points sampled from a prior distribution around the protein onto the water positions observed in crystal structures; at inference, integrating that field yields candidate waters. A confidence model then scores each candidate, low-scoring ones are removed, and nearby survivors are merged, leaving a ranked list of waters with confidence scores. Both stages are parametrized by the same equivariant graph neural network (Jing et al., 2021), trained on ~67,000 structures from PDB-REDO.

Integrating the learned velocity field carries prior samples onto ordered-water positions.

Re-trained on identical sequence- and structure-clustered splits and scored on the same 698-structure holdout, SuperWater reaches a parity F1 of 0.41 at a 1.0 Å acceptance radius; WaterFlow reaches 0.63, and the gap widens at 0.5 Å (0.54 against 0.26). The largest gains are at sub-angstrom localization, the regime that matters for binding free energies and hydrogen-bond geometry.

Precision-recall on the 698-structure holdout. WaterFlow outperforms the re-trained SuperWater at every confidence threshold, at both 0.5 Å and 1.0 Å, and reaches an F1 of 63.4% at 1.0 Å at parity versus 40.7% for SuperWater.

Crystallographic symmetry matters

Many surface waters are coordinated by atoms from symmetry-related copies of the protein in the crystal lattice, so a model that sees only the asymmetric unit sees an incomplete coordination environment. WaterFlow expands the crystal symmetry and adds residues within 8 Å of the asymmetric unit as context nodes in its graph.

Left: one water, seen from the asymmetric unit alone and with its symmetry neighbors. Right: the recall gain from adding symmetry mates is confined to waters within hydrogen-bonding range of a symmetry mate.

This raises F1 at parity from 0.57 to 0.63, and the gain is concentrated in the ~10% of waters within hydrogen-bonding range of the lattice, where recall jumps from 0.29 to 0.60. We see this phenomenon emphasized with one bridging water, which scores 0.004 from the asymmetric unit alone, and 0.71 with its symmetry neighbors present. Almost no other structure prediction method encodes this crystallographic information, and the argument extends well past water: proteins, ligands, and ions also form contacts across symmetry-related interfaces.

Deposited waters are not ground truth

Which waters get modeled into a deposited structure depends on resolution, refinement protocol, and the choices of the crystallographer (Fraser & Murcko, 2024), so two crystals of the same protein rarely agree about their solvent. Clustering deposited waters across large sets of isomorphous structures (lysozyme, endothiapepsin, carbonic anhydrase), we find 13-30% of modeled waters are not reproduced in the other structures of the same protein. Treating the consensus positions as ground truth, the best achievable F1 is 0.87-0.94, a ceiling for any model trained and scored against deposited coordinates, ours included. Consistent with that, filtering low-quality waters out of training did not improve the model; rather, more distinct, high-quality structures were more efficient at reaching the same performance.

Do the predicted waters fit the experimental data?

The more direct test of a predicted water is the experimental density. Measured by EDIA (Meyder et al., 2017), predicted waters fit real-space density as well as deposited ones at a confidence threshold of 0.7, and better above it. Novel predictions, those with no deposited water within 1.0 Å, are nominally false positives, but sit on positive mFo-DFc difference density: the fraction above +3σ grows from 11% to 49% with confidence. A significant share of the apparent false positives are therefore waters the deposition omitted.

The predictions also hold up under re-refinement. Re-refining PDB 4RKW with predicted waters lands R-free within 0.012 of the deposited control, and on an apo/ATP-bound pair of LuxO, the model places the two waters bridging ATP to the protein within 0.1 Å of their deposited positions and leaves the ligand-displaced sites empty. These results bring predicted waters within reach of ligand binding analysis, protein design, and automated model building.

Using WaterFlow

Model setup and prediction commands are in the README. Two selection methods control how many waters are returned. The default is a confidence threshold: every water scoring above the cutoff is kept. The confidence score reflects local disorder, so the model predicts more waters for well-ordered structures and fewer for poorly ordered ones, and the predicted counts fall with resolution much like deposited counts do. Raising the threshold keeps fewer waters with stronger density support; at high thresholds the retained waters are better supported than the deposited ones. A high threshold is appropriate when individual waters matter, for example in QM/MM calculations or as refinement restraints. The second method, density selection, sets the water count from the protein size instead of the confidence score. This is useful for comparing hydration across structures of different quality, and on predicted structures, where errors in the input coordinates lower the confidence scores and a threshold would return too few waters.

WaterFlow is trained on static coordinates, while the experimental data average over heterogeneous protein and solvent, so many of the waters it learns from are partially occupied. Getting past that requires training data with partial occupancies and alternative conformers. WaterFlow paired with an experimentally guided ensemble framework like sampleworks, is one future route to generating it. In the meantime, you can use WaterFlow, follow our progress, and contribute here.

The DiffUSE Project began in July 2025 as an experiment in democratizing the methods of dynamic structural biology. While static models and sequence data have provided incredible insights into biological function and applications, in part due to AI, this data is reaching the limit of what it can explain. The most interesting biological problems cannot be addressed with a single structure. Elucidating the dynamics of transient biological events, understanding allostery, and investigating how binding events reshape the conformational landscape to achieve specificity all require the ensemble. But we are data-poor in this area. To reach the scale and diversity of data needed to be able to leverage, predict, and act upon protein dynamics,

Reaching the scale and diversity of data required to understand and ultimately predict protein dynamics demands a rethink of the entire pipeline, from data collection to interpretation, without the constraints of conventional static structure modeling.

The conventional approach is sequential: collect the data, process it, model it, encode it, then interpret it. But to actually get the data and information needed to tackle the ambitious biological problems of our day, we need to address the full pipeline at once.

But we are not improving one step in an existing workflow. We are changing an entire ecosystem. This work could not be done inside a conventional institutional structure. The bottleneck in dynamic structural biology is the absence of data at scale and the absence of methods that can absorb and collect it. Neither could be solved by an individual group, nor by focusing on individual biological problems. Both are solved by treating methods development as primary scientific work and staffing it accordingly. Our scientific organization is made possible by being a Radial project. Methods developers, platform engineers, and data infrastructure teams work alongside the scientists asking the biological questions. Loose teams form around data collection and processing, modeling, encoding, and interpretation. Each team works within the scope of the problem in front of it, and each output is usable by the others and by the field.

Over the past year, we have made significant progress in every area of the project. We demonstrated reproducibility with diffuse scattering across two beamlines (pub coming soon!); collected diffuse data on 8 different proteins; and advanced theory to better model this data. We also built a robust modeling platform that integrates structure predictors with experimental data and used it to stress-test how much memorization exists in structure predictors. We also recovered latent heterogeneity from deposited X-ray crystallography data in over 60k X-ray structures. We have developed algorithms and methods for improving our ability to model solvent in X-ray crystallography protein structures, a crux for correctly modeling experimental structural biology data. Finally, we have developed new ways to encode and visualize these dynamics. We have done all of this work completely openly, engaging with external groups and the greater structural biology community along the way.

A year in, the different areas of the project are now beginning to converge. We are beginning to integrate diffuse scattering data into our modeling software, and consider how to encode different types of heterogeneity in mmCIF files. As the different areas of the project converge, we are just starting to show the power of tackling the entire pipeline at once: the whole can be greater than the sum of the individual parts.

By the Numbers

4 Scholarly Pubs with 8 different contributors

• sampleworks: A Modular Platform for Experimentally Guided Biomolecular Ensemble Generation
• Supplying a User-Defined Bulk Solvent Map for Refinement
• Recovering Conformational Heterogeneity from the Protein Data Bank at Scale
• WaterFlow: Prediction of Ordered Water Molecule Positions on Protein Structures

16 Logbooks with 18 different contributors

5 Key Software Methods with 13 different contributors

• Mdx2: an open-source toolkit for diffuse data processing written in Python
• sampleworks: a Python framework for integrating generative biomolecular structure models with experimental data
• WaterFlow: a Deep Learning model that predicts the positions of ordered water molecules conditioned on a protein structure
• Goodvibes (General Optimization Of Diffuse halos from VIBrational Elastic network Simulations): a software package that produces a full forward model of the phononic contribution to the diffuse signal
• A Python library and command-line tool for reading, writing, and modifying PDBx/mmCIF protein structure files with support for hierarchical heterogeneity extensions

38 Blog Posts written by 14 different authors!

Making diffuse scattering reproducible

Diffuse scattering was the first experimental data type tackled within the project. In a typical macromolecular X-ray crystallography experiment, only about half of the X-rays scattered by the sample are analyzed. Those photons fall in the Bragg peaks, and they encode the average structure of the molecules in the lattice. Surrounding and between those peaks is a weaker signal, known as diffuse scattering, that captures deviations from the average structure. This signal not only provides one of the few direct measurements of correlated protein motion, which is essential for understanding catalysis, allostery, and ligand recognition.

For decades, diffuse scattering has been treated as background and discarded. This is primarily because the signal is difficult to measure, model, and interpret. Those difficulties have kept the method in the hands of a small number of specialist laboratories. Over the past several years, macromolecular diffuse scattering methodology has advanced substantially, and the measurement and interpretation problems that once defined the field are now considered tractable.

Building the tools and methods to democratize the method would allow rapid scaling by using existing synchrotron infrastructure. This would allow us to potentially reach a throughput of about 10,000 ensembles produced from diffuse scattering per year, providing the scale of data needed to learn about ensemble behavior, including for the training data necessary to predict and act upon protein dynamics. The joint analysis of the Bragg and diffuse components therefore returns two results from a single experiment: a high-resolution atomic model, and a description of how atomic displacements within that model are coupled.

Overall, our goal was to develop the methods and tools to democratize diffuse scattering. We started by aiming to obtain reproducible data from two different beamlines. Working across the Advanced Light Source at Berkeley (led by Kara Zielinski) and the Cornell High Energy Synchrotron Source (led by Steve Meisburger), we collected diffuse data on the same systems at both facilities and compared the resulting maps. Agreement between independently collected maps is the minimum standard for treating diffuse scattering as a quantitative measurement, enabling us to collect and analyze this data more widely. While there were no big surprises in samples and data collection, the lessons learned provide more background to help guide others in collecting data. With samples, we learned that humidity sleeves are essential to prevent dehydration over the course of the experiment, that the dose must be kept low, and that beamline background must also be low. Finally, crystal size matters; the larger the crystal, the better the data collected. We also found that multiple crystals are required for completeness, mostly because dose limits the rotation range available from any single crystal. We also observed that background subtraction is the most sensitive step in the pipeline. These insights have driven improvements in mdx2, our open-source package for processing diffuse scattering. The development of mdx2 is making the software more robust for other datasets. Finally, we can see that good diffuse data requires good Bragg data, with mosaicity and resolution being good proxies for diffuse map quality. Both are available early, which allows samples to be triaged before a full dataset is collected.

Outside of mac1, we also collected data on six other protein systems: NrdE, lysozyme, DNA fibers, ATCase, insulin, and Huwe1. Eighteen contributors collected this data, all logged in DiffUSE Logbook. This work helps democratize the collection, processing, and quality control of high-quality datasets, with the goal of allowing more groups (outside the DiffUSE project) to collect and process their own data. Be on the lookout for our upcoming publication detailing this analysis in much more detail! To model and interpret that data, we have mainly been using molecular dynamics as our primary route to modeling the signal. While this can provide valuable insights into the molecular motion we capture in the diffuse signal, a large gap remains between what the simulations produce and what the experimental data show. Closing that gap for improved modeling, including with machine learning (see below), requires pushing these physical models closer together. Therefore, in parallel, we are pushing further toward a complete forward model of diffuse scattering. Because diffuse scattering comes from multiple sources of deviations from the average structure, including deviations within the unit cell (lattice vibrations) and deviations or movement of macromolecules within the unit cell, we want to separate these two contributions to isolate the biologically interesting motions.

Our first approach is to continue developing GOODVIBES (led by Stephen Thornton), the software package that produces a full forward model of the phononic contribution. We are also working on generating accurate total scattering from crystalline molecular dynamics simulations and comparing it to experimental data (led by Taylor Womack and Mike Wall). Both are critical to determining how we model this signal. By simultaneously collecting and processing high-quality data today, combined with improving the theory of diffuse scattering, we will improve models of the data and biological motion down the road, similar to what has been done with Bragg peak data and cryo-EM data processing.

Machine Learning Modeling of Experimental Data

Machine learning structure predictors are now central to structural biology, in part because they can be used as a search mechanism over conformational space. Predicted models now routinely serve as templates for atomic model building and refinement, and placing predictors in an iterative loop with experimental data accelerates model building while improving prediction quality. One path to advance this idea is to exploit modern predictors’ ability to sample from their learned conformational distribution. Treated as conformational samplers, predictors can be combined with inference-time guidance, also described as scaling or steering, to improve agreement with experimental data. Such methods have been applied to single structures and, more recently, to ensembles. However, comparing results across these approaches or building on them was difficult. This limited our ability to iterate fast to improve modeling.

Developing sampleworks

To enable rapid development and iteration, we developed sampleworks, a platform for modeling experimental structural biology using machine learning. Our initial conditions for sampleworks were that it needed to take in multiple types of experimental data and be robust enough for us and others to build on in the future, as machine learning modeling approaches change. sampleworks provides abstractions for modular combinations of structure predictors, inference-time steering methods, and loss functions, so that new predictors and guidance methods can be implemented and tested against multiple forms of experimental data. It also includes evaluation infrastructure for parallel comparison of predictors and guidance methods, using metrics for ensemble quality and for agreement with experimental data.

We used sampleworks to test whether ensemble predictors have internalized physics or memorized their training distributions. We used alternative conformations deposited as altlocs within a single PDB as the test. They are physically real and experimentally supported, but are stripped from the training data of every public predictor. A predictor that has learned anything resembling physics should recover them, particularly when guided by experimental density. Without guidance, predictions collapse onto whichever conformation dominates the training set. Guidance substantially improves fit to density, raising the fraction of segments above an RSCC of 0.8 from 45.4 to 96.0 percent. However, we observed that the predictors tend to track their training distribution. While guidance reduces this bias, it did not remove it. But we can already see some machine learning tricks or changes to ensemble predictors that point to potential ways forward. sampleworks lets us integrate new experimental data types, such as diffuse scattering, to test new methods and release those capabilities for others to use in modeling structural data. Critically, this lets us integrate the project from data collection through machine-learning-based modeling.

Modeling existing heterogeneity from the PDB

One thing from our analysis with sampleworks is how limited conformational ensemble prediction is by training data. However, one of the largest untapped sources is the experimental data already deposited. Crystallography and cryo-EM measure data from millions of copies of macromolecules and average them. Conventional refinement collapses that average into one set of coordinates. However, the signal on these multiple states exists in the structure factors but needs to be modeled. Therefore, we set out to generate better ensemble data from the latent data in the structure factors.

We ran qFit multiconformer modeling across the high-resolution PDB to extract some of this latent heterogeneity. Starting from 80,876 PDB-REDO entries at 2 Å or better, the qFit multiconformer models fit the data better, with 85.7 percent having a lower R-free than the re-refined deposited comparison. They also contain substantially more conformational heterogeneity, with 9.4 percent of residues gaining at least one alternate conformation, and root-mean-square fluctuation increased by a mean of 0.13 Å. This is only the starting point. We can extract significantly more information from existing data. By combining sampleworks, qFit, and other machine learning approaches, we are extracting even more heterogeneity latent in structure factors. This is a key piece in getting us the ensemble data needed to accelerate generating the diversity we need to predict ensembles.

Solvent Modeling

Solvent occupies about 50% of a macromolecular crystal. X-ray scattering arises from every atom in the crystal, so solvent must be correctly accounted for to model the collected data. Beyond this, water molecules mediate protein stability, ligand binding, and catalysis, making them essential to understanding macromolecular biology. We model ordered waters in the first hydration shells as discrete atoms. However, predicting water molecule positions has lagged far behind protein and ligand structure prediction. Everything beyond them is approximated by a flat, binary mask.

We have tackled this with two approaches, first predicting ordered water molecules with WaterFlow using a deep learning model. The model sets a new state of the art, especially at sub-angstrom localization, where the gains concentrate, and sub-angstrom accuracy is what binding and catalysis applications require. WaterFlow is a two-stage model: a flow-matching generator learns a velocity field that transports prior samples to crystallographic water molecule positions, combined with a confidence model that scores each candidate; the user then sets a threshold that controls how many waters are returned. Both stages share a geometric vector perceptron backbone built on ESM3 residue embeddings. The largest single design gain came from crystallographic symmetry mates.

We demonstrated that we can predict bridging water molecules between proteins and ligands, and that our predicted water molecules fit experimental data better, including novel predictions with no deposited counterpart, sitting on positive difference density far more often than displaced decoys.

We also worked on integrating bulk solvent parameters into refinement software. While bulk solvent is often treated as flat and discrete, this is not what the data is actually capturing. The poor modeling of this solvent is a major contributor to the persistent gap between refinement R values and data quality. Richer descriptions exist, from molecular dynamics, 3D-RISM, and holographic reconstruction; however, getting them into refinement software to test them was a barrier. We developed a patch to integrate these.

Encoding Data for AI and Humans

Modeling heterogeneity is useless if the result cannot be communicated to AI and humans. PDBx/mmCIF represents alternate conformations through altlocs, which mark mutually exclusive atomic positions. It provides no way to state how those states relate. Beyond dynamics, this encoding is critical to communicate protein-ligand interactions. Fragment screening and time-resolved experiments generate exactly this structure of heterogeneity, and both are growing ways for us to understand new biology.

Over the past year, we have worked in the wwPDB and refinement programs to implement a new mmCIF category encoding hierarchical relationships between heterogeneity states, paired with a coexistence table specifying which states may co-occur. To support these changes, we have been developing visualization software and building off of existing modeling software, Bandicoot, to enable model building, visualization, and computing on these heterogeneous structures.

Functional evaluation

Collecting, modeling, and encoding structural data is valuable, but it has limited utility if it does not provide insight into function. However, how to even measure ‘function’ is an open question. A single protein can perform an extraordinary range of biological functions. It can catalyze reactions, bind ligands, and transduce signals allosterically. The impact of each function can change with small perturbations, such as a shift in pH, a macromolecular interaction, or a post-translational modification. Together, these functions are essential for biological processes. Predicting these functions from sequence data and how they change under such perturbations is a central goal of biology. This would enable interpretation of the millions of cataloged protein sequences, understanding the impact of mutations, and designing for function. One challenge of full functional characterization is that protein function is vast and multifaceted, spanning many levels of granularity. Untangling these effects is required to predict mutational consequences, design better therapeutics, and engineer proteins with novel properties.

High-throughput experimental methods now enable the collection of data on more specific, mechanistic functional landscapes at scale. However, each assay often measures a single molecular phenotype under a single engineered selection such as stability, binding, or enzymatic turnover. A variant can be neutral in the assay’s readout yet functionally disruptive through a mechanism the assay never probes. These assays can produce the scale machine learning needs; however, each captures only a subset of functional axes. To characterize models predicting protein function, we developed PLUG, an extensible framework for constructing leakage-free hold-out sets that enables evaluation of prediction models across multiple functional viewpoints. The benchmark compiler is organized so that any new functional benchmark can be added and a rebuilt leakage-free training set can be created against it, giving the community a moving target that resists the gradual contamination that has eroded the value of previous static benchmarks. We propose adopting PLUG as a framework for continual evaluation of protein function predictors, with careful homology detection to assess generalization and performance as a function of training similarity.

We are using PLUG as the baseline for our functional datasets, with our next endeavor looking at how dynamics impact function, focusing on cases where sequence data does not provide the necessary information.

Where are we now?

Beyond progressing on each of these fronts, we are beginning to integrate the pipeline. Diffuse scattering data is being brought directly into Sampleworks. In parallel, we are determining how best to encode Sampleworks ensemble outputs, and we are using conformational heterogeneity already present in the PDB to train better models to input into Sampleworks. We are also building functional outputs to establish where sequence or static structure provides sufficient functional information and where dynamic data is genuinely required, stress-testing our data collection and modeling.

Recurring lessons across the project:

• You can tackle ambitious science projects with the right people
• New organizational structures are needed to accelerate science
• Democratizing tools and methods requires returning to the basics
• Building robust tools that others can readily build upon is essential
• Working openly enables community engagement

We are building on these lessons to expand our experiment into new experimental modalities, new biological questions, and new AI-native approaches to solve these problems.

Most of that effort was unglamorous: making the analysis reproducible, wrapping it in a pipeline, getting it to run the same way across different (and cloud) compute environments, etc. This is the hard part of open science. But doing it forced us to pin down every choice that feeds into a diffuse map, and in the process it deepened our understanding of what the result actually is and how firmly we can stand behind it.

Going into the project, we knew qualitatively that high-quality diffuse data depends on keeping the sample humid, avoiding radiation damage, and using large crystals. Now, we can show that when all three are in place, the maps agree across sites. Our best metric for this is the pairwise correlation coefficient between datasets, compared against a ceiling, or maximum expected correlation, set by the product of each dataset’s CC* (an estimate from its internal consistency (CC½) of how well it tracks the true underlying signal). This lets us ask how similar two datasets are given the noise in each. Across four beamtimes, two at CHESS, two at ALS, every pairwise CC sits at or near its ceiling, meaning the datasets are as similar as their noise allows, with no signature of which site collected them.

Figure legend: Correlation between a CHESS dataset and ALS dataset for the halo (blue) and non-halo (green) relative to the maximum expected correlation (dashed line).

And when one or more of the ingredients is systematically missing (whether on purpose or by accident), the maps clearly diverge, and we can trace the difference back to the piece that was left out. Our focus on reproducibility became the instrument for understanding our physical and computational workflows, setting the stage to help democratize diffuse scattering across more beamlines and users.

In addition to the headline result (diffuse scattering reproducible, yay!), this process gathered a pile of smaller lessons, mdx2 quirks and settings, and beamline configuration changes that we will describe fully in the Stack. Collectively, these lessons allowed us to strengthen our computational methods by identifying and fixing ‘one-offs’ such as a single hot pixel at the detector at CHESS ruining our ability to get quantitative agreement between sites. Without a standard mdx2 workflow, background subtraction can masquerade as signal, or we can accidentally misindex samples.

We are excited to have crossed the threshold of “reproducible diffuse scattering” and the full stack will enable others to join the party. This entire exercise shows that a major value of open science is raising our game to finish the work.

We talk about a big travel month (including Stephanie winning the George Sheldrick Award for Structural Science), federal funding in light of the upcoming turnover of US political representation, pre-prints, and the use of AI with agentic workflows for science and LLMs for writing.

News Links:

• Derek Lowe’s Blog: The Assault On Science Funding Continues
• Senate votes to block rule giving White House more control over research
• Estimating global article processing charges paid to 14 publishers for open access between 2019 and 2025

Preprints/Blogs:

• Atomic modeling of radiation damage in cryoelectron microscopy datasets
• Dynamic view of an allosteric intermediate in a positively cooperative dimer
• Incidental conformational switching in an allosteric enzyme
• Anthropic Protein Design Prompts

• The Rise of Large Language Models and the Direction and Impact of US Federal Research Funding

• Spotify
• Apple Podcasts

What if diffuse scattering data were carried through all steps in data processing and model refinement? What new science would that enable?

Let’s find out! One key step toward this goal is to make diffuse data processing easier. In this post, I’ll introduce our open-source toolkit mdx2, and explain why I think it will change the way structural biologists use X-ray data.

What makes diffuse data tricky to process?

The main task of data processing is to map detector pixels and their intensities into three-dimensional (3-D) reciprocal space, and then to merge and scale photon counts in order to generate a 3-D map of diffuse intensity (Figure 1).

Figure 1. Diffuse data processing can be divided conceptually into two steps: mapping signals from the detector to reciprocal space, and merging them to produce a final three-dimensional (3-D) map. Adapted from Meisburger & Ando (2024).²

In the first step, the precise orientations of the crystal, detector, and X-ray beam are determined, and the photon counts are corrected for various non-ideal effects. Thankfully, this first step is shared with conventional crystallography, and the algorithms have already been worked out. The second step, scaling and merging, requires a different approach because Bragg diffraction and diffuse scattering differ in a few key ways.³

• The diffuse scattering is a continuous signal requiring discretization into reciprocal space coordinates (H, K, L can take any value, not just integers). To fully describe sharp diffuse features like halos, the diffuse map is typically oversampled relative to the reciprocal lattice. Multiple passes of data processing are performed, first producing a coarse map to refine corrections, and then a final, finely sampled map.
• Diffuse scattering is everywhere on the detector and mixed with the background from the experimental apparatus, requiring a separate background dataset measured with the crystal out of the X-ray beam.
• The diffuse signal is very weak compared with the Bragg peaks, with most detector pixels recording zero counts, so algorithms must be robust to count-limited data and carefully designed to avoid artifacts when merging many signals together.

What is Mdx2?

Mdx2 is an open-source toolkit for diffuse data processing written in Python. Mdx2 is built on top of DIALS, an open-source project for conventional crystallography data processing. Mdx2 modules can be imported to access the data as familiar Python objects, or its command-line tools can be run with a DIALS-like syntax

Mdx2 began as a teaching tool for the 2022 International School of Crystallography in Erice, Italy.⁴ Subsequently, the Ando lab’s state-of-the-art scaling and merging algorithms³ from the MATLAB-based mdx-lib were incorporated.² When we began the Diffuse Project last year, it was clear that mdx2 would be the right choice for data processing, since it is open source and easy to use. However, because it was so new, several key features from mdx-lib were still missing, and it had only been tested on a handful of datasets.

Where we’re at

Over the past year, teams at Cornell and UCSF collected hundreds of diffuse scattering datasets over sixteen synchrotron visits (documented online in the logbook). As the teams worked together to process data, sticking points became clear very quickly, and these were addressed through improvements to the code. Here we highlight several major updates.

Reproducible computing. Mdx2 is now packaged on conda-forge. Install the latest version and its dependencies using mamba install -c conda-forge mdx2.

Automated reports. Since version 1.0.4, mdx2 can produce data visualizations, validation statistics, and other plots via the mdx2.report command-line tool. This tool generates Jupyter notebooks from built-in templates and executes them automatically in the mdx2 environment (Figure 2).

Figure 2. Example output generated by mdx2.report. The figure, selected from the visualization.ipynb report notebook, shows a slice through the diffuse map of SARS-CoV-2 macrodomain (mac1) crystallized in the P4₃ space group. Slices are perpendicular to the L direction (the four-fold symmetry axis). The image on the left is the total intensity, which includes scattering from the disordered solvent within the crystal. In the image on the right, the isotropic scattering has been removed to better visualize the fluctuating pattern.

Fine map generation. mdx2.reintegrate implements the final step in data processing, after corrections are refined, to generate finely sampled diffuse maps (Figure 3).

Figure 3. Slices through a finely sampled 3-D diffuse map (same dataset as Figure 2). The scattering was mapped on a 5 x 5 x 11 oversampled grid (H x K x L, relative to the reciprocal lattice) using mdx2.reintegrate. Slices are perpendicular to the L direction. A lattice of intense peaks (red in the color map) appears whenever L is an integer – these are the diffuse halos (Bragg peaks have been removed).

What’s next

Now that mdx2 has been refined through many cycles of data collection, we are turning our attention toward increased automation. We are targeting two applications:

• Providing timely feedback during data collection, which would enable experimenters to make decisions informed by diffuse scattering.
• Batch re-processing accumulated data from the project, for instance when a new algorithm is added to mdx2.

We are keen for others to try mdx2, and we welcome contributions to the project. Tutorials and source code are available at github.com/diff-use/mdx2.

References

Figure 1: A moving slice through reciprocal space of the diffuse signal predicted by a model that includes lattice vibrations of a triclinic lysozyme crystal. The scale is different in each frame so that the diffuse signal between the Bragg peaks is visible.

The brightest features in the diffuse signal occur around the Bragg peaks, and are termed “halos” for their round (but often asymmetric) shape. These halos are well-reproduced by a model that includes thermally excited lattice vibrations, or phonons. Phonons are elastic waves that propagate through crystals and displace proteins from their average positions. Very low-frequency and long-wavelength phonons contain information about the often anisotropic sound speeds and elastic moduli of the crystal, but they do not contain direct information about internal protein motions. High-frequency and short-wavelength phonons can involve correlated motions of proteins in the crystal that are related to the biological function of isolated proteins in vivo.

Figure 2: Selected regions of the diffuse X-ray scattering of a lattice dynamics model (left) and of a real protein crystal (right). The red features are halos. The halos which are surrounded by blue boxes were used to refine the model to the data (image from Meisburger et al. 2020²).

At any temperature, the most populated phonon modes are the lowest-frequency ones. Unsurprisingly, then, these low-frequency phonons which only contain information about long-wavelength crystal properties produce the halos and dominate the diffuse signal³. General Optimization Of Diffuse halos from VIBrational Elastic network Simulations (GOODVIBES) is a software package that produces a full forward model of the phononic contribution to the diffuse signal⁴. It works by modeling a protein crystal as a periodic collection of rigid bodies which are connected to neighbors by springs. From this model and an independently refined map of the electron density, information about the expected diffuse signal contributed by thermally excited phonons can be readily computed. The strengths of springs that connect adjacent rigid bodies are adjusted until the diffuse signal from the model well-fits the bright, halo-like features of the experimental data.

As a full forward model of the phononic contribution to diffuse scattering, the utility of GOODVIBES is many-fold and leads to several open questions:

1. If we model the long-wavelength contribution to the diffuse signal very well, can we subtract it out to extract biologically relevant motions that contribute to the diffuse signal?
2. After the halos are fit, can we understand the more complicated, structured diffuse signal between the Bragg peaks as also coming partially from phonons?
3. Can more sophisticated modeling of the internal motions of the protein be developed so that harmonic modes related to biological function can be fit to the experimental data?

Where we are now

We are re-developing the GOODVIBES pipeline using entirely open-source Python packages. As part of this, we are re-thinking each step in the pipeline, from the structure refinement to the parameterization of the interactions within the asymmetric unit of the crystal. We want the refinement procedure to be flexible enough to handle protein crystals in any space group and of any unit cell size. While we are aiming to refine an elastic network model of Mac1 in the P43 space group (PDB 7tx0⁵) to experimental diffuse scattering data, here we show results based on the refinement of lysozyme in the P1 space group (PDB 6o2h⁶) from an older iteration of GOODVIBES as proof of concept. We treat the entire protein in the unit cell as a rigid body, so no internal protein modes are modeled. With one protein per unit cell, there are consequently 6 phonon bands, which change shape appreciably during the refinement process.

Figure 3: The phonon band structure (left) and density of states (right) of triclinic lysozyme predicted by GOODVIBES during refinement. Each changes dramatically during refinement to fit the halos in the experimental data.

After refinement, the halos are well-fit, and the fully refined harmonic model contains a wealth of easily extractable information that can be compared with other experimental data: the phonon band structure and density of states allow us to determine the independent elastic moduli of the crystal and to calculate the phononic contribution to thermodynamic quantities like entropy and specific heat, which may be anomalous⁷.

Where we’re going next

We have a combination of short- and long-term goals for the future development of GOODVIBES.

1. Demonstrate the full refinement pipeline for the Mac1 protein crystal in the P43 space group (PDB 7tx0⁵), modeling the two independent chains in the asymmetric unit as separate rigid bodies.
2. Incorporate structure refinement into the pipeline - can information contained within the diffuse signal be used to get a better average structure?
3. Rethink the parameterization of the rigid body interactions - are springs between residues found with a neighbor search an optimal choice for this refinement problem?
4. Incorporate flexibility in the rigid body decomposition, perhaps integrating it with other TLS refinement software. This will allow further exploration of how model parameterization affects the portions of the diffuse signal not included in the fit to the experimental diffuse signal.

As we continue to develop GOODVIBES, we will be thinking more deeply about how to integrate information gleaned from a fit to this model which assumes phonons in regular crystals generate the diffuse signal with other models that include contributions that come from motion that cannot be understood as harmonic. Non-harmonic contributions to the signal include conformational heterogeneity (modeled with sampleworks⁸) and highly disordered protein and solvent (found through molecular dynamics simulations (e.g. Wall et al. 2019⁹)).

GOODVIBES is developed in the open. The source code is available at https://github.com/prism-science/goodvibes, and we welcome issues and pull requests.

References:

This display mode, selected in a drop-down menu in BANDICOOT’s Display Manager, generates a hue offset for each altloc (a residue modeled with conformational hetereogeneity) such that all the alternate conformations are visually distinguishable at a glance. The offsets are configurable, so that each user can create the level of visual distinction most comfortable and intuitive to them. The mode works smoothly: removing all alternate conformations automatically reverts the residue to the coloring of bulk molecule, while each added alternate conformation is automatically hue-shifted from the last one.

This is only the first small step in a larger project to implement hierarchical heterogeneity tools and features in BANDICOOT. When complete, users will be able to not only view altlocs in different colors, but visualize the hierarchy, assign hierarchical positions to each alternate conformation, and, importantly, output mmCIF-formatted files with this information.

Bandicoot is an open-source project, forked from Coot 0.9.8.95. Source code and binary releases are available for download from the public repository at https://github.com/fraser-lab/bandicoot. Currently, Bandicoot runs on only on MacOS (tested on Tahoe but runnable on previous MacOS versions as well). It’s a work in progress and thus is a bit rough around the edges. Users are encouraged to report any issues and submit pull requests.

One of the hurdles to this is less scientific than operational: you have to be able to write whatever you are trying to represent down — PDBx/mmCIF. That format is old, rich, and has a very particular structure that a large amount of software reads, writes and depends on. So extending it gracefully is nontrivial. Any extension has to satisfy several kinds of constraint — physical (it must be able to say what is true of the sample), ontological (it must sit sensibly inside the existing dictionary), and social (the surrounding ecosystem should be able to adopt without rewriting too much).

The tool we are putting together is a bit of a playground towards this effort: mmcif-browser. There are three pages: a browser for the mmCIF dictionary itself, a live file inspector paired with a 3D viewer (inspired by the compiler explorer) – source on one side, its consequence on the other – and a written version of the current working proposal that anyone can read and pick apart.

Figure 1: The mmcif-browser file inspector and 3D viewer.

1. The dictionary switcher — flips the definitions the page reads between plain PDBx/mmCIF v50 and v50 plus the proposed heterogeneity extension, so the new categories light up (or don’t).
2. The examples — the set of example files to play with (some real PDB depositions, some hand-crafted scenarios in service of the proposal).
3. The heterogeneity states — where the file contains clearly defined heterogeneity states (either via the existing altloc/occupancy/bfactor mechanisms or via the newly-proposed ones) I provide some QOL UI elements to toggle through them.
4. The settings bar — contains filters and by-category navigation so one may quickly find the part of the cif file that one needs.

The current proposal, briefly

Figure 2: The proposed heterogeneity categories, layered on top of the existing altloc/occupancy/bfactor mechanisms.

mmCIF already records two things per atom: which alternate a given atom belongs to (label_alt_id), and the fraction of copies it is present in (occupancy).

Both are one-atom-at-a-time facts. What they cannot express is which alternates are found together in the same copy — the correlation between sites, the joint distribution rather than its marginals.

The extension aims to add a few optional categories that name alternate states, say which of them exclude one another, and list the combinations that actually co-occur with their joint occupancy.

If that were added, that would bring us one step closer to being able to communicate multiple types of heterogeneity, but it’s very much an open problem as to how to actually compute these joint states perfectly, define or recover them from standard refinement and bundle them together into biologically meaningful states.

All of this is very much a work in progress — the proposal most of all.

Even when crystallized, proteins move. But each copy in the thousands of unit cells don’t all move the same way. Therefore, a protein crystal is in essence a spatial sampling of the thermodynamic ensemble of a protein (within the lattice). These varied internal motions within each unit cell have been identified as a major contribution to the diffuse X-ray scattering intensity. Diffuse scatter is caused by variance in the structure factors between every unit cell in a crystal. X-ray crystallography has therefore always had the potential to detect correlated intraprotein motions, if only we can learn how to interpret the diffuse signal properly¹. These correlated motions are likely to underlie many biologically relevant movements.

This is the challenge that the diffUSE team has been working on for the past year, but there’s a big complication: internal protein motions aren’t the only contribution to the diffuse scattering signal, and they likely aren’t even the majority component². Lattice vibrations known as phonons (basically sound waves) move throughout the crystal lattice and cause long-range disorder that is picked up in the diffuse signal. We can think of these vibrational modes as whole proteins fluctuating around their average positions. Since this disorder is long-range and the vibrations are due to motions of entire unit cells, this type of diffuse scattering mostly shows up as extra intensity concentrated around Bragg peaks, dubbing this feature’s name: halos.

Before we can get to interpreting the signal from the (interesting) internal protein motions, we’d like to isolate this vibrational lattice signal. A great way to remove a signal you’re not interested in is to have a forward model that generates exactly what that signal is, then simply subtract it. This is exactly what the GOODVIBES lattice dynamics simulator developed by Steve Meisburger, David Case, and Nozomi Ando does³. In earlier studies on hen egg-white lysozyme crystals²³, these halo features were effectively simulated by rigid-body elastic network models. Residues, domains, or entire protein molecules were treated as stiff objects where a force (or torque!) applied to one atom causes the whole object to move. These rigid bodies are coupled at contact points by springs and vibrate around their equilibria according to two refineable energy potentials:

where $i$ and $j$ are the indices of the coupled atoms, $\mathbf{u}$ is the displacement vector of an atom from its equilibrium position, the $k_{ij}$s are the respective refineable spring constants between atoms $i$ and $j$, and $\hat{\mathbf{r}}_{ij}$ is the unit vector pointing from atom $i$’s equilibrium position to $j$’s. In this case the complete potential energy of the system is

In the harmonic approximation, and for dimensional indices $l\in\lbrace x,y,z\rbrace$, forces on the $i^{\text{th}}$ atom in the $l$ direction due to displacement of the $j^{\text{th}}$ atom along the $l’$ direction are described as

Rigid body vibrations can be turned into scattering factors through the one-phonon approximation for thermal diffuse scattering⁴. Continued improvements to the GOODVIBES model will not only help isolate unwanted lattice vibrations, but may also help describe biologically important domain movements that are well-approximated by rigid bodies (e.g. catalytically-relevant hinge-bending). This work will be continued by the diffUSE East team especially by the Ando Lab’s new postdoctoral scholar, Stephen Thornton.

Our ultimate goal is a forward model of the complete diffuse scattering map — containing features arising from both rigid-body motions and short-ranged atomistic ones. Molecular dynamics (MD) simulations are still the best tool we have to assess conformational flexibility at the atomic scale. Other diffUSE member (and great mentor!) Mike Wall has developed a structure factor calculation script xtraj.py in the Lunus package (https://www.github.com/lanl/lunus) which generates both $F_{calc}$’s and diffuse intensities from MD trajectories. MD combined with xtraj allows us to model diffuse scatter directly associated with protein internal motions, and the isotropic component of the diffuse signal has been well-characterized using it⁵. However, MD runs are far too computationally expensive to simulate an entire protein crystal. Therefore, MD diffuse maps are missing lattice vibrational features as discussed above, yet they probably aren’t missing all of them. When performing MD simulations of crystals, we need a supercell of several unit cells in order to avoid autocorrelation artifacts. Even in the minimally viable 2x2x2 system, we have a portion of the extended crystal lattice which introduces vibrational elements. To what extent MD recapitulates vibrational features is an open question, and the first system we are using to tackle this question is a 5x5x5 supercell of triclinic lysozyme (6O2H, Figure 1).

Figure 1: (Left) HEW lysozyme 6O2H structure. (Right) Our solvated and equilibrated 5x5x5 triclinic crystal built from 6O2H. This system was used as the basis for production simulations of both restrained and unrestrained dynamics.

So far, I’ve simulated this system over 1μs, calculated the relevant structure factors using xtraj.py, and generated total scattering diffraction series (Bragg + diffuse) using James Holton’s nanoBragg diffraction plotter (Figure 2). I’m working on processing these simulated datasets using the mdx2 diffuse scatter processing pipeline in an attempt to replicate the workflow that we use for experimental data⁶. After that, I plan to fit my simulated diffuse map to a GOODVIBES elastic network model to see which halo features are present. The overall questions I am trying to answer are:

1. How can we generate accurate total scattering from crystal dynamics simulations and correctly compare it to experimental data?
2. What methods most effectively deconvolve lattice dynamics from internal motions?

Figure 2: Total scattering diffraction images generated from MD trajectories of the 5x5x5 6O2H crystal using nanoBragg. (Left) Image from a restrained MD run in which the protein’s heavy atoms were harmonically restrained to their crystallographic coordinates. The Bragg lattice is unrealistically strong and minimal diffuse scatter is observed. (Right) Image from an unrestrained run. The Bragg lattice is weaker and diffuse scatter is evident. It’s possible that targeted restraints and tuning of the well-strength are an avenue to isolating specific motions in the crystal.

If we can identify which vibrations occur in MD, there may be a way to bias the simulation to remove those features while leaving the internal motions unbiased, thus enabling us to simulate a complete diffuse map with MD internal motions and fully realized lattice vibrations from GOODVIBES. Or perhaps constituting the complete diffuse map will be as simple as subtracting out the incomplete vibrational features from MD and adding in the complete ones from a GOODVIBES simulation. These speculations have yet to be clarified, and I am open to feedback and suggestions!

If we can develop an accurate forward model for all the major features of a protein crystal’s diffuse scattering map, we will not only have advanced the study of dynamic structural biology on a theoretical level — we will also enable machine learning and AI tools to use diffuse maps to distinguish between ensembles that produce identical dynamically averaged Bragg data. Tools like the sampleworks platform pioneered by the modeling team within diffUSE. Suffice it to say, diffuse scattering is indeed not merely due to lattice vibrations, but the lattice signal is significant and cannot be ignored; my contribution to the diffUSE project begins with moving us towards developing such a complete forward model.