Fast and Accurate Nanoparticle Characterization Using Deep-Learning-Enhanced Off-Axis Holography

Characterization of suspended nanoparticles in their native environment plays a central role in a wide range of fields, from medical diagnostics and nanoparticle-enhanced drug delivery to nanosafety and environmental nanopollution assessment. Standard optical approaches for nanoparticle sizing assess the size via the diffusion constant and, as a consequence, require long trajectories and that the medium has a known and uniform viscosity. However, in most biological applications, only short trajectories are available, while simultaneously, the medium viscosity is unknown and tends to display spatiotemporal variations. In this work, we demonstrate a label-free method to quantify not only size but also refractive index of individual subwavelength particles using 2 orders of magnitude shorter trajectories than required by standard methods and without prior knowledge about the physicochemical properties of the medium. We achieved this by developing a weighted average convolutional neural network to analyze holographic images of single particles, which was successfully applied to distinguish and quantify both size and refractive index of subwavelength silica and polystyrene particles without prior knowledge of solute viscosity or refractive index. We further demonstrate how these features make it possible to temporally resolve aggregation dynamics of 31 nm polystyrene nanoparticles, revealing previously unobserved time-resolved dynamics of the monomer number and fractal dimension of individual subwavelength aggregates.

N anoparticles play a crucial role in many fields, including pharmaceutical sciences, 1 food production, 2,3 and environmental monitoring. 4 As particle size and composition greatly influence particle function, fast and accurate characterization tools are essential and, ideally, should work in the native environment of the particles. For example, in pharmaceutical applications, the interaction between nanoparticles (e.g., protein aggregates, extracellular vesicles, and viruses) and biological cells depends crucially on particle size and composition, 1,5−8 and studying this relation requires accurate characterization of nanoparticles in the complex extra-and intracellular environments. In food production, nanoparticles are used to stabilize emulsions, improving food texture and shelf life. 2 In environmental monitoring, there is a need to identify and characterize nanoparticles that enter the air, water, and soil as a byproduct of industrial processes and waste disposal. 4 In all of these applications, it is often necessary to determine and monitor particle properties and activity in an environment whose physicochemical properties are unknown.
Traditionally, individual submicron particles in dispersion have been indirectly sized by analyzing their diffusive Brownian motion in a solution of known viscosity (nanoparticle tracking analysis 9,10 ). In this approach, the trajectory of a particle is tracked, its diffusion constant D is determined from the mean squared displacement of the measured trajectory, and finally, the particle radius r is estimated using Stokes−Einstein relation, i.e., r k T D 6 B = πη , where k B is Boltzmann's constant, T is the absolute temperature, and η is the viscosity of the medium. Even though this method is widely employed, it makes several assumptions and presents several limitations. First, as the Brownian motion is stochastic, the particle trajectory needs to be observed over many (>100) time steps to achieve a reliable estimate of the particle radius, which limits the applicability of diffusion-based methods in fast processes (e.g., high-flow sorting) or when the particle sizes change dynamically (e.g., dynamic aggregation processes). Second, the medium must be viscous with a known viscosity η, which prohibits the application of this method in biologically relevant media, which are often viscoelastic with unknown properties.
Third, this method can only be applied to particles close to thermodynamic equilibrium because it implicitly makes use of the fluctuation−dissipation relation, excluding several processes of interest that occur out of equilibrium, such as in living systems and in active matter.
In order to overcome these limitations, several methods have been proposed to determine the properties of the particle by measuring its scattering properties, 11,12 instead of its motion. For example, novel microscopy techniques such as interferometric scattering microscopy (iSCAT 13 ) have made it possible to quantify the optical scattering contrast of nanoparticles close to an interface. For particles larger than the wavelength of light, the angular distribution of the scattering intensity depends strongly on particle size, which has been exploited to characterize the size of particles 14 and protein aggregates, 15,16 both using classical image analysis 17 as well as deep-learning methods. 16 However, quantifying size and refractive index of particles smaller than the wavelength of light from optical scattering patterns in microscopy images remains challenging: The dependence of the angular distribution of the scattering on particle size is very weak, and in the limit of Rayleigh scatterers (r ≪ λ), the angular distribution is independent of size. The situation is further complicated by the fact that the microscope that collects the scattering pattern acts as a lowpass filter, collecting only the light scattered in a limited range of scattering angles determined by the numerical aperture of the objective. These problems make the use of traditional techniques for image analysis challenging; as a result, it has thus far not been possible to use the scattering properties alone to simultaneously characterize the size and refractive index of subwavelength particles.
Here, we demonstrate that holographic imaging combined with deep learning can simultaneously characterize the size and refractive index of subwavelength particles, while using 2 Figure 1. Combining off-axis holography and deep learning to characterize subwavelength particles. (a) Interference pattern is acquired with an off-axis holographic microscope and (b) Fourier-transformed and low-pass-filtered to produce the reconstructed field at the camera plane (here, only the imaginary part of the complex field is shown). (c) These complex field images are used to track the three-dimensional particle position. (d) Region of interest is selected around each detected particle for each time step. (e) Finally, this sequence of observations is used to characterize the radius r and refractive index n using a specialized neural-network design, which we call weighted average convolutional neural network (WAC-NET). (f) Laser beam is split into two paths by a polarizing beam splitter (PBS). The relative intensities of the two paths are regulated by the initial λ/2-waveplate. One of the beams goes through the sample, and the light scattered from the sample is collected by a microscope objective (MO), after which a tube lens (TL) provides a focused image at the camera (C). The paths are recombined using a beam splitter (BS) at a slight offset angle, resulting in an interference pattern at the camera. The particles are suspended in liquid inside a straight microfluidic chip (ChipShop). Reprinted with permission from ref 25 under the Creative Commons license. (g) Schematic overview of WAC-NET. Each observation is downsampled and flattened using a convolutional network. From this flattened vector, a weight and a representation are calculated and multiplied. The weightings are normalized using a softmax transformation. The weighted representations are merged by summation, and the result is passed to a fully connected network that returns r and n. ACS Nano www.acsnano.org Article orders of magnitude shorter trajectories than required by standard methods and without assumptions about the physicochemical properties of the medium. In fact, because of its potential for better, more autonomous performance, 18 deep learning has recently been employed to successfully solve several digital microscopy problems, such as particle tracking, 19,20 anomalous diffusion characterization, 21,22 image segmentation, 23 and image super-resolution. 24 In particular, deep-learning techniques have been shown to outperform traditional image analysis methods for signals with low signalto-noise ratios in the range of 1−10, 19 making them an attractive choice for the problem at hand. In this work, we develop a deep-learning-powered method to determine the radius and refractive index from a series of scattering patterns of subwavelength particles recorded by an off-axis holographic microscope.
We demonstrate that this method requires 2 orders of magnitude fewer observations than traditional methods to reach the same accuracy in size and refractive index. The reduction in the number of observations required for accurate characterization enables significantly faster characterization of subwavelength particles, which, for example, permits us to characterize individual subwavelength particles while they are flowing in a microfluidic device. We also demonstrate that this method correctly characterizes subwavelength particles without requiring knowledge of either particle shape or medium refractive index and viscosity, making it ideal for applications in native environments, whose physicochemical properties are often not known a priori. Finally, we demonstrate the ability of this method to enable applications beyond the state of the art, by continuously monitoring a nonstationary process, specifically the subsecond fluctuations in size and refractive index of aggregates of nanoparticles.

RESULTS AND DISCUSSION
Experimental Setup and Acquisition of Experimental Data. The scattering patterns of a volume of solution containing subwavelength particles are acquired using an offaxis holographic microscope with an illuminating wavelength of λ = 633 nm and an oil-immersion objective with a numerical aperture (NA) of 1.3 (see Figure 1f and Methods for a detailed description of the optical setup). The illuminating beam is split into two beams, one passing through the sample and one used as a reference. Then, these beams are recombined at the camera, which records their interference pattern. A representative example of such a pattern is shown in Figure 1a. This interference pattern contains sufficient information to reconstruct the optical field at the sample with diffractionlimited resolution (Figure 1b) by employing a Fourier transform and a low-pass filter (see Methods for a description of the reconstruction algorithm).
The particles are imaged under constant flow in a microfluidic chip (see Methods for details on the chip). This enables us to gather scattering patterns from a large number of particles in a short amount of time (typically a few hundred particles per minute). In order to extract the scattering patterns from each particle, the particles are first localized in three dimensions using a previously published method 12 (Figure 1c) and subsequently tracked using standard methods (see Methods for a detailed description). A region of 64 × 64 pixels centered around each particle detection is extracted. In this way, we record and store a collection of scattering patterns of each particle (Figure 1d), which is subsequently analyzed using a neural network (Figure 1e).
Structure of the Neural Network. Since the particles are smaller than the wavelength of the illuminating light, the features that are available for determining particle size are contained in the high frequencies of the image spectrum (corresponding to high scattering angles). 26 At low signal-tonoise ratio, it is prohibitively difficult to analytically fit these features and determine the particle properties. Thus, to achieve a good signal-to-noise ratio, it is necessary to average over multiple observations of the same particle. However, direct averaging of observations requires perfect three-dimensional localization, which is in practice impossible to achieve because of the error intrinsic to any localization technique.
In order to overcome this problem, we employ machine learning. While in standard methods the user explicitly defines a set of rules to convert the input data to the desired output, in machine learning, the rules are inferred by providing the machine-learning model with a large collection of input− output pairsthis process is known as training of the machinelearning model. 18 Neural networks are a very successful subset of machine-learning models 27 and consist of interconnected layers which apply some, often simple, transformation to their input and pass the result to some other layer. The transformation each layer applies can be controlled by weights, which are optimized during the training.
The architecture of the neural network we employ is shown in Figure 1g. This architecture is designed to optimally combine the information contained in multiple scattering patterns of the same particle and provide a single prediction of its size and refractive index. As discussed above, the scattering patterns themselves cannot be averaged before being sent to the network, due to imperfect localization. We overcome this problem by averaging the scattering patterns not at the input but at an intermediate layer in the network. This allows the network to produce a representation of the scattering pattern that contains the information that it needs to characterize particle size and refractive index (including the scattering strength and angular spectrum), while disregarding other aspects that are of no importance to the characterization (such as the position of the particle within the image).
An additional complication in the task of characterizing a set of observations of a particle is that each individual observation may not be equally representative (due to, e.g., uncorrelated noise or interference of the scattering patterns of nearby particles). In order to overcome also this problem, the network additionally assigns a score to each observation. These scores are used as weights when averaging the representations of multiple scattering patterns, ensuring that scattering patterns that are corrupted by significant noise contributes less to the overall characterization. We call this neural network architecture weighted average convolutional neural network (WAC-NET).
Training of the Neural Network. In order to train the WAC-NET, we first must generate a training set that is representative of the data produced by the optical setup. Using experimental data for training the WAC-NET is challenging because they are limited in their number and have intrinsic experimental errors. We therefore generate synthetic training data by simulating the complex scattering patterns of subwavelength particles using Mie theory (see Methods for details on the scattering pattern simulation). The scattering patterns are convolved with the optical transfer function of the ACS Nano www.acsnano.org Article microscope, determined by analysis of reference particles of known size and refractive index (see Methods for details on this calibration procedure). We train the WAC-NET to infer particle sizes and refractive indices within a broad range of values by simulating 10 6 scattering patterns cropped to 64 pixel × 64 pixel images from particles having radii uniformly distributed in the range of 115 nm ≤ r ≤ 500 nm and refractive indices uniformly distributed in the range of 1.36 ≤ n ≤ 1.9, dispersed in a medium with refractive index 1.33. This choice captures a range of particle sizes where existing optical characterization techniques struggle to provide accurate sizing results. Specifically, for radii larger than 500 nm, particle sizing can be performed in off-axis holographic imaging by counting the number of pixels the particle occupies in the image. 25 In the opposite limit, when entering the Rayleigh scattering regime, the angular distribution of the scattering pattern does not scale with particle size, prohibiting optical sizing. This consideration sets the lower limit to the sizes included in the training set. In order to make the training set representative of the experimental data, the observations of each particle are corrupted with inaccurate three-dimensional centering of the particle (with a standard deviation of the error of 0.1 μm, similar to the size of individual pixels) and synthetic noise in the form of spatially correlated background noise (with amplitude determined from empty slices of the image). A random number between 5 and 50 such observations are stacked to form the synthetic representation of a traced particle. An additional advantage of using synthetic data is that new data can be continuously generated during the training, eliminating the risk of overfitting due to a limited training set. Taking this approach, the training time of the network is about 12 h on a computer equipped with an NVIDIA GeForce GTX 1650 graphics card. The analysis of a single scattering pattern takes around 4 ms on the same computer. Neural Network Performance on Simulated Data. We first test the trained WAC-NET on simulated data. We simulate the scattering patterns of a set of 2000 stacks of 0.20 μm (radius) polystyrene (PS) spheres (refractive index = 1.58, illumination wavelength λ = 633 nm, and objective NA = 1.3) dispersed in water, each stack consisting of 100 observations of a single particle. The orange symbols in Figure  2a,b represent the estimated values of the radius r est and refractive index n est as a function of the number of time steps during which a representative particle is observed. It can be seen that these estimates converge to the ground-truth values (solid black lines) within only a few observations. For comparison, the characterization based on conventional particle tracking (gray symbols) produces significantly worse results than those obtained by the WAC-NET and do not converge to the ground-truth value even when using 60 time steps.
To compare the network performance with regard to size determination to particle sizing using conventional particle tracking measurements, we simulate 2000 Brownian motions, each having N = 10 4 time steps with time step Δt. To estimate the diffusion constant corresponding to a Brownian motion, we use the standard approach based on the mean squared displacement in a time step, i.e., D N i where Δx i is the length of the ith displacement. To determine the refractive index, there is no standard method for subwavelength particles. Nonetheless, it has previously been demonstrated that measuring either scattering intensity 11 or phase contrast 12 in combination with analysis of the Brownian motion allows simultaneous determination of size and refractive index. 12 In order to compare the performance of WAC-NET with existing methods for the determination of r est normalized by the ground-truth radius r true = 0.20 μm and refractive index difference Δn est normalized by the ground-truth refractive index difference Δn true = 0.25; r est and Δn est are estimated using deep learning (orange symbols) and a conventional diffusion-based approach (gray symbols) as a function of the number of observations of the particle. The deep-learning approach requires fewer observations to converge to the ground-truth values. (c,d) Distributions of the relative errors σ r and σ n for r est and n est , respectively, on the characterization of 2000 simulated particles using five observations. While the standard approach (gray histograms) fails to characterize the particles, the deep-learning approach (orange histograms) characterizes the radius to within 5% and refractive index to within 10% (mean absolute errors). (e,f) Scaling of the absolute relative errors ⟨|σ r |⟩ and ⟨|σ n |⟩ as a function of the number of observations used for the characterization. The dashed line represent the scaling N 1/ , where N is the number of observations. (g,h) Ratio of the absolute relative error made by the deep-learning approach (σ r,DL ) and the diffusion-based approach (σ Diff ) using 20 observations shows that the deep-learning approach provides an order of magnitude more accurate characterization both for particle radius (g) and refractive index (h) in a wide range of particle radii r/λ (where λ is the wavelength of the illuminating light), even on noisy images (signal-to-noise ratio < 10).

ACS Nano
www.acsnano.org Article refractive index, we assume that the phase contrast of the particles is known exactly at each time step and estimate the refractive index as described in ref 12. Figure 2c,d shows the performance of WAC-NET (orange histograms) and diffusion-based characterization (gray histograms) on the ensemble of 2000 particles using five time steps for characterization. While the distributions of estimates produced by WAC-NET are well-defined and centered around the true value, the diffusion-based estimates are scattered in a wide range and are not centered around the true value, demonstrating that WAC-NET performs significantly better than diffusion-based methods when using few time steps. These results highlight the difference in the underlying principle of the two approaches. In contrast to the stochastic nature of the diffusional motion, the scattered field is deterministic, allowing our deep-learning approach to directly infer particle size from the optical scattering properties of the particle. Repeated observations of the same particle are used to refine this inference. Consequently, it is possible to reach high accuracy in particle sizing using only a few observations.
To determine whether the accuracy achieved by the WAC-NET is better than that of the diffusion-based method also when using many time steps for the characterization, we study the scaling of the estimated error of the two approaches (defined as σ(y) = ⟨(y pred /y true − 1)⟩), where y represents either size or refractive index, with the number of time steps in Figure 2e,f. We find that the error of both methods decreases as N 1/ , where N is the number of time steps (dashed black lines), with the WAC-NET consistently producing results more than an order of magnitude better than those of the diffusion-based method. Consequently, the error of both methods can be parametrized as y , where σ 0 is a proportionality constant. In the case of diffusional motion, the determination of the diffusion constant in essence amounts to estimating the variance of the distribution of particle displacements. Assuming that N displacements are known exactly, the uncertainty in the estimated variance of the underlying distribution is given by σ 2 = 2/N. This provides a lower bound on the proportionality constant for diffusional motion that cannot be improved upon by changes to the experimental parameters. 28 In the case of the deep-learning approach, the prefactor is instead determined by the particle type and the quality of the image. Figure 2g,h plots the prefactor σ 0 as a function of the signalto-noise ratio (SNR) of the image and the ratio of the particle radius to the illuminating wavelength (r/λ). This allows us to compare the performance of the network to that of the diffusion-based approach for a wide range of particle sizes and noise levels. We find that the deep-learning-based particle characterization method is more than 1 order of magnitude more accurate than diffusion-based methods for a fixed number of observations in a broad range of particle sizes, extending down to r ≈ λ/3. Owing to the scaling of the errors of the two methods with the number of observations, the diffusion-based method will need more than 2 orders of magnitude more observations to reach the same accuracy in this region of particle sizes. Furthermore, the network shows a considerable increase in accuracy compared to that with a diffusion-based approach even in poor illumination conditions (down to SNR = 2). Also, note that the positions of the simulated particles were known exactly, whereas in any experimental setting, error in the particle localization inevitably induces additional uncertainties in the determination of the diffusion constant; therefore, the stated improvement in accuracy is a lower bound.
The accuracy of the deep-learning method enables distinguishing particle populations that have similar properties. We find that populations differing by only 12 nm in radius or by 0.036 refractive index units can be distinguished using 20 observations for the characterization (supplementary Figure  S1).
It is worth noting that, in principle, any physical characteristic of the particle that alters the optical scattering pattern, such as its morphology or internal refractive index distribution, can be characterized in this way by an appropriate choice of simulation method. As an example, we simulate the scattering patterns of stratified spheres and train the network to characterize the core radius, core refractive index, shell thickness, and shell refractive index based on the optical scattering patterns produced. The results in supplementary Figure S2 demonstrate the applicability of the proposed method also for this more complex scenario.
Neural Network Performance on Experimental Data. To verify that the increase in accuracy of the deep-learning characterization transfers to experimental data, we recorded holographic videos of 0.228 μm radius polystyrene spheres (NIST certified, standard deviation 6.8 nm) flowing through a microfluidic chip (see Methods for further details). In Figure  3a,b, the characterization of a representative particle is shown as a function of the number of time steps used for characterization. We find that the WAC-NET converges to estimates close to the expected values of size and refractive index using only a few time steps (orange symbols). In order to compare the accuracy of the network to conventional approaches, the particles were tracked and their hydrodynamic radii were estimated from their Brownian motion. The characterization based on conventional particle tracking shows considerably larger fluctuations (gray symbols). Similarly, WAC-NET produces predictions consistent with the expected values on the ensemble of particles using only five time steps (orange histograms in Figure 3c,d, expected values of size and refractive index are shown as solid black lines). On the other hand, the diffusion-based approach does not correctly characterize the sample using only five time steps (gray histograms).
When considering the scaling of the error of the predictions with the number of time steps, we find that the accuracy of the deep-learning approach, when applied to experimental data, appears to saturate around a standard deviation of 11.9 nm and 0.03 refractive index units for the size and refractive index, respectively (Figure 3e,f). This saturation partially reflects the distribution of sizes and refractive indices within the sample (the NIST-certified width of the size distribution of these particles is 6.8 nm, measured by disc centrifuge). The remaining variation might stem from spurious contaminants in the sample or false detections, increasing the tails of the size distribution. Indeed, fitting the central peak of the size distribution to a Gaussian function gives an estimated width of 8.2 nm, closely resembling the NIST-certified distribution width of 6.8 nm, also verified by transmission electron microscopy (see supplementary Figure S3). As the particles are imaged under flow, they could not be tracked for sufficiently long to determine a similar saturation point for the conventional particle tracking approach.

ACS Nano www.acsnano.org Article
Characterization of a Multicomponent Sample. As a test of the performance of the network in a more complex and realistic scenario, we analyze a multicomponent sample consisting of PS particles of two different sizes (modal radii = 0.228 and 0.15 μm, refractive index = 1.58) and silica particles (modal radius = 0.21 μm, refractive index = 1.45). This sample is challenging to analyze using standard methods as 0.23 μm polystyrene and 0.21 μm silica are similar in size and are therefore hard to distinguish by diffusion alone, especially when only a few observations are available. In addition, 0.15 μm PS and 0.21 μm silica induce similar phase contrasts, and their scattering patterns are consequently visually similar. This is highlighted in Figure 4a−c, where the phase and amplitude contrasts of characteristic particles within the three subpopulations are shown.
Despite these similarities, the deep-learning approach successfully distinguishes the three subpopulations and accurately determines their modal characteristics (Figure 4e). Specifically, the sizes and refractive indices are estimated by WAC-NET to be r = 0.15 ± 0.01 μm (standard deviation), n = 1.58 ± 0.06 (0.15 μm PS); r = 0.22 ± 0.02 μm, n = 1.44 ± 0.02 (0.21 μm silica); and r = 0.22 ± 0.01 μm, n = 1.58 ± 0.03 (0.23 μm PS). The slightly larger variation observed in the smallest subpopulation is consistent with the discussion above in connection with Figure 2g,h: in our setup, a radius of 0.15 μm corresponds to a ratio r/λ ≈ 0.24, which is within the region where the precision of the WAC-NET starts to decay. In order to understand the correlation between estimated size and refractive index in this population, we note that the scattering amplitude for particles in this size regime scales approximately with V V nn / to lowest order in Δn = n − n m , and where n and n m are the refractive index of the particle and the medium, respectively. 12,26 For particles with r ≲ λ/4, the WAC-NET can still predict the scattering amplitude via the product V·Δn (see supplementary Figure S4a), but the precision by which the network can separate size and refractive index from this product deteriorates somewhat in this size regime. As a consequence, for particles with r ≲ λ/4, the estimated sizes and refractive indices are correlated in such a way as to keep the product V·Δn constant (supplementary Figure S4b).
For comparison, the results obtained from particle tracking and analysis of the Brownian motion of the sample are shown in Figure 4f. Particle tracking does not provide sufficient accuracy in size determination to distinguish 0.15 μm PS and 0.21 μm silica from each other. Thanks to the difference in phase contrast, the 0.23 μm PS population can be distinguished, but the distribution of estimated size and refractive index becomes very broad.
In the context of characterizing biological samples, it is important to characterize samples and separate particle populations having very low refractive index contrasts. As an example, extracellular vesicles, one important class of biological particles, have refractive indices estimated to be in the range of 1.35−1.38. 29 In order to probe the accuracy of WAC-NET in the lower part of this range, we characterize silica particles having a refractive index of 1.42 dispersed in varying glycerol concentration, thereby tuning the refractive index contrast from 0.09 to 0.02, a range of refractive index contrasts relevant for biological particles. We find that WAC-NET accurately characterizes subwavelength particles down to a refractive index contrast of 0.03 refractive index units, demonstrating the applicability of WAC-NET for characterization of biological samples (supplementary Figure S5).
Characterization of Particles in an Unknown Environment. To test the performance of the network in situations where the properties of the solute are not known a priori, we also record holographic videos of 0.23 μm polystyrene particles dispersed in a 50% glycerol/water mixture. The presence of glycerol changes both the viscosity and the refractive index of The manufacturer-provided NISTcertified radius is 0.228 ± 0.0068 μm, and the bulk value for the polystyrene refractive index is between 1.55 and 1.59. The accuracy of the deep-learning approach surpasses the diffusionbased approach with all numbers of observations. (c,d) Distribution of r and n on an experimental sample of 256 particles using five observations. While the diffusion-based approach (gray histograms) fails to characterize either property, the deep-learning approach (orange histograms) provides estimates of the radius with ±16 nm standard deviation and of the refractive index with ±0.05 standard deviation. (e,f) Scaling of the standard deviations σ r and σ n of the measured r and n of the population as a function of the number of observations. The standard deviation of the deeplearning approach (orange symbols) reaches ±11.9 nm in radius and ±0.03 in refractive index units using 60 observations. This performance is much better than that of the standard diffusionbased approach (gray symbols): σ r is comparable to the size variation stated by the manufacturer (±6.8 nm NIST-certified standard deviation, gray dashed line in e), and σ n is within the range of variability of the polystyrene refractive index.

ACS Nano
www.acsnano.org Article the solution, and thus this solute is qualitatively different from the solute used for network calibration. In the case of holographic imaging, a change in the refractive index of the environment primarily changes the scattering amplitude but leaves the spatial structure of the scattering pattern approximately unchanged (Figure 4d, cf. Figure 4c). Thus, we expect the network to correctly recognize the particles having the same sizes in both environments but with a different relative refractive index. Indeed, we found that the particle populations measured in the two solutions overlap in size but are shifted in refractive index (Figure 4e). In order to relate the shift in estimated refractive index of PS particles to the difference between the refractive index of water (n w = 1.33) and that of the glycerol/water mixture (n g/w ≈ 1.40), we note that the shift in scattering amplitude is proportional to n g/w − n w , valid to lowest order in n g/w − n w . Thus, the shift in scattering amplitude is expected to produce a shift in the estimated refractive index of the particles that correspond to the difference in refractive index between the solutions. Indeed, we find that this is the case, demonstrating the capacity of the deep-learning-based method to characterize particles without prior knowledge of the physicochemical properties of their surrounding environment.
Monitoring Clusters of Nanoparticles. In the experiments presented until now, we have evaluated the network performance on spherical particles whose properties do not change over time. In order to test it in a more dynamic scenario beyond the state of the art of what can be done with standard techniques, we image a sample consisting of a solution of 31 nm radius polystyrene nanoparticles freely diffusing in a microfluidic chip and forming dynamic clusters (nanoparticle clustering is induced by adding a droplet (∼20 μL) of saturated NaCl solution to the inlet of the microfluidic chip, which alters the Debye screening length and destabilizes the suspension). These clusters are in general nonspherical, and their size and refractive index fluctuate over time. Since clustering of nanoparticles affects their reactivity, understanding how such clusters form, how they interact, and how their size and morphology evolve in time is important to understand and predict their behavior and performance. 30 For instance, determining whether the formation of clusters is irreversible (aggregation) or reversible (agglomeration) is challenging, presently requiring combining a multitude of measurement techniques. 31 In fact, with the exception of liquid-cell transmission electron microscopy, which suffers from complicated sample preparation and low throughput, existing methods for characterizing nanoparticle clusters only provide snapshots of the aggregation/agglomeration process and thus cannot be used to temporally monitor the clusters. 32 Note that since the clusters are generally nonspherical and inhomogeneous, the radius and refractive index of the clusters are not well-defined concepts. Nonetheless, they will produce a scattered field that is similar to the field generated by a homogeneous sphere of radius r eff and refractive index n eff . In the following, the terms radius and refractive index of the clusters will refer to r eff and n eff .
A signature of cluster formation is the fractal nature of the resulting structure. A fractal cluster can be characterized by its fractal dimension, D f , which dictates the scaling of the size with the number of monomers, N, in the cluster (see insets in ACS Nano www.acsnano.org Article Figure 5a): (r/r 0 ) = k·N f 1/D , where r is the radius of the cluster, r 0 is the radius of the monomers, and k is a proportionality constant (k ≈ 1 for clusters with fractal dimensions D f > 2, 33 so in the following, we set k = 1). For close-packed spheres, the radius of the cluster scales as (r/r 0 ) ∼ N 1/3 . The effective refractive index n of a cluster formed by identical spheres with radius r 0 and refractive index n 0 follows the Maxwell-Garnett relation, L n r r N L n r r L n ( ) ( / ) ( ) ( / ) ( )  34 The latter expression can also be used to directly estimate the fractal dimension of a cluster, given that the radii and refractive indices of both the monomers and the cluster as a whole are known. For diffusion-limited clustering of monodisperse monomers, the fractal dimension is expected to be around D f = 2.5, whereas the value is lowered to D f = 1.75 if cluster−cluster clustering is taken into account. 35 In line with this, previous measurements of the fractal dimension of colloidal clusters have yielded values in the range of D f ∼ 1.6−2.3, with the exact value depending on multiple factors, such as the concentration of salt in the solution and the concentration of monomers. 35−39 Furthermore, the fractal dimension has been predicted to decrease with time, with early clusters having D f ∼ 2.5, decreasing to D f ∼ 1.8 as a function of time. 40 Based on these results, we expect the fractal dimension in our experiment to be close to D f = 2.5. Consistent with this, we find that the fractal dimension of the structures are D f = 2.35 ± 0.1 (Figure 5a). This demonstrates that the deep-learningbased approach is sufficiently accurate even on nonspherical subwavelength particles to estimate their fractal dimension, and that the method does not require the shape of the particles to be known a priori.
The characterization of the clusters using the WAC-NET permits us also to investigate the temporal dynamics of the properties of individual nanoparticle clusters. To do so, we characterize the clusters using a sliding window of 20 time steps (at a frame rate of 30 fps), providing subsecond temporal resolution of size and refractive index of individual clusters. Fluctuations in these properties can, in general, be of two different physical origins: association/disassociation of monomers to the cluster will change the number of monomers in the cluster, without changing the fractal dimension, whereas internal rearrangements of monomers will cause a change to the fractal dimension, at a fixed monomer number. Thus, by monitoring the number of monomers N and fractal dimension D f , the physical origin of fluctuations in size and refractive index can be determined.

CONCLUSIONS
We have demonstrated the potential for deep-learningenhanced optical characterization of subwavelength particles. Going beyond standard approaches, we have shown that the scattering patterns of individual particles recorded in off-axis holographic imaging contain sufficient information to extract size and refractive index of dielectric particles with a radius down to r = 0.15 μm.
With standard approaches, individual characterization of dispersed particles in the subwavelength regime requires the ACS Nano www.acsnano.org Article analysis of the Brownian motion of the particles through particle tracking. By instead using the optical scattering pattern to deduce size and refractive index, our approach offers several advantages compared to traditional approaches. Specifically, our method does not require prior knowledge of the physical properties of the surrounding medium, such as its viscosity or refractive index. This is particularly important in industrial and biological systems, where particles need to be characterized in complex environments. Furthermore, our method provides temporally resolved measurements of size and refractive index of the particles, potentially enabling direct monitoring of interaction kinetics of particles in their native environment.
Taking the approach presented in this article, we achieve an estimated error in particle sizing of <10% using as few as five time steps, orders of magnitude faster than diffusion-based methods. In addition to particle size, our method also estimates the particle refractive index with comparable accuracy, serving as a proxy for particle composition. The refractive index and particle size are estimated from the optical scattering profile, with no reference to particle motion, thus enabling accurate characterization in arbitrary environments.
It is also worth noting that the approach we propose requires only a few observations of each particle to accurately characterize both size and refractive index. Thus, deeplearning-based characterization enables temporally resolving dynamical changes in size and refractive index of individual particles on subsecond time scales. We demonstrate this capability by monitoring the aggregation kinetics of a sample consisting of polystyrene nanoparticles in a high-salt environment. Despite the unknown geometrical shape of the scatterers, our method can resolve changes in the number of monomers as well as the resulting changes in aggregate size and refractive index. This characterization is sufficiently accurate to provide a reliable time-resolved estimate of the fractal dimension of individual aggregates, which, to our knowledge, has not previously been reported.
Taken together, our results show that deep-learningenhanced analysis of holographic scattering patterns allow improved accuracy in particle size and refractive index determination by more than an order of magnitude compared to traditional methods. The characterization is performed without assumptions on the physical properties of the environment and shape of the particle and can be performed with subsecond temporal resolution. In contrast to traditional techniques, this method can temporally resolve size and composition of individual subwavelength particles in their native environment. We anticipate that the drastic improvement in single-particle characterization offered by this technique will find widespread application in any area where subwavelength particles play an important role, ranging from industrial processes to drug discovery and pharmaceuticals.
Off-Axis Digital Holographic Microscope. A sketch of the setup employed in this study is shown in Figure 1f. A 633 nm HeNe laser (Thorlabs) beam is split into two light beams, one passing through the sample (collected by an Olympus 40× 1.3 NA oil objective) and one used as a reference. The two beams are recombined at a slight offset angle, and the resulting interference pattern is recorded by a CCD camera (AlliedVision, ProSilica GX1920). This interference pattern carries information about the optical field at the camera plane, as described below. Video files of particle samples are recorded at 30 frames per second with typical exposure times in the range of 2 to 4 ms.
Image Analysis. The interference patterns, or holograms, are analyzed using the homemade software written using MATLAB (Mathworks Inc.) to extract the amplitude and phase maps using standard methods. 12 In brief, due to the off-axis configuration of our setup, the Fourier transform of the interference pattern contains two off-center peaks, which describe the object field multiplied by a plane wave (exp(±ik ⃗ p ·x⃗ ), where k ⃗ p represents the projection of the wavevector of the reference beam onto the imaging plane on the camera), in addition to the central peak which corresponds to the noninterferometric intensities. In order to isolate the object field, we numerically shift one of the off-center peaks to the center of the Fourier spectrum and apply a low-pass filter. The magnitude and phase of the resulting field correspond to the amplitude and phase of the optical field recorded by the camera. The obtained field is slightly distorted due to optical aberration in the beamline. This is corrected by fitting the phase of the field to a fourth-order polynomial, which is subsequently subtracted from the phase to obtain a close-toaberration-free image of the real and imaginary part of the optical field.
Particle Localization and Tracking. In-plane subpixel localization of detected local extrema in the field amplitude is performed using the radial center method. 41 Particles are subsequently distinguished from background noise by the degree of radial symmetry and the spatial extent of the local extremum. The zposition of the particle is defined as the plane where the standard deviation of the Fourier transform of the field is minimized. 12 The scattering pattern of the particle is stored at this plane. Following a complete tracking of a frame, each observation is associated with a distance metric to particles in previous frames. Specifically, the observations are joined into traces by minimizing the sum of this metric using the Hungarian algorithm. 42 In our setup, the track lengths are typically around 80−100 frames, and all tracks shorter than 20 frames are discarded in the analysis.
Particle Characterization Network Architecture and Training. The problem of characterizing subwavelength particles by their scattering is fundamentally that of noise reduction by averaging multiple observations of the particle and subsequently measuring properties of its scattering pattern, such as the induced phase shift and the radial profile. However, due to small differences in the particle's subpixel position and spatially variable aberrations, observations typically cannot be directly averaged. This motivated the design of WAC-NET (Figure 1g), which consists of two main modules: a representation transforming network and a regression network. The representation transforming network is a convolutional neural network followed by a single fully connected layer. It transforms 64 × 64 × 2 images centered around a particle, where the two feature channels represent the real and the imaginary part of the complex field, into a one-dimensional vector representation with a length of 128. From this vector representation, the model also computes a single scalar that represents the confidence in the correctness of the vector representation using two fully connected layers. The regression network, in turn, receives a stack of vector representations and confidence values, representing all observations of a single particle.
The confidence values are rescaled using the softmax function. The vector representation stack is averaged along the first axis using the rescaled confidence values as weights. This results in a single vector with a length of 128. Three fully connected layers then transform this vector into two scalars representing the radius and the refractive index of the particle. Both modules are trained simultaneously using synthetic data simulated with Mie theory; the point spread function is calibrated to match the microscope used to collect the data. Standard ACS Nano www.acsnano.org Article L1 error is used as loss function, and the Adam optimizer 43 is used with a learning rate of 0.001. Simulation of Training Data. The scattered field from dielectric particles is simulated using the MATLAB package MatScat. 44 The fields are numerically propagated using the angular spectrum method through a lens with NA identical to that of the experimental system (NA = 1.3) and further propagated to the Fourier plane of the lens. The field at this plane corresponds to the Fourier transform of the field at the camera plane. Next, the optical system is modeled via the pupil function P(k x , k y ), which relates the simulated fields F sim (k y , k x ) to the experimentally obtained fields F(k x , k y ), as F(k ⃗ ) = P(k ⃗ )F sim (k ⃗ ). This pupil function is estimated based on experimental images of particles of known size and refractive index (polystyrene spheres with radius 0.23 μm and refractive index of 1.58). The pupil function has support only inside a finite radius, i.e., P(k ⃗ ) = 0 for k x 2 + k y 2 > k P 2 , where k P is set by the numerical aperture of the system as k P = π/λ·NA. In our system, k P = 6.45 μm −1 . This finite support allows considerable dimensionality reduction, as only Fourier components within this radius contain relevant information. This leaves, in our case, 177 pixels in Fourier space.
Transmission Electron Microscopy Analysis. The transmission electron microscopy (TEM) measurements were performed using FEI Tecnai T20 operated at 200 kV. The particle solutions were dropcasted on copper TEM grids with carbon membranes (TED PELLA). A small droplet was placed on a TEM grid, and after approximately 15 min, the excess liquid was removed by slightly touching the droplet with a tissue paper. Thereafter, the grids were air-dried for 15−20 min, after which they were placed in a storage container until used. The TEM image analysis was performed in Matlab, where the image processing was based on the circular Hough transform combined with a global intensity threshold.