Researchers have developed a deep-learning algorithm that can de-noise images to reveal otherwise invisible details.
To enter the world of the fantastically small, the main currency is either a ray of light or electrons. Strong beams, which yield clearer images, are damaging to specimens. On the other hand, weak beams can give noisy, low-resolution images.
“Images taken with low-powered beams can be noisy, which can hide interesting and valuable visual details of biological specimens,” says Shuiwang Ji, associate professor in the department of computer science and engineering at Texas A&M University. “To solve this problem, we use a pure computational approach to create higher-resolution images, and we have shown in this study that we can improve the resolution up to an extent very similar to what you might obtain using a high beam.”
Ji adds that unlike other de-noising algorithms that can only use information coming from a small patch of pixels within a low-resolution image, their smart algorithm can identify pixel patterns that may be spread across the entire noisy image, increasing its efficacy as a de-noising tool.
Instead of solely relying on microscope hardware to improve the images’ resolution, a technique known as augmented microscopy uses a combination of software and hardware to enhance the quality of images. The image processing method holds promise to not just cut down costs but also automatize medical image analysis and reveal details that the eye can sometimes miss.
Currently, a type of software based on a machine-learning algorithm called deep learning has been shown to be effective at removing the blurriness or noise in images. These algorithms can be visualized as consisting of many interconnected layers or processing steps that take in a low-resolution input image and generate a high-resolution output image.
In conventional deep-learning-based image processing techniques, the number and network between layers decide how many pixels in the input image contribute to the value of a single pixel in the output image. This value is immutable after the deep-learning algorithm has been trained and is ready to de-noise new images. However, Ji says fixing the number for the input pixels, technically called the receptive field, limits the performance of the algorithm.
“Imagine a piece of specimen having a repeating motif, like a honeycomb pattern. Most deep-learning algorithms only use local information to fill in the gaps in the image created by the noise,” Ji says. “But this is inefficient because the algorithm is, in essence, blind to the repeating pattern within the image since the receptive field is fixed. Instead, deep-learning algorithms need to have adaptive receptive fields that can capture the information in the overall image structure.”
To overcome this hurdle, Ji and his students developed another deep-learning algorithm that can dynamically change the size of the receptive field. In other words, unlike earlier algorithms that can only aggregate information from a small number of pixels, their new algorithm, called global voxel transformer networks (GVTNets), can pool information from a larger area of the image if required.
When they analyzed their algorithm’s performance against other deep-learning software, the researchers found that GVTNets required less training data and could de-noise images better than other deep-learning algorithms. Furthermore, the high-resolution images obtained were comparable to those obtained using a high-energy light beam.
The researchers note that their new algorithm can easily be adapted to other applications in addition to de-noising, such as label-free fluorescence imaging and 3D to 2D conversions for computer graphics.
“Our research contributes to the emerging area of a smart microscopy, where artificial intelligence is seamlessly integrated into the microscope,” Ji says. “Deep-learning algorithms such as ours will allow us to potentially transcend the physical limit posed by light that was not possible before. This can be extremely valuable for a myriad of applications, including clinical ones, like estimating the stage of cancer progression and distinguishing between cell types for disease prognosis.”
The paper appears in the journal Nature Machine Intelligence. Funding came from the National Science Foundation, the National Institutes of Health, and the Defense Advanced Research Projects Agency.
Source: Texas A&M University
