In the field of image processing/computer vision, everything remains an open research area!

But why is that the case? Do you think that after decades of research, we’d naturally say, “Here’s...”“The issue has been resolved—let’s focus on something else.” To some extent, we can say this—but only for narrow and simple use cases (such as placing a red spoon on an empty whiteboard), not for general computer vision tasks (such as finding a red spoon in every possible scenario, like searching through a large box filled with colorful toys).

Before we delve into what I believe are the main reasons why computer vision is such a challenging field, I first need to explain how machines “see” images. When humans look at an image, we perceive objects, people, or landscapes. But when machines “examine” an image, all they see are numbers that represent individual pixels.

An example can help illustrate this point. Suppose you have a grayscale image. Each pixel is represented by a number typically ranging from 0 to 255 (here I’m abstracting away details such as color spaces and compression). A value of 0 represents black (no color at all), while 255 represents white (full intensity). Any value between 0 and 255 corresponds to a shade of gray, as shown in the figure below.

Therefore, any machine that needs to acquire image content must process these numbers in some way.This is precisely what image/video processing and computer vision are all about—Handle numbers!

Next, we will explain from four aspects the main reasons why solving this problem is extremely difficult.

1.Large data volume

2.Inherent information loss

3.Accompanied by noise

4.It's difficult to understand the meaning of images.

Large data volume

As I mentioned earlier, when it comes to images, all computers see numbers...Lots of numbers!Many numbers imply a large volume of data that needs to be processed in order to be understood.

Let’s take an example to illustrate just how large the amount of data in an image can be. If you have a grayscale (black-and-white) image with a resolution of 1920 x 1080, that means your image is described by 2 million numbers (1920 × 1080 = 2,073,600 pixels). Now, if you switch to a color image, you’ll need three times as many numbers, because typically when you represent a color pixel, you specify not only its red value but also its blue and green components. Then, if you’re trying to analyze images from a video or camera stream—for instance, at a frame rate of 30 frames per second (which is now the standard frame rate)—you’ll suddenly be dealing with:1.8100 million numbers per second（3 * 2,073,600 × 30~ = 180 million pixels/second).This is a huge amount of data that needs to be processed!Even with today’s powerful processors and relatively large memory capacities, the machine finds it difficult to accomplish anything meaningful—processing 180 million numbers per second.

Information loss during the digitization process is another major factor contributing to the challenges in computer vision. The essence of image processing lies in projecting information from the 3D world— or, if we’re dealing with data from a video stream, the 4D world— onto a 2D plane (i.e., a planar image). This means that during this process...A lot of information will be lost..

Our brains can be very...OutstandingInferring what data is missing is an extremely difficult challenge for computers. The figure below shows a messy room.

We can easily see that the green fitness ball is larger and farther away than the black frying pan on the table. But if...The black pan has more pixels than the green sphere.How should machines infer? This is not an easy task. Of course, we can try to simulate the way we see with two eyes by simultaneously capturing two photographs and extracting 3D information from them—this approach is known as...Stereoscopic visionHowever, stitching images together is also not a trivial task, as it remains an open research area.

Noise often accompanies the digitalization process.For example, no camera can capture a perfect, noise-free image of reality—especially when we use the camera on our smartphones. To try to capture our beautiful world, these cameras adjust parameters such as exposure levels and color saturation. Meanwhile, during the image-capture process, “…” may occur.Lens flareIn the case of this phenomenon, we can easily determine what scene lies behind the halo, whereas for a computer, it’s actually quite challenging.

Although many algorithms for removing halos already exist, the field of halo removal algorithms itself remains open.

Moreover, during image compression, the image’s pixel count is reduced or its format is transformed. While such images are easily recognizable by humans, a computer—unless explicitly informed of the compression and transformation operations—will mistakenly identify the compressed image as the original, leading to errors.

Finally, and most importantly, is the content of the image.UnderstandFor machines, this is undoubtedly the most challenging task in the computer vision domain.When we look at images, we rely on accumulated learning and memory.（Known as prior knowledge)Let's analyze it..

For example, we know that we can sit on an exercise ball, and that frying pans are typically used in the kitchen—because these are things we’ve encountered before. Now, if something looks like a frying pan in the sky, it’s probably not a frying pan (unless perhaps Red Wolf has thrown the frying pan he used to beat Grey Wolf up into the heavens). Therefore, we can take a closer look to figure out what the object might actually be—for instance, a frisbee! Or, if someone is kicking around a green ball, it’s most likely a kid’s ball rather than an exercise ball.

But machines don't possess this kind of knowledge. They don't understand our world, the inherent complexity within it, or the myriad tools, goods, and devices that we've created over thousands of years of evolution.Perhaps one day, machines will be able to access Wikipedia and learn information about objects from there.But currently, we are far from such a situation.

Some people would argue that we’ll never reach a stage where machines can fully understand our reality—because consciousness will always remain beyond their grasp.

But who can really say for sure what the future holds?