Gaze, Knowledge, and Threat
The act of looking is passive, but the concept of the gaze implies something active. If the gaze wasn't, then its immobility could hardly pose a threat. The gaze isn't presented as an annoying rock that some people occasionally trip over. It has a movement.
Looking seems to become a gaze when the person turns the information they’re ingesting into something they can exploit. According to the psychologist Alison Gopnik, the transition from exploration to goal directed exploitation is part of what marks adulthood though. A child learns about the world, because it's all brand new. An adult is expected to have some level of comprehension of it in order to function and utilize it's surroundings for survival. Because of this necessary transition to exploitation, a gaze seems to be an inevitability.
When we look at the world, we are processing information so that we can understand life. By doing this, we gain the ability to have knowledge, which is something that helps us navigate existence. But, since knowledge is power, or, more specifically, applied knowledge, the worry or fear about the gaze seems to revolve around the fact that the information gained from looking allows for an exploitation that is destructive to people. And since knowledge is perhaps the only thing that someone truly has the ability to possess, we have a fear about what someone might do with their knowledge. The concept of the gaze, as a result, is our anxiety that revolves around the fact that someone can use their looking ability to exploit the world around them. More specifically, it's the fear that the person will use their looking in a way that treats people as a means to an end.
Part of the increase in anxiety around the concept of the gaze is only escalated by using a Marxist analysis of the world. This is because an individual can only fall into one of two categories: they're either the oppressor or the oppressed. What someone does with their knowledge, then, can't have any other context. Looking, as a result, falls into a binary moral framework. And since there are no thought police, and we can't take knowledge away from people, the only solution is to try and control how people see.