19 — Using Human Brain Activity to Guide Machine Learning

Using data from fMRI signals collected while subjects viewed a large set of images, the authors demonstrate an improvement on computer-vision neural networks by injecting the brain data as additional information in the loss function.

The base model — an SVM classifier — was designed to determine whether an image contained a foreground of a certain type — e.g. “human”, “building”, etc — or not. The loss function for this network was a common hinge-loss:

where $z$ is the correctness of a prediction and $z=y\cdot f(x)$, $f(x) \in \mathbb{R}$ is the prediction, and $y\in\mathbb{N}$ is the truth.

Then, the same network was trained using an amended loss function, an “activity weighted loss” (AWL), where images that were easily recognized by humans (that is, there were robust brain-area activations in the appropriate regions of the brain) are punished more significantly if miscategorized. These correlations are based on existing literature (for instance, the fusiform face area, or FFA, reacts strongly to images of faces, but not to images of buildings). The new loss function includes this information:

where $M(x,z)$ is a function of the activity-weight from the fMRI data.

The authors demonstrate that the neural network that uses the AWL loss function easily outperforms the “vanilla” SVM, suggesting that ease-of-recognition data can play an important role in machine-learning applications.