214 — Neural Network Quine

Quines are computer programs that output their own source code.

One example in Python:

s = "s = %r\nprint(s%%s)"
print(s%s)

These are generally used as coding “puzzles” or as interesting exercises in software engineering (try it — it’s not as simple of a challenge as it seems in many languages!) but Chang & Lipson make a case for using quines as a metric for neural network self-replication.

After all, a neural network that can design a slightly better neural network is a hugely powerful tool, and the first step on the path to that goal is a neural network that can generally another network that is exactly the same amount of good.

And, not to wax too philosophical, but what is biological evolution if not a mechanism for optimizing just that?

The first challenge is figuring out how to “output” a neural network. What does a network look like? Certainly one could output code to construct a net, but there must be a more fundamental representation that is both easier to generate and more generalizable. (If you output Keras Python code, it’s not particularly useful to a Torch user.)

First, the authors propose a Vanilla Quine, which is a neural network that outputs its own weights. They then demonstrate an Auxiliary Quine, which performs the same task as the Vanilla Quine, but also performs a task, such as, in this example, MNIST classification. The auxiliary loss of this network is the self-replication loss-function (sum-squared difference between the actual and expected weights) combined with the auxiliary task loss function, each scaled by some factor to be of similar magnitude.

This is a fascinating project because it takes the concept of a quine out of the context of a single language and makes it a generalized question: Can a concept be replicated between implementations? And if so, what does that look like?