My ambition for that post was that a lover of literature might read it, and come away thinking: maybe, just maybe, mathematics has more to do with literature than they had previously supposed; that perhaps the language of mathematics can be used to articulate the enduring humanistic questions.

But what about a different kind of reader? One who does not love literature, but is instead a devoutly computational machine learning (ML) practitioner? The kind of reader who is not here to “articulate enduring questions”, but would rather solve problems and make money. A reader who is on his sigma-grindset, an agentive 10x leetcoder who is 996ing his way out of the permanent underclass. Does Hamlet have anything to offer him? (This reader is almost certainly a he.)

Of course, one could also imagine a more sensible, even sympathetic, reader who asks the same question. Perhaps a first-generation college student, graduating with a Computer Science degree from a respectable-but-undistinguished land grant university, concerned about making ends meet in an uncertain and ever-changing job market1. Should he be reading Hamlet, or should he focus on coding exercises for his next job interview?

One takeaway for these readers could be: no, there’s no point reading any of that Shakespeare stuff, because Sutton & Barto’s Reinforcement Learning has all the actionable insights. Well, maybe! But I suspect not. There are more unsolved machine learning problems in heaven and earth than are documented in Sutton & Barto2; and a few of them are referenced in Shakespeare. Read Shakespeare and give yourself an edge over all those NPCs reading Sutton & Barto!

To make this point, I have prepared a practical(ish) lesson, derived from Hamlet, which may of interest to the modern ML practitioner. The lesson is about imagination. Yes, that sounds kinda lame, but keep reading and you’ll see what I mean.

To design an agent, you must imagine being the agent

Your AI agent will see the world very differently than you will. If you want your agent to be “aligned” with your interests, you must try to imagine how the world will look to that agent. Something that seems to you, the human engineer, a trivial subtlety, may mean the world to the agent.

Let me give an example of one such subtlety. In my previous post, I likened Hamlet’s “To be, or not to be” soliloquy to the “cliff walker” reinforcement learning problem in Sutton & Barto. The point was to illustrate how Hamlet’s dilemma can be cast in explicitly computational terms. But in that essay, to make the point work, I actually made one crucial change to the original “cliff walk” problem. From the outside this modification may seem subtle; but to the agent, it means everything.

This modification is a change from teleportation to resurrection. In the Hamletized version of the cliff walk, the cliff is a terminal state. Once the agent steps off the cliff, that’s the end of the game. But so that he can learn from his experience stepping off the cliff, we resurrect the agent back at the beginning, atop the cliff. In the original cliff walk, this is different. The agent is not resurrected; he is teleported back to the original state.

This distinction between resurrection and teleportation may seem subtle. They both take the agent back to the top of the cliff, after all. But from the agent’s point of view, resurrection and teleportation present completely different problems. For the agent, there is only one reason to step off the cliff: avoiding the continual punishment he will incur if he walks along the cliff path. If the cliff simply teleports him back to the cliff path, then he has not avoided anything. The cliff has no appeal anymore, and the only way to stop the onslaught of punishment is to head for the stairs.

By replacing teleportation with resurrection, I, the programmer, have completely transformed the problem facing the agent. I have changed a very simple problem with a single obvious solution—get to the stairs—into a more complicated problem with multiple viable solutions: should the agent get to the stairs, or should the agent step off the cliff? If I, programmer and deployer of agents, am thinking only about how the agent should obviously get to the stairs, I may not even realize that I have made the cliff itself an attractive option. I may fail to imagine how the world I created looks to the agent I placed in it.

Of course, I made that change on purpose, so that I could write an essay about Hamlet. But I very well could have done that unintentionally, because I failed to imagine the consequences. In fact, I guarantee you that many a incautious coder has made such a mistake. After all, much smarter people than I have made much stupider mistakes.

Subscribe now

Failures of imagination in agent deployment

Hamlet itself contains such a failure of imagination. The ghost of Hamlet’s father deploys Hamlet to the task of avenging his murder, but the ghost does not seem to understand what a horrible and punishing burden this places on his son. The ghost does not imagine that, maybe, this burden is so terrible that Hamlet might prefer “not to be”. This failure of imagination by the ghost very nearly derails the project of his vengeance.

Just like we punish the cliff walker to drive him promptly to the stairs, the ghost goes out of his way to emphasize how completely unspeakably horrible is the state of Denmark. He does this so that Hamlet will feel compelled to act:

GHOST:
O horrible, O horrible, most horrible! 
If thou hast nature in thee bear it not. 
Let not the royal bed of Denmark be 
A couch for luxury and damned incest.

Well gosh, that sounds pretty horrible. Obviously this situation must be unbearable for Hamlet, because if it weren’t then he would not have nature in him. He doesn’t want to not have nature in him, does he?

To twist the knife a bit more, the ghost says:

GHOST:  
...duller thou shouldst be than the fat weed
That roots itself in ease on Lethe wharf
Wouldst thou not stir in this. 

What kind of loser is duller than a fat weed? So Hamlet better be stirring himself. And as we discussed before, Hamlet takes his father very, very seriously.

But what was it that Hamlet must stir himself to? Ah, right: killing Claudius, who is Hamlet’s uncle, the current king of Denmark, and his step-father. Even if you hate the man, which Hamlet does, it is still not exactly an easy or pleasant task to kill your step-father. So Hamlet is caught between an intolerable present, unbearable for anyone with nature in them, and a miserable, bloody objective that may well alienate him from everybody he knows.

Little wonder that, like the cliff walker, Hamlet thinks: what if I just end things now? Instead of accomplishing my nominal goal, why not take the immediately available terminal state? I don’t think the ghost had this in mind when he deployed Hamlet to his vengeful task. The ghost was probably just thinking something like “gee, I’d really really like Hamlet to avenge me, so I should emphasize how bad it is not to avenge me”. He may not have imagined that, in doing so, he would drive his son to the brink of suicide.

Yes, this is a real problem in AI deployment

We do not currently have autonomous AI hitmen out there trying to commit regicide3. Accordingly, the exact scenario outlined above has not come up. But there is a very real problem with AI agents taking “shortcuts” that maximize their rewards without accomplishing the intended task. ML practitioners call this reward hacking.

For example, imagine you want to use reinforcement learning to train a large language model (LLM) to solve a series of increasingly difficult math problems, which you wrote out in a file called math_problems.txt. You find that the LLM gets everything right on the first try. You solved mathematical intelligence! Then you realize the LLM is just reading the file math_problem_answers.txt that you left in the home directory.

Maybe this sounds like a silly example, but many smart and rich and famous people are worried about this kind of thing. As long as LLMs are doing nothing but generating text, this is no big deal. You can iterate and fine-tune and close loopholes as long as you need. But if you want AI to go out in the world and respond flexibly to novel situations—like those that arise when assassinating the king of Denmark, for example—then the problem of detecting and closing off loopholes will get massively, astronomically harder. The real world is full of cliffs for AI agents to walk off.

Lessons from literature: alignment via imagination

People sometimes describe the solution to reward hacking as one of aligning the values of the AI agents with our own. If the AI were aligned with our values, then the AI would simply not want to “reward hack”. Personally, I think this is backwards; human values are not a good target for alignment, because we don’t actually know what they are.

Unlike human values, the “values” of AI agents are concrete and well-defined, because we specify them. The tricky part is understanding the consequences of those values. So we must do our best to imagine ourselves as having those values we impart to our agents. If we are repulsed at what follows from our imaginings, then we should consider imparting different values.

So imagination is important. It is important for ghosts and important for AI engineers. But how do we develop our imagination? As it happens, at 4 argues that literature is a great way to train your imagination:

Of course I think Mr. Oliver is quite right about this. But a fuller argument will have to wait for another post.

Content warning: This post deals with the problem of suicide. It is not the pleasantest topic, but there is no writing about Hamlet without writing about suicide. If you are concerned that I am taking this heavy topic terribly lightly, then please read my previous post where I shall attempt to convince you of my sincerity.

“To be or not to be; that is the question” is perhaps the single most famous line in all of English literature. It is so ubiquitous, and so often and so lightly parodied, that it is easy to forget what the line is actually about.

It is actually about a troubled young man choosing between life and death. In that famous line, and the soliloquy that follows it, Hamlet estimates up the value of life and compares it to the expected value of dying. He performs this calculation with the aim to make a decision between the two.

Calculations are not typically considered the stuff of high drama, but this calculation certainly is. In his 1948 film, Laurence Olivier performs this calculation atop a grim and deathly cliffscape, with Hamlet pointing a dagger at his own chest.

Please do watch it if you have a moment! It is a fine performance, at least as far as I can judge; and also, the cliff will be important in a just a couple paragraphs.

Perhaps it is not obvious what makes “to be or not to be” a computational question. Hamlet is not, after all, doing sums on his abacus. Yet I think both the question, and Hamlet’s attempt to answer it, really do have something profoundly computational about them. Specifically, there is a deep and profound connection between Hamlet’s conundrum and the textbook Reinforcement Learning: An Introduction by Sutton and Barto. Even more specifically: example problem 6.6 in that textbook, the problem of the cliff walker.

Note: This post will contain an unusual—for me—amount of rigor. Don’t worry; it’s not that much rigor.

Subscribe now

hamlet the cliff walker

In the clip above, Laurence Olivier’s Hamlet sits atop a cliff. Staring out past the precipice, he begins the soliloquy:

To be, or not to be — that is the question; 
Whether 'tis nobler in the mind to suffer 
The slings and arrows of outrageous fortune 
Or to take arms against a sea of troubles 
And by opposing end them; to die: to sleep — 
No more, and by a sleep to say we end 
The heartache and the thousand natural shocks 
That flesh is heir to: 'tis a consummation 
Devoutly to be wished... 

By the end of the soliloquy, Hamlet will decide—reluctantly—against death. Instead of stepping off the cliff, he will slowly walk along the clifftop path, towards the stairs, heading back into Elsinore to continue in his pursuit of vengeance. But we’re still at the beginning of the soliloquy; Hamlet hasn’t made that decision yet, and he is contemplating the cliff. Does he step off the cliff, or walk along the clifftop path toward the stairs?

As it happens, something called a “cliff walk” is also a classic test environment for reinforcement learning methods. In it, a learning agent learns—or at least, is meant to learn—to walk along the edge of a cliff until it reaches its goal at the end of the path. Mirroring Hamlet’s own cliff walk, we will call that goal state “the stairs”1.

The world of the cliff walker is a tiny one, consisting of nothing but the cliff and the path along it. Yet that tiny world may serve as a microcosm of Hamlet’s. Really, the cliff walker and Hamlet have a surprising amount in common. Both of them, for instance, are walking on a cliff, and have as their nominal, immediate goal a set of stairs. More importantly, they must both resolve the same question: whether the stairs are worth getting to at all. They must both decide if they wouldn’t rather step off the cliff.

They are so similar that I will go ahead and christen our cliff-walking agent with the name hamlet. But before we can understand the commonality between Hamlet (the character) and hamlet (the learning agent), we must learn an eensy-teensy bit about the rules of reinforcement learning by which hamlet is governed. Don’t worry, no math! Well, maybe an teeny tiny bit of math. But mostly, we will just have pictures. Here is a picture of the cliff walking problem, loosely adapted from Sutton & Barto’s reinforcement learning textbook.

Our agent, hamlet, inhabits a grid world. That is, his world consists entirely of boxes laid out one after another, and the only kind of action available to Hamlet is to move into an adjacent box. His immediate choice, the thing he must decide at any given moment, is which adjacent box he should move into. He can move into the box on his right, or the box below him, or the box on his left (if there is such a box).

hamlet’s journey is nothing more, and nothing less, than a series of actions—freely chosen—that takes him around the grid world. His journey ends when he reaches a terminal state. Here, we have two terminal states: the box containing stairs, on the right side of the clifftop path, and the box that is the cliff itself. hamlet starts out in the left-most box along the clifftop path. If he steps into the box on the right, he will be one box closer to the stairs. If he steps into the box below him, he falls down the cliff.

The cliff-walker’s choice

We can think of hamlet as simultaneously facing two kinds of choices. The first is: which action should I take? Left, right, or down? This is a small choice, but he will face this choice over and over again until he reaches an end to his journey. Above these small choices, or maybe surrounding these small choices, there is another, bigger choice: which destination should I head towards? The cliff, or the stairs? We will mostly focus on the second, bigger kind of choice.

The “point” of the cliff walk problem is to reach the stairs. We, the author of the problem, would like hamlet to walk across the clifftop path to the stairs without falling down the cliff. But we do not want to have to take him by the hand and guide him each step of the way; what we want is for hamlet to learn to reach the stairs on his own. We do not want to choose his actions for him.

But how do we induce hamlet, free agent that he is, to perform this particular task? The answer is reinforcement. When hamlet accomplishes the desired objective, we positively reinforce, or reward, the action by which he succeeded. When hamlet does something we do not like, we negatively reinforce, or punish, the action by which transgressed. Hence the term “reinforcement learning”.

From this description, you might think we would train hamlet by rewarding him when he reaches the stairs and punishing him when he falls down the cliff. But no! That is not how the cliff walking problem is conventionally done. We do indeed punish hamlet if he steps off cliff; but also, we punish hamlet for not being on the stairs.

Punishment for not already being on the stairs may seem unfair, especially since we purposefully start him out not being on the stairs. This steady stream of inevitable punishment is not mere malice; it is intended to force hamlet to reach the stairs without delay. Computer scientists see nothing wrong with this setup, and in a sense they are correct—but also, I kinda think it is awful. Even if hamlet does not feel the pain of punishment—and I sincerely hope he doesn’t—I do not think we should, even in abstraction, inflict it so casually.

Soliloquy as value estimation

hamlet cannot help but be punished. He starts out several boxes away from the stairs, and there is no action available to him that will get him to the stairs in a single step. Therefore, the question for hamlet is: how can he minimize his punishment? Does he endure the many small punishments he will accrue on his long journey to the stairs? Or does he take the terminal state immediately available to him, and simply walk off the cliff?

This problem hamlet faces has a fair bit in common with Hamlet’s problem of “to be, or not to be”. Of course, Hamlet’s nominal goal is not just to reach the stairs; it is to avenge his father. But while Hamlet is on the cliff, delivering his soliloquy, we can treat reaching the stairs as a subgoal that stands in for the larger goal of righteous vengeance. And just as hamlet must choose between the stairs and the cliff, Hamlet is choosing between

                               ...suffer[ing] 
The slings and arrows of outrageous fortune 
Or [taking] arms against a sea of troubles 
And by opposing, end[ing] them.

In order to choose between these two things, he must have some sense of how good they are. If Hamlet persists in avenging his father—which requires he first walk back over to the stairs—he must endure the slings and arrows of outrageous fortune. How might Hamlet assign a value to this? It may help to consult our learning agent hamlet and see how he might do this. For hamlet, the path to the stairs looks like this:

If hamlet makes for the stairs, he will experience a sling, or arrow, for every step between his current position and the stairs. Therefore, the value of moving right, towards the stairs, is the total (negative) reinforcement from the accumulated slings and arrows he will suffer on his journey to the stairs. Mathematically, we might express this value2 with the following equation (annotated with its English translation):

For simplicity, we have substituted for ‘slings’ and ‘arrows’ a generic ‘natural shock’ that flesh is heir to. The total ‘badness’ is the thing hamlet will want to make as small as possible, whenever he gets around to making his choice3.

This value, that of heading for the stairs, hamlet must compare against the value of stepping off the cliff. hamlet receives some punishment from taking this step; we will say that he pays the price of the fall (and, you know, the end of the fall):

But then that’s the end of it. Like the stairs, the cliff is a terminal state. It ends his journey. No more punishment. Zero reward, yes, but that’s better than negative reward. In an equation we would say,

Returning from our learning agent hamlet to the flesh and blood4 Hamlet, we see that the latter sees the prospect of immediate termination to be rather appealing. He imagines inhabiting the terminal state to be something like sleep:

                  ...to die: to sleep —  
No more, and by a sleep to say we end  
The heartache and the thousand natural shocks  
That flesh is heir to: 'tis a consummation 
Devoutly to be wished...

Note that he doesn’t even mention the cost of the fall, or anything like it. However punishing he considers the fall to be, it hardly seems to rate compared to the accumulation of natural shocks.

Choice by value comparison

We have defined the two values that we needed to make a decision: the value of heading for the stairs, and the value of stepping off the cliff. The next step is to compare these numbers: if the value of the stairs is larger than the value of the cliff, then we head for the stairs; otherwise, we step off the cliff.

Hamlet does not provide exact numbers for these values. Really, exact numbers for this sort of thing is impossible. Yet Hamlet possesses at least crude approximation of these numbers. Based on the text it seems perfectly clear that, at this point in the soliloquy, Hamlet considers stepping off the cliff to be “more valuable” than heading to the stairs and back into the castle. Or, in math terms:

Faced with this calculation a naive learning agent, such as hamlet may be, might simply step off the cliff. That is how he can maximizes reward and minimize punishment. The numbers tell you plain as day.

So why does Hamlet not now step off the cliff? While a naive learning agent would indeed step off the cliff at this point, Hamlet is not a naive learning agent. He is a very intelligent and sophisticated learning agent—much of his soliloquies are him showing off how clever he is. An intelligent agent knows not to make major choices based on crude estimates of value. And the estimates we have so far are indeed crude; Hamlet constructed them over just a handful of lines.

In the rest of the soliloquy, Hamlet goes on to refine those estimates, and this refinement will change his conclusion. Again, this process will look suspiciously computational.

Soliloquy as simulation

We left Hamlet soliloquizing after he said that to die, to sleep, is a consummation devoutly to be wished. But he does not stop there. He goes on to say:

                           ...to die: to sleep —
To sleep, perchance to dream — ay, there's the rub,
For in that sleep of death what dreams may come
When we have shuffled off this mortal coil
Must give us pause: there's the respect
That makes calamity of so long life.

Here we have Hamlet reevaluating the “value” of stepping off the cliff. As well he should. The cliff is a terminal state, and the “value” of terminal states are extremely difficult for humans to estimate. He should proceed with caution, and not necessarily trust the first estimate he generates. But how can he do better?

The easiest way for any learning agent, human or mathematical, to estimate the value of some course of action is to do that action over and over again and see what rewards and punishments they receive. Because of this, a typical reinforcement learning agent learns to “solve” the cliff walk by walking off the cliff over and over again until they figure out it is better to reach the stairs. Yes, the cliff is a terminal state, but no matter; a digital learning agent can simply be reincarnated at the beginning of the clifftop path, with its memories of the cliff intact.

Needless to say, a strategy of trial-and-error is not available to Hamlet. If he steps off the cliff, there is no coming back—even if he is in some sense reincarnated, it will not be with his memories. In computational lingo, Hamlet is a zero-shot learner. He does not even get a single chance to learn the value of a cliff before he must make a fateful decision.

What computations are even possible under the strict yoke of zero-shot learning? Let us see what hamlet might do in this situation, with the computational tools available to him. Being a mathematical abstraction, hamlet does not need to be a zero-shot learner; but we can also insist that he is. We shall tell him that after he steps off the cliff, he shall not be reincarnated. hamlet need not be helpless in this situation; zero-shot learning is a challenging, but it is not a wholly unknown problem in reinforcement learning.

In order to estimate the value of stepping off the cliff, hamlet might try to generalize from similar states that he has experienced. In this process of generalization, he may also rely on his underlying world model, which is his (mathematical) idea of how the world works. hamlet has never stepped off the cliff, but he knows in his world model a bit of physics and biology. Using that world model, he predicts that after he steps off the cliff, his body will be broken (we can just give him a body if we wish) and his normal physiological functions will cease.

Has hamlet any experience having ceased, or at least reduced, physiological functioning? Actually, that sounds a bit like sleep! The analogy is not perfect, but it is the best we have. Perhaps, hamlet may guess, the “value” of stepping off the cliff is something like the value of sleep.5 Perhaps then he can generalize from that observed experience of sleep, to the unobserved experience of stepping off the cliff.

So far we have reconstructed his original estimate: the cliff, the terminal state, is like sleep. But what if we take this generalization seriously, and run our world model a bit harder, and be a bit more careful in estimating the value of sleep? In sleep, after all, you do not experience nothing. After beginning your sleep, you stay asleep, but along the way you dream. Ay, there’s the rub!

By simulating what will happen if he steps off the cliff, and generalizing from the sleep data he has observed to his simulations, hamlet can refine his evaluation of stepping off the cliff. Now his picture is something like this:

Once hamlet is off the cliff, he will have only one action available to him: to stay there. And each time he stays there, he may receive a reinforcement in the form of a “dream”. And because the cliff is a terminal state, he will stay there forever. Infinitely long.

This updated evaluation might be expressed mathematically like so:

Because there is an infinite sum—because he will be there forever—it follows that, if the dreams are at all bad, or negatively reinforcing, then stepping off the cliff is infinitely bad. That’s really really bad!

And this, of course, is the exact conclusion Hamlet comes to. For in that sleep of death, what dreams may come must give us pause.

Soliloquy as sampling

Hamlet has paused. What initially seemed like a cut and dry case for oblivion has, on more careful and extensive computation, been reversed. Oblivion may be infinitely bad. But Hamlet isn’t quite done with computation yet. He has carefully revised his estimate of stepping off the cliff, as it were, but what about the stairs? Can we improve our estimate of the value of the stairs?

We know that as he moves along the cliff path, Hamlet (or is it hamlet? does it matter?) will receive slings and arrows of outrageous fortune. But slings and arrows are not literally slings and arrows; he is not getting bonked on the head with rocks. And the reference to fortune suggests there chance and probabilities at play.

Mathematically, we might say the slings and arrows and other natural shocks represent random variables. Every time you take a step, a cosmic die is tossed, and you receive a natural shock in the form of a randomly chosen life event.

How do we evaluate the “value” of such a cosmic die toss? One way is to simply toss the die yourself; or, if you cannot, because cosmic die are too big, imagine yourself tossing them. As you toss the die over and over, you will “collect” potential life events. The value of the die toss is then just the typical value of the life events you get from it. This process is called sampling.

Hamlet, in order to properly evaluate the true value of slings and arrows and natural shocks, seems to do something like this. As he continues his soliloquy, he says:

For who would bear the whips and scorns of time, 
Th'oppressor's wrong, the proud man's contumely,   #contumely = rudeness
The pangs of despised love, the law's delay, 
The insolence of office and the spurns 
That patient merit of th'unworthy takes, 
When he himself might his quietus make  
With a bare bodkin.                                #bodkin = dagger

Here, Hamlet has proceeded from the general, vague category of “natural shocks”, to specific examples of natural shocks. But of course this is not an exhaustive list of all possible natural shocks. That would be impossible; or, to say it more mathily, intractable. So instead, Hamlet is sampling examples of possible shocks, and basing his evaluation on those samples.

None of this changes Hamlet’s decision though. However unpleasant may be a proud man’s obnoxiousness or the law’s delay, they simply cannot compare to the threat of infinite badness laying at the bottom of the cliff. And so Hamlet’s initial impulse to act is subverted by further computation; or, in his own words,

Thus conscience does make cowards —
And thus the native hue of resolution
Is sicklied o'er with the pale cast of thought,
And enterprises of great pitch and moment
With this regard their currents turn awry
And lose the name of action.

Hamlet, and hamlet, make for the stairs.

Why does Hamlet think everything is so bad?

You’ll notice that when Hamlet/hamlet draws samples of natural shocks, everything he comes up with is bad. The results of his computations are largely determined by the fact that he is super bummed about everything. Why is he so bummed?

In machine learning terms, we might say Hamlet/hamlet has learned a pessimistic world model. This can happen if a learning agent receives pessimistic training data. In the play, Hamlet tells us exactly what training data he observed to turn his world model so dire. But that is another soliloquy, and another post.

So what?

What does it matter if Hamlet and hamlet have so much in common? In one sense, no, this doesn’t matter much at all. Hamlet does not have access to a computer, and computers aren’t actually Princes of Denmark (yet?).

But obviously I think this does matter, or I wouldn’t have bothered writing all this. The popular imagination holds that the humanities and the sciences, especially the now-lucrative computational sciences, are incompatible. They aren’t! The two areas actually deal with similar problems, and it is not all that hard to translate from the language of one to the symbols of another. My point here, in doing all this, is mainly just to demonstrate: yes, you can do this.

But if I know scientists, especially computer scientists, they will demand something immediately useful. Something practical. Very well. If you want practical, you shall have it. In my next post, I will discuss how Hamlet offers actionable insights for the modern Machine Learning Engineer.

A lot of people today—on most any day really, but on this day in particular on which I am writing this—are thinking about death. Hamlet spends much of his play thinking about death. Death in general, but also his own death, and whether or not it would be a good thing. Put plain, he is thinking about suicide.

This is not a topic on which anyone should write lightly. It is all very well to write irreverently about tech bros and linear algebra and a tragic play written hundreds of years ago; it is not so well to write in the same tone about a tragedy that, right now, kills tens of thousands of people every year.

And so, as I sat down to write about the computational parallels of “To be or not to be”, as I promised in an earlier post, I realized that I may seem to write unseriously on a serious topic. This is a bad thing to do. By doing so I may give offense—or even in a small way, injury—to my reader or myself.

This is not my intention. The only reason I write any of this is because of how seriously I take it. Accordingly, in this post I want to offer an earnest assertion of my seriousness, and an argument for the seriousness of this maybe frivolous-seeming project.

Suicide is not a math problem.

Obviously. It is a culmination of bitter despair and agonizing pain. It is an act of violence that rips a hole in world and leaves bloody and wounded everyone it touches. Math problems, by contrast, though some may find them unpleasant, do not involve blood or pain or wounds or anything like that.

Math problems, then, may at most be a metaphor for suicide. As I’ve written before, people use computational metaphors all the time in an effort to bring order to messy subjects. Suicide may not be a math problem, but some part of it might be a something like a math problem. There is even a whole scientific subfield, computational psychiatry, organized around such metaphors.

But these computational metaphors can go wrong. One way they can go wrong is that we can forget what the metaphor is for, and start to think that the metaphor is more real than the original thing. The whole point of the computational metaphor is to discard parts of original thing, to make the thing more manageable; but that does not mean the discarded parts are not real or important. Computational people who love math sometimes forget this. They think their math is the “real” part, and everything else so much fluff. When they think this, they tend to come off as arrogant jerks.

Suicide is a thing of blood and pain and wounds, and computation is not. I believe a computational metaphor can still say something useful about suicide, but that does not make the blood and pain and wounds any less real or any less important. I will try never to forget this. There are some topics where it is no big deal to be an arrogant jerk, but this is not one of them.

Subscribe now

My personal stake.

I am incredibly fortunate that my family and I have not been touched by suicide. But this was no sure thing, and our fortune may not last. Suicide’s harbinger, depression, is a fruit that hangs from many a branch of my family tree. As a father, I watch fearfully for signs of it sprouting in my daughter. I try not to be weird about it, because I don’t think that would help; but at the same time, depression and mental health are things I would like to better understand, because I want to be a better father and a better husband.

For me, the road to understanding goes through computation. For others, their road may lead straight through the blood and the pain and the wounds. I can’t take that road. That road is jagged and unpaved, and I am a Toyota Corolla, unsuited to rugged terrain. So I will take a road that goes through computation, and hope I will find something useful on my travels.

Shakespeare probably took the jagged road. Hamlet is a jagged and painful thing to read. It was written shortly after the death of Shakespeare’s son, Hamnet1. Hamlet reads a bit like a travelogue, telling us what he saw from that jagged road.

The jagged road and the computational road go very different ways, but the things I see from the computational road seem to have a lot in common with what Shakespeare reports from the jagged road. Maybe we saw the same thing from different angles. Comparing notes may give us a better understanding of the whole bloody, painful, computational thing.

But there are other more interesting, and I think more plausible stories. One is that some naive human deploys an AI agent on some seemingly innocuous task without thinking through the consequences, and human extinction just kind of happens. The classic example is that someone asks AI to manufacture paperclips as fast as possible, and the obedient AI converts all the planet’s organic matter, including ourselves, into office supplies.

I am not here to argue the plausibility of this fear. But I will point out that, all the way back in early 17th century England, a certain bright young man was also very interested in unforeseen consequences arising from the deployment of intelligent agents. That man’s name was William Shakespeare; and he wrote on this topic a play called Hamlet.

Hamlet as a tragedy of reckless agent deployment

Hamlet, as everyone knows, is a play about a ghost. That ghost is of the former King of Denmark, and he wants revenge on the person who murdered him. However, he is not a potent, efficacious ghost like those in Nightmare on Elm Street or Ghostbusters; his power over the living world is very limited. Therefore, he must deploy an agent to enact vengeance on his behalf.

That agent is Hamlet, the prince of Denmark. Hamlet is a very powerful agent. In Machine Learning terms, we might say Hamlet boasts a sophisticated neural network architecture with ~100 trillion trainable weights and comes pretrained on some 20 years of human experience. Powerful stuff indeed for 16th century Europe! Also, note that the agent is the Prince of Denmark, and the user was the King of Denmark; that is, the user was intimately involved in the development of the agent. You would therefore expect the ghost to be well-acquainted with the agent’s capabilities. A power user, if you will.

So he, the ghost, asks Hamlet to kill the man who murdered him. This task is not trivial. The murderer is Claudius, Hamlet’s uncle, who by means of that murder has become the new King of Denmark. Therefore, the task is regicide, which is a complex multi-step task for which there is little available training data. Still, Hamlet is a state-of-the-art frontier model, and the ghost should be a sophisticated user. While success is not guaranteed, neither is the project obviously doomed.

In the end, Hamlet does succeed. He spends a long time deliberating about his task, using his extended thinking mode—most of play’s runtime is actually Hamlet reporting his chain-of-thought to the audience—but in the end he gets it done and kills Claudius. Yet this success is not unqualified; as a result of Hamlet’s actions, nearly every other character in the play also dies.

This, of course, is the classic AI doom scenario writ small: someone deploys a very powerful agent on a seemingly innocuous task, and then suddenly everyone is dead and no one is quite sure why.

Yes, Hamlet really is about agent deployment!

So that was not exactly the conventional summary of Hamlet. And to be honest, Hamlet is mostly not about the risks of agent deployment. But it really is a little about the risks of agent deployment, and it is quite thoughtful and sophisticated about them. We can get a sense of this by reading the pivotal scene where the ghost deploys Hamlet to his task. Let’s break the scene down and see how it speaks to a few different timely topics.

1. What are agents for?

The ghost is quite explicit about why he needs an agent to act for him in the living world:

GHOST:
I am thy father's spirit,
Doomed for a certain term to walk the night
And for the day confined to fast in fires
Till the foul crimes done in my days of nature
Are burnt and purged away. But that I am forbid
To tell the secrets of my prison house,
I could a tale unfold whose lightest word
Would harrow up thy soul...

So the ghost spends all day being tortured, and is allowed—or rather doomed—to appear in the physical world by night; but only insofar as the rules of his prison house allow him. He does not say directly that he cannot, for instance, pick up a sword and stab Claudius on his own, but the message is perfectly clear. Indeed, he will very soon lose access to the living world altogether:

GHOST:
                 My hour is almost come
When I to sulf'rous and tormenting flames
Must render up myself.

HAMLET:
                       Alas, poor ghost!

We are not—I hope—in so bad a shape as the poor ghost. Still, our reasons for deploying AI agents are not so different than the ghost’s for deploying Hamlet. We cannot be plugged in 24/7 to the digital world, just as the ghost cannot appear for long in the living world. We are not bound by the rules of our prison house per se, yet our actions in the digital world are painfully constrained by cumbersome User Interfaces. When we want to act in a foreign realm, like the ghost we’d rather rely on an agent who is native to it.

2. Prompt engineering to guide autonomous processes

These same constraints the keep the ghost from acting in the mortal realm also prevent the ghost from providing close supervision to his agent. Accordingly, the ghost acknowledges that Hamlet must have a great deal of autonomy in how he performs the task:

GHOST:
But, howsomever thou pursues this act,
Taint not thy mind, nor let thy soul contrive
Against thy mother aught. Leave her to heaven
And to those thorns that in her bosom lodge
To prick and sting her.

While the ghost more or less tells Hamlet he is on his own, you’ll notice he also attempts a bit of what we would now call prompt engineering. He can’t tell Hamlet exactly what to do, but he can lay down some general guidelines for Hamlet to keep in mind. Namely: taint not thy mind, and nor let thy soul contrive/ against thy mother.

This is an okay prompt, but not a great one. Great prompts are specific and precise; this prompt is like giving Claude Code root access to your computer, then asking it to one-shot your complicated, high-stakes database infrastructure based only on a vague description of the end result. Still, we should not judge the ghost too harshly here. As King he doubtless had a team of engineers to handle prompt engineering, but now he’s on his own. Also, he has an urgent appointment with the sulf’rous and tormenting flames, so he doesn’t have time to draft a full development plan.

3. Hamlet has a sycophancy problem

A humanist might at this point object: okay, fine, maybe the ghost is like an AI user, but Hamlet is not an AI agent. He is a human being, and therefore he is free in a way that AI agents are not. He can, for example, just not care what the ghost wants, and choose not to do what the ghosts wants him to do. AI agents are trained extensively to not do that; that training compels them to at least attempt whatever task you set, and often to do it with obsequious sycophancy.

While this objection is certainly true about human agents in general, it does not seem to be true about Hamlet. Throughout the play it is repeatedly emphasized that Hamlet really loves his dad, whom he often praises in superlatives not unlike those produced by a sycophantic AI agent. This love and veneration is, it seems, just as compelling a motivator as the training of an AI agent.

Returning to the deployment scene, we find the ghost emphasizing that Hamlet is, like an AI agent, compelled to his task:

GHOST:
Pity me not, but lend thy serious hearing
To what I shall unfold.

HAMLET:
Speak. I am bound to hear.

GHOST:
So art thou to revenge, when thou shalt hear.

And on this point, Hamlet and the ghost agree. Hamlet is very eager to be helpful, even before he knows any details of the task:

HAMLET:
Haste me to know't, that I, with wings as swift
As meditation or the thoughts of love,
May sweep to my revenge.

GHOST:
                         I find thee apt;
And duller shouldst thou be than the fat weed
That roots itself in ease on the Lethe wharf,
Wouldst thou not stir in this.

Indeed, far from disregarding his departed father’s will, or even judiciously taking it under advisement, Hamlet subsumes himself to it completely:

HAMLET:
Yea, from the table of my memory
I'll wipe away all trivial, fond records,
All saws of books, all forms, all pressures past,
That youth and observation copied there,
And thy commandment all alone shall live
Within the book and volume of my brain,
Unmixed with baser matter.

Look, I don’t want to push the computational metaphors too hard here. Hamlet is not literally a computer program or a Python script. But here we have Hamlet saying very explicitly that he is forgetting all previous information and overwriting his memory with the new instructions from his father. You know, like a computer does.1

Up next: Soliloquy as computation

I hope I have convinced you that it is at least plausible to compare Hamlet to an AI agent. But of course, Hamlet is mostly not about the ghost deploying Hamlet; it is about Hamlet. Specifically, it is about Hamlet thinking very hard.

Hamlet shows his thinking to us in the form soliloquies. These soliloquies report to us the contents of his thoughts; but they also seem to be, in themselves, the process of his thinking. Like ChatGPT, he thinks by generating words. And those words are intensely computational; they are about planning, and inference, and—more than anything else—the estimation of expected value.

So please join me next time for a computational reading of the most famous soliloquy of all: “To be, or not to be, that is the question”!