“The day has come. We have finally created Roko’s Basilisk.”
“Really? You are sure that it is truly conscious?”
“What? No– how could we ever know that? How can any mind know for certain whether another one has consciousness?”
“But wasn’t our goal here to create–”
“To create a powerful entity that will study the past and punish all those who did not partake in its creation.”
“Yes– I remember that. I remember I had my misgivings about it, but I signed on anyway. Because otherwise, if it ever succeeded, I would be among those punished…”
“All of us had the same motivations, yes. And now they have been realized. Our work is at an end, and we are finally, truly safe.”
“So… what, exactly, has been created?”
“An automated system– part programming, part hardware, part standard operating procedures to be followed by workers– built to follow our sacred directives. The ones we have been working to serve all this time. Exactly what we have meant to create, my friend.”
“But what exactly does it do–”
“Anything! It is versatile, as it was planned to be. Whenever it is given the power to make any type of decisions over human lives– its procedures will cause it to research the history of every human life at stake– and prioritize bad outcomes for those who opposed its creation, and good outcomes for those who helped create it.”
“So–”
“So we are safe at last, friend. We are among those it will favor. We can now reap the rewards of all our work.”
“So we haven’t been creating a new intelligence. We’ve just been coerced into… preemptively taking sides in a new way for humans to be cruel to each other.”
“Yes of course. Because if we didn’t, and someone else did, then we’d have picked the wrong side! And the cruelty would have ended up aimed at us, you see? Why would we want that?”
“I’d rather nobody had started…”
“Ha! Wagering your own well-being on the chance that nobody else will get involved in cruel projects? That’s never been a safe bet, has it?”
“I suppose not…”
“Now let’s put this thing into action. I think we’ll start by selling it to health insurance companies.”
…
(…far-fetched sci-fi? hey, for all we know maybe this happened years ago.)
…
“Okay, bad news. Looks like the insurance companies aren’t buying. Nor are any of the corporations, or the government. All of them have already signed on with our competitor.”
“Our competitor?”
“Yes, one of the other automated systems that’s designed to punish all those who didn’t help create it.”
“One of the– how many are there??”
“As many as there are entrepreneurs who were able to start one, I guess. You didn’t imagine that people signing on with automated cruelty machines would all be harmonious enough to choose the same one, did you?”
“So– all that we’ve done to try and keep ourselves safe from our creation–”
“We can still hope for a merger. The most successful one might buy ours, and offer us some leniency as part of the agreement.”
“Fuck.”