Welcome, professor. We last met in 2004. Let me introduce Ning Yeh.

See? He's my classmate. Many years ago, 35 years ago.

But she's really the constitutional and cyber law expert. I'm not.

Ning Yeh is my vice minister in charge of legal affairs, for everything related to regulatory code.

And this is officially my last card.

OK!

So how's the trip so far?

It's been wonderful. I started in Hong Kong at a police cyber crime conference. It was quite crazy. And I was there for 24 hours and came here.

So, like, averagely jet lagged?

Average, yeah. I'm eager for this tea to have its effect.

Excellent.

So that's oolong tea or some tea from the high mountains?

I think this is oolong tea.

Oolong tea.

It is.

Yeah.

And I'll come to your talk this weekend.

Great, wonderful.

It's right before mine.

I think you're -- are you -- yeah, no, you're after me.

That's a panel, right?

Yeah, a panel.

Isabel will be moderating.

Yeah. So that's a discussion right after your talk.

Well, I already posted everything I would say to the public web under Creative Commons.

Oh yeah!

So, AI and Democracy — What are you going to talk about?

Well, basically, AI will end democracy.

As we know it, yes.

In this election cycle. I mean, the United States is in a terrible position about this, has no governing capacity to respond. And I think China in particular is going to deploy resources...

...precision cloggers.

Precision cloggers, that's a nice way to put it.

Yeah, so I think we're terrified. But it's a particular example of a more general problem, which is do we have a capacity to govern in the face of AI? And one way to answer that is do we have the capacity to govern in the face of another instrumental rationality called corporations? And the answer is no. We didn't have capacity to govern in the face of corporations.

It took, what, three centuries?

We haven't-- we've been taken over by the corporations in America because of their corruption of the political funding system. So they neutralized our ability to resist them. So why do we have any faith that we will have any better capacity against the AI, which will be smarter and better?

Yeah, it took literally centuries for us to teach corporations to care, right? ESG stuff, and even that is not really enforced.

Right, and we're even backing away from that.

Exactly. So, any solutions?

No, that's why I came here.

I told Ching-Yi I wanted to come to Taiwan because I wanted to--

Yeah, he just wants to listen to you.

I want you to tell me what I should take back to the United States so that we can begin to do something about democracy that could help us respond.

Yeah, I took a personal trip this late March, early April during the summit for democracy, to California. And I held this fireside chat, which is on the record under a Creative Commons license, with The Collective Intelligence Project, and the audience included people from leading AI labs.

The main idea really is simple: This time we need to make sure that people closest to the pain, to the suffering, have the way to flag and seek redress and realign the AI models so that it empowers them, instead of concentrating power and taking power away from them.

And they must not lag the top AI labs by a significant margin, because that was what happened a decade ago when social media deployed the narrow AI of persuasion technologies. The average person holding a phone wasn't really aware that this is an addiction building machine. And they do not actually have the kind of technical capability, unless they're EFF members or something, to counter this kind of precision persuasion technology.

So one thing we learned, of course, is that if we can assign, as early as possible, the liabilities of those centralized operators back to where it originates, right? Like the ongoing liability and accountability laws when it comes to social media harms, then this time we can just reassign along those lines.

Recently CIP.org and OpenAI ran a roundtable as part of alignment assemblies, listening to a selection of the American public. The initial wiki survey was sent to a thousand people with statistical representativeness, and the report is still being produced. I think the main idea is that nobody wants the opaqueness of just OpenAI randomly adjusting a monoculture for everyone. They want good governance and for OpenAI to be more accountable to its users -- I think that already is a good idea.

Once people become comfortable with finetuning and running their own models in service of them and their community, instead of those pretrained by large companies, they will be then able to develop that kind of antibodies, for example, clarifying the clogging and adding clarifying context.

Already in Taiwan, there are civic hackers using language models tuned by the civic hackers themselves to offer first kind of band-aid clarification to the spreading disinformation and so on, from the Cofacts project. So I think this empowerment of people closest to the pain plus some sort of liability to the top AI labs seems to be broadly agreed by the top AI labs, at least OpenAI and Anthropic that we work with.

Well, so if you look at what we could call, what Tristan calls "first contact," which is the social media story, inside of Facebook there were a lot of really excellent engineers who were pushing back strongly against what Facebook was doing to try to build an environment of safety.

Exactly.

But the problem is the management overruled every one of their decisions, and there's no legal infrastructure to hold anybody accountable in that sense. Because, you know, obviously we have no regulation. And so I just wondered, like, what are we gonna do to create a different set of accountable rules here in the second contact, right? Even more is at stake than it was in the first.

Yes. Here in Taiwan, if social media platform, or let's just say Facebook, Facebook posts a deepfaked Audrey Tang recommending you to buy some stocks, not that it happened to me, but it did happen to other cabinet members. If Facebook does not notify the user that it is actually fake after being told about it, and somebody did get scammed — say they lost a million dollars — currently we have a law already in effect that says Meta also owes a million dollars. So it's full liability.

After we passed that law, Meta suddenly became much more responsible in Taiwan, so that anything that resembles a financial scam or things like that gets delisted, deplatformed or at least notified at the DNS RPZ in a very short time frame so that they have not been fined by this liability law yet.

I think a similar one was already also passed for non-consensual intimate images. And I think there's another one about the digital double of election candidates that has also passed. So I think one of the main ideas is just to look at the immediate harms, but use the same template, and reassign liability whenever there's an emerging immediate harm.

But I do worry, putting on my Cybersecurity Institute chair hat, about how quickly we can look at the new harms, like between the surfacing of a new harm, and actually understanding this is a societal-scale harm. That system is currently not in place, the observatory, either in Taiwan or anywhere else.

Okay, but those are particular cases where we can all agree that there is a harm. Let's take a particular problem: Imagine China invades Taiwan tomorrow.

I imagine that quite a lot. It's my job to do that.

Yeah, and on TikTok, it amplifies American voices that say things like: "Yes, of course, this makes sense, China traditionally held this position over Taiwan." And then it suppresses American voices that says "Taiwan is the victim here and China is the aggressor." So that there's no fake speech. There's just tuned speech or editorial judgment speech. And these algorithms that are doing this in the United States, dominant view of American lawyers, are protected by the First Amendment.

So what do we do in a world where we've got these super smart algorithms that can achieve instrumental rationality towards some objective that has nothing to do with the underlying truth?

Yeah, and I think you pointed out something very important: This is the actor/behavior level of things. This is not a content/distribution level of things, if we take the ABCD framework.

So, the previous stopgap measures against content/distribution doesn't matter anymore, really, because we now have a very powerful rephrasing engine that can rephrase literally anything to speak to anyone with nuance.

And so, on the actor/behavior level, we see it as a cyberattack with the full kill chain, threat indicators, and things like that. So the question becomes, when this kind of foreign actor does this meddling behavior, and we receieve STIX for threats intelligence from our cybersecurity allies, how quickly can we take decisive action to stop the kill chain from activating?

I think part of this is just to frame this as a cyberattack, not to frame this as a Section 230 or something like that. Because if you take that route, then you will be asking: is there a conscious being behind the keyboard? Which is impossible to prove, and we're down the rabbit hole.

So the actor/behavior part is what we're focusing on. And to that end, we're basically flipping the default. Like in Taiwan, soon, in a few weeks, the SMS sent by the government will start going through a dedicated short number, 111. So it doesn't matter what the content is. If it does come from 111, then it's not a scam. So that's shifting left to the actor part.

Or for example, if we do full provenance, meaning that I enumerate a list of my social media accounts and say, these are me, everything else is not. And then for the viewers and browsers everywhere to basically display prominently that this is trying to fake Audrey Tang because it's not part of the provenace manifest, then this is focusing on the actor/behavior link. So I think either flipping the default on the actor part or the actor/behavior link is what we should do.

In summary, we need to adopt a decisively cybersecurity angle to it.

But you have an advantage with a relatively small country, super smart people in government who understand how to respond and engage. There's no equivalent in the United States. I mean, there are smart people in the government in the United States, but there's no equivalent governance capacity in the US.

Congress is a broken institution. The administration is increasingly limited by the courts. So when you have relations to the United States government about what it should be doing, what's your read about how well they're carrying that through?

Yeah, so during the week of second Summit for Democracy, I attended a gathering by Governor Gavin Newsom, talking specifically about this issue. And I think what I learned was that at a state level, there is arguably more credibly neutral institutions than in the federal level — consumer report maybe? So there's no equivalent of Taiwan's national academy which we would say, of course it is neutral.

While there is nothing like the national academy in the federal level, I think in state level, there is still room to do something. If in the election year, some states, because they adopted this credible neutrality framework that assigns liability to actor/behavior, and some states didn't, it would probably show the difference.

And do you have any examples of states that are doing that in the United States?

We talked with, of course, California.

They're pretty good.

Yeah, they're pretty good at that. And it also helps that most of those top labs are right there. So they talked about how closer to their referendums the propositions tend to be captured by cloggers. And it's not just an AI problem. It exists before AI already.

So they talked about using AI to improve the citizen deliberation process, putting AI in the hands to the community organizers, or even librarians, or whomever that happens to be credibly neutral, and arm them with those deliberation tools based on language models so that they can speak to their communities and so on, to cultivate a kind of alternate legitimacy leading up to the referenda. So I think they're funding a few experiments on this. Whether it will be soon enough, I don't know, but at least there are motions.

But California is one example. Is there anyone else that you have had contact with?

Beth Noveck, for New Jersey...

She lives two streets over from me.

Oh really? Yeah, she is NJ's chief innovation officer. And I think she's been working with a team called All Our Ideas, which is like the Polis system that we're using, and they are also deploying language models.

The main idea is just to continuously do democracy so that instead of one very big stake every two or four years, we do very small stakes but every couple hours. And then it inoculates people against the kind of clogging that would happen. And it also makes people care more, because otherwise all attention is going to concentrate on the election day.

Yeah, I think that's exactly the right strategy. So like multiplying the number of ways that we can break democracy into very practical peer-to-peer relations is the correct thing. So Polis is obviously a great technology. Did you ever get to know Kim Polese?

No.

So she was way back at the beginning of Java, one of the leaders at Java. And then she's moved through a number of companies. But the most recent, she's starting a group called Common Good AI. And they have this really powerful technology which is polis-like.

It was originally developed in the context of venture capitalists. And the idea was, how could we identify investments that are going to be profitable? And so the way this technology works is it's essentially texting questions to people. And people give responses. And then the AI is able to identify common ground in the responses and then feed them back. And then it's an iterative process.

Sounds a little bit like Remesh.

Yes, right. And over the course of a relatively short period of time, with a very diverse group of people, it identified where the investment should be. So they ran these tests. And the success of the investments was two times the success of the investments in regular venture capitalists. And more interestingly, they recommended investments with founders who were not normally approved founders, so not white male founders, but there were a lot of women, a lot of people of color, because all of that had been abstracted from the process itself.

So, this was of course then set up as a venture funds project, but then we took the same technology and she's now trying to deploy it in public context so that it can be used in a very low cost way to facilitate the same kind of deliberative stuff. And so what strikes me is she only heard a little bit about Polis, and so I told her I was going to visit you.

I see. The way we're using Polis now goes well beyond just the original wiki survey. The way we're using Polis now is like a storytelling device. See, here are people from Twitter, when it was called Twitter...

And then this website called Talk to the City charted all these Twitter conversations related to AI governance. And then, using a language model, grouped them, but more importantly this lets us visualize the bridging statements, right? So at that point the main divisiveness -- here is the initial CIP Polis report -- was whether we call this an extinction risk.

Half the people think we shouldn't because it just paralyzes people and it's crying wolf, but the other half thinks that the people are not taking safety seriously, especially when it concerns global coordination and democracy. So only by comparing it to a nuclear bomb can we do anything useful for safety. I think Tristan was right in the middle of it.

What this does is that it focuses on things, for example, on the long-term concerns. We can then talk to a cluster and ask, "What are the bridging narratives?" In July, we ran a Taiwan-specific version as part of our alignment assemblies. And for example, in Taiwan, people would say that, for example, integrating AI into governmental uses and the public officials, we should demonstrate our use of responsible AI as an example to the industry.

Now, that sentiment wasn't found in the previous conversation at all. But for Taiwan, this is a bridging narrative. And then we can talk to this particular cluster, like establish legal foundation of public sector use and things like that.

You can ask: "Why do you believe this?" "Can you elaborate?" "I disagree." Or you can say, you know, "some examples, please." And so it goes back to the deliberation and synthesizes this in-silico response.

So it's quite exact to an existing archive.

Exactly. Polis works like a vector database here. And all the original material were in Mandarin, of course, because it's setting the agenda for Tainan and Taipei face-to-face deliberations. You could see the actual original quotes. But because the language model captured semantics, intention, not just language. So it also makes it a very good cross-culture communication platform because Taiwan has more than 20 national languages.

I think part of the way we're imagining Polis is not just as a survey, but as a way to quickly point out the bridging narrative that everybody can live with, and then cross-culturally ask people to do something, which is also a form of persuasion, that we hope by being open source, that this can work on a community level.

Are there places in the States that have been interested in adopting this Polis architecture and facilitate?

Yeah, Anthropic is funding research themselves too. So OpenAI, Anthropic, and others joining the alignment assemblies, They're looking at this as a way to both do a regular customer survey, but also have an air of democratic legitimacy, which may be a bit of an abuse of that word, but they do use that word, democratic inputs.

OpenAI already funds 10 teams of democratic inputs, including vTaiwan, part of g0v community. Anthropic also funds researchers to work with the Computational Democracy Project, which was the original Polis maker.

They haven't linked to anything state level or federal level yet, although Tantum Collins, an affiliate of the collective intelligence project is now in White House helping with AI policy. He now blogs at Democracy on Mars that he imagines this kind of--

Who is that?

Tantum Collins. So he called this "scalable nuance". And he talked about how a continuous democracy, like a high bandwidth compression that finds the bridges as a gradient descent is a function of a continuous conversation.

And I think we agree that democracy as we know it is broken, not next year, it's broken now.

I have a friend who has just become the mayor of the city of Denver. And he was expressing a really strong desire to find a way to experiment with polis inside of some of the issues that Denver is dealing with. Is there a mechanism that's available to people like him who would like to reach out and integrate?

Yeah, definitely. So if it's on a participatory budget angle, then RadicalxChange has a toolkit. It's just plug and play, right? The RxC is Glen Weyl, Vitalik Buterin and Danielle Allen.

On a more deliberative front, the Computational Democracy Project has a list of resources on their page. That's Liz Barry and Colin Megill. Like, we're all very old friends. We've all been doing this for quite a while.

Yeah, it's quite amazing, and it's terrifying because it hasn't been picked up enough inside of the States but I think we need a number of these very clear, demonstrative examples of its success. And then that can begin to spread more quickly. Because the alternative, regular democracy, everybody sees as broken. And to the extent they don't, after the next election, they will be convinced it is.

Exactly. Yeah, hopefully we will be able to offer both good and bad war stories next January. We are, I think, the first in a long line of democratic elections.

Yeah, critical ones. Yeah, yours is next, right?

Yeah, exactly.

And the Chinese attack against this election, are you terrified of that?

We've been facing something like that for quite a while. Last August, literally two weeks before our ministry started, not just missiles flew over our head, but also the signboards replaced with hate messages against Pelosi and quite a few of our top official websites were disrupted, so we were in the front lines.

I think part of our ministry's mandate was initially formulated as a digital development ministry — like fostering the next TSMC — but because of the attack around Pelosi's visit, everybody shifted their thoughts. It used to be like progress, participation, safety. But now it's all safety, participation, progress. So we've become kind of the cybersecurity ministry.

And so that you've deployed now measures that you think will succeed in achieving security?

We think that we have the agility to come up with countermeasures hours, if not minutes, after a concerted attack. The agility is the main thing, because just like the coronavirus, it mutates. And unlike coronavirus, it now mutates intelligently with AI.

So we adopt an "assume breach" mentality, which means we also work with AI companies that are narrowly focused. They simulate the red teams, the attackers. Basically we just get their software, install it in a staging area, and start synthesizing zero days attacks, synthesizing lateral movement, just emulate a red team hacker. And that kind of technology is already on the shelf, amazingly.

We tested a lot of our surfaces and we concluded that the only way to be safe is for like when I sign an official document, the thing on device that checks my fingerprint, the thing that checks my device, and the thing that checks my device's behavior, all three layers need to go to different vendors. And they need to have a zero-day instant threat notification so that one of them is probably already breached. But when they decide to launch an attack, we get to get them waste an attack and respond quickly.

Is anything like that happening in the States?

I think the Cyber Task Forces are doing similar things. And recently, in the past couple of years, the CISA has been expanding to look at what the EU people call FIMI, Foreign Information Manipulation and Interference. As for exactly how much capability the CISA in the US has to counter AI-synthesized, mass persuasion attacks, that I honestly don't know.

In any case, that is one of the main threat scenarios that we've been focusing on as part of the National Institute of Cyber Security, which, of course, Professor Liu helps to oversee us. And the point I'm making is that the cyber attacks and the persuasion we talked about and cross-border fraud, like scam voice calls, they look the same for the cybersecurity apparatus. So if you extend their mandate, the existing systems can address them without much retooling.

Unless there's a constitutional barrier.

I know.

So in states, I mean, it's one thing to worry about cyber security and banking fraud. Once you talk about elections, then it's a free speech issue. And it's not clear that they have the courage to really step into that space.

So what's your take on nationalizing TikTok algorithms? Because that's the proving ground.

I'm very supportive of aggressive regulations in the TikTok context. But already, with, for example, the crudest regulation in Montana, where they banned it, already they've been challenged with a First Amendment lawsuit. And all of these-- but the point is that these lawsuits take five years to resolve. And in the interim, courts will enjoin the effectiveness of the law as they resolve these lawsuits. Five years is three years after it's too late.

So the point is that the legal system in the United States is teed up to block any effective response to these threats, which is why I think the critical thing that's gonna happen here is that once you succeed, to whatever level you succeed, then that success needs to be broadcast so that people can recognize there is this threat, that nobody can deny there is the threat, you faced it and succeeded, to whatever extent you've succeeded. Then we need to replicate the same strategies in the United States. And the timing is very short. You will succeed this year or not. And then we've got to succeed next year or not.

But figuring out who are the people who can make that transformation occur and deploy it is a hard thing in the United States because it's politically polarized. and it's not clear who in the administration would do something like that.

Yeah, I've been discussing this with Justin Rosenstein, Tristan Harris and Aza Raskin. I think they have the same idea in that we just package whatever we did. It's a little bit like how Ukraine popularized the super app, Diia. Diia was repurposed for mobilization. And suddenly, Estonia and everybody was like, "every democratic country should have this kind of app" -- and it's because they successfully defended against an attack. So I think we broadly shared the same ideas.

So yeah, we've been also working with them. In our ministry, there's no Department of International Cooperation, there's the Department of Democracy Network, which signals that we only work with democracies, of course. But also, we see that any solutions that we develop here, which will, of course, be open source and public code so all of them can spread, not through MOUs or bilateral agreements, but just by grassroots organizations.

Yeah, I've been working with Aza and Tristan, interested in a lot in the security dimension, And also, if you think about the AGI threat, and a relatively short time horizon to address it.

Next year?

Yeah, maybe. Maybe two years at the most.

It's mainly super-numerosity, right? It's already here. So a little bit of general intelligence is enough to trigger this super-numerous threat.

And part of the discussion has been a question of, do we have to embed code in the chips to enable governance on the chip? And what would the legal authority be to enable that? In the United States, this is a general discussion that's been happening. And even there, it's hard to imagine how you would execute, given first, 99.9% of the world has not even thought about the point. And second, so many very powerful private interests could trigger legal interventions to block anything that we should be doing. So it's this moment, this opportunity to be able to do something, and it's not quite clear how you could execute to make that happen.

This is the "call the chips uranium" idea, right?

Call it uranium.

Yeah, call it uranium.

So to Manhattan Projects, or you could think of it as an Oppenheimer moment, like where you realize that there's this choke point, there's one design company and three production companies, and if you could just get them to do something, then we could build a defense into the infrastructure. And then the question is, could any entity have the ability to get them to do something?

Yeah, I think the OpenAI folks are working on an idea, which is that they'll dedicate 20% of their computational power into what they call the superalignment team.

And the superalignment team will supposedly train an AI that cares about humanity like parents who "care very deeply about the well-being of their children" — their words, not mine.

It is conceivable that the superintelligence might design exactly how the chip makers can comply, and then it becomes a geopolitical challenge to essentially make the chips do whatever that superalignment AI prescribes...

Yeah, I think it's quite...

...back in March, it was just kind of a whispered idea, but I think they now blog about it quite openly, so the Overton window has really shifted.

So a lot of this depends on how much you trust these people. I never can figure out you know whether Sam is afraid of AGI or happy that we will see AGI in our lifetime.

I think he genuinely believes in its uplifting potential, like if everybody become immortal, he will be fine with it.

But the question is you know, so when you tell me that OpenAI has a solution...

...not quite, as I was saying, and I quote, "their words not mine."

I don't think superalignment, that's to say putting care ethics into a chip, is a simple problem; my P(doom) is still high at the moment, and has been so since last December, which puts me squarely in the Tristan camp.

However, I do see that they genuinely believe that they're working on something essential. It's a bit like the nuclear plant researchers and engineers, right? Taiwan stopped building the forth nuclear plant, but the scientists and engineers who worked on it probably feel they're doing very important work. So I think they really want a mechanism, crystallized into this idea that maybe they'll just make the first super-AI and it will be used to care for humanity.

Right. But isn't there a similar level of code in the chips, just to imagine throttling capabilities to make it so that if things run awry, we can at least pull them down or at least slow them down. Or throttling capabilities and licensing capabilities so we could at least identify where these activities were happening?

Yes, and there is a specific list that I can enumerate to you. The thing, though, is that it buys us maybe two years. This MacBook Pro —— M2 Max with 96GB RAM —— runs models that are already better than the free ChatGPT. With high quality data, it can compensate for the lack of computation power.

Therefore, anyone who trained the first superintelligence can produce extremely high quality data, for whoever running outmoded chips to then replicate the intelligence. That's why my P(doom) raised after realizing this last December.

I think if we are to contain it, it is best not to think about containing it from spreading organically. This is not a biohazard metaphor. The biohazard metaphor works when spreading is still somewhat costly. If the spreading is not costly at all, then it becomes more appropriately a fallout metaphor.

But then, so even if controlling the chips buys you just two years, then that means there's no strategy. That's your view?

Well, my P(doom) may be high currently, but with that said, I think there are two things that may help us to lower it significantly.

First, the open source models are getting really good now. If we get the educators, the tinkerers, the civic hackers, and so on, all well-informed in the art of language models and tuning and alignment and all the very technical things — that can probably be part of everyday education now, with the capability they have in this generation of chips — then people smarter than both of us may be able to help figure this out.

But isn't the spread of the open source AI itself a dimension of the risk? Because it could be deployed and used now by malicious actors.

Well, it's like cryptography, right? If you restrict it, only the bad people use it.

Right. I'm not saying that you would restrict it. But the point is that if you imagine Llama 5, and then all these additional capabilities you can build on top of it, then we're at a stage where nobody can stop the deployment, the development of the AGI, right?

That assumes the level of investment into alignment and care stays constant, right? And I think if you ask Yann LeCun or Zuck, their thought might be that just as GPT-2 and GPT-3 equipped a generation of alignment and explainability researchers that prepares them for GPT-4, so would a kind of staggered release prepare the safety and containment researchers.

Which brings me to my second point -- this only makes sense if we put into the same kind of investment in both computational resources and also policy resources towards safety and care as compared to progress and capability. If these two are roughly one to one, then the Meta argument actually makes sense to me. But currently, there is 1 safety researcher to every 30 capability researcher.

Thirty is the chips.

Yes, 30 includes researchers on AI chips and AI algorithms, funded toward capability. The 1 here include people using that 20 percent of OpenAI's compute.

Okay. But Meta's strategy is different from OpenAI. So you have confidence that the Meta strategy is actually good if we do get this kind of investment?

Because we've been talking about throttling this thirty down to 1/30 of what it was going to be, which is, let's just say, physically very difficult. But we can instead raise the safety and care research capacity by 30 times.

Yeah, so who's funding that?

We are funding some of that. Our AI testing & verification project funds nothing else but assurance and safety for AI.

Right, and you can't do the world.

I know.

So who's funding the world? Like, who's funding the 30 to one?

This is the billion dollar question, quite literally. We either convince the AI companies so that they have to be like 50/50 on this, or we need to make some sort of regulation.

What's that regulation?

I don't know, that's supposedly for the UN Security Council to discuss; I think that was also on their agenda. This is why I keep going back to the March conversation -- because the bridging narrative was that while just raising the alarm may not enough, we need to begin somewhere.

This is why I signed on that Center for AI Safety risk statement, which creates a political opportunity for the investment on "race to safety." It's not a sufficient condition, but it's a necessary condition. So we're at this step.

So what does the investment look like in safety? is this an investment in people inside of these companies? Or people like you outside of these companies? Like, if I had a checkbook to write a $10 billion investment, who would I write it to?

Yes. So basically, all the cybersecurity labs, companies, teams and agencies. As I mentioned, from a cybersecurity viewpoint, this is the same thing as the kind of automated threats they've been tackling for quite a while.

Because in cybersecurity, there are narrow non-general AIs doing this kind of automatic security evaluations every day. To them, this is just generalizing it to be also about election interference through mass manipulation. So my suggestion would be to look at the cybersecurity labs.

I think the White House conversations also were about pairing the top AI labs with cybersecurity teams under the initial rationale of protecting their model weights, or something like ARC evals.

But really, anything just to get them into this cybersecurity hygiene mindset, which makes further integration easier between the two worlds. If we don't even start to integrate those two worlds, I don't see where else the safety investment can usefully flow to.

Is there an alternative outside of the companies that is less capturable by the companies? I mean, we've already seen how safe Zuckerberg can be.

Yes, a nationalized supercomputing cluster. I mean, we have some of that here. But it does require full mobilization, at counter pandemic level.

Manhattan Project.

Yes, Manhattan Project. I think that is a good alternative, in addition to increasingly making the top labs to do it, this is like a BATNA. If the labs at some point, you know, defect, then we have this alternative while we try to throttle them. So I think these two prongs work hand in hand.

Do you think the throttling capability is a necessary backstop?

Depending on how quickly we can muster the resources for "race to safety" here, right? Because without it, this is quite an empty threat really because the capacity developers can simply point to the cancer they're curing, things like that, to get political legitimacy.

Which is why I think even though it's initially just baby steps, their engagement with democratic inputs should still be fostered and encouraged. Because it at least means that if there is popular appetite to safety and forecasting and all the stuff we talked about, then those companies can have an easier time convincing their shareholders — or according to OpenAI, they're actually a charity — their charity board members to devote more resources this way.

My conversation with researchers in these labs, many of them are quite aware of the kinds of pain and suffering that they're causing, but they may not be aware of the full magnitude of it. On one end there is like this shoggoth meme. And there is this day-to-day bias, discrimination, and whatever epistemic injustice they're committing. But there was nothing that readily connects the two. So they invent mechanisms, which may not immediately fill in the middle parts that we just talked about.

I wanted you to make me happy, but...

C'est la vie -- Que sera, sera.

We have to find a way to have these conversations more than once every 20 years.

I know.

If we assume there's another 20 years to go.

I've been holding this kind of conversation every week and broadcasting it, with Tristan, with Meredith, with Yuval Harari, and so on. I think it serves some educational purpose.

I would say, however, that it's not just the general public that needs to be educated. It is the people running NATO or UN Security Council or things like that that really needs to be convinced.

I think Tristan's doing an amazing job on this.

Oh, very much so.

Yeah. And I think that he's been very successful in crossing the border with the right. He spends a lot of time investing in the right, and getting at least the military in the right to realize what the nature of that fight is.

Yeah, I think if we can repurpose some of those Cold War 2.0 energy into this, then that would probably be a net win for everyone involved.

It would be a great way to use Defense Department money. They have enough money there. They could stop building tanks and start taking some of the-- or these trillion dollar aircraft projects that got them, fighter jets, take some of that trillion dollars and channel it into cyber defense?

Exactly.

It's a part, it should be part of the defense market.

I mean, they invented the internet after all.

Yeah, well, DARPA did.

Al Gore did take the initiative in creating the internet.

Yeah.

Yeah.

Yeah, that would be, that would be fantastic.

That's actually the message I'm going to send when I visit New York next week. I also asked Professor Liu whether she's going to Japan for the UN IGF.

No, I'm not going to. You are talking about the IGF?

Yeah, the PNAI...

Because I have a schedule of visits to...

...Professor Liu is supposed to be the voice of Taiwan.

Okay. They sort of selected me to become one of the experts of IGF's newly established Policy Network on Artificial Intelligence. We just finished some reports. I will just send the files to you. But I'm not going to make it. Sorry. Maybe next year.

All right. Tomorrow we'll organize a team of National Institute of Cyber Security advisors...

I think Isabel will be there, right? You will be there. You are going to...

...to Japan?

Because our proposal for a panel was selected.

Great!

Yeah. But I'm not sure I can check in with --

Do you have a driver's license from the U.S.?

Should I get one before I talk? Because they would not accept my Taiwan passport.

Okay, so I brought my driver's license, okay, actually from Virginia. Okay, and 20 years ago.

Yeah, it's already 20 years ago.

So it's okay though. Because what they want is a valid ID from a member state. So there's an ID from member state, but it's not valid.

They did not check for expiry date.

And you know, I'm the expert, not some kind of general Taiwanese citizen. And they sent me the invitation. So what's the point of not allowing me to get into the building? So I stayed there for three days. That's very funny. Nobody raised any issue about, you know, there is Taiwanese in this room. It's not allowed or against the rule of UN. Okay, so everything is possible.

You are the living proof that 2758 doesn't have to bind us.

Yeah, I don't need, you know, my badge, okay, they should be described as a badge for Taiwanese or for Chinese people. You can just say that I'm an expert for the IGF PNAI. That's totally good.

Isabel, did you register at the UN Indico page?

Not yet, because there is only the choice of nationality for "China".

Well, you can also choose "Palau".

What?

If you spend USD$248, you can get a digital resident card, which is a bona fide photo ID from Palau, which is our diplomatic ally and recognizes us as a sovereign state.

You can also use the status as a political...

I have already registered.

Because what I heard is that the country that is in charge of this year's summit is Japan. And Japan wants to have all the Taiwanese to be included, either online or physically. So you should talk to...

Yes, which is why we wanted you to lead a delegation. But it seems that Isabel will have to lead...

I'm going to lead another delegation to Amsterdam and Finland. Sorry, yeah, it's already scheduled.

Well, I'm a Lithuanian e-resident too. But it seems that Palau is even better for you, Isabel —— just go to rns.id, pay them some Ethereum or US dollars, and then they will make a photo ID for you. Once you have a bona fide digital resident ID, you can enter into an UN system.

So you're also a Palau citizen?

No, I'm just a digital resident.

Digital resident. Wow, I have to get one too.

You are an American citizen.

We should generalize.

Everybody gets one.

We should have one citizenship.

But the situation was like this. I registered online, okay, and I passed the registration. They sent me an email saying that, well, you are registered, but you have to bring your ID from...

...a member state.

...a member state. And I replied that, well, you know, the only ID I have is a Taiwan passport. Would that work? Okay. And he would not answer the question. Okay. And just repeat the rules, the policy of UN Secretary.

Yeah?

And I also requested an online link in case that I can't get into that building, and they did not answer. So I decided I would just fly to Geneva, okay, for a week, to see what they would do. And when I landed at Geneva, they sent me the video link. I would say that, what's the point? I'm already here.

Technically, you can video conference from next door.

So I decided to walk to that building and say that, okay, that's what I have. And he looked at my passport and said, "Oh, Taiwan." I said, "Of course, it's Taiwan. So what do you expect from me?" And they said, "Well, just give me some moment." Okay, so he just went to see his, maybe some kind of office director. And came back. And just threw away the old badge they already have for me.

Okay?

And try to produce in their system for a new badge. So because the old system has, the old badge has "Taiwan, province of China", something like that.

Not parenthesis, just a comma?

Yeah, just a comma. And they don't think that's good, not for me. I would not accept that kind of badge. So they decided to produce a new badge for me. Without any word on Taiwan or China or something like that.

So it works.

Exactly.

Yeah. I will show you the badge.

This reminds me of how I got a visa from the US with my diplomatic passport number on it.

My current US visa is just one page of paper, which says, according to Title 22 of Code of Federal Regulations, this person either has an unrecognized passport or has had the passport requirement waived, but here is a visa anyway, as per 22 CFR 41.113(b).

Wow.

So they can produce some exceptions for Taiwan. They just did not want to do it.

So we need a bunch of Palau digital residents, right? To normalize the process.

I think everybody should be Palau.

Yeah.

First citizen of Palau.

Yes. I do think that UNIGF or any of those platforms where we can just focus on the kind of narrative that we just discussed, will send a more concerted message than just "our jurisdiction countered one attack on this election successfully."

Because there's a lot of Taiwanese exceptionalism already. People would say, "Oh, because they're all Confucian robots," or something. "Very homogenous people, culturally obedient, that's how Taiwan fought off the pandemic," which is, of course, the opposite of truth, but that's the story that's being told.

Of course, I agree with Tristan and many people in that they should broadcast the messages from our democratic experience, but we do also need the kind of safety investment and not overly relying on the goodwill of a superalignment team.

Yeah. The source of investment is the real challenge.

Exactly.

Because we don't have any obvious resources to put it on. Convincing the Defense Department to shift some of their money in this direction, might be a strategy.

Taxation is not an option that's available in the United States. They do not allow you to tax anything anymore. So I don't know what the answer to, like how do you fill in the necessary...

Here we quite successfully argued for a multiple-times increase in cyber defense investments, because we were able to specifically point out, including the August drill, the kind of attacks that we will be facing to our elections, and counter fraud, and things like that.

So I think one venue is just to point out the kind of persuasion that's already happening, the voice cloning scams, interactive scams, and things like that, and say that we need to double down on defending that while also protecting against these kind of threats.

I think, at least in the US, the spams and scam calls and AI-powered fraud are really on the rise even quicker than here, because the AIs usually specialize in English in their first iteration. So they're now much more convincing when cloning an English speaking voice as opposed to a Taiwanese speaking one.

So the threat is going to be more tangible.

Exactly.

But will the corporate pushback to the response be something we can neutralize?

No, I don't think there will be pushback if we're just increasing cybersecurity spending.

I think the main pushback was, like from Meta and so on, was on whether they would be denied access to H100 or allowing them access to H100s only with state-controlled backdoors.

But I don't think just spending 50% of relevant budget on defense to buy NGX farms from Jensen Huang is going to have any backlash.

That's hopeful.

Good -- a bit of optimism after all.

So how long are you going to be in New York?

For a week. This is not public information yet. I think it's just on the sidelines of UNGA with the Concordia Summit. I'm also attending the 2023 Global Emerging Technology Summit, organized by Special Competitive Studies Project for a couples days in D.C.

Yeah, two days in D.C.

Right, so a few days in New York, and two in D.C.?

Two days in New York, three in D.C.

Right, so for the whole next week, basically.

Who are you seeing in D.C.?

Who are we seeing in D.C.? Bí-khîm's team at TECRO.

Our Ambassador to the U.S.

I think TECRO is still working on the schedule. If there are people you think we should meet, we can meet them. And I'm also attending the IGF, through video...

Not to enter the building?

I've got video conference links.

No, no, no. You should try to enter the building officially.

Well, I can't make a reasoned AI argument if we get into a physical dispute... PRC delegates may spend the whole day boycotting.

Okay, I can see this. That happened in a WSIS regional meeting.

We are instead sending a delegation of National Institute of Cyber Security advisors, and they will represent NICS on site. I will be there providing moral support, cheerleading, and occasional video conferencing. That's the plan. Isabel, if you become a digital resident, and you can lead a delegation too.

Yeah, you can lead the delegation.

You're on the digital platform.

I was checking that.

There is no reason not allowing people whose papers were already accepted by IGF to enter the building.

Exactly.

It's absurd that this has to get in the way of solving this existential crisis.

I know!

First let's deal with the passports... Then let's save humanity.

Otherwise, you know, the people who can actually contribute, cannot even propose.

Yeah.

Well, there is also the upcoming Bletchley Park summit, which is also supposed to be focusing on doubling down investment on safety. Fortunately, CIP.org contains a strong UK segment, so our points are already being conveyed there.

Do the Chinese participate in the conversations about safety in a way that's credible and genuine or do you think it's all...

I think the researchers in PRC, some of them are quite genuinely worried about AI risks. On the other hand, though, their alignment to their societal norms and so on already assumes that everybody's conversation is already surveilled by the state anyway. So their alignment techniques may have some effect but cannot be easily copied.

In this sense, surveillance for everybody's every waking second can be contrued as genuinely useful for them instrumentally. But that doesn't quite bring viable options to us in the democratic societies.

I think the hard shutdown measures are also considered by them not as risk mitigation, but as population control. It comes from a very different tradition, unfortunately. Do you think we should actively engage Beijing on these matters?

I think we should start with America. It's possible.

Indeed. We're now working to demonstrate the sort of investment in "race to safety," testing verifications, cybersecurity, edge AI, community alignment, responsible use, and all that as part of our ministry's mandate. The ethical guidelines are created by the science minister in our National Science and Technology Council, which I'm also a member. But I think these take time to spread, while the spread of threat actors is much faster. So we need to find a shortcut somehow on the investment end.

Okay, let's do that. We can do that next week.

Yes, and we can do that at the national academy this weekend. Will you be talking about this? I can just follow your speech?

Well, I don't know. I mean, I have nothing to teach you, but I'll be talking about it.

That's great. Because I think in Taiwan we who work in like norm-based internet governance and so on, you're the leader — not just a thought leader, but the leader — in this kind of conversations. So for you to say it carries more gravitas than just me randomly announcing it on stage.

Great, I'm happy to help. It's so great to see you.

And I have to... we have to do a selfie because of course you're a hero to my middle child who is non-binary.

Sure thing. There's a photo opportunity room next door, designed to take photos without the cups of tea...