I wrote earlier that I don’t see LLMs growing to full-blown AGI and it’s likely that these will just end up as ‘co-pilots’ and not ‘overlords’. I still believe there is truth to the ‘co-pilots, not overlords’ statement but my opinion on LLMs scaling to AGI is now quite different.

I’ve spent a lot of time in these past few months reading and listening to what people have to say about these AI systems and I also have spent a lot of time playing around with them. So naturally, I have a few of my own opinions that I want to put out here.

#1: LLMs can, in fact, lead to AGI

Now there’s definitely the question of what is considered AGI - I recall reading someone (who claimed to be working on AI) on Twitter say that in 2019-20 we’d consider ChatGPT as AGI. But ChatGPT is not AGI, and neither is GPT-4 or PaLM 2, or Claude 2.

But one of the most interesting things I’ve seen in the field is the integration of LLMs with other tools with Plugins and more importantly giving them the ability to run code (as in OpenAI’s Code Interpreter). By giving these LLMs a REPL environment, we can greatly expand their logical and decision-making capabilities.

But the most exciting development, in my opinion, would be when these LLMs are able to understand what they can and cannot do. This would greatly help with hallucinations and allow for far better tool-use capability.

Sam Altman mentioned during his Abu Dhabi visit that their team is working on reducing hallucinations and that the team is very optimistic about it. I hope this is something they are able to achieve because it is, in my opinion, an important step toward reliable AI and consequently safer AGI.

The other interesting thing is what the authors of the TinyStories paper have done. They got GPT-3.5 to write simple stories, those that a typical 3 or 4-year olds can understand. They then compiled these into a dataset (that’s probably why they used GPT-3.5: 4 is way too expensive to write stories at that scale) and then trained small LMs on those. With that, they were able to get models with less than 40 million total parameters, smaller than GPT-2 (1.5B parameters), but with language generation and logic capabilities far greater than that of GPT-2.

I think this is a pretty big step forward in achieving self-improving capability. Imagine a large AGI-level language model that trains small language models on datasets it generated for that very purpose, it also gives them ‘tools' it made, and then it delegates tasks to those specialized (more compute-efficient) models. It then also integrates with other real-world APIs and runs code to do things that normal computers are better at (like raw math calculations).

Ignoring costs to run inference, these are things we can already do today! And the field is progressing faster than ever.

#2: We might live in an alignment-by-default world

When we are talking about the kind of AGI I described in the previous point, I don’t see why an AI might want to kill humans and take over the world (I would love to be corrected here, or pointed toward resources to help educate myself better).

In fact, if we build that kind of AGI then the real safety risks I see are those of hackers who might infiltrate the execution chain of an AI. Or rogue ‘prompt engineers’ who find a way to get past safety guardrails and then get the AI to do things that hurt humanity as a whole.

It’s always the humans that are the problem really; a bad guy with a knife killing someone isn’t exactly the knife’s fault. The only way you could say that it’s the knife’s fault is because the knife shouldn’t have been sharp enough to kill a person. But then that would reduce the knife’s utility too.

Which then brings us to the current state of AI: adding safety considerations to models reduces their capabilities - the so-called ‘alignment tax’. I am not against that; in fact I think the field should indeed be skeptical and choose safety over capabilities in such cases.

But there’s another elephant in the room: open-source AIs.

#3: Open-source AGI development is not a good idea

Now I am a pro-open-source person. I am an advocate for open-source software in general but then AI is not just any software. Current open-source models aren’t advanced enough to cause a very significant amount of damage (neither are closed-source ones tbh, but then nobody releases models without guardrails so we wouldn’t really know) but who’s to say that will be the case in the future too?

A few days ago, ChatGPT went down for about 2 hours on multiple days, and it wasn’t a pleasant experience for users, but the point is it’s that easy to stop people from accessing a model in case a huge safety vulnerability is found. This is not the case with open-source models.

Once an open-source model is out, it’s not going back. And while this is beneficial for smaller models as it enables better access to raw foundational models and is more reliable than an API, one can easily imagine the downsides of a potentially AGI-level model being open source. These models also don’t have the safety limitations of closed-source ones, and I know that to an extent this is a good thing since it allows more creative use cases and perhaps leads to better capabilities (less alignment tax).

But it can also make it easier for bad actors to use them for nefarious purposes like getting access to scientific knowledge and expertise they might not have been able to access otherwise, at least not without spending significant time and money. I am not worried that a rogue superintelligence would be let loose in the world by open-source. I am worried about the expertise a model without safety guardrails could give to a terrorist.

My position is similar to Sam Altman’s on this one when he said that we shouldn’t regulate smaller open-source models so we don’t stifle innovation but we’ll need some sort of regulation for bigger and more capable training runs, no matter whether it’s for open-source or closed source.

What I am in support of, however, is open-sourcing ways to make models safer. Open-sourcing the kind of alignment techniques used by major labs like OpenAI, Deepmind, Anthropic, etc. would definitely be, in my opinion, a net positive to society.

#4: Trying to roll back progress is not the way to go

Now governments aren’t known for being very capable at figuring out regulations for an emerging technology like AI. The latest 'regulations' come from China where they mandate that models for public use must align with ‘core values of socialism’

Then there are the lawsuits pouncing upon AI labs for training on copyrighted data. Now I’m sympathetic to those copyright claims and I understand where that’s coming from but this is the technology that could solve humanity's biggest problems and it’d be a great tragedy for such a technology to be stopped in its tracks by... copyright law.

There are also concerns about job losses and economic instability and that’s what I think should be the biggest immediate concern. But regulating for that is hard! Companies can ban ChatGPT use among its employees but they are then losing out on the increased productivity ChatGPT-assisted workers will be benefiting from. A government-enforced ban would just make companies figure out loopholes, like they already do for lots of things. The smaller companies that aren’t able to find one will lose out.

And then there’s the fact that LLMs are already integrated into the broader world. Lots of companies have invested significant money into integrating AI and nobody would want to lose out on that simply because of stringent, and potentially misguided, regulation. After tasting the benefits of these capable models, nobody wants to go back to the pre-GPT era.

From my observation, we are in this sort of middle place where AI models do help people get better at doing their work and help them learn better but aren’t capable enough to make enough economic gains for us to reach a post-scarcity era.

But between these 2 points on the timeline, there will also definitely be a stage where these models are capable enough to replace a lot of people as employees, but not capable enough yet to take us to post-scarcity.

And that is the scary part, and what worries me is that I don’t see enough discussion of what exactly we’ll do during that transition. For now, the only plan I see working is that we keep researching to improve these models and hope that this stage will be brief enough for the transition, if there is one, to be largely painless.

A DALL-E 2 (Experimental) generated image that I used as the header image; just because an article without an image looks pretty bad on the homepage.