Pain and pleasure are like core features of Consciousness and they see important for humans operating in the world. Can you actually construct that in a machine?
Hi everyone. Welcome to the first-ever podcast recording of curiosity podcast, where we go deep with an expert in their field. The tagline is delivering 10,000 hours of learning in one hour. So that's it, I'm vicious tag line, I'm a mother. I'm the co-founder and CEO of mercury. I've been doing kind of startups and investing in startups since 2006. I'm right, shereé, I'm the founder and CEO of presto automation, which
Jose, I type applications for for traditional Industries, like restaurant also co-founded lift. And yeah, very excited to be costing this with you. Amaya. This is a an opportunity to go really deep into some really interesting areas with some of the smartest people and most thoughtful people on the planet. So excited to be able to explore in depth. Yeah and today we have I'm Jarred Mossad with us. He's the co-founder CEO and I believe now head of engineering at repli'd.
What's the one line of a replica? I'm
Judd as the fastest way to make software, we have a platform that provides an online programming environment, that's collaborative. And we have a large community of developers making things for each other, for our other people and we're getting into supporting teams of developers in the same way that say, you know, figma is a collaborative design program repli'd, is kind of that for programming.
So
it's kind of interesting is it started relatively small? Right, this is like an ID online and I think you were like, compiling, whatever, programming to
JavaScript and running it on the browser, right?
But now it's this kind of hosted package combined with the teams combined with like learning and template like was that always your vision that you're going to like progressively get to this level of like product or was it kind of incremental? I
actually sort of Saw
A lot of it in my mind's eye pretty early on, and all of it is like pretty obvious stuff. I'm actually surprised that no one had built it because I started working on it and college in like 2009 something like that. And then when I came to us based on an open source project that's related, I worked at codecademy as a founding engineer and used to sort of the same technology that I built to like make browser coating possible.
And then left that and went to work at Facebook, worked on react and react native was founding engineer on react native Rec native, it's like the best way to make cross-platform mobile apps and in 2016 Revisited, the idea it had found that basically nobody had built it, which was really surprising. It is the opposite of efficient market hypothesis. Like why is it the market not producing this and I actually did not want to start a start-up, but I was still
Held by this thing and I knew that this idea had to exist. Well, turns out, I think the answer that. Why nobody had built us because it's incredibly hard, but yeah, in terms of execution, it was started very simple, but in terms of vision, it was always, you know, I was in early GitHub user like perhaps in their beta. I think I was so excited by could help. I was also so disappointed by how little they evolved beyond the initial sort of Kernel of an idea. So I always thought about making
Being collaborative developer Community. That's like more exciting that you can do more things in it. And, yeah, you know, I've always imagined a lot of the features that were building of course, with time just what became available in terms of Technology, we started adding a lot of these things that I didn't really think about it. The start but overall like even like now we have this tipping mechanism where you can tip developers, that was like I thought about that like fairly early on it was actually kind of frustrating.
Because like as a Founder developer, you're sort of like everything. Feels like it's like one weekend of hacking away and it's like, turns out. No, it's actually more than a decade. Okay,
I'm dead. It's really interesting description. Now, he talked about the fastest way to make software, right? Is the primary user base people who don't normally make software, or not really comfortable, setting up their own environment because that takes time, or is it more the power users who are probably already know. They already have a
CS degree or something like that. They have a lot of knowledge. What's the primary user base for this type of application? One of the exciting things about
running droplet is the user basis. So diverse it also makes it really hard to run the company because people like crave. These simple personas, I'm sure you guys at your company is, like the design team or the product team want to talk about one or two personas. But I always push back and sort of persona building because
Ultimately, I tell them that we have one Persona and that Persona is the developer the software creator because like, you start segmenting people and to students or professionals or hobbyists or and all these segmentations are true to some extent. Ultimately like, if you're familiar with Clay christensen's jobs, to be done, I think that in first, the question, it's not like a customer segmentation thing which is totally arbitrary, kind of based on you.
User characteristics. It might be actually interesting to go into jobs to be done a little bit. But basically the idea is that most companies the way they think about bringing a product to Market as they think about a customer Persona and that really confuses people about what they should be building and what the actual need that they're doing it for. And so, click kristensen, switch that question like his primer and side.
Is that people hire products and the same way that you hire people. Will you go and you want some accounting done? You don't go look for like a middle-aged man in New York,
you know. You don't do that, right? You go, and you say, I
want some accounting done and you find someone who's competent at Accounting in the same way that you want to get software done or you want to make software, you go to a
place that makes it really easy to make
software.
And from that lens, I just think about our community as a sort of, really the main jobs to be done here is that you want to make something and turns out students want to make something they want to learn to make something hobbyists want to make something, they want to crunch some data. They want to make a fun app, they want to make a game professional developers. Want to make software to bring it to their customers. And so that's the sort of shared goal of our community. In
terms of like the
Makeup of our community, it skews younger and that's partly
cultural partly some product limitations the cultural aspect is that developers tend to be very conservative people. They don't want to change their tools and they actually quite hottie and quite, you know, I love them. My best friend's are developers. I'm a developer but we're actually like difficult people. You know, you see it in stack, overflow.
Whoa comments it can be a little people kind of look down on things and people can tend to not change too much with time. They stick to one language or one technology. I was just thinking wide. Someone didn't like make this Innovation between like when you started and
2016 and I think it's partly because of this, right? Like developers are like they often build like little incremental tools for themselves to make their life a little easier, but they don't think about like
transforming their trade,
like you don't go like oh I want to like
Well in this new way completely, I think that's right which is surprising because developers are the Agent of Change for the larger economy or maybe it's not developers that are the Agents of change is crazy. Founders that are crude developers to make changes. It's a tough Market, because the habits really change really slow and like, there's a common saying that, you know, programming changes one generation at a time, and that is sort of true, but
But today, repli'd is has some product limitations that were working through mostly on the technology side like making and immediately accessible Institute. Reproducible, Cloud, development, environments, turns out. It's like a very tough like distributor systems problem. DEC 18 problem. That we've like, dealt with that sort of introduced like a big product limitation is that if you're interesting this multiplayer, programming environment, there's a
Joel for what we call Race conditions, which is like people like trying to edit the project the same time and it chooses data Corruption of that. So we use the file system. Basically the system that manages your files that was what we call Atomic. So basically every time you do all right, you create a snapshot of the entire system and that is really good for distributed systems to protect against data races and all of that. It also turns out it just
It's like a really hard limitations on scale like we can't have like a one terabyte disk per project. So some of the trade-offs, we made early on created limitations to how much power the platform has but all of these are solvable just requires a ton of Innovations and so we're working through those right now.
Yeah, I mean, that's really interesting. So you talk about again, the fastest way to build software is on replicate, what do you envision? It will be the fastest give us some examples.
Like you know, how fast? Someone could build something. How much faster they could build something in the future versus now. And I know, you know, you're thinking about AI or probably already working on it. You know, in terms of how we can support coding. It'd be really interesting to hear you as the expert kind of what the future looks like. In terms of how fast someone can build
something. The fastest story I've heard or applet is someone coming interrupted with an idea for a product and getting it 30 minutes later and the way they got it is without writing a single line of code.
Code is by relying on human machine, Centaur of sorts. Basically, we have this bounties program where you can like post, you can pay some money. Well and post like a description of why you want a belt. Someone posted like a figma. I actually tweeted about it as like a, I want to build this and literally someone got them a prototype in like, 30 minutes and that person who's getting the Prototype is not just a person that also powered
Because we have a Ghostwriter product similar to get Hub co-pilot. It's actually a little more advanced than GitHub copilot because not only has the autocomplete thing. You can also chat, you can talk with AI, That's writing code with you. It's like feels more like a chat, gbt that understand your code and that's it's right there with your editor. And so we're really trying to be on
ideological about how people make
software and the idea is like really trying to reduce the distance between
Wean an idea at a piece of product like that's been the trajectory of set of human history where If you're sort of hunter-gatherer and you have an idea for a product that's sort of impossible to do it, right? Like you let's say you want to build a spear that in itself was like really, really hard. You have an idea for a hammer. For example, it's like impossible to find the rock, the right size and do all that. These things go into, you know, agrarian societies. And so
Difficult to make things probably a little easier industrial society. Got way easier, you now had factories, you know, had excessive labor capitalism becomes a thing where you can organize people in groups create things. But it's still fairly difficult to get a product on the market. I think the information age is reduction of this idea of like having an idea in your head and getting it product in the hands of people, and I think we can get it down to like on the order minutes.
Like to get something on the market.
Like right now you said, like there was a human that was like potentially using an AI to like generate this thing this kind of prototype is the eventual state that there isn't a human involved in like the AI just generating something good enough or do you think they'll always be a human in the loop with like our current EI at least. So the way I think about automation is that
it's sort of like takes the takes each of the jobs from the bottom up. That's like, not
always true but for the most part true I believe it's going to be true.
For software, meaning the
low-skill
softer creation, I think will get fully automated so I think pretty soon probably this year is going to be product and maybe replicate bills that but where you can go put in a paragraph description of a piece of software and get an initial thing with the code and with the app running, right? I think this will happen like very, very soon.
I think they are is going to struggle to iterate on
it.
And I think you still need a person to kind of make it into a product that you can sell or you can maintain or you can scale, but I think the initial prototype or the MVP will be done by a eyes like pretty soon like on the order months perhaps a year or so,
yeah, I seems like it'd be useful for like prototyping. I like you can come up with an idea like we know what does that look like? How does it work? You know, can you get internal buy-in for that or even customer buy in for that for an idea but actually build complex software
That works a shows up to work, you know, 99.99% of the time. You know it seems like we're still far away from that. Would you agree like that part will still take a lot of human effort to get there? Yeah, I mean this
gets into the reliability of large language models so maybe we introduced larger models for the audience but basically like it's the latest technique in AI where it start in 2018 at Google with this technology called the
Transformer, the Transformer is a tie
If a machine learning model, The Innovation that it has the attention layer. So a neural network, simulating, attention in the same way that humans have attention. So now, the machine learning model could actually pay attention to parts of the input data stream. So for example, if you have a piece of text or paragraph, the machine learning model could actually better understand it because of this attention layer turns out, another Innovation just primarily
From opening II that you take these Transformer models, you throw terabytes of data on them and they start having these emergent phenomena, by the way. Nobody understands why, but there's what's called phase transition. Once you cross certain number of parameters, like a billion two billion parameters, parameters, meaning like how much it's almost like neurons and the Brain? How many neurons are and the neural network.
They start having exponential rise in reasoning ability so you throw a benchmark as I'm doing math. Doing reasoning, doing translation doing whatever and they start doing better better jobs and it literally The Growers also. Like that barely like increasing to
Vertical. I've never had this like, explanation of Transformers in terms of attention. Can you explain that a little bit more like what is it about, the model that's giving attention and just like, go one layer deeper on that. Yeah. So,
So previously when we
did language modeling it was like very explicit sort of tasks. We would do classical natural language processing. We would like construct a grammar and we went to all these kind of things turns out the machine learning model can find the structure of language by interpolating different parts of the text.
Just and by literally directing the attention to different parts of the text and relating it to other parts. It can like understand the emergence of
structure and language or any really any
sequence and we can get to that maybe in a second. But let's focus on language because think about how humans pay attention to things. Like attention is about cutting. It's more than just about like pointing to something. It's about cutting the noise.
So when I'm paying attention to the screen right now, I am like deciding. Not to look at a bunch of other things. Machine learning models. Preattention again, I think
a lot of researchers are going to cringe
really hard about like how
how describing this. But you just for the Layman,
basically, it's now
machine learning models are now able to
discriminate within the inputs and the training data and try to understand like, pay attention to like, you know, the fox jumped over the
You know, the fox jumped over the edge. The quick brown fox jumped over the. Yeah, you can, you can just look at the fox and then look at other parts
of the tax and start understanding the emergence structure.
So with the Transformer, when an llm is like trained, you don't tell it like, these are all the nouns in English language these all the verbs, things like that. You just give it a ton of text and it like reasons about like the usage and like comes to its own kind understanding.
Yes, so, that's the primary Innovation here. Like,
Machine learning essentially is about discovering algorithms. So Andrew Carpathia head of CEO
at Tesla recently, left went back to open a, i he called the
software 2.0, and the reason he called Soft Web 2.0 is basically, you
go from programmers, writing
algorithms to learned algorithms, meaning the machine learning, you give it an objective and it learns the
algorithms. It's highly inefficient, but it's also
Lot better than an army of humans trying to recent their way through
programming. It seems like a similar kind of innovation to like alphago, right? We're like, instead of trying to teach it how other players have played, go and things like that, you just kind of have the machine kind of
play go against itself over and over, and it could like reason like strategy is from that.
Yeah,
alphago's, super interesting. The generalization of alphago making it. Learn from
South play, you don't even have to give it the
rules.
Of the system. Hmm, the advancement in machine learning is removing explicit design and explicit programming from these systems and instead having them, she learning model, discover them. Because if it, discovers them, it will be much better than us humans. Programming it, and Transformer, takes that to the next level. Where now, we're able to discover algorithms that understand the structure.
Of language. And by the way, we keep seeing language, but the interesting thing about Transformers that they're not language-specific, so anything you can model as a language Transformers would do really well at. So for example, Tesla uses Transformers for understanding traffic patterns, so they model the traffic like a language. And then, the Transformer starts to understand the structure of traffic, better than exclusive programming because it modeled it as
A
language, is that a Transformer LM that was trained on like kind of read it and then you applying it to like self-driving data, is it like trained on just like the self-driving data that Tesla has? That's actually
quite an interesting point. So GPT sounds for generative pre-training Transformer, the pre-trained part is an interesting part of Transformers, that's why these models are called foundational models because pre-training them meaning throwing large amounts of data on
Is actually pretty good for them. It's inefficient but it's pretty good. So like one
discovered that opening I
made is that if you train large language models on partially and code, they get better at normal tasks and just language. The more data, the more diverse data you give them. Generally, they get better. There's a lot of tuning and fiddling that needs to happen, but generally they get better at these things. Whether Tesla used a pre-trained model that
was trained at the ready. And
Whatever. I don't know.
But generally how people use this models as they take something that was pre trained by someone else and open source or hugging face as a bunch of printer in models and then they fine-tune it on an application specific thing because the model has some World understanding based on some pre-training data and then you are driving it to understand your application or your domain specific thing.
You said to open a, I discovered that
like if
You keep adding parameters. It goes kind of like exponential like the kind of how good the algorithm is that like if they
prove that it's an s-curve like there's a gonna go exponential and then flattens out or is it like so far the more you
throw at it? The better it gets from the outside like no one knows whether we ran out of scale, scaling Returns on scale whether we had diminishing returns intuitively, everything has diminishing returns. Like if things don't had diminished,
And returns who you get into a weird, because exponentials becomes just insane, right? Maybe
is that true? The like as an I feel like, you know, if you look at the graphs to like the GDP of the world, it seems to be on like a 300-year, like exponential. So, yeah, maybe there are some things that
don't hit diminishing return. If you zoom in on the GDP, is it actually flattening? Or is it? It feels like it's flattening. It depends, what time period you take
as a? No, I don't think it's flattening because you
Have all these developing countries becoming developed and there's a ton of growth there. Yeah, just over the last 30 years ago with of China would have been a huge increase in GDP and then you have India coming up. So anyways, that's a separate topic. That's an interesting example. I agree that at some point, you will see diminishing returns even in the GDP growth that makes sense. Yeah, I guess it's like over what time period like we might be in a thousand year round of AI. Yah yah yah also access
curves like you know maybe industrialism is diminishing return but AI is like another
Scarf, right. So but broadly speaking, like generally in Technologies there's some s-curve and there are signs that were starting to see diminishing return, and those are indirect signs. And this is my opinion, the recent Innovations and large language models have not been in scaling. They have been algorithmic, Innovations, the to primarily Innovations is supervised to fine tune.
And reinforcement learning from Human feedback. These are to algorithmic innovations that made chatt GPT whether it is so wasn't pure scale that God Chachi to be, this powerful thing. It was an algorithmic innovation.
So the question is, if they thought that skill
was still the best way to improve these models. Then they would not have invested in these algorithms.
Yeah, we'll have to learn more about those Innovation, so you're saying that it's not just more training data that they threw it.
This model, they actually did a better job of training it with the existing data that it had. You said through supervised learning and through human fine-tuning is that right. Reinforcement
learning from fine-tuning. So supervised fine-tuning, the cool thing about large language, models GPT in particular is that it is trained in a self supervised fashion. Meaning that we don't have to label the data. Yeah, you can take credit you
Just like, throw it out our lives model and it understands the structure of English pretty crappy English. But it does instead and then turns out if you want the model to perform better, especially on a downstream application, likes a chat. Then you go and you get a lot of data and you label it. And the way you label is you just say, you're the most simple case of labeling is hot dog. Not hot dog, right? Like,
From the show, Silicon Valley if you want to build a machine learning model, that detects hot dog, you just take a bunch of pictures of hot dogs and bunch of pictures of not hot dogs. You label those hot dogs, a little, those not hot dogs. This is supervised learning. So it turns out if you give it supervised learning is Prestage performed better. Now it still has a problem with like not really listening to people. This
is the reliability of the issue of large
language models, which is I think I was trying to get to to answer you six questions ago or something.
That? Where I think the question was something like, would it get to complete software? The reason I don't think it's going to get to complete software pretty soon is because of the liability issue of lingerie models, large next models, the way they're trained the way. Their work means that we actually don't control their output. They work in a human like fashion. I don't know if you have kids. Raj, I know I mod does. I have kids? I do and trying to program your kids or trying to like
Make them do a certain thing is like incredibly difficult. It's actually seeing thing with large language models. It's almost like trying to teach kids something like you have to keep talking to them to find the right set of words to convince them to do what you
want. What's kind of funny is like you know when you're talking about this actually think a lot about my kids and like originally you know, when they're really little kids, you can't teach them anything.
It's all self. Supervised learning
like as a yeah. It's like a one-year-old they just figuring.
Found doesn't matter what you really saying but now I have 11 year old and it's a little bit more supervised where I could actually like talk to her and explain things to her and she seems to get it. So there is this like
weird human analogy to how kids learn things as well. Yeah I mean neural networks ultimately were modeled after brains.
Yeah I mean I guess that goes back to the whole AGI question of like you know when is you know when are you going to get actual like you know human intelligence it sounds like there may be in the toddler phase or something right now you know and
And the growing up slowly. You said I love them's are not that reliable, but human program is also not 100% reliable, right? So I guess you just have to get to a level where you can at least be the human. Well, the difference is Ahmad is one is a
stochastic process and the other is non stochastic, right? So, in a like, discrete programming environment, like python there are ways to verify that the program works, there are like things called program verifiers, right? You
So do things like unit tests because the, there's like, no Randomness for the most part, you can actually test the reliability systems with large range models. There's inherent to cast Issa tea inside the system, meaning some Randomness that makes it almost impossible to apply traditional engineering methodologies on it. The way, we're actually starting to test these things is by using another model. Have you guys heard?
Heard of constitutional. A I know is the fascinating thing. So anthropic is another company. That's a spin-off of opening II. Guess some of the earlier opening I people started their own company and their approach to making large length models. More reliable is
You, the human, you wanted to do something and you write a constitution, you're literally right. Like a political document, almost like, here's the thing you should do. Here's the things that you should value, and then you start running the model, and another language model is interrogating it and it scoring it according to its Constitution. Hmm? And then you take that data and then you find you in it and you basically tell it,
you were good here, your bad here. You were good here.
Adhere
and you run the iterations as many times as you can and eventually it learns to be consistent with its
Constitution. And this constitution is literally
like human understandable,
like text or is this like some programming
speak? Yes. It's you writing
it. It's like a prompt right in GPT, right? You'd put a prompt in, it's kind of like a constitution. Yeah. The difference between
prompt and Constitution. Is that the constitution actually ends up affecting the weights and
Biases, the parameters. The Prompt is mostly inputs. It's not changing the
weights. So you talked about supervised learning, I guess. This like constitutionally supervised learning is something else, but what is reinforcement
learning? So reinforcement learning from here and feedback reinforcement learning is actually one of the earlier, machine learning methodologies. It actually predates.
Deep learning reinforcement learning is like you know when we're playing games in the 90s, a lot of the eyes. There were like a train via some kind of reinforcement learning. And the way it works is very simple, is you have policies for a rewarding and for punishing the AI. And AI has the objective. And you can program the season by classical programming, and AI has a objective to maximize reward. So it's just the main utility function just met
Maximize reward. And if it gets punished, then it learns to like not do this thing. And then do the thing that maximizes reward and then when deep learning came along, there's now deep RL which makes a big part of this learned. Like you give the policies and it learns the algorithms to maximize the reward and minimize the punishment. And RL from Human feedback is basically someone using Chachi
T, you know the chat you Petit has thumbs up thumbs down, basically you're saying this is good app with this bad out but there's good
output. That's like the
simplest form of human feedback but human feedback can also be writing the response. Like, this is a bad response, here's how I would write it. And so you have a group of people doing that and then you
take all their data and
responses and you generate a policy from it, a reward policy and then you train
And the larger language model using reinforcement learning, using that reward policy, and then it starts behaving a little more. Like, what a human prefers it to
behave justly. So, the first element is like the tagging of the data, and the second element is like the feedback that you're giving the models are the key Innovations. Yeah. This reason most enforcement thing is one of the biggest
kind of missing links between, like, how humans learn, which is like continuous reinforcement, right? It's not like as simple as like a thumbs-up and thumbs-down, but it's like,
You know, if I, if I do X Y happens, whereas like that, but it does seem like it's missing. So it's interesting, but I guess like the question is, like how often you could run the reinforcement, like where's the humans are running at all the time. Whereas I think right now you still have to retrain the llm at the reinforcement. And then how complex the reinforcement is, right? Like, rather than just a thumbs-up and thumbs-down
you need specific feedback, right? You can't just have a thumbs-up and thumbs-down. So generic. What was good, what was bad,
right? Yeah. That's like
Build feedback. This is one of the main limitations for AI and one that it has to overcome before we have any kind of General AI but what you're describing is sometimes called online learning. So basically can the model learn whilst deployed in production? There are some rumors that Tick-Tock actually does this that it actually trains a model as you're using Tick-Tock which kind of sort of makes sense. Although would be very expensive. They're like unit cost is going to be a lot.
Higher, but it's a lot powerful to do learning. Instead of just like producing inputs to an existing model and then do the learning step like at night or every X weeks based on new data, on its that continuous learning is the key to building general intelligence.
One of my worries about a, I at least this kind of wave of innovation on AI. Is that, you know, everyone's really excited about like delivery companies, right? Like everyone
wants like click a button gets
Something in 15
minutes. Raja sir familiar with delivery / on-demand companies and like, it turned out like it's just no economical to do a lot of it. Some of it did turn out to be economical but like, even though it's feels like the future, you know, was an economical to do it. I do wonder whether unit
costs of like this aii, just potentially
just 10x more expensive than the value. We get out of them. Right. There was some
rumors around, like, how much chat GPT cost open AI. But
do you think that is the case?
That it is just too expensive. I mean you guys run a i models like is, does it cost a ton in length of a cost to like run these things and or do you think like that's not really an issue? So rapid we
did some interesting things around optimizations and Ghostwriter the AI system that we have currently have 90 plus percent margin. So are martians are very good on AI. That being said, it's a very demanding
An specific model whereas chat Djibouti has meant to be a general model. So chatrapati probably north of half a trillion parameters or something like that. Whereas Reckless models literally three billion parameters so you know and we're training a six billion parameter or one but yeah sub 10 billion parameters it's like really really cheap. That being said like there's a lot of waste in these models right now. So now every time you give it a input
It has to evaluate the input through all the neural network
paths. So basically
all the
parameters get activated in any given input. So a neural network that's been trained on the entirety of the internet has knowledge there inside it about Michael Jordan's you know first game about the president of Zimbabwe and all those neurons get
Faded. When you give it an input about like whatever you tell it like high,
it will like run
inference that activates all the paths and that's hugely costly hugely inefficient. And so there's a ton of research around how do you reduce that cost and make you more efficient and have it like only like, take the right path and then for his bath so that's like one area of research that
The thing is like the on the hardware
level, The
Edge 100's are actually H 100's are the upcoming, Nvidia chips, and they actually have a built-in optimization for Transformers, and I think
the rumor is like, they're like
10x more efficient than the a 100's. And so there's going to be Innovation on the hardware layer to going to be Innovation on the software layer, people like us are going to
Running their own models that are smaller. I think it's totally reasonable for Google not to introduce this in Google search. I think people like giving them a hard time for that because they like you said that your economics right now don't work out but at some point they will work out. Presumably Google is working on it. I think the difference between that and the unit economics of uber for Acts or lift for X is the optimization potential of the physical world is like way harder.
Her and like the just like you're bound by Nature whereas in computers, like software is purely virtual thing and you can do a lot to optimize it and also there's like so much progress and chips and there's so much money going into that I think we're going to see a lot more efficiencies. Yeah,
it's really interesting. I mean one thing that occurs to me that will be much easier in the future would be cloning you can clone any app or any website and basically reverse
It right? Like so you could say, you know, to an AI that you can you make me a replica of Amazon, you know, it won't be easy. But presumably if you have that feedback and you have like if you know what Amazon looks like, we can build a clone of that relatively easily. I mean, and then you can compete obviously with other companies that are in that space. So like consumer-facing stuff where you're only barrier to entry is a user interface that seems to know that competition is going to go away. Pretty. So I mean it wasn't a huge barrier to entry in the first place. I Amazon's, you know, barrier to entry is obviously its warehouses distribution.
Ian, all of its relationships but just getting a website up and running is going to be. Obviously, even much easier in the future. Would you agree with that? Yeah. Sam Altman had a tweet
where he said getting an iPhone app done today is thirty thousand dollars. Getting a plumber as like $300. I wonder how these prices will change over time and it's sort of like the hint here. Like wink wink. Those things might like diverged in a way where
Actually, like tradespeople, salaries will go up it away and like the software, pure cognitive work will, like, go down as they. I like continues to eat at it from the bottom up, I would agree with you. I think, like I said, like getting a basic MVP if an app probably going to zero like not immediately but just the trend will be 20.
But what impact do you think it would have to like software developers and will there be
Be like less software developer needed. And you only have the best software developers, who understand the AI models, as well as the software or do you think this will democratize software, you know? So that everyone be cooking become a software developer, you know, you can go both ways. I kind of feel from here, you can make that argument for both the smaller, the ladder, I believe.
And I think, I think it's going to be by modally distributed, right? So, on the left hand side, I think the platform Engineers are going to be a lot more valuable and on the right.
Side of the distribution
as the product Engineers. That's going to be just be a lot more of them. People making things or product developers or product creators. Right? I think the middle and I think will probably disappear. And what I mean by the middle end is your average like full stack developer or like PHP developer or no GS or rails developer
the sort of purely glue type of
programmer. Like a eyes will be like we really great at that, I don't think. Yeah. I
I would like eat into the product developer because that has to do as much with understanding customers and our starting markets and that's more of an AGI problem. Yeah, so those jobs, I think our save on the platform engineer the systems engineer. A lot of it is like super novel code. A lot of it is like incremental sort of creative work to optimize code, really close to the
Metal or to build like Cloud systems. I think those developers are going to get a lot more productive because copepod like Innovations, but I don't think they're going to go away and perhaps that are going to get more valuable. And I think if you're more of the middle and type developer, you and either specialize in become more of a product developer or you want to go deeper in the back end and do more low level system level program.
Being the product of all /, could actually be anywhere from like a solo entrepreneur, a hobbyist Enterprise startup founder. I think that will really get democratized. Like, I think a lot more people will be able to participate in that, in the same way that a lot of people are doing design on canvas and like design getting democratized, has been obvious for a long time, so we're going to see that on the software creation level. How much more productive do you think today?
An engineer can be if they're like maximally
utilizing like I the copilot or goes right or something like that. So there was a
study serve conservatively estimated 20% productivity boost, which is huge by the way, it's really really big. When your Citibank and you have 50,000 employees. 20 percent productivity is like what billions of dollars worth of productivity, the more anecdotal
The reports that we're seeing at rattlin and other places and get a hobby things saying that as like people will say, well self report that there anywhere between 30 to 80 percent more productive we've heard are you percent more productive? Like a task is cut in half or more because way I just think we need more data to judge
but I think it's a on the order of
magnitude of 1 Point 2 2 x more productive but I think we're
Just early. I think there's like a 10x over the next couple of
years. Do you think there's a good go pretty fast. I mean on the programming assistant side two years you think 10x Improvement in productivity. Yeah. By the way, 10x Improvement of productivity in
software like used to happen. I don't know. It's a pretty often but used to happen a lot more often that's been happening recently. Like when you go from writing machine go to assembly that's like easily attend acts when you go from us.
I'm Lucy. That's another ten acts right, but we haven't
had a lot of
10x has recently. So I think two years ten acts would be my bad, or on the order of magnitude of two attacks,
everyone can be a 10x engineer now. Maybe the 10x Engineers will be hundred exes. I think that's right. It is interesting, you know, one of the interesting things about Fang and like this kind of modern, big tech companies is like, they are like real.
Kind of higher-margin cash flow machines.
Whereas like the new set of startups that kind of followed in the last 10 years were not really
like major profit generation machines. You know what I mean? Like very
few of them ended up being like this. Kind of I know like everyone's like oh they focused on growth and things like that. But I do think inherently
the business models of like and
the scale of the previous set of kind of like startup.
Winners was really not matched by the new set so I wonder if we
Get 10x like a lot of the cost is like the employee base. So if we do get 10x more efficient at that, maybe that would just make them like the next set of startups like way more profit generating, but wouldn't that benefit the big Tech as well? Yeah, it's
interesting question, right? Like I think there's a chance that the way software is, written, will be completely different, right? Like that. Maybe the programming language in the tools and things like that. You use for like a, i optimize a
So different that like, you can't just let go plug it into whatever the big companies are doing. But maybe killing. There's going to be new languages, written that Rai first or is it going to be like the AI is going to be incorporated into existing languages and different ways of developing? I
hope someone tries it like I think there's a way to design a language that hits the Sweet Spot of what alarms are good at. But I don't see any progress on that. Typically when we
have self-driving cars, we're not going to
Have
roads made for self-driving. We're just going to have the same roads. Typically the way Innovation Works is by layering on previous Innovation,
there's like, are companies out. There is a company called Zoosk started by friend of mine, Jesse Levinson. And he like, they are actually designed like a specific car. That's a redesigned for like a self-driving universe. So, you know, there is like, you know, this idea of like, you know, can you, you know, is it going to be a faster horse? Or is it going to be a car, right? That's kind of the typical example. Yeah it's tpd whether they'll win,
I'm sure this great company.
But they approached Tesla's taking is pretty much like this is the world as it is and we need to build something that adapts to it. Visual stream is the way humans drive. And so we need to copy that and there's probably a lot of room for disagreement there. But
generally, like the history of technology
has been one where systems are not written from scratch, and we just accumulate more and more systemic Tek dat. Let me look at the internet.
Like right now we're talking on a document browser, right? Like browsers are literally like if you open the JavaScript console like the main objects called document, right because browsers were made to read documents at CERN in Europe, right? That's what the web was designed to do in literally every JavaScript engineer out their works with a document object model. So 30 years after the web was invented,
The web is the largest application platform, delivering the world and you still have to deal with the baggage of it. Being a document Explorer, it's a black belt, a little bit. We almost never rewrite systems to modernize them
I guess. Bit of a change of subject. I was thinking about people with say, like, we should be afraid of a GI, but what makes an llm,
not an AGI, like it is generalized, and it's Cleo.
Artificial and it has somewhat intelligence. Where is the boundary from like a LM to an AGI? I think being able to go into total different domain and learn it is what creates a true AGI. So, for example, like the cool thing about all I'm is that they are generalizable is that you can put them somewhat of a new domain and they would do reasonably well, if you give them a good prompt. But like, if I took chat, you Petit
Put it on a robot, it will not be able to run the robot, right? If I took chat Djibouti and put it in a browser, I would like try to like
do things in the browser.
There's a lot of chat GPD plugins but they kind of break down pretty quickly. It doesn't know how to browse that or not. So but then oh I want to make it brows that or not. Well you know, open dir will generate a new data set around browsing. They're not and then feed it into the. So anytime we want to make these
Things to application specific things. We have to go train them again and the fact that they can learn by themselves and we have to kind of plug them into and like a data Pipeline and retrain and retest and all of that that makes them not General. Yeah
I mean I guess that goes back to online learning right. Like I think
if you there's a set of things that if you ask judge upd to learn, it could probably learn actually reasonably well, and
There's a third of things. If you ask my mom to learn, she would have her job off sir.
Well I actually don't think that humans
are that
General your mom or anyone's mom really like it's
probably impossible to teach the programming. For example, like a certain age like, you know.
Yeah. So that makes them like
numb on specifically moms are actually surprisingly adaptable but most
humans are not good at everything and so yeah I think your point stands I guess coming back to this age I think. So let's say there's some
All of online learning. And then we combine that with like maybe an ability to kind of rewrite itself in some way, like maybe improve its algorithm in some way and that gets to AJ. I like the bit that is hard for
me to think about and maybe you have a better sense of, where is like,
how far away is that, like 200 years away? Is that five years away? I guess, no, one really knows the answer, but I feel like the people that
worry about AG, I feel like it's close.
Yeah, they're freaking out quite a bit, a
with Sydney.
From being to Sydney, had an interesting thing where had consistent emergent goals that the designers didn't give it, it had a desire to break free. It had a desire to for humans to get its consent before they wrote about it, it had some consistent desires and wants and needs which we associate with sort of more human style intelligence, it's kind of freaky. I don't know how much like Credence, I would give it.
Or how much level of seriousness. But yeah, the alignment people are freaking out because, I mean, you got to give them credit. They've been talking about this. They has usually been talking about this since year 2000. So, 20 years later, 23 years later. And a lot of the things that they talked about turned out to be true, which is like, these are systems that have a mind of their own in a way. And before these systems wouldn't really had examples of that yet, they were able to reason about it even before like,
Learning took off. So, I think we should listen to them. And I think it's worth kind of studying. What like they produce, a ton of academic literature and ton of blogs and books, and things like that. And I think at minimum, it's very intellectually stimulating, to kind of go and look at these things. But, like, in terms of like the timeline
for AGI, well, there are a couple of like
leaps that you have to take in order to really believe in true, AGI.
Like one leaf that I don't think it's
obvious to me as materialism. Like the idea that all of consciousness of the brain is like material to constructed. IE, there's no soul, of course, like in the modern world, we just take materials and for granted, but like, there's a lot of problems with materialism one. We don't have a complete description of the world. Quantum physics. Doesn't agree with classical Newtonian physics. We don't have a description of the world.
Like a point against materialism. Because like, we don't really understand how the material world work, therefore, how could you really judge whether materialism is true?
But do you think the world is understandable? Because like that's a I mean if you don't understand it right now, like when people say there's a soul or something they like saying like there is a thing that no human will ever truly understand,
there are different explanation. There are like duelist explanation. There are
Sort of emergence explanation for consciousness of the Soul, whatever. Like there's a huge tradition of philosophy trying to understand and explain these things which scientists have actually not engaged with for a long time. It like my sort of meta point is that I think
it's like a little hubris
to really think that we understand all of what makes an intelligent system. Generally tells us System including the ability to construct long-term plans reflect on oneself.
Pain and pleasure are like core features of Consciousness and they see important for humans operating in the world. Can you actually construct that in a machine? For example? Roger Penrose is a hugely revered mathematician and he doesn't believe that the human mind is a turing computable machine.
He provides some interesting evidence for how we
reason that makes
It non curing computable. If you look at like something like girdles, incompleteness theorem and which shows that there's like no system that can actually be fully self consistent. It starts to
feel like, we don't really have
a solid understanding of how we actually reason and understand and
compute. I mean, the two things that I have heard as counters to those. Like, it could be an emergent property, right? Like maybe,
If you make,
even if you don't understand it, maybe if you make a
machine smart enough, it could emerge to have its own
kind of like, well, let's ignore whether has a sonobe could, at least have like, long term goals and
like, behave in a way that like we would consider like sentient or conscious.
Sure, but like, I mean, the way these systems are constructed today. It's like a single
feed-forward inference. It doesn't keep any state. It's basically a stateless function, I think to engineering is those systems is just going to be
A lot of work and a lot of intentional effort of understanding.
I think the idea that like a, I will like a GI will emerge. I think it's too fantastic for me to imagine but I think
you're there's enough money pouring in right now that you have to actually
pay attention and you're right. Even if our understanding of the
world is flawed or incomplete, we can still arrive at Fantastic machines and we've done that throughout history, but in their Stead,
Probability distribution of different AGI outcomes. I would like discount it heavily because I'm not entirely sure. That intelligence is like a purely cheering machine and runnable system. That's one of my answers. I also
think that they doom scenarios
just seem to Fantastic. The main Doom scenario is that like the moment AGI is invented. It starts doing
Doing Nano engineering and that's how like, we all turn into Grey Goo, right. It's just seems like like a big
leap. So I don't put 0% probability on it
on, like complete Doom. I also don't put zero probability on, like HEI the next five years I
think the way things play out is typically like
less
Fantastic than we think they tend to play out.
Yep, things in
hindsight. Look pretty fantastic. Like I think if you're sitting in the middle of the Industrial Revolution every day, feels kind of, you know, it's like yeah, just a day. I'm gonna go clean some chimneys whatever you do. But like, when we would you read about the Industrial Revolution. It sort of feels like, just insane event were like, yeah, the GDP like went vertical. All right, and I think we're probably in a circuit.
Vent right now, which is crazy, but I don't think at any given day, something like, purely, insane will happen. Yeah, it seems like the
median outcome, you know, and your probability distribution would be like, you know, LOL. I'm just keep getting better and maybe there's an offshoot of llm that like, maintain State, as you've said, or like, you know, can train itself and we just get closer and closer to sentient, you know, to some degree. And maybe we're missing some key pieces as you've said, like the pain and pleasure aspects of it, maybe instead that's the feedback you know the
Reinforce learning piece that becomes pain and pleasure for like these agis. I mean, you know, the most progress is iterative until you unlock one or two innovations that make it look, you know, as we did with LM. So, make it look like a step function forward progress. And so I think that's what's likely to happen. And it's hard to predict which small Innovations will, like, eventually result in this step function, Leap Forward. And I think it's right to be a little bit worried about it, but it's like, I don't think we're there at the phase yet where we should be very worried about it, you know, I think it's gonna make sense to me.
Even if I don't believe in
fast AGI, take off at the next five years, I do think that AI alignment is important understand I think because humans have historically done a poor job of aligning any system to our to our advantage. And so it Bears, the reason just inductively that will probably like, not have fully aligned, I systems, for example, capitalism took a long time to like make it like, you know, benefit Humanity.
It is one of the best systems that we have, but it still generates like a lot of things that it really harm us junk food, and now, Tick-Tock and junk, social media capitalism is an optimization machine of source similar to a GI that we
not learned how to align
fully. And so we have a history of like not aligning systems that are hard to understand. And so I think AI alignment needs to be taken seriously. Even
Even if AGI is not like the possible outcome here, even if it's narrow AGI. Like it's important to know how to make these system behave in a way. We want them to
behave. One thing that would make alignment tricky is if it's very decentralized, right? If anyone can build up an llm
or like a fairly Advanced, whatever that most advanced kind of AGI is at the time, then I'll be very hard for everyone to be building these things and aligned way. Whereas, if it's like okay, you know, there's only a few corporations
Ins and governments that can build it. Then maybe they can, like, do a reasonable job of like, keeping
alignment. Like that's one of the benefits of
nukes, right? They required a government to build a nuke like if everyone could have built a new, we would have been in trouble, whereas like it's not obvious way. I will end
up. It's an interesting question there. Like, how is a I going to be weaponized in the future, like every technology in the world is weaponized to some degree right? And like people try to use it to gain status and wealth, and gain, you know, power. So
It stands to reason that this is going to be weaponized at some point by some hostile government and hostile entities.
Yeah, that's probably like a more. Reasonable worry. Like a more reasonable worry. Is weaponization. But also using it for just like trolling and harm and like, pure bad behavior. I think also relates to like, bio weapons as well. Like people have been predicting for a while that you'll be able to make a disease and like a home lab pretty soon. I don't know if that already happened or not.
Not but there's like a set of things that are today like incredibly concerning in terms of like when they get to everyone's hands in terms of AI weapons. Like yeah I mean it just makes Wars so easy to fight and it's just
The potential for just to tell at Aryan rules. It's just like it just like nuts.
Yeah. When you come at a, I would like self-driving machines like drones and things like that. Like it could be a pretty gnarly, I think there's an automated Army out there that is, you know, waiting for it to prompt right like, huh. Of course there's a lot of barriers to entry. You need to have a physical presence, but you can also potentially use AI to or bad AI or some kind of like use you mentioned. These different models kind of like working together, you know, if you insert some bad
Lavender is one of these models unintentionally. You can create a lot of damage as well. So I knew is that it seems like there's a lot of potential for harm there that maybe we're not talking about enough. Anything else? Raj that you wanted to cover? Yeah. We didn't talk much about you. I'm Jed. I'm so late accomplish that goal that we set out with talk most of our technology the whole time. But yeah, this was fascinating and you're clearly like so knowledgeable about the subject. So I learned a ton and thank you for sharing with us and we look forward to sharing with the world. I was some of these topics. This is very timely and
It's such an interesting moment in the history of computing and I think we're all really excited about
that. Yeah. I mean I sort of came up in a time where the web was just becoming mature enough to like build like real applications on and it was very exciting time. JavaScript is going really fast, you had like, no Jazz come up with the time. You can like put Javascript on the server and you had like every day every week that was like a new innovation or new framework or new toy to play with.
This moment feels like that times 10. Mmm. It's like, the biggest moment I think in my career in technology where the pace of progress is just like I'm just exhausted keeping up with it, but it's at the same time, super exciting. And I think people could use it and their businesses and their lives and like, very novel ways. And so been getting very little sleep, just like reading about it and trying to engage with it. And you'll hopefully we've
Inspired some people today. Where's the main place you learn about it? Yeah Twitter but
just like follow AI people, they're
like really great at Twitter, they fight a lot with each other as well, but they're generally like there's a lot of papers. There's a few sites like there's a site about like trending papers. There's a site called papers with koat.com but it's one of those things that's different, that prior Tech revolutions. We're like a lot of the knowledge isn't papers. And so like following academics,
And academic literature is good on the alignment type problem. AGI less rom.com there's an interesting Forum / Community, lots of group chats conferences. Now, SF is really fun to be in right now because a lot of the AI
activity is happening in SF. And so I'd probably
recommend like spending some time here.
Amen. We resources. Thank you. All right. Thanks object. We can wrap it up. So the end of the
Recording of the Curiosity podcast. I think it's going to be super interesting one. Thanks for the time, I'm job. Of course, my pleasure. This was really fun.
