George Hotz: Comma.ai, OpenPilot, and Autonomous Vehicles | Artificial Intelligence (AI) Podcast
the following is a conversation with George Hotz he's the
founder of comma AI a machine learning based vehicle automation company he is
most certainly an outspoken personality in the field of AI and technology in
general he first gained recognition for being the first person to carry or
unlock an iPhone and since then he's done quite a few interesting things at the
intersection of hardware and software this is the artificial intelligence
podcast if you enjoy it subscribe on YouTube give it five stars on iTunes
supported on patreon or simply connect with me on Twitter at lex friedman
spelled fri d-m a.m. and i'd like to give a special thank you to Jennifer from
Canada for her support of the podcast on patreon merci beaucoup Jennifer she's
been a friend and an engineering colleague for many years since I was in grad
school your support means a lot and inspires me to keep this series going and
now here's my conversation with George Hotz do you think we're living in a
simulation yes but it may be unfalsifiable what do you mean by unfalsifiable so
if the simulation is designed in such a way that they did like a formal proof
to show that no information can get in and out and if their hardware is
designed to for the anything in the simulation to always keep the hardware in
spec it may be impossible to prove whether we're in a simulation or not so
they've designed it such there's the closed system you can't get outside of the
system well maybe it's one of three worlds we're either in a simulation which
can be exploited we're in a simulation which not only can't be exploited but
like the same things too about VMs I'm a really well-designed VM you can't even
detect if you're in a VM or not that's brilliant so where it's yeah so the
simulation is running in a virtual machine but now in reality all VMs have
wasted the fact that's the point I mean is it yeah you've done quite a bit of
hacking yourself and so you should know that really any complicated system will
have ways in and out so this isn't necessarily true going forward I spent my
time away from comma I learned cock and said dependently typed like it's a language
for writing math proofs and if you write code that compiles in a language like
that it is correct by definition the types check it's correct and so it's
possible that the simulation is written in a language like this in which case
yeah yeah but that can't be sufficiently expressive a language like that all weekend
it can be yeah okay well so all right so the simulation doesn't have to be
tiring complete if it has a scheduled end date looks like it does actually with
entropy and you know I don't think that a simulation that results in something
as complicated in universe would have a formal proof of correctness right as as
possible of course we have no idea how good their tooling is and we have no
idea how complicated the universe computer really is it may be quite simple
it's just very large right it's very it's definitely very large but the
fundamental rules might be super simple yeah Conway's gonna live kind of stuff
right so if you could hack it so imagine the simulation that is hackable if you
could hack it what would you change about the you know like how would you
approach hacking a simulation the reason I gave that talk I by the way I'm not
familiar with the talk he gave I just read that you talked about escaping the
simulation yeah like that so maybe you can tell me a little bit about the theme
and the message there - it wasn't a very practical talk about how to actually
escape a simulation it was more about a way of restructuring an us-versus-them
narrative if we continue on the path we're going with technology I think we're
in big trouble like as a species and not just as a species but even as me as an
individual member of the species so if we could change rhetoric to be more like
to think upwards like to think about that we're in a simulation and how we
could get out already we'd be on the right path what you actually do once you do
that while I assume I would have acquired way more intelligence in the process
of doing that so I'll just ask that so the the thinking upwards what kind of
ideas what kind of breakthrough ideas do you think thinking in that way could
inspire and what did you say upwards upwards into space are you thinking sort
of exploration in all forms the space narrative that held for the modernist generation
doesn't hold as well for the postmodern generation what's the space narrator
we're talking about the same space the dimensional space like going a little
ace is like building like yuan mosque like we're gonna build rockets we're
gonna go to Mars we're gonna colonize the universe and the narrative your
friend was born in the Soviet Union you're referring to the race to space the
race to space explore okay that was a great modernist narrative it doesn't seem
to hold the same weight in today's culture I'm hoping for good postmodern
narratives that replace it so think let's think so you work a lot with AI so
the eyes one formulation of that narrative there could be also I don't know how
much you do in VR and they are yeah that's another eye I know less about it but
every time I play with it and our research is fascinating that virtual world
are you are you interested in the virtual world I would like to move to a
virtual reality in terms of your work no I would like to physically move there the
apartment I can rent in the cloud is way better in the apartment I can rent in
the real world well it's all relative isn't it because others will have very nice
departments too so you'll be inferior in the virtual world that's not how I
view the world right I don't view the world I mean it's very like like almost
zero-sum issue a to view the world say like my great apartment isn't great
because my neighbor has one - no my great apartment is great because like look
at this dishwasher man yeah you just touch the dish and it's washed right and
that is great in and of itself if I have the only apartment or if everybody had
the apartment I don't care so you have fundamental gratitude the the world
first learned of Geo ha George Hotz in August 2007 maybe before then but
certainly in August 2007 when you were the first person to unlock carry unlock
an iPhone how did you get into hacking what was the first system you discovered
vulnerabilities for and broke into so that was really kind of the first thing I
had I had a book in in 2006 called grey hat hacking and I guess I realized that
if you acquired these sort of powers you could control the world but I didn't
really know that much about computers back then I started with electronics the
first iPhone hack was physical card work um you had to open it up and pull an
address line high and it was because I didn't really know about software
exploitation I learned that all in the next few years and I got very good at it
but back then I knew about like how men chips are connected to processors and
he knew about software and programming he didn't didn't know I'll really see you
the view of the world and computers was physical was the most hard work actually
if you read the code that I released with that in August 2007 it's atrocious the
language was it a C say yes and in a broken sort of state machine SC I didn't know
how to program man so how did you learn to program what was your journey cuz I
mean we'll talk about it you've live streams from your programming man this is
a chaotic beautiful mess how did you arrive at that years and years of practice
I interned at Google after the summer after the iPhone unlock and I did a contract
for them where I built hardware for for Street View and I wrote a software
library to interact with it and it was terrible code and for the first time I
got feedback from people who I respected saying you know like don't write code
like this now of course just getting that feedback is not enough the way that I
really got good was I wanted to write this thing like that could emulate and
then visualize like armed binaries because I wanted to hack the iPhone better
and I didn't like that I couldn't like see what that I couldn't single step
through the processor because I had no debugger on there especially for the low
level things like the boot ROM in the bootloader so I tried to build this tool
to do it and I built the tool once and it was terrible I built the tool second
times it was terrible I built the tool third time this by the time I was at
Facebook it was kind of okay and then I built the tool fourth time when I was a
Google intern again in 2014 and that was the first time I was like this is
finally usable how do you pronounce this kira-kira yeah so it's essentially the
most efficient way to visualize the change of state of the computer as the
program is running that's what I mean by debugger yeah it's a timeless debugger
so you can rewind just as easily as going forward think about if you're using
gdb you have to put a watch on a variable if you want to see if that variable
changes and Kure you can just click on that variable and then it shows every
single time when that variable was changed or accessed think about it like get
for your computers uh the run lock so there's like a deep log of of the state
of the computer as the program runs and you can rewind why isn't that maybe it
is maybe you can educate me what isn't that kind of debugging used more often
ah because the tooling is bad well two things one if you're trying to debug
chrome chrome is a 200 megabyte binary that runs slowly on desktops so that's
going to be really hard to use for that but it's really good to use for like
CTFs and for boot roms and for small parts of code so it's it's hard if you're
trying to debug like massive systems what's the CTF and what's the boot ROM the
boot ROM is the first code that executes it's the minute you give power to your
iPhone okay and CTF were these competitions that I played capture the flag to
capture the flag I was going to ask you about that what are those LaVette I
watched a couple videos on YouTube those look fascinating what have you learned
about maybe at the high level of vulnerability of systems from these
competitions the like I feel like like in the heyday of CTFs you had all of the
best security people in the world challenging each other and coming up with new
toy exploitable things over here and then everybody okay who can break it and
when you break it you get like there's like a file on the server called flag
and then there's a program running listening on a socket that's vulnerable so
you write an exploit you she'll and then you cat flag and then you type the
flag into like a web-based scoreboard and you get points so the goal is
essentially to find an exploit in the system that allows you to run shell to
run arbitrary code on that system that's one of the categories that's like the
PO noble category vulnerable yeah horrible it's like you know you pwned the
program you are it's a program yeah yeah you know for personally I apologize
I'm gonna I'm gonna say it's because I'm Russian but maybe you can help educate
me some video game like misspell to own way back in the Mia and there's just I
wonder if there's a definition I'll have to go to urban dictionary for it okay
so what was the heyday seat yeah by the way but was it what decade are we
talking about I think like I mean maybe I'm biased because it's the era that
that that I played but like 2011 to 2015 because the modern CTF scene is
similar to the modern competitive programming scene you have people who like do
drills you have people who practice and then once you've done that you've
turned it lesson to a game of generic computer skill and more into a game of
okay you memorize you you drill on these five categories and then before that
it wasn't it didn't have like as much attention as it had I don't know they
were like I won $30,000 ones in Korea for one of these competitions oh crap
they were they were that so that means I mean money's money but that means
there was probably good people there exactly yeah are the challenges human
constructive or are they grounded in some real flaws and real systems usually
they're human constructed but they're usually inspired by real flaws what kind
of systems are imagined is really focused on mobile like what has vulnerabilities
these days is it does primarily mobile systems like Android everything does No
yeah of course the price has kind of gone up because less and less people can
find them and what's happened in security is now if you want to like jailbreak
an iPhone you don't need one exploit anymore you need nine nine chained
together what women yeah Wow okay so it's really so what's the but what's the
benefit speaking higher level philosophically about hacking I mean it sounds
from everything I've seen about you you just love the challenge and you don't
want to do anything you don't want to bring that exploit out into the world and
doing the actual let it run wild you just want to solve it and then you go on
to the next thing oh yeah I mean doing criminal stuffs not really worth it and
I'll actually use the same argument for why I don't do defense for why I don't
do crime if you want to defend a system say the system has ten holes right if
you find nine of those holes as a defender you still lose because the attacker
gets in through the last one if you're an attacker you only have to find one
out of the ten but if you're a criminal if you log on with a VPN nine out of
the ten times but one time you forget you're done because you're caught okay
because you only have to mess up once to be caught as a criminal yeah that's
why I'm not a criminal but okay let me uh that's having a discussion with
somebody just at a high level about nuclear weapons actually why we're having
blowing ourselves up yet and my feeling is all the smart people in the world
look at the distribution of smart people smart people are generally good and
then this other person I was talking to Sean Carroll the physicist and you were
saying no good and bad people are evenly distributed amongst everybody my sense
was good hackers are in general good people and they don't want to mess with
the world what's your sense I'm not even sure about that like I have a nice
life crime wouldn't get me anything but if you're good and you have these skills
you probably have a nice life too right like you can use the father things but
is there an ethical is there some is there a little voice in your head that says
well yeah if you could hack something to where you could hurt people and you
could earn a lot of money doing it though not hurt physically perhaps but
disrupt her life in some kind of way it is there a little voice that says um what
two things one I don't really care about money so like the money wouldn't be an
incentive the thrill might be an incentive but when I was 19 I read crime and
punishment right that was another that was another great one that talked me out
of ever really doing crime Oh cuz it's like that's gonna be me I'd get away
with it whatever just went in my head even if I got away with it you know and
then you do crime for long enough you'll never get away with it that's right in
the end that's a good reason to be good I wouldn't say good I just say I'm not
bad you're a talented programmer and a hacker in a good positive sense of the
word award you've played around found vulnerabilities in various systems what
have you learned broadly about the design of systems and so on from that from
that whole process you learn to not take things for what people say they are but
you look at things for what they actually are yeah I understand that's what you
tell me it is but what does it do man and you have nice visualization tools to
really know what it's really doing oh I wish I'm a better programmer now than I
was in 2014 I said Kira that was the first tool that I wrote that was usable I wouldn't
say the code was great I still wouldn't say my code is great so how was your
evolution as a programmer except practice he went he started with C at which
point did you pick up Python because you're pretty big and Python though now
yeah in uh in college I went to Carnegie Mellon when I was 22 um I went back
I'm like I'm gonna take all your hardest CS courses we'll see how I do right
like did I miss anything by not having a real undergraduate education took
operating systems compilers AI and they're like a freshman reader math course
and operating says some of these some of those classes you mentioned actually
they're great at least one the 2012 circuit 2012 operating systems and compilers
we're two of the best classes I've ever taken my life because you write an
operating system and you write a compiler I wrote my operating system in C and
I wrote my compiler in Haskell but classical well somehow I picked up Python
that semester as well I started using it for the CTS actually that's when I
really started to get into CTF and CTF you're all to race against the clock so
I can't write things and say oh there's a clock component so you really want to
use the programming language you can be fastest than 48 hours pone as many of
these challenges you can pone yeah you got like a hundred points a challenge
whatever team gets the most you were both the Facebook and Google for a brief
stint yeah well the project zero actually at Google for five months where you
develop kara what was project zero about in general speak what what just
curious about the security efforts in these companies well product zero started
the same time I I went there what what years are there 2015 2015 so that was
right at the beginning of project it's small it's Google's offensive security
team I'll try to give I'll try to give the best public facing explanation that
I can so the idea is basically these vulnerabilities exist in the world nation
states have them some high powered bad actors have them sometime people will
find these vulnerabilities and submit them in bug bounties to the companies but
a lot of the companies don't really care it only fix the bug there's no it
doesn't hurt for there to be a vulnerability so project zero is like we're
gonna do it different we're going to announce a vulnerability and we're going
to give them 90 days to fix it and then whether they fix it or not we're gonna
drop the drop the zero day oh wow we're gonna drop the weapon that's so cool
that is so cool I love that deadlines though that's so cool give him real
deadlines yeah and I think it's done a lot for moving the industry forward I
watched your coding sessions on the stream downline you code things up basic projects
usually from scratch I would say sort of as a programmer myself just watching
you that you type really fast and your brain works in both brilliant and
chaotic ways I don't know if that's always true but certainly for the live streams
so it's it's interesting to me because I'm more I'm much slower and systematic
and careful and you just move I mean probably an order of magnitude faster some
curious is there a method to your madness is this just who you are there's pros
and cons there's pros and cons to my programming style and I'm aware of them like
if you ask me to like like get something up and working quickly with like an
API that's kind of undocumented I will do this super fast because I will throw
things at it until it works if you ask me to take a vector and rotate it 90 degrees
and then flip it over the XY plane I'll spam program for two hours and won't
get it all because it's something that you could do with a sheet of paper think
through design and then just you really just throw stuff at the wall and you
get so good at it that it usually works I should become better at the other
kind as well sometimes I'll do things pathetically it's nowhere near as entertaining
on the twitch streams I do exaggerate it a bit on the edge games as well the
twitch streams I mean what do you want to see a game or you want to see actions
permit me right I'll show you a PM for programming yes I recommend people go to
I think I watched I was probably several hours you put like I've actually left
you programming in the background while I was programming because you made me
you it was it was like watching a really good gamer it's like energizes you
because you're like moving so fast it so it's it's awesome it's inspiring and
so it made me jealous that like because my own program is inadequate in terms
of speed Oh as I was like so I'm twice as frantic on the live streams as I am
when I code without oh it's super entertaining so I I wasn't even paying
attention to where you were coding which is great it's just watching you switch
windows and VAM I guess is driven screen I've developed a workflow Facebook and
talk about how do you learn new programming tools ideas techniques these days
what's your like methodology for learning new things so I wrote for comma the
distributed file systems out in the world are extremely complex like if you
want to install something like like like Saif Saif is I think the like open
infrastructure to should be a file system or there's like newer ones like seaweed
FS but these are all like 10,000 plus line projects I think some of them are
even 100,000 line and just configuring them as a nightmare so I wrote I wrote
one um it's 200 lines and it's it uses like nginx to the live servers and has
low master server that I wrote and go and the way I go this if I would say that
I'm proud per line of any code I wrote maybe there's some exploits that I think
are beautiful and then this this is 200 lines and just the way that I thought about
it I think was very good and the reason it's very good is because that was the
fourth version of it that I wrote and I had three versions that I threw away
you mentioned you see go I ready go yeah and go so is that a functional
language I forget what goes they go is Google's language right I'm a functional
it's some it's like in a way it's C++ but easier it's it's strongly typed it
has a nice ecosystem erotic when I first looked at it I was like this is like
Python but it takes twice as long to do anything yeah now that I've open pilot
is migrating to sea but it still has large Python components I now understand
why Python doesn't work for large code bases and why you want something like Oh
interesting so why why doesn't Python work for so even most speaking for myself
at least like we do a lot of stuff basically demo level work with autonomous
vehicles and most of the work is Python yeah why doesn't Python work for large
code bases because well lack of type checking is a big errors creeping yeah and
like you don't know the compiler can tell you like nothing right so everything
is either you know like like syntax errors fine but if you misspell a variable
and Python the compiler won't catch that there's like linters that can catch it
some other time there's no types this is really the biggest downside and then
will Python slow but that's not related to it well maybe the kind of related to
its that's lacking so what's what's in your toolbox these days is a Python what
else go I need to move on something else but my adventure interdependently type
languages I love these languages they just have like syntax from the 80s what do
you think about JavaScript yes thanks Nick tomorrow typescript javascript is
the whole ecosystem is unbelievably confusing NPM updates a package from zero
to two to zero to five and that breaks your babble linter which translates your
es5 into es6 which doesn't run on so why do I have to compile my JavaScript
again huh it may be the future though if you think about I mean I've embraced JavaScript
recently because just like I've continually embraced PHP it seems that these
worst possible languages live on for long is that cockroaches never die yeah
well it's in the browser and it's fast it's fast yeah it's in the browser and
compute mites they become you know the browser it's unclear what the role the
browser's in terms of distributed computation in the future so javascript is
definitely here to stay yeah interesting if Tom's vehicles will run on
JavaScript one day I mean you have to consider these possibilities well all our
debug tools are JavaScript we actually just open-source them we have a tool
Explorer which you can annotate your dis engagements and we have tool cabana
which lets you analyze the canned traffic from the car so basically any time
you're visualizing something about the log you using javascript yeah well the
web is the best UI toolkit by far yeah um so and then you know what you're
voting in JavaScript we have a react guy he's good he acts nice let's get into
it so let's talk to Thomas vehicles you found it comma a let's at a high level how did you get into
the world the vehicle automation can you also just for people who don't know
tell the story of comma yeah sure so I was working at this AI startup and a
friend approached me and he's like dude I don't know where this is going but
the coolest applied AI problem today is self-driving cars I'm like well
absolutely do you want to meet with UI mosque and he's looking for somebody to
build a vision system for auto pilot this is when they were still on ap one
they were still using mobile I kneel on back then was looking for a replacement
and he brought me in and we talked about a contract where I would deliver
something that meets mobile eye level performance I would get paid twelve
million dollars if I could deliver it tomorrow and I would lose 1 million
dollars for every month I didn't deliver yeah so I was like ok this is a great
deal this is a super exciting challenge you know what even if it takes me 10
months I get two million dollars it's good maybe I can finish up in five maybe
I don't finish it at all and I get paid nothing and I'll work for twelve months
for free so maybe I just take a pause on that I'm also curious about this
because I've been working on robotics for a long time and I'm curious to see a
person like you just step in and sort of somewhat naive but brilliant right so
that's though that's the best place to be because you basically full-steam take
on a problem how confident how from that time because you know a lot more now
at that time how hard do you think it is to solve all of autonomous driving I
remember I suggested to Elon in the meeting I'm putting GPU behind each camera
to keep the compute local this is an incredibly stupid idea I leave the meeting
10 minutes later and I'm like I could have spent a little bit of time thinking about
this problem was I would just send all your cameras to one big GPU you're much
better off doing that oh sorry you said behind every camera you have a small
GPU I was like oh I'll put the first few layers of my comm there Oh like why
did I say that that's possible it's possible but it's a bad idea it's not
obviously a bad idea pretty obvious but whether it's actually a bad idea or not
I left that meeting with Elon like beating myself up I'm like why did I say something
stupid yeah you haven't given I'm at least like thought through every aspect
yes he's very sharp too like usually in life I get away with saying stupid
things and then kind of course alright right away he called me out about it and
like usually in life I get away with saying stupid things and then like people
will you know people a lot of times people don't even notice and I'll like
correct it and bring the conversation back but with Elon it was like nope like
okay well that's not at all why the contract fell through I was much more
prepared the second time I met him yeah but in general huh how hard did you
think it is like 12 months is uh-oh is it tough timeline oh I just thought I'd
clone mob like you three I didn't think I'd solve level five self-driving or
anything so the goal there was to do lane-keeping good good link keeping I saw
my friend showed me the outputs from a mobile I in the office from a mobile I was
just basically two lanes at a position of a lead car mm-hm like I can I can
gather a dataset and train this net in in weeks and I did well first time I
tried the implementation of mobile I and the test I was really surprised how
good it is it's quite incredibly good because I thought it's just because I've
done a lot of computation I thought it'd be a lot harder to create a system
that that's stable so I was personally surprised you know have to admit it because
I was kind of skeptical before trying it because I thought it would go in and
out a lot more it would get disengaged a lot more and it's pretty robust so
what how how hard is the problem we need to when you tackled it I think a p1
was great like Elon talked about dis engagements on the 405 down in LA we'd
like the lane marks were kind of faded and the mobile eye system would drop out
uh like I had something up and working that I would say was like the same
quality in three months same quality but how do you know you you say stuff like
that yeah confidently but you can't and I love it but well the question is you
can't you're kind of going by feel because he not solely absolutely like like I
would take I hadn't I borrowed my friends Tesla yeah I would take ap one out
for a drive yeah and then I would take my system out for a dry and seems
reasonably like the same so the four or five how hard is it to create something
that could actually be a product that's deployed I mean I've read an article or
you on this respondent said something by you saying that to build autopilot is
is more complicated than a single George Hotz a level job how hard is that job
to create something that would work across the globe Lee what are the global
ease the challenge but Elon followed that up by saying it's gonna take two
years in a company of ten people yeah and Here I am four years later with a
company of twelve people and I think we still have another two to go two years
so yeah so what do you think what do you think about the hottest is progressing
with autopilot v2 v3 I think we've kept pace with them pretty well I think
navigator autopilot is terrible we had some demo features internally of the
same stuff and we would test it and I'm like I'm not shipping this even as like
open-source software to people what do you think is do Consumer Reports does a
great job of describing it like when it makes a lane change it does it worse
than a human you shouldn't ship things like autopilot open pilot they Lane keep
better than a human if you turn it on for a stretch of highway like an hour
long it's never gonna touch a lane line human will touch probably a lane line
twice you just inspired me I don't know if you're grounded and data on that I
read labor okay but no but that's interesting uh I wonder actually how often we
touch Lane lines in general like a little bit cuz it is okay I could answer
that question pretty easily with the common data side yeah I'm curious I've
never answered it I don't know yeah I just - is like my person it feels right
that's interesting because every time you touch the lane that's the source of a
little bit of stress and kind of lane-keeping is removing that stress that's
all to me the big the biggest value-add honestly is just removing the stress of
having to stay in lane and I think honestly I don't think people fully realize
first of all that that's a big value add but also that that's all it is and
that not only I find it a huge value add I drove down when we moved to San Diego
I drove down our Enterprise rent-a-car and I missed it so I missed having the
system so much it's so much more tiring to drive without it it's it is that
Lane centering that's the key feature yeah and in a way it's the only feature
that actually adds value to people's lives and autonomous vehicles today way
mode does not add value to people's lives it's a more expensive lower slower
uber maybe someday it'll be this big cliff where it adds value but I don't
usually do this vessei I haven't talked to is that this is good because I
haven't I have intuitively but I think we're making it explicit now I I
actually believe that really good lane-keeping is a reason to buy a car will be
a reason to buy a car is a huge value add I've never until we just started
talking about it haven't really quite realized that that I've felt with elan
chase of level four is not the correct chase it was on because you should just
say Tesla has the best as if from a testing perspective say Tesla has the best lane-keeping
coming I should say coming I is the best link keeping and that is it yeah yeah
does do you think well you have to do the longitudinal as well you can't just
Lane keep you have to do a cc but a cc is much more forgiving than lanky
especially on the highway oh by the way are you uh calming eyes camera only
correct oh no we use the radar we from the car you were able to get to open it
um we can't do a camera only now it's gotten to the point but we leave the
radar there is like a it's it's fusion now okay so let's maybe talk through
some of the system specs on the hardware or what it what's what's the hardware
side of what you're providing what's the capabilities in the software side
would open pilot and so on so open pilot as the the box that we sell that it
runs on it's a phone in a plastic case it's nothing special we sell it without
the software so you're like you know you buy the phone it's just easy it'll be
easy setup but it's sold with no software open pilot right now is about to be
0.6 when it gets to 1.0 I think we'll be ready for a consumer product we're not
gonna add any new features we're just gonna make the lane-keeping really really
good so what do we have right now it's a snapdragon 820 say so many IMX 298
forward-facing camera driver monitoring camera and she's a selfie cam on the
phone and a can transceiver biffle's little thing calls pandas and they talk
over USB to the phone and then they have three canvases that they talk to the
car one of those campuses is the radar CANbus one of them is the main car
CANbus and the other one is the proxy camera CANbus we leave the existing
camera in place so we don't turn a DB off right now we still turn a TV off if
you're using our longitudinal but we're gonna fix that before 1.0 you got it
wow that's cool so in its can both way so how are you able to control vehicles
so we proxy the vehicles that we work with already have Lane Keeping Assist
system so Lane Keeping Assist can mean a huge variety of things it can mean it
will apply a small torque to the wheel after you've already crossed a lane line
by a foot which is the system in the older Toyotas versus like I think Tesla
still calls it Lane Keeping Assist where it'll keep you perfectly in the center
of the lane on the highway you can control like you would in joystick the cars
these so these cars already have the capability of drive-by-wire so is it is it
trivial to convert a car that it operates with it open pile is able to control
the steering Oh a new car or a car that we so we have support now for 45
different makes of cars what are one of the cars general mostly Hondas and
Toyotas we support almost every Honda and Toyota made this year and then a
bunch of GM's bunch of Subarus which it doesn't have to be like a Prius it
could be Coral as well okay the 2020 Corolla is the best car with open pilot it
just came out there the actuator has less lag than the older Corolla I think I
started watching video with your eye the way you make videos is awesome
literally the dealerships streaming stream for an hour yeah and basically like
if stuff goes a little wrong you're just like you just go with it yeah I love
it what's real yeah that's real that's that's it's that's so beautiful and it's
so in contrast to the way other companies would put together a video like that
how do I like to do it like good I mean if you become super rich one day is successful
I hope you keep it that way because I think that's actually what people love
that kind of genuine oh it's all that has value to me yeah my money has no if I
sell out to like make money and I sold out it doesn't matter what do I get
yacht I don't I got and I think Tesla's actually has a small inkling of that as
well with autonomy day they did reveal more than I mean of course there's
marketing communications you can tell but it's more than most companies will
reveal which is I hope they go towards a direction more other companies GM Ford
oh Jessa Tesla's gonna win level 5 they really are so let's talk about it you
think you're focused on level 2 currently currently we're gonna be one to two
years behind Tesla getting to level five okay we're interested right we're into
it you're in I'm just saying once Tesla gets it we're one to two years behind I'm
not making any timeline on when Tesla's that's right you did that's brilliant I'm
sorry Tesla investors if you think you're gonna have an autonomous robot taxi
fleet by the end of the year yes that's all bet against that so that what do
you think about this the most level four companies are kind of just doing their
usual safety driver during full autonomy kind of testing and then Tesla does
basically trying to go from lane-keeping to full autonomy what do you think
about that approach how successful would it be a ton better approach because
Tesla is gathering data on a scale that none of them are they're putting real
users behind the behind the wheel of the car it's I think the only strategy
that works the incremental well so there's a few components to test approach
that's that's more than just incrementally you spoke with is the one is the
software so over-the-air software updates necessity I mean way more ease have
those - those aren't but there was differentiating from the automaker's right
no link keeping assist systems have no cars with lane keeping system have that
except Tesla yeah and the other one is the data the other direction which is
the ability to query the data I don't think they're actually collecting as much
days people think but the ability to turn on collection and turn it off so I'm
both in the robotics world in the the psychology human factors world many people
believe that level to autonomy is problematic because of the human factor like
the more the task is automated the more there's a vigilance decrement you start
to fall asleep you start to become complacent start texting more and so on do
you worry about that because if we're talking about transition from
lane-keeping to full autonomy if you're spending eighty percent of the time not
supervising machine do you worry about what that means to the safety of the
drivers one we don't consider open pilot to be 1.0 until we have 100% driver
monitoring you you can cheat right now our driver monitoring system there's a
few ways to cheat it there pretty obvious we're working on making that better
before we ship a consumer product that can drive cars I want to make sure that
I have driver monitoring that you can't cheat what's like a successful driver monitoring
system look like it's keep its is it all buzz just keeping your eyes on the
road um well a few things so that's what we went with it first for driver
monitoring I'm checking I'm actually looking at where your head is looking but
cameras know about my resolution eyes are a little bit hard to get well head is
this big I mean that is good and actually a lot of it just as psychology wise
to have that monitor constantly there it reminds you that you have to be paying
attention but we want to go further we just hired someone full-time to come
onto the driver monitoring I want to detect phone in frame and I want to make
sure you're not sleeping how much does the camera see of the body this one not
enough not enough the next one everything what's interesting fish Atkins we
have we're doing just data collection that real-time but fish eye is a
beautiful mouth being able to capture the body and the smartphone is really
like the biggest problem I'll show you I can show you one of the pictures from
from our finder system awesome so you're basically saying the driver monitoring
will be the answer to that um I think the other point that the original paper
is is good as well you're not asking a human to supervise a machine without
giving them meat they can take over at a time right our safety model you can
take over we disengage on both the gas or the brake we don't disengage on
steering I don't feel you have to but we disengage on gas or brake so it's very
easy for you to take over and it's very easy for you to re-engage that
switching should be super cheap yeah the cars that require even autopilot
requires a double press that's almost I said I like that yeah and then then the
cancel um to cancel in autopilot you either have to press cancel which no one
knows where that is so they press the brake but a lot of things you don't you
want to press the brake you want present ass so you should cancel on gas or
wiggle the steering wheel which is bad as well wow that's brilliant I haven't
heard anyone articulate at that point I like what this is all I think about it's
because I think I think actually Tesla has done a better job than most automakers
at making that frictionless but you just described that it could be even better
I love super cruise as an experience once it's engaged yeah I don't know if
you've used it but getting the thing to try to engage him yeah I've used this
of Germany's super cruise a lot so what's their thoughts on the super Cruiser
system in June disengage super cruise and it falls back to ACC so my car's like
still accelerating it feels weird otherwise when you actually have super cruise
engaged on the highway it is phenomenal we bought that Cadillac we just sold it
but we bought it just to like experience this and I wanted everyone in the
office to be like this is what we're striving to build GM pioneering with the
driver monitoring you know you like their driver monitoring system it has some
bugs if there's a sun shining back year it'll be blind to you by overall mostly
yeah that's so cool you know the stuff that's uh I don't often talk to people
that because it's such a rare car unfortunately they bought one yes possibly
for us we lost like by 25k the deprecation but a Philips worth it I was very
pleasantly surprised that GM system was so innovative and really that wasn't
advertised much wasn't talked about much yeah and I was nervous that it would
die that they would disappear my eyes did they put it on the wrong car they
should've put it on the bolt and not some weird Cadillac that nobody bought I
think that's gonna be into they're saying at least is going to be into their
entire fleet so what do you think about it if as long as we're on the driver
monitoring what do you think about you know I must claim that driver monitoring
is not needed normally I love his claims that one is stupid that one is stupid
and you know he's not gonna have his level five fleet by the end of the year
hopefully he's like okay I was wrong I'm gonna add driver monitoring because
when these systems get to the point that they're only messing up once every
thousand miles you absolutely need driver monitor so let me play Delta because
I agree with you but let me play devil's advocate so one possibility is that
without driver monitoring people are able to monitor the self-regulate monitor
themselves you know that so your idea is seeing all the people sleeping in
decimals uh yeah well I'm a little skeptical of all the people sleeping in
Tesla's because I have I've stopped paying attention to that kind of stuff
because I want to see real data there's too much glorified it doesn't feel
scientific to me so I want to know you know what how many people are really sleeping
in Tesla's vs. sleeping I've I was driving here sleep-deprived in a car with no
automation I was falling asleep I agree that it's high P it's just like you
know what if you under I've am wondering I think I rented a my last autopilot
experience was I rented a model three in march and drove it around the wheel
thing is annoying and the reason the wheel thing is annoying we use the wheel
thing as well but we don't disengage on wheel for Tesla you have to touch the
wheel just enough you should trigger the torque sensor to tell it that you're
there but not enough as to disengage it which don't use it for two things you
disengage one wheel you don't have to that whole experience Wow beautiful put
that all those elements even if you don't have driver monitoring that whole experience
needs to be better driver monitoring I think would make I mean I think super
cruise is a better experience once it's engaged over autopilot I think super
cruise is our transition to engagement and disengagement are significantly
worse yeah so there's a tricky thing because if I were to criticize super
cruise is uh it's a little too crude and uh I think it's like six seconds or
something if you look off-road you'll start warning you it's some ridiculously
long period of time and just the way it I think it's basically it's a binary
chili adapter it yeah it's it just needs to learn more about you and used to
communicate what it sees about you more like I'm not you know Tesla shows what
it sees about the external world it would be nice the supercruise would tell us
what it sees about the internal world it's even worse than that you press the
button to engage and it just says super cruise unavailable yeah why why yeah
that transparency is good we've renamed the driver monitoring packet to driver
state service state we have car state packet which has the state of the car
driver state packet which I stay the driver so what does itah make their BAC must
be do you think that's possible with computer vision absolutely so to me it's
an open question I don't haven't looked into too much they actually had quite
seriously looked at the literature it's not obvious to me that from the eyes and
so on you can tell you might need to stuff from the car as well yeah you might
need how they're controlling the car right and that's fundamentally at the end
of the day what you care about you but I think especially when people are
really drunk they're not controlling the car nearly smoothly as they would look
at them walking right there the car is like an extension of the body so I think
you could totally detect and if you could fix people who drunk distracted
asleep if you fix those three yeah this is that's huge so what are the current limitations
of open pilot what are the main problems that still need to be solved um we're
hopefully fixing a few of them in 0-6 we're not as good as auto pilot at stop
cars so if you're coming up to a red light at like 55 so it's the radar stopped
car problem which is responsible to auto pilot accidents it's hard to
differentiate a stopped car from a like signpost yes that ecology um so you
have to fuse you have to do this visually there's no way from the radar data to
tell the difference maybe you could make a map but I really believe in mapping
at all anymore um really what you don't believe in mapping no so you basically
the open pilot solution is saying react to the environment is just like human
doing beings and then eventually when you want to do navigate on open pilot
I'll train the net to look at ways all runways in the background I'll train a
car using GPS at all we use it to crown trees we use it to very carefully
ground treat the paths we have a stack which can recover a relative to 10
centimeters over one minute and then we use that to ground truth exactly where
the car went in that local part of the environment but it's all local how are
you testing in general just for yourself like experiments stuff all right were
you were you located San Diego San Diego yeah okay Oh what you basically drive
around there then collect some data and watch on Florence we have a simulator
now and we have our simulators really cool our simulator is not it's not like a
unity based simulator our simulator lets us load in real estate what I mean we
can load in a drive and simulate what the system would have done on the
historical data ooh nice interesting so what yeah right now we're only using it
for testing but as soon as we start using it for training what's your feeling
about the real world versus simulation do you like simulation for training if
this moves to training Chuck we have to distinguish two types of simulators
right there's a simulator that light is completely fake I could get my car to
drive around in GTA mm-hmm um I feel that this kind of simulator is useless
you're never there's so many my analogy here is like okay fine you're not
solving the computer vision problem but you're solving the computer graphics problem
right and you don't think you can get very far about creating ultra realistic
graphics no because you can create ultra realistic graphics of the road now
create alter a realistic behavioral models of the other cars oh well I'll just
use my self-driving no you won't you need real you need actual human behavior
because that's what you're trying to learn the dead driving does not have a
spec the definition of driving is what humans do when they drive whatever way
mode does I don't think it's driving right well I think if you win more than
others its if there's any useful reinforcement learning I've seen it used quite
well I study pedestrians a lot too is try to train models from real data of how
pedestrians move and try to use reinforcement learning models to make
pedestrians move in human-like ways by that point you've already gone so many
layers you detected a pedestrian did you did you hand code the feature vector
of their state did you guys learn anything from computer vision before deep
learning well okay you know I feel like this is a perception to you is the
sticking point does that mean what what's what's the hardest part of the stack
here there is no human understandable feature vector separating perception and
planning that's the best way I can I can put that there is no so it's all
together and it's it's a that's a joint problem so you can take localization
localization and planning there is a human understandable feature vector
between these two things I mean okay so I have like three degrees position
three degrees orientation and those derivatives maybe those second derivatives
right that's human understandable that's physical the between perception and
planning um so like way Moe has a perception stack and then a planner um and
one of the things way matters right is they have a simulator that can separate
those two they can like replay their perception data and test their system which
is what I'm talking about about like the two different kinds of simulators
there's the kind that can work on real data and is the kind of can't work on
real data now the problem is that I don't think you can hand code a feature
vector right like like you have some lists of like well here's my list of cars
on the scenes here's my list of pedestrians in the scene this isn't what humans
are doing what are humans doing global some something you're saying that's too
difficult to handle I'm saying that there is no state vector given a perfect I
could give you the best team of engineers in the world to build a perception
system and the best team to build a planner all you have to do is define the
state vector that separates those two I'm missing the state vector that
separates those two what do you mean so what is the output of your perception
system I'll put it the perception system it's theirs okay well there's several
ways to do it one is this lamp components localization the other is drivable
area drivable space drivable space and then there's the different objects in
the scene and different objects in the scene over time maybe to give you input
to then try to start modeling the trajectories of those objects sure that's it
I can give you a concrete example of something you missed what's that so say
there's a bush in the scene humans understand that when they see this bush that
there may or may not be a car behind that bush drivable area and a list of
objects does not include that humans are doing this constantly at the simplest
intersections so now you have to talk about occluded area right right but even
that what do you mean by occluded okay so I can't see it well if it's the other
side of a house I don't care what's the likelihood that there's a car in that
occluded area right and if you say okay we'll add that I can come up with 10
more examples that you can't add certainly occluded area would be something
that simulator would have because it's simulating the entire you know occlusion
is part of it a part of a vision stack pleasures that what I'm saying is if you
have a hand engineered if your perception system output can be written in a
spec document it is incomplete yeah idem you know certainly it's it's hard to
argue with that because in the end that's going to be true yes I'll tell you
what the output of our perception system is was that it's a thousand it's a
thousand twenty four dimensional vector training underling oh no not it's a
thousand twenty four dimensions of who knows what because its operating on real
data yeah yeah and that's the perception that's the perception stake right
think about a think about an autoencoder four phases alright if you have an
autoencoder four phases and you say it has 256 dimensions in middle and I'm
taking a face over here and projecting it to a face over here yeah can you hand
label all 256 of those dimensions well no but those are generated automatically
but they but even if you tried to do it by hand could you come up with a spec
for your and between your encoder and your decoder no no because that's not it
is it wasn't designed but there no no but if you could design it if you could
design a face Reconstructor system could you come up with a spec no but I think
we're missing here a little bit I think the the you're just being very poetic
about expressing a fundamental problem of simulators that they're going to be missing
so much that the feature vector would just look fundamentally different from in
the simulated world in the real world I'm not making a claim about simulators
I'm making a claim about the spec division between perception and planning and
planning even in your system just in general right just in general if you're
trying to build a car that drives if you're trying to hand code the output of
your perception system like saying like here's a list of all the cars in the
scene here's a list of all the people here's a list of the included areas
here's a vector of drivable areas insufficient and if you start to believe that
you realize that what Wayman crews are doing is impossible currently what we're
doing is the perception problem it's converting the scene into a chessboard you
yeah and then you reason some basic reasoning around that chessboard yeah and
you're saying that really there's a lot missing there first of all why are we
talking about this cuz isn't this a full autonomy is this something you think
about oh I want to win self-driving cars so you're really your definition of
win includes level of fool five level five I don't think level four is a real
thing I want to build I want to build the alphago of driving so so alphago is
really end to end yeah is uh yeah it's end to end and do you think this whole
problem is those that also kind of what you're getting at with the perception
and the planning is that this whole problem the right way to do it is really to
learn the entire thing I'll argue that not only is it the right way it's the
only way that's going to exceed human performance well certainly true for go
everyone who tried to hand code go things built human inferior things and then
someone came along and wrote some 10,000 line thing that doesn't know anything
about go that beat everybody it's 10,000 lines true in that sense the the open
question then that maybe I can ask you is uh driving is much harder than go the
open question is how much harder so how because I think the AH mosque approach
here with planning and perception it's similar to what you're describing which
is really turning into not some kind of modular thing but really do formulate
is a learning problem and it solves a learning problem of scale so how many
years put one is how many years would it take to solve this problem or just how
hard is this freaking problem well the cool thing is I think there's a lot of
value that we can deliver along the way I think that you can build lane-keeping
assist actually plus adaptive cruise control plus okay looking at ways extends
to like all of driving yeah most of driving varies oh your adaptive cruise
control treats red lights like cars okay so let's jump around with you you mentioned
that you didn't like navigate an autopilot yeah what advice how would you make
it better do you think as a feature that if it's done really well it's a good
feature I think that it's too reliant on like hand coded hacks for like how
does navigate an autopilot do a lane change it actually does the same lane
change every time and it feels mechanical humans do different lane changes
human sometime will do a slow one sometimes do a fast one navigate an autopilot
at least every time I used it it did the identical language how do you learn I
mean this is a fundamental thing actually yeah is uh the braking and an
accelerating something that's still test the probably does it better than most
cars but it still doesn't do a great job of creating a comfortable natural
experience and navigate on autopilot just lane changes an extension of that so
how do you learn to do natural lane change so we have it and I can talk about
how it works so I feel that we have the solution for lateral but we don't yet
have the solution for longitudinal there's a few reasons longitudinal is harder
than lateral the lane change component the way that we train on it very simply
is like our model has an input for whether it's doing a lane change or not and
then when we train the end-to-end model we hand label all the lane changes
because you have to I struggled a long time about not wanting to do that but I
think you have to because you order the training data for the train data right well
we actually we have an automatic ground truth or which automatically labels all
the lane changes was that possible to automatically label interest yeah and
detect the lane I see when it crosses it right I don't have to get that that
high percent accuracy but it's like 95 good enough now I set the bit when it's
doing the lane change in the end-to-end learning and then I set it to zero when
it's not doing a lane change so now if I wanted to do a lane change a test time
I just put the bit to a 1 and I'll do later yeah but so if you look at the
space of lane change you know some percentage not a hundred percent that we
make as humans is not a pleasant experience because we messed some part of it
up yeah it's nerve-racking to change even look at the seizure des accelerate
how do we label the ones that are natural and feel good you know that's the
because that's your ultimate criticism the current Oh navigate not apologies
doesn't feel good well the current navigator on autopilot is a hand coded
policy written by an engineer in a room who probably went out and tested it a
few times on the 280 probably a more a better version of that but yes that's
how we would have written it a comment yeah Tesla they tested it and it might
have been two engineers yeah no but so if you learn the lane change if you
learn how to do a lane change from data just like just like you have a label
that says lane change and then you put it in when you want to do the lane
change it'll automatically do the lane change that's appropriate for the
situation now to get it the problem of some humans do bad lane changes we haven't
worked too much on this problem yet it's not that much of a problem in practice
my theory is that all good drivers are good in the same way and all bad drivers
are bad in different ways and we've we've seen some data to back this up well
beautifully put so you just basically if that's true yeah hypothesis then you
know task is to discover the good drivers um the good drivers stand out because
they're in one cluster and the bad drivers are scattered all over the place and
your net learns the cluster yeah that's uh so you just learned from the good drivers
and they're easy to cluster we learned from all of them and that automatically
learns the policy that's like the majority but we'll eventually probably
afterthought so if that theory is true I hope it's true because the the counter
theory is there is many clusters maybe but rarely many clusters of good drivers
because if there's one cluster of good drivers you can at least discover a set of
policies you can learn a set of policies which would be good universally yeah
that would be a nice that would be nice if it's true and you're saying that there
are some evidence that let's say lane changes can be clustered into four clusters
right right there's this finite level of I would argue that all four of those
are good clusters all the things that are random are noise and probably bad and
which one of the four you pick or maybe it's Tanner maybe it's twenty you can
learn them it's context dependent it depends on the scene and the hope is it's
not too dependent on the driver yeah the hope is that it all washes out the
hope is that there's that the distribution is not bimodal the hope is that it's
a nice gas man so what advice would you give to Tessa how to fix how to improve
navigate an autopilot the lessons you've learned from Kamiya the only real
advice I would give to Tesla is please put driver monitoring in your cars with
respect to improvement you can't do that anymore I said to interrupt but you
know there's a practical nature of many of hundreds of thousands of cars being
produced that don't have a good driver facing camera the model 3 has a selfie
cam is it not good enough did they not have put IR LEDs for night that's a good
question but I do know that the is fisheye in its relatively low resolution so
it's really not this I he wasn't it wasn't designed for Arman you can hope that
you can kind of scrape up and and and have something from it yeah but put it in
today put it in today today every time I've heard Carpathia talk about the
problem and talking about life software 2.0 and how the machine learning is
gobbling up everything I think this is absolutely the right strategy I think
that he didn't write navigate on autopilot I think somebody else did and kind
of hacked it on top of that stuff I think what Carpathia says wait a second why
did we hand code this lane change policy with all these magic numbers we're
gonna learn it from data they'll fix it they already know what to do there well
that that's that's Andres job is to turn everything into a learning problem and
collect a huge amount of data the the reality is though not every problem could
be turned into a learning problem in the short term in the end everything would
be a learning problem the reality is like if you want to build alpha vehicles
today it will likely involve no learning and that's that's the the reality is
so at which point does learning start it's the crutch statement that lidar is a
crutch on which point will learning get up to part of human performance it's
all over human performance and imagenet classification under ivan is the question
still it is a question I'll say this I'm I'm here to play for 10 years I'm not
here to try to I'm here to play for 10 years and make money along the way I'm
not here to try to promise people that I'm gonna have my l5 taxi Network up and
working in two years do you think those mistake yes what do you think there was
the motivation behind saying that other companies are also promising alpha
vehicles with their different approaches in 2020 2021 2022 if anybody would
like to bet me that those things do not pan out I will I will bet you even
money even money I'll bet you as much as you want so are you worried about
what's going to happen because you're not in full agreement on that I was going
to happen when 2022 21 come around and nobody has fleets of autonomous vehicles
no you can look at the history if you go back five years ago they were all
promised by 2018 and 2017 but they weren't that strong of promises I mean Ford
really declared pretty that I think not many have declared as as like
definitively as they have now these dates well okay so let's separate l4 and l5
do I think that it's possible for way mo to continue to kind of like like hack
on their system until it gets to level 4 in Chandler Arizona yes knows no
safety driver Chandler Arizona yeah but by OSI which year are we talking about oh
I even think that's possible by like 2020 2021 but level 4 Chandler Arizona not
level 5 New York City level 4 meaning some very defined streets it works out
really well very defined streets and then these streets are pretty empty if
most of the streets are covered in way MOS we mo can kind of change the
definition of what driving is hmm right if your self-driving network is the
majority of cars in an area they only need to be safe with respect to each
other and all the humans will need to learn to adapt to them now go drive in
downtown New York oh yeah that's already you can talk about autonomy in like
like fun farms it already works great because you can really just follow the
GPS line so what does success look like for comm AI what what are the
milestones like where you can sit back with some champagne and say we did it
boys and girls well it's never over yeah but don't be let's drink champagne
everything straight so what is a good what are some wins um a big milestone
that we're hoping for by mid next year is profitability of the company and
we're gonna have to revisit the idea of selling a consumer product but it's not
gonna be like the comma one when we do it it's gonna be perfect open pilot has
gotten so much better in the last two years we're gonna have a few a few
features we're gonna have a hundred percent driver monitoring we're gonna
disable no safety features in the car um actually I think it'd be really cool
we're doing right now our project this week is we're analyzing the data set and
looking for all the AEP triggers from the manufacturer systems we have a better
data set on that than the manufacturers how much does how many does Toyota have
ten million miles of real-world driving to know how many times they're AUB
triggered so let me give you cuz yes right financial advice yeah cuz I work
with a lot of automakers and one possible source of money for you which I'll be
excited to see you take on is basically selling the data so which is something that
most people are not selling in a way we're here here at automaker but creating
we've done this actually at MIT not for money purposes but you could do it for
significant money purposes and make the world a better place by creating a
consortium where automakers would pay in and then they get to have free access
to the data and I I think a lot of people are really hungry for that and would
pay significant amount of money for it here's the problem with that I like this
idea all in theory he'd be very easy for me to give them access to my servers
and we already have all open source tools to access this data it's in a great
format we have a great pipeline but they're gonna put me in the room with some
business development guy mm-hmm and I'm gonna have to talk to this guy and he's
not gonna know most of the words I'm saying I'm not willing to tolerate that
okay but I think I agree with you I'm the same way but you just tell them the
terms and there's no discussion needed if if I could just tell them the terms
yeah and then like all right who wants access to my data I will sell it to you
for let's say you want to go on a subscription I'll sell you 400 a month any
100k mo 100k month I'll give you access to the data subscription yeah yeah I
think that's kind of fair came up with that number off the top of my head if
somebody sends me like a three line email where it's like we would like to pay
a hundred K month to get access to your data we would agree to like reasonable
privacy terms of the people who are in the data set I would be happy to do it
but that's not gonna be the email the email is gonna be hey do you have some
time in the next month where we can sit down and we can I don't have time for
that we're moving too fast yeah you could politely respond to that email but
not saying I don't have any time for your bullshit yeah you say oh well
unfortunately these are the terms and so this is we try to we brought the cost
down for you in order to minimize the friction of education after here's the
whatever it is 1 2 million years dollars a year and you have access and it's
not like I get that email from like but okay am I gonna reach out am I gonna
hire a business development person who's gonna reach out to the automaker's no
way yeah okay if they reached into me I'm not gonna ignore the email I'll come
back with something straight yeah if you're willing just pay honeycomb all the
facts they don't man I'm happy to to set that up that's what my engineering time
but actually quite insightful view you're right yeah probably because many of
the automakers are quite a bit of old-school yeah there will be need to reach
out and they want it but they they'll need to be some some communication you
right mobile eye circuit 2015 had the lowest R&D; spend of any chip maker
like purpur and you look at all the people who work for them and it's all
business development people because the car companies are impossible to work
with yeah so you're you have no patience for that and you're you're legit
Android huh I have something to do right like like it's not like it's not like
I don't like I don't mean to like be a dick and say like I don't have patience
for that but it's like that stuff doesn't help us with our goal of winning
self-driving cars if I want money in the short term if I showed off like the
actual like the learning tech that we have it's it's somewhat sad like it's
years and years ahead of everybody else's not so maybe not Tesla's I think Tesla
has similar stuff to us actually yeah I think Tesla's similar stuff but when
you compare it to like what the Toyota Research Institute has you're not even
close to what we have no comment but I also can't I have to take your comments
I ain't into ative Lee believe you but I have to take it with a grain of salt
because I mean you you are an inspiration because you basically don't care
about a lot of things that other companies care about you don't try to bullshit
in a sense like make up stuff so to drive a valuation you're really very real
and you're trying to solve the problem and admire that a lot what I don't
necessarily fully can't trust you on I do respect it's like how good it is right
I can only but I also know how bad others are and so I'll say I'll say two things
about don't trust but verify right I'll say two things about that one is try
get in a twenty twenty Corolla and try open pal 0.6 when it comes out next
month I think already you'll look at this and you'll be like them this is already
really good and then I could be doing that all with hand labelers and all with
with like like the same approach that like Mobileye uses when we release a
model that no law has the lanes in it that only outputs a path mm-hmm then
think about how we did that machine learning and then right away when you see
and that's gonna be an open pilot that's gonna be an open pilot before 1.0 when
you see that model you'll know that everything I'm saying is true because how
else did I get that model good one of the things too about the simulator oh yeah
yeah this is super exciting that's super exciting and uh but like you know I
listened to your talk with Kyle and Kyle was originally building the the after
market system and he gave up on it because of technical challenges yeah because
of the fact that he's gonna have to support twenty to fifty cars we support
forty five because what is he gonna do when the manufacturer ABS system
triggers we have alerts and warnings to deal with all of that in all the cars
and how is he going to formally verify it well I got ten million miles of data
it's probably better it's probably better verified than the spec yeah I'm glad
you're here talking to me this is I'll remember this day is this interesting if
you look at Kyle's from from Cruz I'm sure they have a large number of business
development folks and you work with he's working with GM you could work with
agro a I working with Ford it's interesting because chances that you fail
business-wise like bankrupt are pretty high yeah and and yet it's the Android
model is you're actually taking on the problem so that's really inspiring I
mean well I have a long-term way for kamma to make money too and one of the
nice things when you really take on the problem which is my hope for autopilot
for example is things you don't expect ways to make money or create value that
you don't expect will pop up oh I've known how to do it since kind of 2017 is
the first time I said it well which part to know it to know how to do which
part our long-term plan is to be a car insurance company insurance yeah I love
it yeah yeah what I make driving twice is safe not only that I have the best
date is that you know who statistically is the safest drivers and oh oh we see
you we see you driving unsafely we're not going to insure you and that that
causes a like bifurcation in the market because the only people who can't get
common insurance or the bad drivers Geico can insure them their premiums crazy
higher premiums are crazy low would win contracts take over that whole market
okay so if we win if we went but that's I'm saying like how do you turn comma
into a ten billion dollar company is that that's right so you you know a musk
who else who else is thinking like this and working like this in your view who
are the competitors are there people seriously I don't think anyone that I'm
aware of as seriously taking on lane-keeping you know like to worse a huge
business that turns eventually into full autonomy that then creates yeah like
that creates other businesses on top of it and so on thinks insurance thinks
all kinds of ideas like that do you know who anyone else thinking like this not
really that's interesting I mean it my sense is everybody turns to that in like
four or five years like Ford once the autonomy doesn't feel fall through but at
this time Elon to the iOS by the way he paved the way for all I was not i OS
true I would not be doing comma AI today if it was not for those conversations
with Elon and if it were not for him saying like yeah I think he said like well
obviously we're not gonna use Leiter we use cameras humans use cameras so what do
you think about that how important is lidar everybody else is on l5 is using lidar
what are your thoughts on his provocative statement that lidar is a crutch see
sometimes we'll say dumb things like the driver monitoring thing but sometimes
we'll say absolutely completely 100% obviously true things yeah of course lidar
is a crutch it's not even a good crutch you're not even using it they're using
it for localization yeah which isn't good in the first place if you have to
localize your car to centimetres in order to drive like yeah they're not drive
it currently not doing much machine learning I thought polite our data meaning
like to help you in the tasks of general tasks of perception the main goal of
those light hours on those cars I think is actually localization more than
perception or at least that's what they use them for yeah that's true if you
want to localize two centimeters you can't use GPS the fanciest GPS in the world
can't do it especially if you're under tree cover and stuff flatter I can do it
pretty easily see really they're not taking on I mean in some research they're
doing they're using it for perception but and they're certainly not which sad
they're not fusing it well lay vision they do use it for perception I'm not
saying they don't use it for perception but the thing that they have vision
based and radar based perception systems as well you could remove the lidar and
and and keep around a lot of the dynamic object perception you want to get
centimeter accurate localization good luck doing that with anything else so
what should Cruz lame-o do like what would you be your advice to them now
anyway Mo's actually there's I mean they're doing they're serious way mo out of
all of them equate so serious about the long game if everybody fell five is a
lot is requires fifty years I think when will be the only one left standing at
the end with the forgiving the financial backing if they have Google box um
I'll say nice things about both lame-o and Cruz let's do it nice is good way mo
is by far the furthest along with technology way mo has a three to five year
lead on all the competitors um if that if the way mo looking stack works mm-hmm
maybe three year lead if the way mo looking stack works they have a three year
lead now I argue that way mo has spent too much money to recapitalize to gain
back their losses in those three years also self-driving cars have no network
effect like that yeah goober has a network effect you have a market you have drivers
and you have riders self-driving cars you have capital and you have riders
there's no network effect if I want to blanket a new city in self-driving cars
i buy the off-the-shelf Chinese knockoff self-driving cars and I buy enough up from
the city I can't do that with drivers and that's why Ober has a first mover
advantage that no self-driving car company will can you uh disentangle that a
little bit uber you're not talking about uber the autonomous vehicle number you
talked about the uber cars okay yeah I'm over I open for business in Austin Texas
listen I need to attract both sides of the market I need to both get drivers my
platform and riders on my platinum and I need to keep them both sufficiently
happy right riders aren't going to use it if it takes more than five minutes
for an uber to show up drivers aren't gonna use it if they have to sit around
all day and there's no riders so you have to carefully balance a market and
whenever you have to carefully balance a market there's a great first mover
advantage because there's a switching cost for everybody right the drivers and
the riders would have to switch at the same time let's even say that you know
um let's say Luber shows up in Luber somehow you know agrees to do things that
add a bigger you know you know we're just gonna we've done it more efficiently
right Luber is only takes five percent of a cot instead of the ten percent that
Hooper takes no one is gonna switch because the switching cost is higher than
that five percent so you actually can in markets like that you have a first
mover advantage yeah autonomous vehicles of the level five variety have no
first mover advantage if the technology becomes commoditized say I want to go
to a new city look at the scooters it's gonna look a lot more like scooters every
person with a checkbook can blanket a city in scooters and that's why you have
10 different scooter companies yeah which one's gonna win it's a race to the
bottom it's terrible market to begin because there's no market for scooters and
scooters don't get a say and whether they want to be bought and deployed to a
city or not right so the yeah we're gonna entice the scooters with subsidies
and deals so whenever you have to invest that capital that's it doesn't it
doesn't come back yeah that they can't be your main criticism over the way mo
approach oh I'm saying even if it does technically work even if it does
technically work that's a problem yeah I don't know I if I were to say I I
would I would say you're already there I haven't even thought about that but I
would say the bigger challenge is the technical approach so way most cruises
and not just the technical approach but of creating value I still don't
understand how you beat uber the the human driven cars in terms of financially
it doesn't it doesn't make sense to me that people want to want to get an
autonomous vehicle I don't understand how you make money in the long term like
real long-term but it just feels like there's too much capital investment needed
oh and they're gonna be worse than ubers because they're gonna they're gonna
stop for every little you know thing everywhere um actually a nice thing about
Cruz that was my nice thing about wait another three years that it wasn't nice
oh that's three years technically ahead of everybody their tech stack is is
great my nice thing about Cruz is GM buying them was a great move for GM for 1
billion dollars GM bought an insurance policy against way mo they put Cruz is
three years behind way mo hmm that means Google will get a monopoly on the
technology for at most three years and technology works you might not even be
right about the three years it might be less might be less crews actually might
not be that far behind I don't know how much way mo has waffled around or how
much of it actually is just that long tail yeah okay if that's the best you
could say there's some nice things it that's more of a nice thing for GM that
that's a smart insurance policy it's just more insurance policy I mean I think
that's how I I can't see crews working out any other for crews to leapfrog way
mo would really surprise me yeah so let's talk about like the underlying
assumptions of everything is we're not going to leapfrog Tesla Tesla would have
to seriously mess up for us because you're okay so the way you leapfrog right
is you come up with an idea or you take a direction perhaps secretly that the other
people aren't taking and so cruise way mo even Aurora no Aurora tzuke's is the
same stack as well they're all the same codebase even and they're all the same
DARPA urban challenge codebase so the question is do you think there's a room
for brilliance and innovation there that will change everything like say okay
so I'll give you examples it could be if revolution and mapping for example
that allow you to map things do HD maps of the whole world all weather conditions
really well or revolutionist simulation to where the the what you said before becomes
incorrect that kind of thing I knew room for breakthrough innovation um what I
said before about oh they actually get the whole thing well I'll say this about
we divide driving into three problems and I actually haven't solved the third
yet but I have an idea how to do it so there's the static the static driving
problem is assuming you are the only car on the road right right and this
problem can be solved 100% with mapping and localization this is why farms work
the way they do if all you have to deal with is the static problem and you can
statically schedule your machines right it's the same as like statically
scheduling processes you can statically schedule your tractors to never hit
each other on their paths all right because then you know the speed they go at
so so that's the static driving problem Maps only helps you with the static
driving problem yeah the question about static driving yeah you just made it
sound like it's really easy that was really easy how easy how well because the
whole drifting out of lane when when Tesla drifts out of lane is failing on the
fundamental static driving problem Tesla is drifting out of lane the static
driving problem is not easy for the world the static driving problem is easy
for one route and one route in one weather condition with one state of lane
markings and like no deterioration no cracks in the road I'm assuming you have
a perfect localizer so that's all for the weather condition and me the lane
marking condition that's the problem is how could you how do you have a perfect
you can build perfect localizers are not that hard to build okay come on now
with with wood lighter why don't ya wood lighter okay yeah but you use lighter
right like use lidar build a perfect localizer building a perfect localizer
without lidar it's gonna be it's gonna be hard you can get ten centimeters
without liner you can get one centimeter with lidar maybe concern about the one
or ten centimeter I'm concerned if every once in a while you're just way off
yeah so this is why you have to carefully make sure you're always tracking your
position you want to use light camera fusion but you can get the reliability of
that system up to a hundred thousand miles and then you write some fallback
condition where it's not that bad if you're way off right I think that you can
get it to the point it's like özil D that you're you're never in a case where
you're way off and you don't know it yeah okay so this is brilliant so that's
the static static we can especially with lidar and good HD maps you can solve
that problem easy no you just the static static very typical for you to say
something's easy I got it it's not as challenging as the other ones okay well
it's okay maybe it's obvious how to solve it the third one's the hardest well
where do we get and a lot of people don't even think about the third one and
even I see it as different from the second one so the second one is dynamic the
second one is like say there's an obvious examples like a car stopped at a red
light right you can't have that car in your map yeah because you don't know
whether that car is gonna be there or not so you have to detect that car in
real time and then you have to you know do the appropriate action right also
that car is not a fixed object that car may move and you have to predict with
that car will dim alright so this is the dynamic problem yeah do you have to
deal with this um this involves again like you're gonna need models of other
people's behavior do you are you including in that and I don't want to step on
on the third one oh but if I are you including in that you're influenced and
people I guess the third okay that's the moon we call it the counterfactual
yeah I believe that I just talked to Judea pearl who's obsessed with counterfactuals
oh yeah yeah so the static and the dynamic yeah our approach right now for
lateral will scale completely to the static a dynamic the counterfactual the
only way I have to do it yet they don't give you thing that I want to do once
we have all these cars is I want to do reinforcement learning on the world I'm
always gonna turn the exploiter up to max I'm not gonna have them explore but
the only real way to get at the counterfactual is to do reinforcement learning
because the other agents are humans so that's fascinating that you break you
down like that I agree completely I've set my life thinking about this
beautiful they're so and part of it because you're slightly insane because not
my life just the last four years no no you have like some some nonzero percent
of your brain has a madman in it which that's a really good feature but there's
a safety component to it that I think when this sort of counterfactuals and so
on that would just freak people out how do you even start to think about just
in general I mean you've you've had some friction with Nitza and so on I am frankly
exhausted by safety engineers the the prioritization on safety over innovation
to a degree where it kills in my view kills safety in the long term so the
counterfactual thing they just just actually exploring this world of how do you
interact with dynamic objects and so on how do you how do you think about safety
you can do reinforcement learning without ever exploring and I said that like
so you can think about you're in like a reinforcement learning it's usually
called like a temperature parameter and your temperature parameter is how often
you deviate from the Arg max I could always set that to zero and still learn
and I feel that you'd always want that set to zero on your actual system got
you but the problem is you first don't know very much and so you're going to
make mistakes so the learning the exploration happens to ready yeah but okay so
the consequences of a mistake yeah open pilot and autopilot are making mistakes
left and right yeah we have we have we have 700 daily active users a thousand
weekly active users open pilot makes tens of thousands of mistakes a week these
mistakes have zero consequences these mistakes are oh it I wanted to take this
exit and it went straight so I'm just gonna carefully touch the wheel humans
the humans catch them and the human disengagement is labeling that
reinforcement learning in a completely consequence-free way so driver
monitoring is the way you ensure they keep yes they keep paying attention how
is your messaging say I gave you a billion dollars you would be scaling and now
oh my fact it's guy couldn't scale with any amount of money I'd raise money if
I could if I had way to scale yeah you're not focusing I don't know I don't
know how to do Oh like I guess I could sell it to more people but I want to
make the system better better I don't know I mean but what's the messaging here
I got a chance to talk to you on and and he he basically said that the human
factor doesn't matter you know the human doesn't matter because the system will
perform there would be sort of a sorry to use the term but like a singular like
a point where it gets just much better and so the human it won't won't really matter
but it seems like that human caching the system when it gets into trouble is
like the thing which will make something like reinforcement learning work so
how do you how do you think messaging for Tesla for you should chant for the
industry in general should change I think my messaging is pretty clear at least
like our messaging wasn't that clear in the beginning and I do kind of fault
myself for that we are proud right now to be a level 2 system we are proud to
be level 2 if we talk about level 4 it's not what the current hardware it's not
gonna be just a magical OTA upgrade it's gonna be new hardware it's gonna be
very carefully thought-out right now we are proud to be level 2 and we have a
rigorous safety model I mean not like like okay rigorous who knows what that
means but we at least have a safety model and we make it explicit is in safety
MD and open pilot and it says seriously though safety dot MD Android so well
this is this is the safety model and I like to have conversations like if like
you know sometimes people will come to you and they're like your systems not
safe okay have you read my safety Doc's would you like to have an intelligent
conversation about this and the answer is always no they just like scream about
it runs Python okay what so you're saying that that because pythons not
real-time Python not being real-time never causes disengagement disengagement
SAR caused by you know the model is QM but safety dad MD says the following
first and foremost the driver must be paying attention at all times I don't can
I do I still consider the software to be alpha software until we can actually
enforce that statement but I feel it's very well communicated to our users two
more things one is the user must be able to easily take control of the vehicle
at all times mm-hmm so if you step on the gas or brake with open pilot it gives
full manual control back to the user or press the cancel button step 2 the car
will never react so quickly we define so quickly to be about one second that
you can't react in time and we do this by enforcing torque limits braking limits
and acceleration limits so we have um like our torque limits way lower than
Tesla's this is another potential if I could tweak autopilot I would lower
their torque limit or would a driver monitoring um because autopilot can jerk
the wheel hard yeah open pilot can it's we we limit um and all this code is
open source readable and I believe now it's all misery C compliant misra is
like the automotive coding standard um at first I you know I've come to respect
I've been reading like the standards lately and I've come to respect them
they're actually written by very smart people yeah they're brilliant people
actually they have a lot of experience there's sometimes a little too cautious
but in this case it pays off miss was written by like computer scientists and
you tell them as a language they use you can tell by the language they use they
talk about like whether certain conditions in misra are decidable or
undecidable you mean like the halting problem and yes well all right you've
earned my respect I will tell you carefully what you have to say and we want to
make our code compliant with that all right so you're proud level two and
reform so you were the founder and I think CEO of comm AI then you were the
head of research what the heck are you know what's your connection to come AI
the president but I'm one of those like unelect unelected presidents of like
like a small dictatorship country not one of those like elected presidents oh
so you're like Putin when he was like yeah I got sure so there's uh what's the
governance structure what's the what's the future of commie I finance I mean as
a business do you want you just focused on getting things right now making some
small amount of money and mean to and then one that works it works in each scale
our burn rate is about 200k a month and our revenue is about 100k a month so we
need to 4x our revenue but uh we haven't like tried very hard at that yet and
the revenue is basically selling stuff online yeah we sell stuff shopped a
comment at AI is there other well okay so you you'll have to figure out that's our
that's our only see but to me that's like respectable revenues yeah we make it
by selling products to consumers we're honest and transparent about what they
are most actually level for companies right because you could easily start
blowing up like smoke like over selling the hype and feeding into getting some
fundraisers oh you're the guy you're genius because you hacked the iPhone oh I
hate that I hate that yeah I can trade my social capital for more money yeah I
did it once I almost regret it doing the first of it well on a small tangent
what's your you seem to not like Fame and yet you're also drawn to fame what
were you on we're on you where are you on that currently have you had some introspection
some soul-searching yeah I actually I've come to a pretty stable position on
that like after the first time I realized that I don't want attention from the
masses I want attention from people who I respect who you respect I can give a list
of people so are these like Elon must have characters yeah well actually you
know what I'll make it more broad than that I won't make it about a person I
respect skill I respect people who have skills right and I would like to like
be I'm not gonna say famous but be like known among more people who have like
real skills who in cars doers do you think have skill not do you respect Oh
Kyle vote has skill a lot of people away mo have skill and I respect them I I
respect them as engineers like I can think I mean I think about all the times
in my life where I've been like dead set on approaches and they turn out to be
wrong so I mean this might I might be wrong I accept that I accept that there's
a decent chance that I'm I'm wrong and actually I mean having talked to Chris Urmson
sterling anderson i those those guys I mean I deeply respect Chris I just
admire the guy he's legit can you drive a car through the desert when everybody
thinks it's impossible that is that's legit and then I also really respect the
people who are like writing the infrastructure of the world like the linus
torvalds and the chris lab they're doing the real work I know they're doing the
real work this every dog that Chris Ladin you realize especially when they're
humble it's like you realize oh you guys were just using your oh yeah all the
hard work they did him that's incredible what do you think mr. Anthony lowendahl
ski what do you he's a he's another mad genius sharp guy oh yeah what do you think
he might long-term become a competitor Oh tu cama well so I think that he has
the other right approach I think that right now there's two right approaches
one is what we're doing and one is what he's doing can you describe I think
it's called pronto a certain you thing did do you know what what the approaches
actually don't know embark is also doing the same sort of thing the idea is almost
that you want to so if you're I can't partner with Honda and Toyota Honda and
Toyota are uh like four hundred thousand person companies it's not even a
company at that point like I don't think of it like I don't personify it I
think of it like an object but a trucker drives for a fleet maybe that has like
some truckers are independent some truckers Drive for fleets with a hundred trucks
there are tons of independent trucking companies out there start a trucking
company and drive your costs down or figure out how to drive down the cost of
trucking another company that I really respect is uh not oh I should I respect
their business model no auto sells a driver monitoring camera and they sell it
to fleet owners if I that's right if I owned a fleet of cars and I could pay
you know 40 bucks a month to monitor my employees this is gonna like reduces
accidents 18% yeah it's it's so like that in the space that is like the
business model that I like most respect is there creating value today yeah
which is uh that's a huge one is how do we create value today with some of this
then the link keeping things huge and it sounds like you're creeping in or full
steam ahead on the driver monitoring - yeah which I think actually were the
short-term value if you can get right I still I'm not a huge fan of the
statement that everything is to have driver monitoring but will I agree with
that completely but I'm that statement usually misses the point that to get the
experience of it right is not trivial oh no not at all in fact like so right
now we have I think the time out depends on speed of the car but we want to
depend on like the scenes day if you're on like an empty Highway it's very
different if you don't pay attention then if light you're like coming up to a
traffic light and long-term it should probably learn from from the driver
because that's to do I watched a lot of video we've built a smartphone detector
just to analyze how people are using smartphones and people are using it very
differently and there's a it's a texting styles there's videos yeah like I got
billions of miles of people driving cars in this moment I spent a large
fraction of my time just watching videos because it's never fails to to learn
like it never I've never failed from a video watching session to learn
something I didn't know before fact I usually like when I eat lunch I'll sit
especially when the weather is good and just watch pedestrians with an eye to
understand like from a computer vision I just to see can this model can you
predict what are the decisions made and there's so many things that we don't understand
this is what I mean about the state vector yeah it's I'm trying to always think
like Gamma understanding in my human brain how do we convert that into how hard
is the learning problem here I guess is the fundamental question so something
that from a hacking perspective this is always comes up especially with folks
well first the most popular question is the trolley problem right so that's not
a sort of a serious problem there are some ethical questions I think that arise
maybe will you want to met you or do you think there's any ethical serious ethical
questions that we have a solution to the trolley problem Akane aye well so
there is actually an alert in our code ethical dilemma detected it's not
triggered yeah we don't we don't how you have to detect the ethical dilemmas
but we're a level two system so we're going to disengage and leave that decision
to the human you're such a troll hey no but the trolley problem deserves to be
trolled yeah that's a beautiful answer actually I know I gave it to someone who
was like sometimes people ask like you asked about the trolley problems like
you can have a kind of discussion about it like boo you get someone who's like
really like earnest about it because it's the kind of thing where if you ask a
bunch of people in an office whether we should use a sequal stack or no sequel
stack if they're not that technical they have no opinion but if you ask them
what color they want to paint the office everyone has an opinion on that and
that's why the trolley problem is that's it I mean it's a beautiful answer yeah
we're able to detect the problem and were able to pass it on to the human yeah
I've never never heard anyone say it nice escape route okay but proud level -
I'm proud level - I love it so the other thing that people cope you know have
some concern about with AI in general is hacking so how hard is it do you think
to hack a nataas vehicle either through physical access or through the more
sort of popular now these adversarial examples on the sensors be adversarial
examples one you want to see some adversarial examples that affect humans hmm
right oh well there used to be a stop sign here but I put a black bag over the
stop sign and then people ran it all right adversarial yeah right like like
like there's tons of human adversarial examples - um the question in general
about like security if you saw something something just came out today I'm like
there are always such high P headlines about like how navigate on autopilot was
fooled by a GPS spoof to take an exit right at least that's all they could do
was take an exit if your car is relying on GPS in order to have a safe driving
policy they're doing something if you're relying and this is why v2v is such a
terrible idea v2v now relies on both parties getting communication right this
is not even so I think of safety security is like a special case of safety
right safety is like we put a little you know piece of caution tape around the
hole so that people won't walk into it by accident security is I put a 10 foot
fence around the hole so you actually physically cannot climb into it with
barbed wire on the top and stuff right so like if you're designing systems that
are like unreliable they're definitely not secure your car should always do
something safe using its local sensors and then the local sensor should be
hardwired and then could somebody hack into your can boss and turn your
steering wheel on your brakes yes but they could do it before common AI too so
let's think out of the box and some things so do you think teleoperation has a
role in any of this so remotely stepping in and controlling the cars no I think
that if safety if the safety operation by design requires a constant link to
the cars I think it doesn't work so that's the same argument using for v2i VTV well
there's a lot of non safety critical stuff you can do with v2 I like v2 I liked
v2 I weigh more than V B because Vita I is is already like I already have
internet in the car right there's a lot of great stuff you can do with v2 I um
like for example you can well where I already have v2 Waze is V die right ways
can route me around traffic jams that's a great example of v2 I mm-hmm and then
okay the car automatically talks to that same service like improving the experience
but it's not a fundamental fallback for safety know if any of your if any of
your if any of your things that require wireless communication are more than qm
like have a nozzle rating you should you previously said that life is work and
then you don't do anything to relax so how do you think about hard work well
what is it what do you think it takes to as great things you know there's a lot
of people saying that there needs to be some balance you know you need to in order
to accomplish great things you need to take some time off each of reflects and
so on now and then some people are just insanely working burning the candle at
both ends how do you think about that I think I was trolling in the Siraj
interview when I said that off camera right before I spoke a little bit we'd
like get out spot this is a joke right like I do nothing it relaxed look where
I am I'm at a party right yeah that's true so no of course I I don't um what I
say that life is work though I mean that like I think that what gives my life
meaning is work I don't mean that every minute of the day you should be working
I actually think this is not the best way to maximize results I think that if
you're working 12 hours a day you should be working smarter and not harder well
so it gives work gives you meaning for some people other source of meaning is
personal relationships yeah like family and so on you've also in that interview
of Sirach or does the the trolling mentioned that one of the things you look
forward to in the future is AI girlfriends yes so at the topic that I'm all
very much fascinated by not necessarily girlfriends but just forming a deep connection
with AI what kind of system do you imagine when you say AI girlfriend whether
you were trolling or not know that one I'm very serious about and I'm serious
about that on both a shallow level and a deep level I think that VR brothels
are coming soon and are gonna be really cool it's not cheating if it's a robot
I see the slogan already but there's I don't know if you've watched it just
watched the black mirror episode i watch the one year yeah yeah oh the the
Ashley - one way da no where there's two friends were having sex with each other
and mo in the VR game your game it's just two guys but yeah one of them was was
a female and yeah there's another mind-blowing concept that in VR you don't
have to be the form you can be to animals having sex weird I mean I'll see you
I said the software Maps the nerve endings right yeah yeah they they sweep a
lot of the fascinating really difficult technical challenges under the rock
like assuming it's possible to do the mapping of the nerve endings then I wish
yeah I saw that the way they did it with a little like stim unit on the head
that'd be amazing so wanna know on a shallow level like you could set up like
almost a brothel with like real dolls and oculus quests right some good
software I think it vehicle novelty experience you know on a deeper like
emotional level I mean yeah I would really like to fall in love with with with
the machine do you see yourself having a long-term relationship of the kind
monogamous relationship that we have now with a robot with a a AI system even
not even just the robot so I think about maybe my ideal future when I was
fifteen I read eliezer yudkowsky early writings mmm-hmm on the singularity and
like that AI is going to surpass human intelligence massively he made some
Moore's law based predictions that I mostly agree with and then I really
struggled for the next couple years of my life like why should I even bother to
learn anything it's all gonna be meaningless when the machines show up right
maybe maybe when I was that young I was still a little bit more pure and really
like clung to that and I'm like wow the machines ain't here yet you know and I
seem to be pretty good at this stuff let's uh let's try my best you know like what's
the worst that happens but the best possible future I see is me sort of merging
with the Machine and the way that I personify this is in a long-term monogamous
relationship with a machine oh you don't think there's room for another human
in your life if you really truly merge with another machine I mean I see
merging I see like the best interface to my brain is like the same relationship
and to merge with an AI right does that merging feel like I see yeah I've seen
couples who've been together for a long time and like I almost think of them as
one person like couples who spend all their time together and that's that's how
you're actually putting what does that merging actually looks like it's not
just a nice channel like a lot of people imagine it's just an efficient link
search link to Wikipedia or something I don't believe in that but it's more
you're saying that there's the same kind of the same kind of relationship you
have one other human that's a deep relationship is that's what merging looks
like that's that's pretty uh I don't believe that link is possible um I think
that that link so you're like oh me to download Wikipedia right to my brain
yeah my reading speed is not limited by my eyes my reading speed is limited by
my inner processing locally and to like bootstrap that sounds kind of unclear
how to do it and horrify but if I am with somebody and I'll use a somebody who
is making a super sophisticated model of me and then running simulations on
that model I'm not gonna get into the question whether the simulations are
conscious or not I don't really want to know what it's doing um but using those
simulations to play out hypothetical futures for me deciding what things to say
to me to guide me along a path and that's how I envision it so on that path to
AI of superhuman level intelligence you've mentioned that you believe in the
singularity that singularity is coming yeah again could be trolling could be
not could be part I'm all trolling his truth in it I don't know what that means
anymore what is the singularity yeah so that's that's really the question how
many years do you think before the singularity what form do you think it will
take does that mean fundamental shifts and capabilities of AI does it mean some
other kind of ideas um maybe this is just my roots but so I can buy a human
beings worth of compute for like a million bucks that I it's about one TPU pod
v3 I want like I think they claim a hundred peda flops that's being generous I
think humans are actually more like twenty so that's like five humans that's
pretty good Google needs to sell their teepees um but I could buy I could buy I
could buy GPUs I could buy a stack of like by 1080 tea eyes build data center
full of them four million box I can get a human worth of compute but when you
look at the total number of flops in the world when you look at human flops
which goes up very very slowly with the population and machine flops which goes
up exponentially but it's still nowhere near I think that's the key thing to talk
about when the singularity happened when most flops in the world are silicon and
not biological that's kind of the crossing point like they are now the dominant
species on the planet and just looking at how technology is progressing when do
you think that could possibly happen you think go to happen in your lifetime oh
yeah definitely my lifetime I've done the math I like 2038 because it's the UNIX
timestamp rollover yeah beautifully put so you've you said that the meaning of life
has to win if you look five years into the future what does winning look like
so [Music] hi there's a lot of I can go into like technical depth to what I
mean by that to win um it may not mean I was criticized for that in the
comments like doesn't this guy want to like save the penguins in Antarctica or
like you know listen to what I'm saying I'm not talking about like I have a
yacht or something I am an agent I am put into this world and I don't really
know what my purpose is but if you're a reinforcement if you're if you're an intelligent
agent and you're put into a world what is the ideal thing to do well the ideal
thing mathematically you go back to like Schmitt Hoover theories about this is
to build a compressive model of the world to build a maximally compressive to
explore the world such that your exploration function maximizes the derivative
of compression of the past mid Hoover has a paper about this and like I took
that kind of as like a personal goal function so what I mean to win I mean like
maybe maybe this is religious but like I think that in the future I might be
given a real purpose or I may decide this purpose myself and then at that point
now I know what the game is and I know how to win I think right now I'm still
just trying to figure out what the game is but once I know so you have you have
imperfect information you have a lot of uncertainty about the reward function and
you're discovering it exactly the purpose is that's that's the better way to
put it the purpose is to maximize it while you have it a lot of uncertainty around
it and you're both reducing the uncertainty and maximizing at the same time
yeah and so that's at the technical level what is the if you believe in the universal
prior yeah what is the universal reward function that's the better way to put
it so that when it's interesting I think I speak for everyone in saying that I
wonder what that reward function is for you and I look forward to seeing that
in five years in ten years I think a lot of people who do myself right and
cheering you on man so I'm I'm a happy you exist and I wish you the best of
luck thanks for talking today man thank you this is a lot of fun you English
(auto-generated)