March 27th of this year Comma.ai made a blog post on Medium entitled “Self-Driving Car For Free”. In the blog post they detail how using the webcam you’ve already got, and a few other pieces connected to your laptop you can essentially turn your car (if it’s one of the 40 that are compatible with openpilot) into a self-driving car. Pretty neat.
The openpilot repository on Github currently has 3.7k forks, and over 15k people have starred it. One of those people is 15 year old Leon Hillman. Leon was able to use two computers, a webcam, and openpilot, to drive a Honda Civic in the 7-year-old Grand Theft Auto V. But it wasn’t as simple at pointing a camera at the screen and pressing start.
The way the openpilot system works is using a collection of cameras and sensors to understand where the car is in space, and to observe the driver. There are two cameras, one that faces forward, and one that points at the driver. Then, there are two infrared cameras but that’s only for seeing at night.
In order to actually control the car, the openpilot system needs to have access to the physical controls of the car. It has to be able to steer, accelerate, and brake. To do this you can use a car harness, along with Comma.ai’s black panda. But this brings up a complication, what happens when the car your driving doesn’t have a steering wheel or even a gas pedal? This is the case if you’re trying, like Leo, to control your car in Grand Theft Auto.
In addition to this, the car has to be one that is compatible with openpilot’s software. So the tasks for Leo were to first, build a car inside Grand Theft Auto that would be compatible with openpilot. So he chose a good candidate, a car “based” on 2018 Honda Civic. Cars in the real world relay information about their status through what’s called a CANBUS. Cars in GTA don’t have one of these “[so] rewriting that took me around 2 weeks, keep in mind I started with that project in the end of march. After those two quite frustrating weeks I had a working output for the steering angle and the gas and the brake.”
The next task would be to determine an input to control the car with openpilot. He first figured out a way that openpilot was essentially using the W, A, S, D, keys to control the car which is pretty good! But that wasn’t enough for Leo, in his words “I wouldn’t even consider this an average 13 year old GTA online player, so I wanted to have continuous input”. So he set out to use a PlayStation controller after 5 days deciding that wouldn’t work he decided on an Xbox controller. But now he had to figure out to adjust the openpilot output, to match the Xbox controller input. “Turns out you need to multiply the steering output with 2,5 and add 1600 to it”.
Once he had that figured, it was showtime. Connecting a webcam to openpilot on his laptop, and then running Xbox on a separate PC with the webcam pointed at the display he was off to Vespucci Beach, Grand Senora, or wherever his 2018 self-driving Honda Civic could take him. Leo shared a video on his twitter @littlemountainman. You can see the whole setup, and even see him demonstrate that it’s not recorded as he wiggles his mouse to change the camera set up.