Self-Driving RC Car

This summer, upon completing the Machine Learning MOOC offered by Stanford University on Coursera, I wanted to enjoy the fruits of my labour through a side-project. I achieved this by recreating Hamuchiwa’s AutoRCCcar project and programming my own neural network using the python library Keras. Learning Keras was greatly simplified by reading the book “Deep Learning With Python”.

SDC Configuration(1).png


The Raspberry Pi is responsible for obtaining distance data from the sensors, and images from the PiCamera. The Raspberry Pi then sends this data to the Laptop using TCP/IP. At the laptop, image processing is done and the trained deep learning model makes a prediction on what button to press in the form of a one-hot-encoded array (e.g. [0 0 1] to move right). The laptop then sends instructions to the Arduino which is interfaced with the RC controller to send the signal to move.






Deep Learning

Data Collection and Preprocessing

With the limitations of my RC car, training was a made a lot more difficult. The turning radius of the car prevented me from training on a circular track. Thus, I had to train on a linear track, resulting in having to manually pick up the car to bring it back to the staring line. By the end of the training, I had  gathered about 1000 samples, yet it still wasn’t enough. So, I used a trick I learned from the Machine Learning MOOC to generate more data by flipping all the images and labels in the x direction.

Building The Model

The next step was to actually build the Keras model. We first split the data into training and testing sets. Our model consists of 3 Dense layers, but we add some Dropout between layers to prevent the model from picking up random patterns. The final activation is a softmax function which is necessary for our multi-label classification problem.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)

# define model
model = Sequential()
training_start = time.time()
print("Training data...")
model.add(Dense(100, input_dim=38400, init='uniform'))
model.add(Dense(100, init='uniform'))
model.add(Dense(3, init='uniform'))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)

# Fit the model
history =, y_train,
          validation_data=(X_test, y_test))


The final result is cross validation accuracy of 77% with loss under 0.3.






Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s