Kevin Alejandro Roundy with Dr. Michael A. Goodrich, Computer Science
Original Project Goals
The purpose of this research project was to further Nissan Motor Company’s research in technologies designed to improve driver comfort and safety. In particular I looked into the possibility of using force feedback on a steering wheel to help a human driver steer more safely and more comfortably. We hoped both to help distracted drivers stay in their lanes and to warn them when they choose to make unsafe lane changes. We did our work with a driving simulator, but recent technology makes it possible to detect both of these situations in the real world by using a video camera and placing sensors around the perimeter of the automobile. A similar existing application helps truckers by turning on a warning light come on when they are following another car too closely. Unfortunately, visual warning systems can actually aggravate the situation by distracting the driver and overloading the mind’s visual channel. The nice thing about using force-feedback in a steering wheel is that it can provide a corrective force in addition to a warning signal, allowing the driver’s reaction times to be very good.
Another item on our research agenda was to experiment with a new algorithm in the branch of machine learning known as reinforcement learning. Dr. Goodrich named this particular algorithm “satisficing Q-learning,” because it integrates the principle of satisficing into the wellknown Q-learning algorithm. In particular, we wanted to develop this algorithm and show that it would allow the computer to learn what forces should be applied to the steering wheel under different conditions.
Because real cars are expensive and dangerous, we used a driving simulator to conduct this experiment. Fig. 1 shows a screenshot of TORCS (http://torcs.sourceforge.net), an opensource driving simulator written in C++. The driver sits down and controls the car with a Microsoft Sidewinder Force Feedback steering wheel and gas and brake pedals. Because the steering wheel is a Microsoft product, it was made with Windows in mind, and doesn’t work well with other platforms. This meant that I had to make lots of changes to the simulator before I was able to use the force-feedback characteristics of the steering wheel.
I implemented the satisficing Q-learning algorithm to control the forces applied to the steering wheel and we experimented primarily with two types of forces: vibrations and forces to one side or the other. Preliminary experiments showed that vibrations are strictly less helpful than forces from one side to the other because they are simple warnings that communicate no information. For example, when the driver is about to drift into the left-hand lane, a vibration tells him that he is leaving the lane, but not whether he has drifted left or right. A nudge to the right, on the other hand, provides both a warning signal and a corrective force, resulting in an immediate corrective reaction. The state space of the Q-learning algorithm was a discretized measure of time-to-lanecrossing. The action space consisted of forces of different magnitudes. We rewarded low steering-entropy (steering entropy is a measure of driver distraction) and penalized for lane departures.
We conducted our experiments with human test subjects and told them to drive around a course in the right-hand lane for 50 minutes. We used three different force-profiles to see which was the most effective. First was the control, where the human subject drove the simulator with no force at all. The second was a fixed force profile that offered a nudge towards the center of the lane just as the car began to drift out of the road. Lastly was the variable force-profile that the machine learning algorithm produced on its own. For half of the time we made the test subjects drive while completing a secondary task. We chose to flash simple arithmetic problems across the top of the screen and have the driver respond by clicking buttons on the steering wheel.
The results showed that force-feedback can indeed assist a driver in maintaining lane position. Drivers performed best with the force profiles given by the machine-learning algorithm. These forces proved to be very different from those of the fixed force profile that we had provided. Normal steering wheels exhibit a return-to-center force, i.e. the steering wheel, wants to go to the center. The machine-learning algorithm essentially learned to adjust this return-to-center force to the left on left turns and to the right on right turns. As a proof of concept we also showed that the car can actually learn to drive itself without the aid of a human at the wheel.
We also learned lots of things about how the algorithm behaves with human drivers. Some attentive drivers did not need the forces at all and were able to keep on task without it. For these individuals the algorithm would eventually learn that very few intervening forces were necessary. Worse drivers caused the algorithm to learn quickly and settle on a very strong forceprofile that guided the car along inside the lanes.
Dr. Goodrich presented some of our results to Nissan in the summer of 2003 and got very positive feedback. This contract with Nissan is ongoing and was taken over from me by BYU student Morgan Quigley. Further experiments will involve driving with erratic drivers on the road and with better driving simulators.