Skyler Greenman and Dr. Bret Swan, Department of Information Technology
I am reporting to the MTC on August 15th. I am therefore submitting my final report a few months early of the December deadline. However, I have put nearly a year’s amount of work into this project as I started researching early knowing that there were many different ideas and concepts that I had to understand before I could begin experimenting with the hardware. For example, I had to understand how an Inertial Measurement Unit (IMU) when attached to a user’s hand could navigate a phone like a mouse and how to overhaul the Android operating system to create a fake cursor.
I soon realized the largest hurdle to this project would be to figure out how to create an absolute navigational system that was not tied to any external docks (e.g. infrared, GPS, etc.) The technology—to move a mouse with one’s hand in a 3-D space— has been achieved by only a few companies most notably, Nintendo and Gyration who use a mixture of accelerometers and gyroscopes (essentially an IMU) to determine the hand’s position. In theory, the derivative of the accelerometer’s data needs to be taken twice to achieve it’s position. This leads to exponential drift as the constant grows uncontrollably due to the inaccuracies of consumer accelerometers. The solution would lie in obtaining a type of filter to reset the accelerometer data which could be as simple as averaging the current coordinate with the last ten—or as complex as the byzantine graduate-level Kalman Filter. I discovered many sub-par mathematical filters which provide less than satisfactory results. After many hours of research on the subject, I realized there were some hardware companies offering an accurate solution built into their micro-controllers. They used an algorithm dubbed “sensor fusion” which allowed the use of Direct Cosine Matrices and Euler’s Angles to manipulate the accelerometer and gyroscope data to essentially reset each other. However, after 30+ hours of initial research, I found no open source code for this “sensor fusion.” I know understand it to be proprietary and doubt it will be available to the public any time soon. Nonetheless, I found a board, the ArduIMU v3 which claimed it would release its source code for the algorithm by early Spring 2012. However, the company never followed up on its claim.
An alternative solution had to be found. I learned navigational math from my professor including how to calculate correct positioning through double integration of accelerometer data. We were able to create a simple “fusion sensor” code that worked in theory. Nevertheless, things in theory rarely work in reality. The algorithm failed and I was forced to use only gyroscope angles for the gloves’ movement. This rendered the idea of making the glove mimic a keyboard impossible.
Additionally, I had to decide how I was going to control the glove’s mouse through an Android app. I had learned many of the fundamentals of creating Android apps through tutorials online; however, I soon realized the magnitude of the project was entirely above my level of knowledge. I would have to hack Arduino’s system kernel in order to take control of the phone’s OS navigational system (each app, is contained in its own virtual space—which makes it hard to interact universally with the phone). This solution was not scalable as it would force us to modify each phone’s operating system in order for it to work. In order to get around this problem, I began to research alternative solutions. I found out that the company Arduino had just recently launched a new series of micro-controllers which emulated HID devices (keyboards, mice, joysticks, etc.). This was revolutionary because previously, you had to purchase a $2,000 vendor license and program your own USB protocol, which usually involves teams of highly-skilled programmers to create it.
Additionally, I researched the possibility of extending the smart phone onto video glasses to increase the screen size and allow a user to wear my gloves. I was shocked to find out that even the high end glasses costing at $3500 a piece would not be able to read this document (12-point font) (I tested this fact in Professor Swan’s lab). The most common commercial resolution for glasses in the $200 range is around 320×240 pixel/inch. The pixel ratio must increase tremendously before this idea becomes viable in any manner.
I have a working navigational glove that uses hand tilt motion to control cursor position by accessing the ArduIMU board’s yaw and pitch gyroscopes to smoothly move the cursor on a computer screen. I was able to get the left, middle and right mouse buttons to work. Additionally, those buttons can double as keyboard input buttons. However, due to the tight vendor restrictions placed on Android 4.0, I cannot pair the glove with an Android smart phone as the micro-controller’s USB vendor ID is Arduino based—which has no presence in the tech world. The only solution is to wait for Android to loosen its grip on USB vendor restrictions.
I was unsuccessful at finding an algorithm which could provide accurate data from both gyroscope and accelerometer sensors. However I learned how to communicate between two Arduino micro-controllers through the serial protocol. I learned how to effectively use the I2C protocol and was able to successfully connect the Arduino Leonardo (Master) with the ArduIMU v3 (Slave). I learned the different uses and functions of the pins on a micro-controller and how they fundamentally worked in tandem with the CPU through logic gates. I learned the fundamentals of creating Android apps and how you interact with the different levels of app events. This allows me to create heavily-interactive apps in the future. All of these skill sets have given me a critical understanding of the amount of effort and planning needed to accomplish a project of this magnitude. There are basic criteria that any individual needs to figure out before he actually begins a project. I experienced this multiple times and suffered many lost hours by going down dead ends.
The basic reality: Due to many technology restrictions, the only solution left is to wait for video glasses technology to increase their pixel/inch ratio while waiting for Android to also open its door to allow more USB vendor classes. Moreover, the “sensor fusion” algorithm must be made readily available to the public if one is to incorporate actual movement (accelerometers) instead of just tilt.
