Enabling fast and effortless customisation in accelerometer based gesture interaction

Transcription

1 Enabling fast and effortless customisation in accelerometer based gesture interaction Jani Mäntjärvi P.O.Bo 1100 Juha Kela P.O.Bo 1100 Panu Korpipää P.O.Bo 1100 Sanna Kallio P.O.Bo 1100 ABSTRACT Accelerometer based gesture control is proposed as a complementar interaction modalit for handheld devices. Predetermined gesture commands or freel trainable b the user can be used for controlling functions also in other devices. To support versatilit of gesture commands in various tpes of personal device applications gestures should be customisable, eas and quick to train. In this paper we eperiment with a procedure for training/recognizing customised accelerometer based gestures with minimum amount of user effort in training. Discrete Hidden Markov Models (HMM) are applied. Recognition results are presented for an eternal device, a DVD plaer gesture commands. A procedure based on adding noise-distorted signal duplicates to training set is applied and it is shown to increase the recognition accurac while decreasing user effort in training. For a set of eight gestures, each trained with two original gestures and with two Gaussian noisedistorted duplicates, the average recognition accurac was 97%, and with two original gestures and with four noisedistorted duplicates, the average recognition accurac was 98%, cross-validated from a total data set of 240 gestures. Use of procedure facilitates quick and effortless customisation in accelerometer based interaction. General terms Human computer interaction, input technolog, mobile devices Kewords Gesture recognition, gesture control Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To cop otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MUM 2004, October 27-29, 2004 College Park, Marland, USA. Copright 2004 ACM /04/10... $5.00" 1. INTRODUCTION Gestures are an intuitive communication channel, which has not et been full utilised in human-computer interaction. Mobile devices, such as, PDA s or mobile phones, and wearable computers provide new possibilities for interacting with various applications, but also introduce new challenges with I/O devices, e.g., small displas and miniature input devices. Gesture input devices (containing accelerometers for detecting movements) could be integrated to clothing, wristwatches, jeweller or mobile terminals, for providing a means for interacting with various of devices and applications located in surroundings. These input devices could be used to control, e.g. home appliances, with simple user definable hand movements. In related work we focus on movement sensor based approaches, which utilise different kinds of sensors, e.g. tilt, acceleration, pressure, conductivit, capacitance, etc. to measure movement. An eample of an implementation is GestureWrist, a wristwatch-tpe gesture recognition device using both capacitance and acceleration sensors to detect simple hand and finger gestures [11]. Accelerometer based gesture recognition is used for eample in a musical performance control and conducting sstem [12], and a glove-based sstem for recognition of a subset of German sign language [3]. In wearable interface called Ubi-finger acceleration, touch and bend sensors are used to detect fied set of hand gestures, and infrared LED for pointing a device to be controlled [14]. Yet another gesture based interaction device is XWand [15]. It utilises both sensor based and camera based technologies capable of detecting the orientation of the device using a 2-ais accelerometer, a 3-ais magnetometer and a 1-ais groscope, and position and pointing direction using two cameras. The user can select a known target device from the environment b pointing, and control with speech and a fied set of simple gestures. Gesture recognition has been also studied for implicit control of functions in cell phones, e.g. for answering and terminating a call, without user having to eplicitl perform the control function [1], [7]. In implicit gesture control, the main problem is that users usuall perform the gesture ver differentl in different cases, e.g. picking up a phone can be done in ver man different was depending on the situation. Eplicit control directl presupposes that gestures trained for 25

2 performing certain functions are alwas repeated as the were trained. As a complementar interaction modalit, acceleration based gesture command interface is quite recent, and man research problems still require solving. The topic is ver wide in scope, since there are a ver large number of possible suitable gestures for certain tasks, as well as man tasks that could potentiall be performed using gestures. Obviousl, the recognition accurac for detecting the gesture commands should be high. Nearl 100% accurac is required for user satisfaction, since too man mistakes cause the users to abandon the method. Moreover, in some applications the control commands need to be trained b the user. If training is too laborious, it ma again cause users to abandon the interaction method. Therefore, the training process should be as effortless and quick as possible. As a sensing device, SoapBo (Sensing, Operating and Activating Peripheral Bo) is utilised in this work. It is a sensor device developed for research activities in ubiquitous computing, contet awareness, multi-modal and remote user interfaces, and low power radio protocols [13]. It is a light, matchbo-sized device with a processor, a versatile set of sensors, and wireless and wired data communications. Because of its small size, wireless communication and batter-powered operation, SoapBo is eas to install into different places, including moving objects. The basic sensor board of SoapBo includes a three-ais acceleration sensor and other sensors for monitoring environment [13]. Various statistical and machine learning methods can potentiall be utilised for training and recognising gestures [1], [7]. This stud applies discrete Hidden Markov Models (HMM), a well-known method, e.g., from speech recognition [10]. HMM is widel used in speech and hand-written character recognition as well as in gesture recognition in videobased and glove-based sstems. In our previous work we have found that people usuall prefer defining their own gestures, concluding that gesture control should be customisable. Moreover, it should be customisable which functions are performed with gestures [6]. This result leads to the important problem, which is the abilit to train and recognise free form gesture commands. The users should be able to carr out training of personal gestures with as few repetitions as possible, since making man repetitions can be a nuisance. The novelt and main contribution of this paper is that the amount of required repetitions performed b a user, thus user s effort in training can be decreased, when discrete HMMs are applied. According to the results, this is feasible based on a procedure where noise-distorted signal duplicates are used in training. It is shown that with the procedure applied the amount of required repetitions is decreased while good recognition accurac is maintained. Organisation of the paper is the following. Gesture interface basic concepts are first defined and categorised. Methods for gesture training and recognition and for decreasing amount of training repetitions are presented. The eperiments and results are provided. The eperiment is based on a DVD gesture control prototpe, which demonstrates the practical feasibilit of gesture recognition. Finall, discussion and suggestions for future work are given together with conclusions. 2. GESTURE CONTROL The main purpose of this paper is to present a procedure for decreasing user effort in customising accelerometer based gesture control when discrete HMM:s are applied. As discussed in previous section, accelerometers are utilised in implementing various tpes of interfaces. To clarif the differences between various approaches we clarif the tpes of movement sensor based user interfaces b the categorisation, Table 1. Table 1: Categorisation and properties of movement sensor based user interfaces Interface Operating Customis-ation Compleit tpe principle 1. Measure & control Direct measurement of tilting, rotation, - Ver low 2. Discrete gesture command 3. Continuous gesture command or amplitude Gesture recognition Continuous gesture recognition Machine learning, freel customisable Machine learning, freel customisable High Ver high Direct measurement and control sstems are not considered as gesture recognition sstems since in their operating principle, measurement of tilt, rotation or amplitude is mapped directl to control. In this paper, we refer gestures to as user hand movements collected with a set of sensors in a handheld device. Hand movements are modelled b machine learning methods in such a wa that an movement performed can be trained for later online recognition. Furthermore, a gesture based device control command is eecuted based on hand movement recognised. In discrete gesture command start and stop of a gesture is defined, e.g., with a button while in continuous gesture command recognition of gestures is carried out online from a flow of hand movements. Handheld devices with gesture recognition enable the control of applications located in a handheld device or in eternal devices in vicinit. Concerning sensor based hand movement interfaces the paper hence focuses on discrete gesture command interfaces (second categor in table 1). There eist multitudes of simple measure & control applications e.g. tilt and rotation based actions. We consider them to belong to the categor one in table 1. 3 GESTURE TRAINING AND RECOGNITION According to our previous stud, users prefer intuitive user definable gestures for gesture-based interaction. This is a challenge for gesture recognition and training sstem, since both online training and recognition are required. To make the usage of the sstem comfortable, low number of repetitions of a gesture is required during the training. On the other hand, a good generalisation performance in recognition must be achieved while maintaining good recognition accurac. Other requirements include; a recogniser must maintain models of several gestures, and when a gesture is performed, training or recognition operations must not take a long time b a sstem. 26

3 This section presents methods used in online gesture recognition in our prototpes. In accelerometer based gesture interaction, sensors produce signal patterns tpical for gestures. These signal patterns are used in generating models that allow the recognition of distinct gestures. We have used discrete Hidden Markov Models (HMM) in recognizing gestures. Main motivation for choosing HMM for our purposes is that the method a modelling tool that can be applied for modelling time-series with spatial and temporal variabilit. The HMM has also been utilised in other eperiments for gesture and speech recognition [7], [11]. Acceleration sensor based gesture recognition using HMM has been studied for eample in [3], [4], [7]. The recognition sstem works in two phases: training and recognition. A schematic block diagram of the sstem is presented in figure 1. Common steps for these phases are signal sampling from three accelerometers to 3D sample signals, preprocessing, and vector quantisation of signals. Repeating the same gesture produces variation of measured signals, because the tempo and the scale of the gesture can change. In preprocessing, data from gestures is first normalised to equal length and amplitude. The data is then submitted to a vector quantiser. The purpose of the vector quantiser is to reduce the dimensionalit of the preprocessed data to one-dimensional sequences of discrete smbols that are used as inputs for the HMMs in training and in recognition. One vector quantiser is designed using an etensive set of gesture data. The vector quantiser is used to quantise all data in our eperiments. Our procedure for gesture training/recognition includes adding various tpes of noise to data. The procedure is eplained in more detail in net section. Figure 1: Block diagram of a gesture recognition/ training sstem 3.1 Preprocessing The preprocessing stage consists of interpolation or etrapolation and scaling. Gesture data is first linearl interpolated or etrapolated if data sequence is too short or too long, respectivel. Then amplitude of data is scaled using linear min-ma scaling. The same parameters are used both in the training and in the recognition phase. 3.2 Vector quantisation The vector quantisation is used to convert preprocessed three dimensional data into one dimensional prototpe vectors. The collection of the prototpe vectors is called a codebook. In our eperiments the size of the codebook is selected empiricall to be 8. Vector quantisation is done using k-means algorithm [2]. 3.3 HMM Hidden Markov Model is a stochastic signal modeling method. HMM is an etension of Markov process. The output of Markov model is the set of states at each instant of time, where each state corresponds to a phsical, observable event. The HMM includes the case where the observation is a probabilistic function of the state. Hence, the HMM is a double stochastic process with an underling stochastic processes that is not observable, but can be observed through another set of stochastic processes that produce the sequence of observation. Formall a HMM can be epressed as λ = (A,B,π) (1), where A denotes the state transition probabilit matri, B is the observation smbol probabilit matri and π is the initial state probabilit vector. The specification of the discrete HMM involves a choosing number of states, a number of discrete smbols, and definition of the three probabilit densities with matri A, B, and π. Usuall, three basic problems must be solved for the real applications: the classification, the decoding and the training. In this stud onl classification and training are relevant and are solved using Viterbi and Baum-Welch algorithms, respectivel. The global structure of the HMM recognition sstem is composed of parallel connection of each trained HMM ( λ 1, λ2,... λm ), where λ i indicates a trained HMM model for each gesture and M is a number of gestures [16]. Hence adding a new HMM or deleting the eisting one is feasible. The recognition of the given unknown gesture is performed b finding an inde of the discrete HMM which produces the maimum probabilit of the observation smbol sequences. In this paper an ergodic topolog, was utilised. Left-to-right model is a special case of ergodic model and often preferred when modelling time-series whose properties sequentiall change over time. However, in the case of gesture recognition from acceleration signals, both ergodic and left-to-right models have been reported to give similar results [3]. The alphabet size, which is the codebook size, used here is 8 and number of states in each model is 5. In [3] it has been concluded that an ergodic model with five state works best for gesture set the use. It has been reported that the number of states does not have significant effect on gesture recognition results [8]. 3.4 Decreasing user effort in training The user eperience in accelerometer based gesture interaction should be as positive as possible and making several training 27

4 repetitions can be a nuisance. Thus, approach for tring to decrease the amount of training repetitions is well justified. It has been shown that adding noise increases detectabilit in decision making in certain conditions [5]. In this paper, we appl the idea of adding noise to eamine whether the idea can be used in decreasing the amount of training repetitions done b the user when discrete HMM:s are applied for the training and recognition task. The approach is to generate new training data, the three dimensional noise-distorted gesture signal duplicates i + n i, b coping the original gesture data vector i and adding random noise vector n i into the cop. We consider two random noise distributions in our eperiments: functions and corresponding gestures used in DVD control prototpe. Table 2: Gestures used in DVD prototpe. Origin of the gesture is presented as a dot and arrow indicates the trajector of the gesture. X,-co-ordinates indicate that gesture is drawn in the air in,-plane in front of the user. Pla Stop Net Previous Uniform distribution. Random samples are generated between [-a, +a], mean µ = 0, variance σ 2 = a 2 /3. Gaussian distribution. Random samples are generated from normal distribution, mean µ = 0, variance σ 2. Increase Decrease Fast forward Fast rewind Various signal to noise ratios (SNR) are eperimented with. SNR is determined as ratio of signal variance to noise variance. 4 PROTOTYPE We have implemented a prototpe based on wireless handheld sensor bo, SoapBo, and PC software for practical eamination of gesture based interaction, figure 2. SoapBo acceleration sensors (ADXL202) measure both dnamic acceleration (e.g. motion of the bo) and static acceleration (e.g. tilt of the bo). The acceleration is measured in three dimensions and sampled at a rate of 46 Hz. The measured signal values are wirelessl transmitted from the remote SoapBo to a central SoapBo that is connected to a Windows PC with a serial connection. The gesture start and end are marked b pushing the button on the SoapBo at the start of the gesture and releasing it at the end, which then activates either training or recognition algorithm. All signal processing and pattern recognition software runs in a PC. Recognition results can be mapped to different control commands and transmitted to the control target using e.g. infrared control signals or TCP/IP socket communication. The mapping between gestures and output functions is done in the training phase b naming the gestures using specific command names, e.g. DVD Pla, for each gesture. The recognition results are mapped to infrared remote control signals and sent to DVD plaer b means of IR transmitter. This prototpe utilizes onl discrete gesture commands. 5 EXPERIMENTS AND RESULTS 5.1 Eperiments Eight gestures were selected to control DVD plaer. Recognition capabilit of HMM based gesture recognition sstem was tested using gestures in table 2. For each gesture, 30 distinct three-dimensional acceleration vectors were collected from one person, and thus the total test data set consisted of 240 repetitions. Length of the three-dimensional acceleration vectors varied depending on the duration of the gesture, mean length l of gestures was 25 samples l [13,45], with σ = 9,85. Figure 3 illustrates three-dimensional acceleration vector for gesture Pla. Figure 2: Conceptual overview of DVD plaer prototpe 4.1 Gesture control of DVD plaer Gesture recogniser is trained with eight popular gestures and used to control basic functions of DVD plaer. The gesture control is provided as an alternative control method to the eisting remote controller. Table 2 presents the control 28

5 Amplitude (mv) ais -ais z-ais Length Figure 3: Three-dimensional acceleration vector for gesture Pla Each three-dimensional vector was either interpolated or etrapolated to a length of 40 samples. Thereafter, vector quantizer was used to map three-dimensional vectors into onedimensional sequence of codebook indices. The codebook was generated from collected gesture vectors using k-means algorithm. After the vector quantization, the gesture was used either to train HMM or to evaluate recognition capabilit of HMM. The eperiments include the following tests (printed with Italics): Eamination of the optimal threshold value to determine the convergence of the HMM. Training of the HMM is an iterative process, which will continue until convergence. Here, the convergence of the HMM was determined using threshold value and following form f ( t) f ( t 1) < threshold avg (2), where ( f ( t) + f ( t 1) ) avg = (3) 2 and f (t) is log-likelihood of the HMM at iteration t. When testing threshold value, the number of training vectors was kept in 5. Eamination of the recognition accurac using different amount of training repetitions. In this test, the recognition rate for each gesture is first calculated b using 2 data vectors for training and the remaining 28 for recognition. The recognition accurac for each gesture is the result of cross validation, so that in the case of 2 data vectors, there are 15 training sets, and the rest of the data (28) is used as the test set 15 times. The procedure is repeated for each gesture, and the result is averaged over all the 8 gestures. This procedure is then repeated with 4, 6, 8, 10 and 12 data vectors for training to find out the recognition accurac when onl original data is used and how man training vectors is required for reaching a proper accurac. Eamination of noise SNR levels to find a noise level leading to best results. In this test, two actual gestures are used for training, plus two cop gestures, in total four. Tests are carried out with Gaussian and uniforml distributed noise. Also in this test the accuracies are the result of cross-validation. Eamination of the effect of using noise distorted signal duplicates in training. In this test, we stud the procedure for decreasing the required user effort in training b coping the original gesture data and adding noise into the cop. The noise distorted cop is then used as training data. Eperiments are carried out with varing number of original + noise distorted signal duplicates. Also in this test the accuracies are the result of cross-validation. 5.2 Recognition results and discussion With user definable gestures, the training has to be done b the user. This means that training situation should be as eas and as quick as possible. Thus, it is important to keep the number of repetitions required from user. Eamination of the optimal threshold value to determine the convergence of the HMM. When testing threshold value, the number of training vectors was kept in 5. Figure 4 shows the recognition rate for different threshold values. There is some variance between gestures, but on average the best result was achieved with threshold value 1,0e -03. This value keeps the number of training iterations between 10-30, depending on gesture. With additional training iterations models do not seem to learn more, but instead overfit and recognition capabilit suffers. Eamination of the recognition accurac using different amount of training repetitions. Effect of the number of the training vectors for recognition rate is shown in figure 5. Recognition accurac over 90% is achieved alread with four training vector while one and two training vectors reach to accuracies below 90%. However, it can be seen that recognition results get better as the number of training vectors increases. With si original training vectors the accurac is over 95%. But training eight gestures, for eample, with si training repetitions requires alread the user to make 48 repetitions which is obviousl a nuisance. Figure 4: Recognition accuracies for various threshold values 29

6 Figure 5: Recognition accuracies for various number of training vectors (original data) Eamination of noise SNR levels to find a noise level leading to best results. Recognition accuracies with various SNRs for Gaussian and uniforml distributed noise are calculated using two original and two noise distorted signal duplicates (2+2) as training vectors. Recognition accuracies with various SNRs are shown in figure 6. It shows that best accurac 97,2% is obtained with Gaussian distributed noise with SNR = 3 while best accurac with uniforml distributed noise 96,3% is obtained with SNR = 5. This suggests that it is reasonable to use those relative noise levels in further eperiments. Eamination of the effect of using noise distorted signal duplicates in training. The recognition accuracies for various number of noise distorted training vectors for uniforml and Gaussian distributed noise are shown in figure 7. It shows that with both tpes of noise recognition accurac over 96% can be achieved with one original and three nois copies (1+3) with both tpes of noise. However, performance with Gaussian noise is slightl better; almost 98% accuracies with two original and two/four nois copies (2+2, 2+4). Figure 6: Recognition accuracies for various relative noise levels for Uniform and Gaussian distributed noise. Compared to the situation where HMMs are trained using onl one original training vector (Fig. 5), the gain achieved b adding one nois cop (1+1) to training set is over 11% with both noise tpes, and the gain achieved b adding three nois copies (1+3) to training set is over 15% with both noise tpes. Furthermore, compared to the situation where two original training vectors are used in training (Fig. 5), the gain achieved b using two originals and nois signal duplicates is at least ~11% percent when Gaussian noise is used. With uniforml distributed noise results are also good. It must be noted that accuracies obtained b using nois copies with 1 to three original data vectors (1+3, 2+2, 2+4, etc.) are all better than accurac obtained using si original training vectors (Fig. 5). Adding the nois vectors to the original training set improves natural variation of the gesture and it becomes better captured, and thus the new training set is more representative sample of the vectors describing the gesture. These results show that good accelerometer based gesture recognition accuracies can be achieved using noise distorted signal duplicates and low amount of original data vectors in training. This decreases user effort considerabl because one or two training repetitions of a gesture instead of si is adequate for reaching proper functionalit of gesture interaction with this dataset and discrete HMMs. Figure 7: Recognition accurac versus various number of noise distorted training vectors for Uniform and Gaussian distributed noise. 6 CONCLUSIONS AND FUTURE WORK An approach for enhancing customisation in accelerometer based gesture interaction was presented. Eperiments were conducted to evaluate a procedure for training/recognizing accelerometer based gestures with minimum amount of user effort in training. Discrete Hidden Markov Models were applied. Recognition results were presented for a DVD plaer gesture commands. A procedure based on adding noisedistorted gesture signal duplicates to training set was applied and it was shown to increase the recognition accurac while decreasing the user effort in training. For a set of eight gestures, each trained with two original gestures and with two Gaussian noise-distorted duplicates, the average recognition accurac was 97%,and with two original gestures and with four noisedistorted duplicates, the average recognition accurac was 98%, cross-validated from a total data set of 240 gestures. Use of the procedure facilitates quick and effortless customisation in accelerometer based interaction. For some tasks, gesture control can be natural and quick. However, man targets of future work 30

7 remain. The results of user dependent recognition should be etended to user independent recognition, as is the goal in speech recognition. Other sensors, e.g., groscopes, should be studied. Furthermore, the feedback of the gesture interface, e.g., b means such as vibration or audio should be studied. ACKNOWLEDGEMENTS We gratefull acknowledge the support of our partners in the Ambience project and Tekes for the funding. REFERENCES [1] Flanagan J, Mäntjärvi J, Korpiaho K, Tikanmäki J (2002). Recognizing movements of a handheld device using smbolic representation and coding of sensor signals. Proceedings of the First Intl. Conference on Mobile and Ubiquitous Multimedia, pp [2] Gersho A, Gra R.M (1991). Vector Quantization and Signal Compression. Kluwer. [3] Hoffman F, Heer P, Hommel G (1997). Velocit Profile Based Recognition of Dnamic Gestures with Discrete Hidden Markov Models. Proceedings of Gesture Workshop 97, Springer Verlag. [4] Kallio S, Kela J, Mäntjärvi J (2003). Online Gesture Recognition Sstem for Mobile Interaction. IEEE International Conference on Sstems, Man & Cbernetics, Volume 3, Washington D.C. USA pp [5] Ka S. (2000) Can detectabilit be improved b adding noise? IEEE Signal Processing letters, Vol 7 No 1. pp [6] Korpipää, P., Häkkilä, J., Kela, J., Ronkainen, S., Känsälä, I. Utilising Contet Ontolog in Mobile Device Application Personalisation. To appear in proc. International Conference on Mobile and Ubiquitous Multimedia [7] Mäntlä V-M, Mäntjärvi J, Seppänen T, Tuulari E (2000). Hand Gesture Recognition of a Mobile Device User. Proceedings of the International IEEE Conference on Multimedia and Epo, pp [8] Mäntlä V-M (2001). Discrete Hidden Markov Models with Application to Isolated User-Dependent Hand Gesture Recognition. VTT Publications. [9] Peltola J, Plomp J, Seppänen T (1999). A Dictionaradaptive Speech Driven User Interface for Distributed Multimedia Platform. Euromicro Workshop on Multimedia and Telecommunications, Milan, Ital. [10] Rabiner L (1998). Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, Vol. 77, No. 2. [11] Rekimoto J (2001). GestureWrist and GesturePad: Unobtrusive Wearable Interaction Devices. Proceedings of the Fifth International Smposium on Wearable Computers, ISWC [12] Sawada H, Hashimoto S. (2000). Gesture Recognition Using an Accelerometer Sensor and Its Application to Musical Performance Control. Electronics and Communications in Japan Part 3, pp [13] Tuulari E, Ylisaukko-oja A (2002). SoapBo: A Platform for Ubiquitous Computing Research and Applications. First International Conference, Pervasive 2002, pp [14] Tsukada K, Yasumura M (2002). Ubi-Finger: Gesture Input Device for Mobile Use. Proceedings of APCHI 2002, Vol. 1, pp [15] Wilson A, Shafer S (2003). Between u and i: XWand: UI for intelligent spaces. Proceedings of the conference on Human factors in computing sstems, CHI 2003, April pp [16] Yoon H.S (2001). Hand Gesture Recognition Using Combined Features of Location, Angle and Velocit. Pattern Recognition 34, pp