VRSafe: A Secure Virtual Keyboard to Mitigate Keystroke Inference in Virtual Reality

Ghost Password Generation. Many models such as RNN-based seq2seq model (Sutskever et al., 2014), transformer based LLM models (Vaswani et al., 2017; Achiam et al., 2023) are capable of generating a similar password sequence based on a given password. However, they are inadequate for this task as they cannot guarantee the subsequence constraint (i.e. original password $S$ is a subsequence of ghost password $S^{\prime}$) is satisfied on every input/output pair. To address this limitation, we need to employ a method that can reliably embed the characters of the original password into the ghost password. In VRSafe, we adopt a pointer-based method with two actions: (i) copy action, which appends the current pointed character in the original password to ghost password, and (ii) inject action, which appends a random ghost character. By progressing the pointer from the first to the last character of the original password, this approach ensures that the subsequence constraint is always satisfied. To decide between the two actions, we model the process as a finite sequence of Bernoulli trials: with probability $P$, the model performs an inject action and with a probability $1-P$, it performs a copy action. Furthermore, to prevent the ghost password from becoming excessively long or identical to the original password, we impose a maximum limit on the number of consecutive ghost characters and a minimum requirement on the total number of injected ghost characters.

Choosing Ghost Characters. We further elaborate the noise injection process (Inject() in Algorithm 1) in the generation of ghost passwords. A straightforward baseline approach is to uniformly choose a ghost character from an alphabet set of all legal characters. However, a selection based on uniform distribution remains vulnerable to simple human inspection. For instance, suppose that the original password contains a date of birth, and if an alphabetic letter is added as a ghost character in between, then the attacker can easily exclude this letter from the inferred text. In order to make the ghost character look similar to a real character, we select ghost characters using simple language models that can capture linguistic connections between characters to make the process context-aware. Unlike neural networks that often require significant computational resources and introduce additional latency, the Markov model offers a lightweight and efficient alternative for ghost character selection. By definition, a $k$-order Markov model assigns a probability distribution over the next character conditioned only on the preceding $k$ characters:

where $count(x_{i-1},x_{i})$ refers to the number of occurrences of string $x_{i-1}x_{i}$ in a given dataset. We test the Markov model using the configuration used in  (Ma et al., 2014) with different orders and find that the 3-order Markov does not incur any noticeable delay in the ghost character generation process and yet provides sufficient context-awareness and randomness.

Implementation. The keyboard in VRSafe is built using Unity based on a publicly available QWERTY keyboard template from the MRTK toolkit by Microsoft(Microsoft, 2024). While entering the password, if the next character is a ghost character, the keyboard will disable all other keys on the keyboard except the ghost character and notify the user via text prompts, ensuring the user’s next action injects the intended noise. After typing the ghost character, the keyboard layout will be restored to normal QWERTY and the user can continue typing. In a normal keyboard, every key fired by the user click will be appended to the textbox. We add a private variable to track whether each character is a real or a ghost character, and only real characters are forwarded to the text output. The ghost password is stored alongside the real password in the textbox GameObject, ensuring that it persists even if the keyboard is closed. We also create a login interface together with the VRSafe keyboard as a VR application using Unity 2022.3.24f1 and deploy it on Meta Quest2 for the user study. The screenshot of the application is shown in Figure 3.

Overhead. Ghost characters enhance the security of the original password, however, they also incur longer text entry time. Therefore, it is critical to carefully consider the overhead introduced in the process. Although entry time can be influenced by various factors such as the user’s experience with the device and typing proficiency, we focus on reducing the overhead introduced by the additional characters, and we further discuss the subjective factors of users’ perceived overhead in the user study (Section 4.3).

Based on the above formula, the overhead can be calculated as the difference in the distance between the original password and the ghost password. Our objective is to reduce the overhead without significantly compromising the enhanced security achieved by VRSafe. We can either reduce the number ghost characters or use weighted selection instead of uniform random choice. The former can be adjusted by choosing different randomness levels discussed in Algorithm 1. Here, we primarily focus on the latter. In natural languages, certain bi-grams occur with similar frequencies, yet their corresponding cursor traversal distances on the QWERTY keyboard can differ substantially. For instance, the bi-gram ”PA” shows up 10963 times and ”PO” appears 11965 times in one million English words (Solso et al., 1979). This indicates that both bi-grams are equally plausible from a linguistic standpoint, selecting either “A” or “O” as a ghost character offers no meaningful advantage to an attacker. However, typing ”PO” results in a much smaller distance cost than ”PA”. Therefore, we can choose ghost characters that are located closer on the keyboard to reduce the moving overhead and yet achieve a similar protection performance. We implement this constraint in two ways, namely hard constraint and soft constraint. With hard constraint, we remove the characters that are unacceptably far from the preceding character by defining a threshold distance $\tau$ and assigning zero probability to any candidate that exceeds the threshold. In contrast, the soft constraint does not explicitly eliminate distant characters from the candidate set. Instead, it adjusts their probabilities of being selected based on the distance. We compute the distance $d$ between each candidate ghost character $c_{i}$ and preceding character $c_{i-1}$, and adjust the selection probability using a softmax weighted function where closer characters receive higher probabilities:

Higher $\lambda$ or lower $\tau$ will both result in smaller overhead with a more compact key distribution. However, it might increase the risk of weakening the enhanced security. We discuss the performance of VRSafe for different values of $\lambda$ and $\tau$ in Section 4.

Honeyword Selection and Storage. Without any doubt, the ghost password should be included as a honeyword, as it is highly likely to be selected by an attacker in a login attempt. To broaden the detection surface, we should include as many honeywords as possible, however, introducing too many honeywords may greatly increase verification time and impair the performance. To maximize detection rate while using a reasonable number of honeywords, we rank all possible guesses in descending order of probability from the password guessing model and choose the top $n-1$ guesses excluding the ghost password and the original password, making $n$ honeywords in total. As for false alarms caused by the user due to typographic errors, since the ghost password has a requirement for the minimum number of injected characters, the likelihood that a user’s typo accidentally matches the ghost password should be minimal.

We now discuss how to store the honeywords at the server. We note that the ghost password is generated from a probabilistic model which varies across login sessions, and the honeywords are derived based on it. As a result, each time a user logs in using VRSafe, we might get a different honeyword set and the total number tends to increase as the user continues to log in multiple times over time. To reduce the storage overhead associated with the growing number of honeywords, we use bloom filter (BF)(Bloom, 1970), a compact data structure that tests whether an element is a member of a set with possible false positives but guarantees that there are no false negatives. Using BF has two advantages over storing honeywords as individual hashes. First, it provides insert and lookup operation with constant time $O(k)$ regardless of the number of elements already inserted, where $k$ is the number of hash functions of BF defined during initialization. Second, the space cost of BF is fixed and it does not grow with the number of stored elements. However, BF also has its limitations. For a given BF consisting of $k$ hash functions and size of the bit array $m$, there would be a maximum expected elements $n$ given a certain false positive rate (FPR) $p_{fp}$:

At the same time, the bloom filter does not support deletion operation, which means that once the number of inserted elements approaches the expected maximum $n$, we can only rebuild a new filter, otherwise the FPR will become higher than expected. Therefore, we need to have a good estimate on the expected growth trend of the password logins by the user in order to avoid rebuilding the BF frequently. According to a study on web password habits (Florencio and Herley, 2007), on average, a user logs into a website approximately 3.22 times per day (i.e. roughly 1,000 times per year). Such estimates can be used to determine the expected interval (e.g., 6 or 12 months) for rebuilding a BF and the expected maximum number of elements.

#1: Targeted Guessing vs. Simple Typo Correction. We first illustrate that evaluating the effectiveness of our defense under the password guessing model is a more realistic setup. To simulate passwords inferred from keystrokes, we randomly sample 20,000 passwords in the test set. Specifically, each character is replaced with one of its adjacent keys on the QWERTY layout with a probability of 5%, which aligns with a character level inference accuracy of approximately 95% in state-of-the-art studies (Yang et al., 2023). We then copy all passwords into a Google Doc, refining passwords with ”spelling and grammar” tools. Some suggestions will split the word into phrases (e.g. ”iloveyou” to ”i love you”), we only accept the suggestions that retain a single word. At the same time, we input the passwords into the guessing model and pick the top-1 guess of each password. We find that the refined passwords from Google Doc have similar performance (64.55%) compared to targeted password guessing model (64.69%) when no ghost characters are injected, but it quickly falls behind once a small amount of noise is introduced (27.52% vs. 32.78% with an injection probability $P=0.1$). In real-world cases, it is also likely to have false negative or false positive keystrokes, which will result in larger divergence between inferred passwords and original passwords. Therefore, using the password guessing model can more accurately reflect the performance of our defense against realistic attackers.

#2: Accuracy Analysis. We sample passwords in different categories to simulate users with different password choosing habits. First, each character in a password can be classified as either digits, letters or symbols (Weir et al., 2009), and prior work has shown that the number of classes is one of the key factors affecting password guessing performance (Melicher et al., 2016; Wang et al., 2023; Kelley et al., 2012). We categorize passwords based on the number of classes they consist of, denoted as Class-X where $X$ denotes the number of classes of characters in the password. For instance, the password string ”Jamesbond007” belongs to Class-2 passwords. In addition, password length is another important factor that influences password guessing performance, with longer passwords generally being more difficult to guess accurately. In  (Wang et al., 2023), the researchers found that their model outperformed prior approaches for passwords with lengths between 10 and 16 characters. Based on this, we categorize passwords into Short ($<10$), Medium ($10-16$) and Long ($>16$) passwords to provide comprehensive evaluations of our defense method. We input passwords to our noise injection algorithm using $p_{0}=0.5,\Delta p=0.05,\alpha=0.1$ at various randomness level $r$. Then we forward the ghost passwords to Pass2Edit password guessing model to determine the final adversary’s guesses. Fig. 6 shows password guessing accuracy under different password lengths and number of classes at different number of guesses ($10,100,1000$), which are common numbers evaluated in prior research (Wang et al., 2023; Pal et al., 2019; Wang et al., 2016). The x-axis represents different randomness levels $r$ as described in Algorithm 1 and y-axis represents guessing accuracy under top-k guesses. Overall, using the Markov model in selecting ghost characters results in a better performance compared to uniform selection, especially for long passwords (see Table 3). The password accuracy of Class-2, Long passwords using Markov model achieve 5% to 15% lower accuracy compared to uniform selection under the same setting when 1,000 guesses are allowed across all randomness levels (Fig. 6(e)). With different number of allowed guesses, we observe that the adversary’s guessing accuracy has an approximately linear relationship with the logarithm of the number of guesses, and the accuracy under 10 guesses is only around 20% (see Fig. 5). When the number of guesses increases, an attacker typically needs to generate ten times as many candidates in order to achieve approximately twice the accuracy, and the improvement is even smaller for more complex passwords.

#3: Overhead in Typing. While increasing the number of ghost characters consistently reduces the attack accuracy, it also introduces additional overhead which can potentially impact usability. As shown in Table 5, overhead measured in moving distance always exhibits a trade-off relationship with guessing accuracy. For instance, when the randomness level is fixed at 0.4, increasing $\lambda$ from 0 (i.e. no constraint) to 0.2 and 0.5, the overhead reduces from 34.5% to 24.2% and 18.1%, respectively, but simultaneously increases adversary’s guessing accuracy to 1.12% and 3.5%, respectively. We also see that, the main factor affecting accuracy is the randomness level, and applying constraints does not cause guessing accuracy to deviate significantly from the baseline accuracy at the same randomness level without constraints. Moreover, as more ghost characters are injected, our method achieves greater overhead reduction, thereby improving usability. These results demonstrate that VRSafe can be flexibly configured in different methods and parameters for different user and application needs.

#4: Detecting Malicious Logins. Previous experiments show that injecting ghost characters cannot prevent the adversary from guessing the original password if we allow a large number of guesses. In this experiment, we build a malicious login detector with bloom filter with maximum expected element, $n=10^{6}$ elements, expected $FPR=10^{-30}$ and SHA-256 as hash function to ensure enough capacity to store honeywords with low risk of false positive. We believe this is a reasonable setup for most websites that enforce a mandatory password renewal policy not exceeding one year. Given that a typical user is likely to log in thousands of times in this period (Florencio and Herley, 2007), the chosen configuration ensures that a sufficient number of honeywords can be generated without exceeding the bloom filter’s maximum expected capacity. We generate a total of 20 honeywords including the ghost password using the same password guessing framework, but with only 1/10 of the data size, which is already sufficient for detecting malicious login at high accuracy. As shown in Table 4, on average, 52.04% of malicious attempts can be detected in the first guess, and 83.97% are identified within 10 malicious login attempts. If we expand the number of honeywords to 100, the detector can achieve 72.43% and 96.05% detection rate under 1 and 10 login attempts, respectively.

#5: Computing Resource Cost. We evaluated the computing resource usage for VRSafe. The time and memory costs of the malicious login detection bloom filter are summarized in Table 6, with a storage space cost of approximately 17.55 MB. We further measured the CPU and memory cost for running VRSafe using Android Debug Bridge (ADB), a widely used development toolkit recommended in official Meta documentation (Meta, 2024). Over a 10-minute profiling session on Meta Quest 2, VRSafe with ghost character added during typing has a CPU utilization averaging 48% (with a range of 35-62%) and memory averaging $1.77$ $\mathrm{GB}$. In comparison, typing without the ghost character (i.e. using the regular QWERTY keyboard) showed a similar average CPU utilization of 48% (with a range of 35-52%) and a slightly lower average memory usage of $1.75$ $\mathrm{GB}$. These results indicate that enabling the ghost character introduces only marginal memory overhead and no noticeable increase in average CPU utilization.

Methods. Our study is approved by our Institutional Review Board (IRB). To simulate user account logins using password within a VR environment, we developed a VR application using Unity 2022.3.24f1 that uses VRSafe as the keyboard for typing (Fig. 3). Participants in our study were seated in a chair and instructed to complete the password typing tasks, and their actions were recorded by a camera to simulate video-based keystroke inference attacks (see Fig. 8 in Appendix A). All participants were informed of the recording and provided consent. To protect participants’ privacy, facial information was excluded from the recordings, and the passwords used for the typing tasks were selected from the publicly available leaked datasets. We did not initially disclose that the recording simulated an attack scenario, to avoid influencing participants’ natural typing behavior. After completing the study, we fully debriefed participants on the experiment design, including the purpose of the video recording, and addressed any concerns or questions. Each participant received 10 USD as appreciation. Participants were first introduced to the essential knowledge for using the VR device (e.g., moving the cursor, typing), as well as the differences between a standard keyboard and the VRSafe keyboard. They were then given five minutes to explore the application freely. The main experiment consisted of ten password typing tasks. The first two tasks involved standard keyboard input without ghost characters, serving as a baseline. The remaining eight tasks incorporated ghost characters, simulating the enhanced security mechanism of VRSafe. After completing each task, the participant notifies the researcher and clicks the “Next” button on the login page to proceed to the next task. Upon completion of all tasks, participants were asked to complete a post-study questionnaire regarding their experience. We used MediaPipe (Zhang et al., 2020), a popular hand-tracking framework to extract the keystrokes from the collected video clips.

Results: We recruited 15 participants (mean age = 25.6, sd = 4.22; 7 males, 8 females) in this study through printed flyers and announcements in weekly newsletters. All participants were over 18 years old. There were no restrictions regarding student status or prior VR experience, and the only requirement was for the participant to have a normal or corrected-to-normal vision when wearing the VR headset. Among the 15 participants, 9 do not have any prior experience with VR headset, and the remaining 6 use VR devices less than once per month. A total of 150 video clips were recorded, of which 6 were discarded. ²²2Mediapipe failed to recognize hands in 2 videos since the participant had long wearable nail extensions and 3 videos were discarded since the participant misunderstood the ghost character typing process as a bug and terminated the task. 1 video was discarded because the hand tracking was lost and the participant could not proceed with the task. The average password length across all tasks was 8.35. The average number of keystrokes detected in the baseline tasks (without ghost characters) was 10.5, while the average in the remaining eight tasks (with ghost characters) was 16.2.