project-13-a SOFTENG-702-Android-Cursor

Android library for pointing and clicking with a floated cursor using sensors

See releases for apk download: GitLab, GitHub

For GitLab, if the link doesn't work, go to Wiki > Releases: http://gitlab.nectar.auckland.ac.nz/literate-chickens/project-13-a/wikis/Releases

System Requirements

• Android phone
• SDK at least 24 (Android 7.0)
  • Target SDK is 27 (Android 8.1)
• You need to allow installation of apps from sources other than the Play Store

Use

Interactions

• Volume Down: Click
• Long Press Volume Down: Dragging (e.g. drag refresh, scroll, drag and drop)
• Volume Up: Traverse through different cursors or changing sensitivity (depending on a setting).
• Long press Volume Up: Focus mode (cursor moves slowly while the button is held down).
• Volume Up & Volume Down: Calibrate to re-centre cursor at current pitch and roll angles

Points to note

• Each page has it's own cursor, thus cursor selection, sensitivity, and calibration is for that page only
  • In FindFood, the restaurants list is a page, and every time you click a restaurant it's a new page
• Back-tapping is an external service and is not 100% stable. Sensitivity also depends on the specific device, and may be too sensitive or too insensitive
  • We have also configured it so that double-tapping on the back of the device also performs a click, as double-tapping may be more sensitive than single-tapping

Use Cases

• Single-handed use
• Phone use without occlusion of the screen
• Projecting phone screen onto a display and interacting (e.g. for demos)

FindFood

This app uses the internet to find restaurants around you. For each result, you can add a note about the restaurant, which will be saved across sessions.

Inputs:

• Your location will be retrieved via your device’s GPS.
• You can type in your notes following the instructions onscreen.

Outputs:

• You will be displayed with a list of restaurants around you (sorted by location proximity)
• You will be displayed the restaurant information available when you click into the restaurant's details

Workflow:

1. open the app, it will retrieve your location
2. select a restaurant
3. click the button to add a note
4. the first 4 lines of the note will be displayed on screen (as preview)
5. click on the note to view or edit it
6. long click the note and you'll be able to remove it
7. you can add multiple notes for the same restaurant
8. these notes will be saved across sessions (stored in a database)

User Study App

Workflow:

1. This app comprises of 8 buttons
2. You can select what the volume up button (short press) does with the settings radio buttons, and use the physical buttons to configure the cursor
3. Start the study by clicking any button
4. All but one of the buttons are greyed out
5. Click that button that's "lit up"
6. Once you've successfully clicked the button, another button will "light up"
7. When all the buttons light up (after 20 clicks), the study has finished
8. Logs can be retrieved from the phone's internal storage at Android/data/nz.ac.auckland.usabilitytest/logs
   • the first few lines list the buttons and print their IDs respectively
   • the following 20 lines are the button ID and device system time (ms) at which they were clicked
9. We take the difference between these times to accurately calculate the time taken to click a button
10. We will perform data analysis on this data to see times taken with different pointing and clicking methods, and whether button locations impact on performance

Physical Buttons

Currently, as a proof of concept, the volume up button can be configured to toggle cursor sensitivity or switch cursor. This is an example of settings the application developer using our library can provide, to give their users more fine-grained control.

User Study app provides this setting, so users can set VOL_UP to change cursor sensitivity, and select the mode that they prefer (see Cursor Sensitivity Levels below).

FindFood does not provide this setting, and the developer (us) has set in the code that VOL_UP changes the cursor appearance. The user therefore cannot change the cursor sensitivity.

Cursor Sensitivity Levels

• DWELL: if we think you intend on dwelling on the same location, we will keep the cursor stationary from jitter
• SMOOTHEST: lowest sensitivity to change in angles, but may appear laggy
• NO_DWELL: same sensitivity as dwell, but won't keep cursor stationary, for fine-grained control

Note: FindFood is set to SMOOTHEST

Back-Tapping to Click Functionality

In order to utilising tapping the back of the device for clicking, the βTap Service application must be installed on the device. Simply install the application from Google Play - Download βTap Service. Then, the application that uses the cursor should be able to perform the clicking function via a back tap.

For developers

Install

Install the library cursor by importing it using New -> Import Module and selecting the cursor module in master

Run gradle clean installDebug locally

Customise

You want to extend the current Activity using the Cursor Activity class. This class allows your view to use the cursor. You can bind to onSensorChanged's super method to listen to sensor changes.

• setCursorSensitivity(CursorSensitivity sensitivity) - sets the current cursor's sensitivity
• setVolUpTogglesSensitivity(boolean togglesSensitivity) - Set what vol up button does. Currently either toggles cursor sensitivity or switches cursor
• toggleNextCursorSensitivity() - toggles to the next cursor's preset sensitivity
• toggleNextCursor() - cycles between different cursor appearances
• simulateTouchMove() - simulate a touch move event on the current position
• simulateTouchDown() - simulate a touch down event on the current position
• simulateTouchUp() - simulate a touch release event on the current position

See FindFood for an example of a complete app that has our library imported

Project Plan vs. Implementation

Project Plan	Implementation	Is achieved
Identify alternate ways to perform pointing without using a finger, and evaluate suitability and feasibility	Movement (pitch and roll) of device to control cursor	Looked at the different methods of pointing - decided on a cursor
Identify alternate ways to perform clicking without occlusion of the screen, and evaluate suitability and feasibility	Included back tapping and side button click (volume down button).	Completed using multiple clicking techniques
Identify available embedded mobile phone sensors to provide alternate input (e.g. accelerometer, gyroscope, audio, etc. and combinations), and evaluate suitability and feasibility	Used a combination of the accelerometer and the magnetometer for cursor movement. For back-tapping, the service we used utilised a combination of accelerometer, gyroscope, gravity, and microphone [1].	Combinations of sensors identified and utilised
Identify the ways a user can input information into a smartphone device by using an accelerometer	Pointing is achieved by a moving cursor controlled by the movement of the device. Different click methods are included through the volume down button and back tapping	Done with a point & click method.
Implement mapping onto phone coordinates for pointing	Pitch and roll angles are mapped onto y and x coordinates of cursor, respectively.	Obtaining coordinates is achieved, with a multiplier that depends on the size of the display
Evaluate the possibility and suitability of rebinding physical buttons	Volume Down: Click Long Press Volume Down: Dragging (e.g. drag refresh, scroll, drag and drop) \ Volume Up: Traverse through different cursors or changing sensitivity (depending on a setting). Long press Volume Up: Focus mode (cursor moves slowly while the button is held down). Volume Up & Volume Down: Calibrate to re-centre cursor at current pitch and roll angles	Managed to rebind physical buttons for certain tasks
Identify thresholds of movement to detect an input or action	Using different sensitivity values for movement and dwell detection. The cursor sensitivity level of DWELL has a threshold where if we think you intend on dwelling on the same location, we will keep the cursor stationary from jitter.	Experimented with the different threshold values
Investigate between having our tool as a library that can be added into applications, and adding it as a separate application that runs on the phone independent of the current application	Made android cursor an independent library that can be utilized in other sample applications (i.e. in FindFood and Sample App). We provide methods where they can customise parameters accordingly (e.g. in our sample apps we have settings for users to select their own button rebinding).	Created a library so developers can use the point and click method. We cannot be a system application to overlay other applications due to security reasons, and our cursor also can't draw over pop ups for the same reason.
Consider and implement the toggling this tool on and off	Developers can choose whether to include a cursor in their app.	Because it's imported on code level, it is up to the app developer to decide whether they want to import our library, and/or whether they want a toggle in their app to change whether or not to utilise what we provide in our library
Consider cancellations for misinputs and reversibility	Works exactly the same way as a tap, so just move the cursor and click whatever cancellation means is provided by the app.	No longer applicable, as the cursor can just click the back buttons, in the same way as tapping the touch screen.
Implementation Plan
Set up architecture, repository, and testing framework	Overall set up with the library, sample application and etc is included.	Completed
Research algorithms for target selection (magnetic/sticky cursor, triangulation of motion, etc.)	We implemented mapping to coordinates, low pass filtering to smooth movement, dwelling, and other thresholds and parameters.	Completed, with some presets that can be put into settings
Prototype algorithms as a small demo	Improved filtering and parameters throughout development.	Completed
Calculate mapping phone sensor events to change in target selection/pointer location on the screen	Calculation performed with parameters set based on pilot studies.	Completed
Simulating a click event	The library allows for different click methods that do not require a touch on the screen. This is through the volume down button and the back taper (if configured correctly).	By exploring alternative click methods a click event is achieved.
Recentering/calibration	The cursor can be recentered according to the current pitch and roll angles.	Achieved
Toggle on/off	Developers can decide whether to include this library in their applications.	The app developer has the flexibility in deciding whether they want to include this technique in their application.
Demo application	Utilized FindFood for using the point and click method for a realistic application. Also implemented a sample application which would provide a suitable indication for the usability study	It has been applied to standalone applications.

Note to the Marker: Teamwork and Workflow

Since we have experience working in this team, we have a preferred workflow which includes using peer/mob programming as well as squashing and merging branches. Additionally, we have started using our implementation using GitHub before the GitLab had been set up. As a result, commit history that was on branches which got squashed and merged will not be immediately visible on GitLab, and commit numbers and commit counts may not reflect the exact amount of contribution each team member has made. Our master branch with all the final implementation and master history is on GitLab, and we have confirmed with Danielle that this is fine. We also confirm that we have made equal amounts of contribution to this project (25% each).

References:

[1] E. Granell, L. A. Leiva (2016). Less Is More: Efficient Back-of-Device Tap Input Detection Using Built-in Smartphone Sensors. In Proc. ISS '16. ACM