How to Control your Android Device Using Facial Gestures

What to know

You can use your phone’s front camera to control your Android device when you set up the “Camera switches” accessibility feature.
To enable Camera switches, go to Settings > Accessibility > Switch Access and turn on the Use Switch access toggle.
You can then configure this accessibility function to control your device, navigate using facial expressions, and assign different functions to different gestures.

Google offers an Accessibility feature on Android which it calls Camera switches. The feature allows you to control your Android phone’s screen using your device’s front camera and your face.

In this post, we’ll explain how to set up facial gestures and how you can use them to control your Android device.

How to set up Facial Gestures on Android

You can follow the steps below to set up and enable Facial gestures on your Android phone.

Step 1: Turn ON Switch Access

You can start setting up facial gestures by turning on the Camera Switches functionality on your Android device.

Open the Settings app on your Android device and select Accessibility.
Inside Accessibility, tap on Switch Access.
On the next screen, turn on the Use Switch access toggle.
You should now see a new prompt on your screen that asks whether you want Switch access to have full control of your Android phone. Since the Camera Switches feature requires permission to read the content on your screen as well as the ability to perform actions, you can go ahead and give full control of your device to Switch access. To do that, tap on Allow inside the prompt.
Since this is the first time you’re setting up Facial Gestures on your Android device, the Switch access setup guide will now open up on its own. The first thing to do here is to choose Camera Switch as your desired option when you’re asked to Choose a switch type. After you select Camera Switch, tap on Next at the bottom.
You will be prompted a box that asks whether or not you want Android Accessibility Suite to take pictures or record videos. Inside this prompt, select While using the app since the Android system may need access to your Camera time and again to perform facial gestures.

Step 2: Choose the number of switches

Once that’s done, you will be asked to choose the number of switches on the next screen. This may be tricky as you may not be able to decide if you need one switch or two switches to perform facial gestures on your device.

Android recommends that you use two switches so that you can use one gesture to start highlighting an item and another gesture to select it. If you select a single switch, you will have to make do with the same gesture to start highlighting an item as well as selecting an item on the screen.

If you still don’t understand what it means to select the number of switches, it’s best that you stick with Google’s recommendation of Two switches (the second option) and then tap on Next at the bottom. We’re also sticking with the same option for the sake of demonstrating the feature to you.

Step 3: Select your scan type

Next, you’ll have to select how you wish to move through the actionable items on your screen through your facial expressions.

Choose from any of the following options to select how you want Android to scan through options:

Linear scanning: When you select this option, each of your predefined gestures will move from one item to the other in the forward direction. Since you’re moving between items one at a time, it may take a long time to get to your preferred option. This option won’t, however, be active when the keyboard is open on your screen.
Row-column scanning: Selecting this option will first scan a row of items and then move on to the next row. Once you’ve finalized the row that has your preferred item, you can select it and then move through the items in that row to get to your item. This is also applied when you have the keyboard open on your screen. Row-column scanning will work in your favor on screens that have a lot of options to choose from.
Group selection: When this option is selected, Android will assign a color to all the items on your screen and your facial gestures will be used to narrow down which color you want to select before you choose your preferred item on the screen.

To keep things simple, we’ll choose Linear scanning in this instance and go ahead from there. Once you’ve selected your scanning option, tap on Next at the bottom right corner.

Step 4: Assign gestures for various actions

Once you’ve selected your preferred mode of scanning, it’s time to assign gestures for three actions – Next, Select, and Pause. You can choose from any one of the following gestures to set as default for a specific action – Open mouth, Smile, Raise eyebrows, Look left, Look right, and Look up. Of the six available gestures, you can choose multiple gestures for a certain action and once a gesture has been selected for an action, it won’t be available for selection in the upcoming steps, meaning a gesture you select can only be used for one particular action.

In this instance, we’re selecting the Raise eyebrows gesture for the Next action and confirming our choice by tapping on Next at the bottom.
On the next screen, you’ll be asked to choose a facial gesture for the action – Select. Here, you should be able to see that the gesture you selected for the previous action “Next” is greyed out, meaning you can no longer use that gesture for a different action.
Now, choose the gesture you want to use for selecting items on the screen from the list of remaining gestures. We’re selecting Open mouth but you can select your preferred method and then tap on Next.
After you assign gestures for the first two actions, you’ll have to select one for the action – Pause on the next screen. Google recommends that you set a gesture for the Pause action as it may help you avoid accidental triggers by letting you assign a gesture that toggles Camera Switches on and off.
On the screen select a gesture to set for the Pause action (we selected Smile) and then tap on Next.
There you go, you’ve successfully set up facial gestures for three basic actions and you can use it to move around your Android device. Once you’ve finished the setup process, you’ll be taken to the Switch Access settings screen.

Step 5: Try out various gestures

On the Switch Access settings screen, you can test out various gestures and how sensitive they are to your face by tapping on Try it at the top.
When you do that, a camera preview will be visible above the Use Camera Switches toggle. You should also see a blue square icon (the one resembling a face) at the top of your screen. When this square icon is blue, it means your face is detectable.
If this icon is red, it means Android cannot detect your face using your front camera.
Now, you can perform a different facial gesture that we mentioned before and check whether your Android device can detect them.

Here are some of the successful attempts that we were able to make:

How to use your phone with Facial Gestures [10 ways]

Now that you have successfully set up Facial Gestures on Android, you may still need to tweak and modify some of its settings so that you can fully take advantage of the feature on your phone. That’s because, the initial setup process that we went through above was only able to accomplish assigning gestures for three actions – Next, Select, and Pause. Since you will want to perform more actions using your facial expressions, you will need to follow the tips below to fully customize your facial gestures experiences on Android.

1. Add a new action for a gesture or Change the action

If you wish to change an action you had previously selected for a gesture or if you want to assign an action for an unselected gesture, you can do that by following the steps below.

Go to Settings > Accessibility > Switch access > Settings and select Camera Switch settings.
On the next screen, select the gesture you want to assign an action to from the “Face gesture settings” section.
When you select a gesture, scroll down on the next screen and select Edit assignment under “Set assignment”.
In the overflow menu that appears, select the action you want to assign for this gesture and then tap on Save.

2. Change gesture size and duration

If you have already selected an action for gestures, you can adjust its size and duration so that the selected action is triggered only when you want it to be triggered and not on any other occasion.

To adjust the size and duration for a gesture, open the Settings app and go to Accessibility > Switch access > Settings > Camera Switch settings.
On the next screen, select the gesture whose size and duration you want to customize.
To adjust how big a gesture you’d have to make with your face for a certain expression, drag through the slider under “Gesture size” or tap on either Smaller or Bigger repeatedly under you reach your desired gesture size.
On the same screen, tap on Gesture duration to set how long you may want to hold a gesture to trigger it.
In the overflow menu that appears, select your preferred duration from the six available options and then tap on OK to confirm.

3. Add gestures for more actions

In one of the previous sections, we helped you assign new actions for a gesture. Now, we’ll help you assign gestures if you wish to perform a certain action.

Open the Settings app and go to Accessibility > Switch access > Settings.
Here, select either Assign switches for scanning or Assign switches for global actions depending on what you want to assign a gesture to.
If you selected ‘Assign switches for scanning’, you’ll be able to assign gestures for Movement & selection (actions include Auto-scan, reverse auto-scan, select, next, and previous) and ‘Actions on highlighted item’ (actions include Touch and hold, scroll forwards, and scroll backward).
If you selected ‘Assign switches for global actions’, you’ll be able to assign gestures for Back, Home, Notifications, Quick Settings, and Overview; all of which will be visible under ‘Global action’.
Once you select an action you want to assign a gesture to, tap on Add Camera Switches under “Assign Camera Switches”.
In the overflow menu, select the gesture you want to assign for this action and tap on Save.
The assigned gesture will now be visible for this particular action.

4. Get visual/audio feedback when Facial Gesture is used

When Camera Switches is active, you may not realize if your face is getting scanned or if an action has been triggered unless the screen changes. If you want to get better feedback for your gestures, you can enable visual and audio feedback every time you make a gesture. When you enable these options, you may be able to see an alert that tells you how long you held a gesture and you may also hear a sound when a gesture is triggered.

You can enable visual and audio feedback for your gestures by going to Settings > Accessibility > Switch access > Settings and selecting Camera Switch settings.
Scroll down on the next screen and turn on the switches adjacent to Enhanced visual feedback and Enhanced audio feedback under “Additional settings for Camera Switches”.

5. Keep your screen turned ON for Facial Gestures

Since facial gestures require your phone to remain unlocked and active, you may want your screen to stay ON for longer than usual. Camera switches offer a better way to approach this as it lets your phone’s display remain turned on for as long as you lock it so you can use facial gestures at your will.

To keep your phone’s screen turned on, open the Settings app and go to Accessibility > Switch access > Settings > Camera Switch settings.
Here, scroll down on the screen and turn on the Keep screen on toggle under “Additional settings for Camera Switches”.

6. Avoid repeated gestures from getting triggered

If Android is triggering your facial expressions with action on repeated occasions within a short duration, you can ignore these repeated triggers by specifying a duration within which repeated triggers will be counted as one trigger.

Open the Settings app and go to Accessibility > Switch access > Settings > Camera Switch settings.
Here, scroll down on the screen and tap on Ignore repeated Camera Switch triggers.
When an overflow menu pops up, enter the number of seconds within which you want your gestures to be counted as a single trigger and then tap on OK.

7. Remove gestures for an action

If you wish to remove a gesture from one of the actions you configured earlier, you can do so by following the steps below.

Go to Settings > Accessibility > Switch access > Settings.
On this screen, choose either Assign switches for scanning or Assign switches for global actions from under “Assign switches” depending on which action you want to clear gestures from.
On the next screen, select the action you want to remove a gesture from.
To clear the assigned gesture for this action, tap on the X mark adjacent to the gesture you want to remove under “Assign Camera Switches”.
You will now see a prompt on the screen asking whether or not you want to clear the assigned switch for the action. Tap on Clear switch within the prompt to remove the assigned gesture.

The selected gesture will now be removed from the specified action.

8. Customize scanning for gestures

Android lets you personalize the way it scans for your facial gestures even after you have selected your preferred method during the setup process.

You can customize gesture scanning for your device by going to Settings > Accessibility > Switch access > Settings and then scrolling down to the Customize scanning section.
From here, you can:
Enable auto-scan so that Android starts highlighting items automatically so you can select them.
Change Scanning Method from Linear scanning, Row-column scanning, and Group selection.
Enable Point Scan to use moving lines to select a specific horizontal and vertical location.
Turn on Automatic Start Scanning so that Android may start scanning without needing to press a switch.
Enable Auto-select to choose a highlighted item automatically.

9. Change scan highlight color and style

When you enable Switch access, Android shows you a visual indication whenever it detects your face doing a gesture. The moment a gesture is triggered, the edges of your device’s display will blink in blue color.

You can change this highlight color and the highlight style by going to Settings > Accessibility > Switch access > Settings and selecting Scanning highlight style under “Display & sound”.
On the next screen, you can change Highlight color of this visual indicator.
Now, choose from Green, Orange, Red, Blue, and White.
Next, you can tap on Highlight line style.
From here, you will be offered to choose your preferred option between Thin, Medium, and Thick solid lines as your highlight style.

10. Adjust audio and vibration feedback

Android also lets you modify the way your device reacts to your triggers via audio and vibration.

You can access these feedback options by going to Settings > Accessibility > Switch access > Settings and selecting Speech, sound & vibration under “Display & sound”.
On the next screen, you can choose whether or not you want to use Spoken, audio, or vibration feedback, allow speech to finish before highlighting the next item, and the volume of the feedback sound.

How to turn off Facial Gestures on Android

If you have tried using facial gestures on Android and you don’t like it, you can turn it off by following the steps below.

Open the Settings app, go to Accessibility > Switch access > Settings, and select Camera Switch settings.
On the next screen, turn off the Use Camera Switches toggle.

The facial gestures you configured earlier will no longer work on your Android device.

FAQs

While Android’s facial gestures are pretty simple in terms of their action, you may still have your doubts regarding the feature. The following FAQ should help you resolve your doubts about Facial Gestures on Android.

How do Facial Gestures work on Android?

Android will now be able to look for visual keys from your face that it will detect using your phone’s camera and then use that information to perform an action that you assign for the facial expression you made.

In addition to selecting a screen action with your facial expression, Camera switches also let you adjust the gesture sensitivity of your facial expressions. This exists to make sure gestures aren’t triggered at the slightest of your expressions but are sensitive enough to trigger actions when you intend to trigger them.

Facial Gestures on Android can be beneficial for those of you who may find it hard to interact with some touch controls on your phone. It’s also less annoying than voice commands with Google Assistant as you don’t need to shout to your phone in public every time you want to get things done.

What gestures can you use to control your Android device?

During the setup process, you can activate actions that will be assigned based on a handful of expressions that you make with your face. These expressions include:

Open Mouth
Smile
Raise Eyebrows
Look Left
Look Right
Look Up

What actions can you perform using Facial Gestures?

The aforementioned gestures can be used to perform any action that you can otherwise interact with on your screen with touch. These include:

Toggling Auto-scan
Reversing auto-scan
Going to the next item
Going to the last item
Selecting an item
Tapping and holding on an item
Scrolling forward
Scrolling backward
Going Home
Going Back
Accessing Notifications
Pulling Down on Quick Settings
Accessing Overview
Pausing the Camera Switches option

Are videos and images captured on my phone sent to Google?

Although Android uses your phone’s front camera to detect your facial expressions, Google has promised that none of the images that are captured from your device are sent to its servers. This means that all the computations that happen to determine your facial gestures take place within your phone without any online help from Google.

Do Facial Gestures affect my battery life?

Since facial gestures require continuous access to your phone’s camera, your device’s battery life will be substantially reduced. Google recommends that you keep your device plugged in and charging when you use facial gestures on it.

Can my phone get slower than usual when using facial gestures?

As we explained above, facial gestures use your phone’s camera as well as its other resources. This may make your device slower than other times when on battery. If you wish to use your device with facial gestures without slowing it down, you can do that while your phone is plugged in and charging.

Can I use the same gesture for multiple actions?

No. Since the same gesture may clash with several actions within Android, the system won’t let you assign multiple actions for a single facial gesture. When you select a gesture under an action, this gesture will remain greyed out when assigning gestures for other actions.

Can I use multiple gestures for the same action?

Yes. Android lets you assign more than one gesture for the same action. This allows you to keep several facial expressions for triggering the same operation on your Android device so that you have different ways to trigger the action when your device isn’t able to detect one of your assigned gestures.

That’s all you need to know about using Facial gestures on Android.

RELATED