As technology advances and cross-platform applications become more common, your Android app is no longer limited to running on a flat-screen touchscreen device. Richer interactions enable users to use your application with more complex inputs. So as a developer, it’s time to consider providing robust input support for a wide variety of devices. This article is ready for you to share more extensive and powerful input support, you are welcome to read.

If you prefer to see this in video, check it out here.

For Android devices, input is one of the key factors in determining the user experience. Developers often want user interaction to be as simple and intuitive as possible, but if you buy a new foldable device that comes with a keyboard and your favorite app doesn’t support standard keystrokes, it can be a frustrating experience.

Non-touch input is now built into many Android devices, such as Chromebook keyboards, or even provided as standard on some devices. So among the various Types of Android devices like phones, foldable devices, tablets, Chromebooks, Chromeboxes with external displays, Chromebases with built-in displays, Android TV, and more, How can developers ensure that different input methods work for their applications?

▽ Multiple Android devices

Not all users use a touch screen to interact with your application. Some users may use a keyboard, stylus, etc., or even have accessibility requirements. So every developer needs to spend some time thinking about how to make an app as enjoyable as possible for as many users as possible.

Enhanced Input

△ Standard input mode and enhanced input mode

For each type of input device, we can divide the functionality of the application into standard use cases and enhanced use cases:

  • The techniques used to handle standard use cases, which include features expected by users such as selection, text entry, long and right-clicking, are simple to implement and, in some cases, automatic. If you want to provide more unique and enhanced support for your application, you need to think about it.

  • In enhanced use cases, some features are not just better, but may be required, such as a mobile game that doesn’t support gamepads or a text editor that doesn’t support standard copy-and-paste shortcuts.

Thinking about adding unique and supportive features beyond the basics is how developers can really make their apps stand out. For example, eDJing, as shown below, adds support for keyboard DJS, trackpad twists, and MIDI DJ controllers that instantly transform a user’s phone or Chromebook into a DJ workstation.

△ Application scenario of eDJing

Cubasis, a premier digital audio workstation app, has released a new version optimized for the Chrome OS operating system that enhances functionality and usability with the advantage of a larger screen and the ease with which a MIDI controller can connect to a Chromebook.

△ Cubasis application scenarios

Drawing apps are more focused: Bluetooth and USB drawing pads work continuously, and low-latency stylus apis are available for drawing and painting apps in Chrome OS.

△ Drawing application scenarios

In short, users want a unique, enjoyable, intuitive experience in your app, and you make it happen!

Keyboard input support

Keyboards are built into Chromebooks or become part of the everyday experience of using detachable devices, tablets, foldable devices and TVS. The good news is that most basic keyboard input can usually be used directly, unless you’re working on building your own on-screen keyboard or writing your own text input view from scratch.

Send key support (KEYCODE_ENTER)

The developer needs to create a new line for the Enter key in the standard EditText view. If your application has chat, feedback forms, newsletter sign-ups, or anything that requires you to send text, the default newline behavior is certainly not what you expect, and don’t worry that the expected send functionality is easy to implement.

Chat and send text

The developer needs to create a new line for the Enter key in the standard EditText view, which shows the code for pressing the Enter key. The complete code looks like this:

override fun onKeyUp(keyCode: Int, event: KeyEvent?).: Boolean {
    when (keyCode) {
        KEYCODE_ENTER -> {     // Undo onKeyUp and listen on KEYCODE_ENTER
            sendMessage()
            return true     // Make sure to return true if the keystroke event has been processed by the application}}// If the event is not processed, it is returned to the system for processing
    return super.onKeyUp(keyCode, event)  // If not, the event is passed to super
}
Copy the code

Media Key Support

If your application wants to support media playback, you also need to include media key support. To do this, replace KEYCODE_ENTER in the Enter key code with the media key code you want to support, such as MEDIA_NEXT and MEDIA_PREV. Also, don’t forget to use SPACE to control play and pause with the SPACE bar. The complete code is shown below:

override fun onKeyUp(keyCode: Int, event: KeyEvent?).: Boolean {
    when (keyCode) {
        KEYCODE_SPACE, KEYCODE_MEDIA_PLAY_PAUSE -> {
              playOrpauseMedia(); 
              return true 
        }
        KEYCODE_MEDIA_NEXT -> {nextTrack(); return true }
        KEYCODE_MEDIA_PREV -> {prevTrack(); return true}}// If the event is not processed, it is returned to the system for processing
    return super.onKeyUp(keyCode, event)
}
Copy the code

The game key support (KEYCODE_W | A | S | D)

For game applications, you might want to include support for arrow keys and the FOUR keys W, A, S, and D, which is equally simple. You just need to find the correct key code in the Android documentation and listen for those keys. The complete code is shown below:

when (keyCode) {
    KEYCODE_W, DPAD_UP -> { goUp(); return true }
    KEYCODE_A, DPAD_LEFT -> { goLeft(); return true }
    KEYCODE_S, DPAD_DOWN -> { goDown(); return true }
    KEYCODE_D, DPAD_RIGHT -> { goRight(); return true}}Copy the code

Note that onKeyUp is usually listened for when keyboard support is provided, so you don’t have to worry about sending duplicate onKeyDown events when you hold down a key. In addition, if you want to ensure millisecond time response, you can listen for onKeyDown and handle repeated keystroke events yourself. The complete code is shown below:

override fun onKeyDown(keyCode: Int, event: KeyEvent?).: Boolean {
    when (keyCode) {
        KEYCODE_F -> {
            shootFireball()
            return true}}return super.onKeyUp(keyCode, event)
}
Copy the code

For more information, visit chromeos.dev

Shortcut key support (Ctrl+)

In addition to basic keyboard keys, one thing to consider is configuring Ctrl-based shortcuts, such as copy, paste, undo, redo, etc. Common shortcuts are suitable for many applications. Excellent keyboard support will help your apps achieve more functions, and some apps even go one step further to put advanced functions within users’ reach. For example, when using The eDJing app, users can use the touch pad to twist disks by holding down the Ctrl key.

The code for the Ctrl+Z shortcut for undo is shown below, similar to the onKeyUp and onKeyDown codes above, but using dispatchKeyShortcutEvent to indicate the metaskey combination. Note that this action can be triggered by pressing Alt, Ctrl, or Shift. The complete code is shown below:

override fun dispatchKeyShortcutEvent(event: KeyEvent): Boolean {
    return when (event.getKeyCode()) {
        KEYCODE_Z -> {
            undoAction() // [Ctrl]+ Z, [Shift]+ Z, or [Alt]+ Z trigger
            true
        }
        else- > {super.dispatchKeyShortcutEvent(event)
        }
    }
}
Copy the code

If you only want to respond to specific meta-key combinations, you can use Meta_Alt_On, or you can use methods like isShiftPressed() or isCtrlPressed(). The complete code is shown below:

when {
    event.isCtrlPressed && event.isShiftPressed -> {
        redoAction(); true / / [Ctrl] + [Shift] + z trigger
    }
    event.isCtrlPressed -> {
        undoAction(); true / / [Ctrl] + z trigger
    }
    else- > {// Pass it to the system
        super.dispatchKeyShortcutEvent(event)
    }
}
Copy the code

According to this code, undo is only performed when Ctrl+Z is simultaneously pressed, and redo is only performed when Ctrl+Shift+Z is simultaneously pressed, and Alt is not used.

Mouse/trackpad input support

As with keyboards, most mouse and trackpad inputs usually work without any extra code. Still, it’s important to use the mouse to test all features of your app to see if there are any omissions.

Delta mouse

Right click support

Right clicking is one of the most common omissions. A common example of touch is holding down a screen to perform context-based actions, but holding down a mouse click is not intuitive. If you want to support right clicking, there are several options. The code to add the context menu is shown here. The complete code looks like this:

RegisterForContextMenu (myView) // First register one or more views for the context menu, which automatically handles both long and right click operations. . override fun onCreateContextMenu(menu, view, menuInfo) { super.onCreateContextMenu(menu, view, MenuInfo) menuInflater. Inflater (R.m emu. MyContextMenu, menu) / / and then withdraw overrideonCreateContextMenu and extend the right menu layout, the same menu can be used in multiple views. } override fun onCreateContextMenu(item: MenuItem): Boolean { return when(item.itemId){ R.id.myMenuItemId -> { menuItemAction(); True} else - > {return. Super onContextItemSelected (item)} / / in the end, set onContextItemSelected instructions to select a specific menu item needs to perform the operation. }}Copy the code

For right-click actions other than the context menu, you can set the view with onContextClickListener, simply by using it to invoke the same method used in the long-press use case. The corresponding code is as follows:

myView.setOnContextClickListener {
       performContextAction()
       true
}
Copy the code

Hover response support

When using a mouse or trackpad, users often expect the interface to respond in some way. For example, when the mouse cursor hovers over the clickable view, visual feedback will be generated, as shown in the figure. The pointer icon may change, or some other visual indications may appear, which can be intuitively felt by users. Here is the code for when the pointer hovers over the view. The complete code looks like this:

myView.setOnHoverListener { view, MotionEvent -> // Set onHoverListener when (motionEvent.action) {ACTION_HOVER_ENTER -> {view.pointericon = PointerIcon.getSystemIcon( appIicationContext, Pointericon.type_hand) view.setBackgroundColor(color.blue) true} // Listen for the HOVER_ENTER event and perform the corresponding operation. The display pointer icon will change to a hand shape and the background color will change to blue. ACTION_HOVER_EXIT -> { resetUI(); } else -> {false} // Don't forget to set the icon and background color to reset when HOVER_EXIT occurs. }}Copy the code

Pointer capture support

Pointer capture is another common mouse and trackpad enhancement that is not only essential for some games, but also adds specific functionality to some applications. The application can capture the mouse cursor through the pointer capture function, so that the cursor does not appear on the screen, so that it can receive relative pointer events without moving the cursor to the edge of the screen. First-person games like Minecraft: Education Edition are good examples.

Minecraft: Education Edition

To support pointer capture, you can call requestPointerCapture and then releasePointerCapture to release captured Pointers. In the code can add OnCapturedPointerListener use receive a pointer to the data, and use the pointer position relative changes to achieve some great features. The complete code is shown below:

view.requestPointerCapture()
...
view.releasePointerCapture()
...
view.setOnCapturedPointerListener { _, motionEvent ->
    // Calculate the increment of coordinates since the last motionEvent
    mouseChangeX = motionEvent.x
    mouseChangey = motionEvent.y

    // Calculate the displacement since the last motionEvent
    val totalDistance = hypot(mouseChangeX, mouseChangeY)
    doSomethingCool(totalDistance)
}
Copy the code

Gesture support

The Android API also provides advanced pointer handling, including gestures, two-finger extension and zooming support, as shown below.

If you want to learn more about the Android API, see the Getting Started guide on the Android Developer website: Using The Touch Power.

Stylus input support

If you’ve added excellent pointer support to your application, the stylus usually works as expected for most use cases. Some of the stylus enhancements are notable, such as stylus tilt and press support on some devices, which can help you add some nice controls and features to your painting or drawing applications. There is also a low-latency stylus API that provides minimal latency display responses in painting or drawing applications and configurable stroke prediction to create a pen on paper drawing experience. To see this in action, check out an app similar to Concepts on a supported Chromebook or Android device.

△ Concepts app support for stylus

  • For implementation details, see the documentation on AXIS_PRESSURE and AXIS_TILT on the Android developer website.

  • See GitHub for the low-latency Stylus API library and demo application.

Use the stylus in the Android emulator

We’ve worked with Microsoft to bring console stylus support to the Android emulator. If you’re optimizing your app to provide more advanced stylus support, you’ll be able to test precise tilt and pinch control using the Android emulator on a supported host.

The Microsoft Surface Duo developer Experience team has partnered with Google to support advanced input development and testing, such as multi-touch analytics and stylus support. So how do you use the stylus to test your app when running it in the emulator?

△ Test the stylus in the simulator

As you can see, the Microsoft Duo 2 emulator running on Surface Studio has two apps running at the same time. In the right pane is a sample app that allows you to test stylus press sensitivity, pen orientation, erase pen tips, and other stylus buttons. In the left pane are Microsoft OneNote applications that can be drawn, noted, or erased on the OneNote canvas using the emulator.

We’re very excited that the Android emulator now supports multi-touch on touchable PCS as well, allowing you to test gestures that require the use of multiple fingers to interact with your app, such as two-finger extension, zooming, and other touch interactions.

Use gestures in Google Maps

The foldable simulator, built into Android Studio, is running Google Maps, zooming in and out with just two fingers. And these emulators are updated to support not just two fingers, but up to 10 touch points if your hardware allows.

None of the changes you see are unique to the Surface Duo emulator, and they work with other foldable emulators as well. Microsoft has also been using these simulator updates to develop and optimize its own apps, such as testing stylus interaction on a variety of device types, including Surface Duo, large screens and other foldable devices.

Handle input support

Add gamepad support if you have a game application. Use the corresponding key code to determine whether you want to perform an operation on onKeyUp or onKeyDown.

▽ Gamepad

The gamepad arrow keys have the same key code as the keyboard arrow keys, and you can process both by listening for the KEYCODE_DPAD event. Gamepad buttons also have their own key codes, and you can listen for these buttons as you did here for the X button. The complete code is shown below:

when (keyCode) {
    BUTTON_X -> {
        engageNitro(); return true
    }
    DPAD_UP, KEYCODE_W -> {
        goForward(); return true
    }
    DPAD_LEFT, KEYCODE_A -> {
        turnLeft(); return true
    }
    ...
}
Copy the code

MIDI input support

When specialized hardware is involved, support for devices and use cases is required. MIDI support is especially important for music and creative tools, allowing for a wide range of expressive input signals, from pressure-sensitive piano keyboards to devices with sliders, knobs, keyboards, and many other different input triggers.

If you want to use or learn more about MIDI, the Android Media Samples library provides some handy open source examples to help you get started. Take a look at the MidiSynth project, specifically the Synth Engine module in that project. Note that starting with Android 29 you can also use the NDK MIDI API.

review

Android devices with larger screens have emerged and are becoming more common, and providing good input support on Android has always been important, but it’s especially important for foldable devices, tablets, and the Chrome OS. Think about your app’s input processing and how you can add interaction, unlock new features, and improve the app experience. We look forward to your efforts to build great Android apps and add great input processing to them!

Please click here to submit your feedback to us, or share your favorite content or questions. Your feedback is very important to us, thank you for your support!