Remote Input shell scripts for your Android Device, or my screen is cracked

My Android has a cracked screen. Help!

I, too, had this happen to me. I had TitaniumBackup Pro on my device, but when the new version of KitKat came out, I lost root access because of the update.  I never (smack forehead) re-rooted my device. Therefore I didn’t have everything backed up. I could use the lower third of my Nexus 7 … sometimes. I tried to rotate my tablet so I could press certain spots on the uncracked screen. I made little progress, and I am sure provided a lot of entertainment to anyone who was watching me.

This is how I recovered my data, and what tools I wrote to make this easier. Luckily I had USB debugging  enabled.

I can remotely install applications using GooglePlay, but I needed to do more as some of these applications require permissions set using the device. And that requires tapping on the (broken) screen.

What didn’t work

I tried to use AndroidScreenCast. This showed me the screen, but I was unable to send any input to the device, despite the work-arounds I tried.

I also tried installing Droid VNC Server. I started the server, and I was able to click the mouse remotely on the screen, but the VCN client on my Linux machine, Remmina Remote Desktop Client, did not show me the screen, so I was clicking blindly.

In turned out that the best way to remotely tap on the screen was to use adb. This  works very reliably, but is inconvenient because you need to type dozens of different commands to do anything useful.

Install TowelRoot, SuperSU, and Helium

Install Towelroot to root the device

I first had to root my device. I used other methods before, but this time I used geohot’s TowelRoot. I downloaded the apk, and used adb to install it:

adb install tr.apk

I had to click on the screen to permit untrusted applications to be installed. I used

adb shell input touchscreen xxx yyy

where xxx and yyy were the coordinates of the spot I wanted to press. By rotating the screen and trying several XY combinations, I was able to approve this action. But it wasn’t easy. I wanted a better workaround. But compared to other methods for re-rooting the device, this worked

Let me briefly describe my XY script, as I use it a lot. My Nexus 7 has a screen of 1200 by 1920. I could use the touchscreen option to input, mentioned above, but I wanted an easier way to type in locations. My home screen has a layout of 5 by 5 icons for each application, with a row on the bottom for the apps on each screen. I wanted to be able to type “XY 2 1” to click on the application that was over 2, and down one, on this grid. My script converts this 2 and 1 location into the command “adb shell input touchscreen tap 300 350”

To make my script more flexible, the second version allows fractional values, so “XY 2 2.5” is the position on the second column, midway between icons 2 and 3 down (between rows 2 and 3).

Launching SuperSU

After launching SuperSU, I had to click on the OK button to grant permission:

adb shell input touchscreen tap 700 1200 
# or perhaps
XY 4 4.5

To be frank, once I had SuperSU working, I was in good shape. I could remotely log into the device, execute “su”, and change the permission of /data/dalvik-cache. Once this was done, I could use “adb pull /data/dalvik-cache” and extract nearly everything. But I wanted to have multiple backups.

Installing Helium Backup

I also installed Helium by ClockworkMod. I installed the Linux version on my Ubuntu system. Once I started the application on my Linux machine, by executing ./run.sh

I launched the Helium app, and I had to grant SU access. I used the command

XY 5 4.5

using my script XY, described below, I hit the Helium Menu button, and selected the PC Download:

XY 6 0.1 # select the menu option of Helium
XY 6 0.3  # select PC Download

This launched a web server on the Android device, and gave me the URL to type into my browser. I was then able to backup my applications using my browser on my Linux machine.

The ADB shell scripts

I wrote several independent bash shell scripts.

These are based on the adb shell input command.  There are several variations to this command, as the usage shows:

usage: input ...
 input text <string>
 input keyevent <key code number or name>
 input [touchscreen|touchpad] tap <x> <y>
 input [touchscreen|touchpad] swipe <x1> <y1> <x2> <y2>
 input trackball press
 input trackball roll <dx> <dy>

If one wanted to, one can type “adb shell” and then just type combinations of the above input command. But that’s a lot of typing. I wanted something quicker.

Here is the list of bash shell scripts in my collection below. You can combine them, extend them,  modify them, and add new commands.  As an example, suppose I want to power up the device. unlock it, go to the home screen. swipe right, launch the application at 6,2, and turn up the volume, I can use the following bash shell script

#!/bin/bash
Power
Unlock
Home
SwipeRight
XY 6 2
sleep 5 # seconds
VolumeUp

If you know the locations of your icons, you can “automate” several scripts using a series of bash scripts. Or combine scripts together to make more sophisticated scripts.

Here is a summary of the shell scripts I wrote:

Back - The Back Button
Notification - Show the list of notifications
CloseNotification - Close the notification list
GetOrientation - get the orientation of the device. 
GetScreenshot - Take a screen show and stores it in screen.png
GetXY - get the X and Y dimensions of the screen
Home - The Home button
Menu - The menu button
Power - Turn power on/off
Recent - Show recent applications
SwipeDown - Swipe down
SwipeLeft - Swipe left
SwipeRight - Swipe right
SwipeUp - Swipe Up
Text - Type text - including spaces
Unlock - Unlock the device
Widgets - Show the widget
XY - tap at the X,Y location
VolumeUp - turn volume up
VolumeDown - turn volume down

Command that start with “Get” return values that can be  used by other shell scripts.

Here is the source of each one.  Each one is a small script, and some could be shell functions or aliases, but I find it easier to make each one a separate shell script. For conciseness, I don’t show the complete script with the bash invocation and comments. I just show the meat of the matter. As always – your mileage may vary

Back – The Back Button

This is uses a lot because it undoes the last input. If you get the location wrong, this usually returns you to the previous screen, so you can try a different location.

adb shell input keyevent 4

Notification – Show the list of notifications

adb shell input swipe     10 10 10 1000

CloseNotification – Close the notification list

adb shell input swipe     10 1000 10 10

GetOrientation – get the orientation of the device. Returns 0, 1, 2, or 3

adb shell dumpsys input | grep SurfaceOrientation |  awk '{ print $2 }'

GetScreenshot – Take a screen show and stores it in screen.png

This was taken from shevtsov’s blog.

# http://blog.shvetsov.com/2013/02/grab-android-screenshot-to-computer-via.html
adb shell screencap -p | perl -pe 's/\x0D\x0A/\x0A/g' > screen.png

GetXY – get the X and Y dimensions of the screen

I use this for bound checking in my XY script. It returns the maximum X and Y positions.

adb shell dumpsys window | \
sed -n '/mUnrestrictedScreen/ s/^.*) \([0-9][0-9]*\)x\([0-9][0-9]*\)/\1 \2/p'

Home – The Home button

adb shell input keyevent 3

Menu – The menu button

adb shell input touchscreen tap 600 1775

Power – Turn power on/off

adb shell input keyevent 26

Recent – Show recent applications

adb shell input touchscreen tap 800 1850

SwipeDown – Swipe down

The names of the swipe commands can be confusing. I use Left and Right to indicate that I want to see what is on the Left or Right, not what direction my finger moves.

adb shell input swipe 800 400 1200 400

SwipeLeft – Swipe left

adb shell input swipe 400 400 800 400

SwipeRight – Swipe right

adb shell input swipe 800 400 400 400

SwipeUp – Swipe Up

adb shell input swipe 800 400 400 400



Text – Type text – including spaces

Because you cannot type spaces using the adb shell test commands, I use sed to convert spaces into “%s”

#!/bin/bash
str="$@"
nstr=$(echo "$str" | sed 's/ /%s/g')
echo adb shell text "$nstr"

Unlock – Unlock the device

If you have a password or pin, you are on your own.

adb shell input keyevent 82

Widgets – Show the widgets screen

Note this is the same event and the “unlock” event.

adb shell input keyevent 82

VolumeUp – turn volume up

adb shell input keyevent 25

VolumeDown – turn volume down

adb shell input keyevent 25

XY – tap at the X,Y location

This is the most complicated routine. The first version only accepted integer values. I added the capability to input floating point numbers to get more precise control.

The positioning is a guestimate. I got the maximum X and Y values using GetXY. I divided these values by the number of icons to get the XDIM and YDIM values, which represent the size of each icon. (Improvements are welcome). I had to add an OFFSET to compensate for the first row.

I also added a warning if the requested position is outside of the maximum dimension.

#!/bin/bash                                                                                                                         

# Bruce Barnett Wed Sep 17 13:43:51 EDT 2014                                                                                                      
export PATH=$PATH: # make sure GetXY is in the searchpath                                                                           
xmax=$(GetXY | awk '{print $1}')
ymax=$(GetXY | awk '{print $2}')

x=`echo $1 | awk '                                                                                                                  
BEGIN {                                                                                                                             
    XOFF=-100                                                                                                                       
    XDIM=200                                                                                                                        
}                                                                                                                                   
{                                                                                                                                   
     newx=XOFF + ($1 * XDIM)                                                                                                        
     if (newx > XMAX) {                                                                                                             
             print "Warning: ", newx, " > ", XMAX >"/dev/stderr"                                                                    
}                                                                                                                                   
     printf("%4.0f \n", newx )                                                                                                      
}' XMAX=$xmax`
y=`echo $2 | awk '                                                                                                                  
BEGIN {                                                                                                                             
    YOFF=100                                                                                                                        
    YDIM=250                                                                                                                        
}                                                                                                                                   
{                                                                                                                                   
     newy=YOFF + ($1 * YDIM)                                                                                                        
     if (newy > YMAX) {                                                                                                             
             print "Warning: ", newy, " > ", YMAX >"/dev/stderr"                                                                    
}                                                                                                                                   
     printf("%4.0f\n", YOFF + ($1 * YDIM))                                                                                          
}' YMAX=$ymax`

echo adb shell input touchscreen tap $x $y
adb shell input touchscreen tap $x $y

Future work

If there is interest, I can make these available in a git repository.

References

Advertisements
Gallery | This entry was posted in Hacking, Linux, Shell Scripting and tagged , , , , , , , . Bookmark the permalink.

4 Responses to Remote Input shell scripts for your Android Device, or my screen is cracked

  1. raul says:

    Very good tutorial. I wonder if there is a command to do a long press.

  2. mobileqablog says:

    adb shell input keyevent 83
    – can be used to open and close notifications.

    if screen is PIN protected, you can run:
    adb shell input text && adb shell input keyevent 66
    however it doesn’t work for all devices.

  3. mobileqablog says:

    Recent apps: adb shell input keyevent KEYCODE_APP_SWITCH

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s