IOS remote real machine control based on STF, step pit experience sharing

introduce

STF (or Smartphone Test Farm) is a web application for debugging smartphones, smartwatches and other gadgets remotely, from the comfort of your browser.

Making portal

In simple terms, a solution that allows you to remotely operate a real machine from a browser is provided by a group called Device Farmer. The STF itself does not support iOS devices, so the “Device Framer” provides a separate project called “STF_IOS_Support”. This article describes the deployment process of this project and the problems encountered.

The principle of

In fact, the stF_IOS_support real machine operation is also based on the WebDriverAgent project, the basic want to remote control iOS devices, all rely on the WebDrviverAgent project. Due to domestic iPhone port is closed, so the external can’t directly access, so rely on wdaproxy mobile phones port mapping, PC can access directly through http://localhost:7000/status, the corresponding equipment operation. Stf_ios_support is a WebSocket Server that listens to device connections and transmits instructions from the STF Server. Of course, there is the most important, the real machine screen is through iOS_Video_enabler, FFMPEG implementation.

start

material

• Docker Desktop container
• Mac laptop
• Xcode
• VPN (Scientific Internet access, not necessary)
• Apple developer account and signature certificate

Step 1: Project initialization

Check out the stf_iOS_support directory to the local Mac directory, CD the stf_iOS_support directory, and run the./init.sh file. Before executing the script, you need to install Xcode and configure the developer information correctly (Xcode configuration is not described here, you have configured successfully by default). If the network is not good, it is recommended to get online scientifically, otherwise many components will not be able to pull down.

git clone https://github.com/DeviceFarmer/stf_ios_support.git;
cd stf_ios_support
./init.sh
Copy the code

If successful, the following information is displayed:

zhudezhen@zhudezhendeMacBook-Pro stf_iOS_support %./init.sh Xcode 13.0 installed jq => version 1.6 graphicsmagick => Version 1.3.36 Zeromq => Version 4.3.4 Protobuf => Version 3.17.3 yasm => Version 1.3.0_2 PKg-config => Version 0.29.2_3 Carthage => version 0.38.0 automake => version 1.16.3_1 autoconf => version 2.71 libtool => version 2.4.6_3 wget => Version 1.21.1 Go => Version 1.16.5 node@12 => version 12.22.3 libsodium => Version 1.0.18_1 CZMQ => Version 4.2.0 Jpeg-turbo => version 2.1.0 nanomsg => version 1.1.5 libgcrypt => Version 1.9.3 gnutls => Version 3.6.16 MobileDevice => Version 2.0.0 libplist-head already installed libplist-installed HEAD is version 2.2.1 (==2.2.1) libusbmuxd - HEAD already installed LibusBMUxd - Installed HEAD is version 2.0.3 (==2.0.3) zhudezhen@zhudezhendeMacBook-Pro stf_ios_support %Copy the code

Step 2: Install the STF service

First, you need to install the Docker DeskTop client container. After the installation, ensure that the Docker container is configured correctly.

Then, copy the server folder in stf_iOS_support to another target directory, CD it to the cert directory in server, and run the gencert.sh script. This service is the STF server under Docker.

cp stf_ios_support/server xxxx/xxxx/test/;
cd xxxx/xxxx/test/server/cert/;
./gencert.sh
Copy the code

The self-signed certificate is complete if the following information is displayed:

zhudezhen@zhudezhendeMacBook-Pro cert % ./gencert.sh Generating a 2048 bit RSA private key ..................... + + +... +++ writing new private key to 'server.key' ----- zhudezhen@zhudezhendeMacBook-Pro cert %Copy the code

Then, modify the directory folder server/. Env file to change PUBLIC_IP and HOSTNAME to the local IP, for example: 192.168.199.191 here

PUBLIC_IP = 192.168.199.191 SECRET = SECRET RETHINKDB_PORT_28015_TCP = TCP: / / rethinkdb: 28015 HOSTNAME = 192.168.199.191 STF_IMAGE=livxtrm/devicefarmer:latestCopy the code

Then, cut back to the root directory of the target folder server, run the docker-compose up command (scientific Internet access), install the STF server in the Docker container, start the command line, and wait for the completion.

zhudezhen@zhudezhendeMacBook-Pro server % docker-compose up Creating network "server_default" with the default driver Creating volume "server_rethinkdb" with default driver Creating volume "server_storage-temp" with default driver Pulling Rethinkdb (rethinkdb: 2.3)... Pulling from Library/RethinkDB 2746a4a261C9: Download complete 4c1d20cdee96: Download complete 0d3160e1d0de: Download complete c8e37668deea: Download complete b7d1bf5200eb: Download complete f6b24861af7c: Download complete 677900de5c00: Download complete 7c6f2faef424: Download completeCopy the code

If the installation is successful, the following information is displayed:

Note: If any of the services in the container are not working, it is recommended to delete the contents of the container (Containers/Apps, Images, Volumes) and run the docker-compose up command again.

At this point, you can access the STF service via the browser http://192.168.199.191, and you will find that the browser cannot access the STF service, as follows:

You need to import server/cert/server.crt from the target folder into your system’s login keystring and change it to “always trust”.

At this point, you refresh the browser, you will find the advanced options, more than a familiar operation, click continue to 192.168.199.191 (not safe), you can access. (Enter account/email at will)

Step 3: Compile the STF_IOS_SUPPORT server

Go back to the stf_iOS_support directory, copy config.json.example to config.json, and edit the information in it. The above section of JSON is the simplest configuration case, the following section is a complete configuration case, we can copy the following section.

You need to modify “xcode_dev_team_id”, “STF”, “install”, “bin_paths” in the JSON field. Do not modify the other information as follows:

{...// iOS Developer Organization Teamid, this developer certificate is related to the organization ID, which can be viewed in the string
  "xcode_dev_team_id": "xxxx"."stf": {
    // The IP address of the STF service is the IP address configured in the server/. Env environment above
    "ip": "192.168.199.191"."hostname": "192.168.199.191"
  },
  "install": {
    // Change it to the current stf_iOS_support directory
    "root_path": "xxxxx/xxxx/stf_ios_support"."config_path": "xxxxx/xxxx/stf_ios_support/config.json"."set_working_dir": false
  },
  "bin_paths": {
    // 这里需要修改 osx_ios_video_enabler => video_enabler
    "video_enabler": "bin/video_enabler",}}Copy the code

After the configuration is complete, run the make command on the CLI to compile the STf_ios_support server. Wait until the compilation is complete. The following information is displayed:

zhudezhen@zhudezhendeMacBook-Pro stf_ios_support % make GIT_COMMIT="12aabb5d3ac0b9b6fce56de4a4c7368bff695f27" GIT_DATE = "1619067214" GIT_REMOTE = "https://github.com/DeviceFarmer/stf_ios_support.git" EASY_VERSION = "1.0" /Applications/Xcode13-beta.app/Contents/Developer/usr/bin/make -C coordinator go get go get . go build -o .. /bin/coordinator -ldflags "-X main.GitCommit=12aabb5d3ac0b9b6fce56de4a4c7368bff695f27 -X main.GitDate=1619067214 -X main.GitRemote=https://github.com/DeviceFarmer/stf_ios_support.git - X main. EasyVersion = 1.0 "git clone https://github.com/nanoscopic/ios_video_stream.git repos/ios_video_stream Cloning into 'repos/ios_video_stream'... remote: Enumerating objects: 1923, done. remote: Counting objects: 100% (1923/1923), done. remote: Compressing objects: 100% (649/649), done. Identifiers objects: 29% (574/1923), 11.18 MiB | 1.09 MiB/s sCopy the code

At that time, actually compile cannot succeed, first you will encounter the first error, information is as follows:

o build -o ios_video_pull .
go: github.com/nanoscopic/ios_video_pull: package github.com/google/gousb imported from implicitly required module; to add missing requirements, run:
	go get github.com/google/[email protected]
make[1]: *** [ios_video_pull] Error 1
make: *** [repos/ios_video_pull/ios_video_pull] Error 2
Copy the code

Repos/iOS_video_pull /go.mod: / iOS_video_pull /go.mod: / iOS_video_pull /go.mod: / iOS_video_pull /go.

1.14 the require module github.com/nanoscopic/ios_video_pull go (github.com/danielpaulus/quicktime_video_hack V0.0.0-20200514194616-c4570b6b687c // ******* BEGIN Add the line ****** github.com/google/gousb V0.0.0-20190812193832-18 f4c1d8a750 / / * * * * * * * END to add this line can be * * * * * * / / indirect github.com/nanomsg/mangos V2.0.0 + incompatible github.com/sirupsen/logrus v1.6.0 go.nanomsg.org/mangos/v3 v3.0.1 nanomsg.org/go/mangos/v2 v2.0.8 / /  indirect )Copy the code

Then, re-execute the make command and proceed, and you will see a second error:

go build . cp repos/wdaproxy/wdaproxy bin/wdaproxy go get github.com/fsnotify/fsnotify go get github.com/sirupsen/logrus  go build view_log.go view_log.go:13:5: no required module provides package github.com/fsnotify/fsnotify: go.mod file not found in current directory or any parent directory; see 'go help modules' view_log.go:14:5: no required module provides package github.com/sirupsen/logrus: go.mod file not found in current directory or any parent directory; see 'go help modules' make: *** [view_log] Error 1 zhudezhen@zhudezhendeMacBook-Pro stf_ios_support %Copy the code

The go.mod file cannot be found. Run the following command:

go mod init view_log.go;
go mod tidy ;
Copy the code

Continue to make, continue to go down… WebDriverAgent, then encountered an error:

/bin/sh: ./Scripts/bootstrap.sh: No such file or directory
make: *** [repos/WebDriverAgent/Carthage/Checkouts/RoutingHTTPServer/Info.plist] Error 127
Copy the code

This is because WebDriverAgent’s latest project is available without bootstrap.sh, so we need to manually change the configuration and modify the script file.

First of all, the CDS to stf_ios_support/repos/WebDriverAgent directory, use the Xcode IDE open this project

Change the Target to the BundleId property of the WebDriverAgentLib and the description file available to you.

Change the Target to the BundleId of WebDriverAgentRunner and its own available description file, so that the BundleId is the same as the previous value.

Finally, you can modify the Target Target, and the Target machine you are building, which you can test by pressing Command +U.

If the following information is displayed, it is successful.

Then we go back to stF_IOS_support and find the stF_IOS_support /makefile and find the following line:

repos/WebDriverAgent/Carthage/Checkouts/RoutingHTTPServer/Info.plist: | repos/WebDriverAgent
    cd repos/WebDriverAgent && ./Scripts/bootstrap.sh
Copy the code

Modify it as follows:

repos/WebDriverAgent/Carthage/Checkouts/RoutingHTTPServer/Info.plist: | repos/WebDriverAgent
    pwd
Copy the code

Then, go ahead and make… , basically here, ok, the following information:

Signing Identity: "Apple Development: De Zhenzhu (5LLQQ4S25F)" Provisioning Profile: "epoint_develop" (99f3dc52-90a2-4a39-ba42-2d4f14987b3e) /usr/bin/codesign --force --sign A7EAFF73719AECC112E9E746E0659D08F20BD121 --entitlements /Users/zhudezhen/Library/Developer/Xcode/DerivedData/WebDriverAgent-dvupaxwdbymikpdqfnltimkvxrvr/Build/Intermediates.noi ndex/WebDriverAgent.build/Debug-iphoneos/WebDriverAgentRunner.build/WebDriverAgentRunner.xctest.xcent --timestamp\=none --generate-entitlement-der /Users/zhudezhen/Library/Developer/Xcode/DerivedData/WebDriverAgent-dvupaxwdbymikpdqfnltimkvxrvr/Build/Products/Debug-ip honeos/WebDriverAgentRunner-Runner.app /Users/zhudezhen/Library/Developer/Xcode/DerivedData/WebDriverAgent-dvupaxwdbymikpdqfnltimkvxrvr/Build/Products/Debug-ip honeos/WebDriverAgentRunner-Runner.app: replacing existing signature ** TEST BUILD SUCCEEDED ** zhudezhen@zhudezhendeMacBook-Pro stf_ios_support %Copy the code

The compiled output is as follows:

Step 4

Plug in your device and run the following command:

stf_ios_support/run;
Copy the code

The console will have the following output:

zhudezhen@zhudezhendeMacBook-Pro stf_ios_support % ./run INFO[0000] auto network interface set; using en0 interface_name=en0 type=default_iface INFO[0000] Process start - device_trigger binary=bin/ios-deploy proc=device_trigger type=proc_start INFO[0000] Process start - stf_ios_provider binary=/usr/local/opt/node@12/bin/node Client_hostname = zhudezhendeMacBook - Pro. Local client_ip = 192.168.199.191 location = macmini/zhudezhendeMacBook - Pro. Local Proc = STf_IOS_PROVIDER Server_hostname =192.168.199.191 server_IP =192.168.199.191 type=proc_start INFO[0000] Device object  created dev_ios_port=9240 dev_name=iPhone15 dev_uuid="***401E" type=devd_create usbmuxd_port=9920 vid_port=8000 vnc_port=5901 wda_port=8100 INFO[0000] Device connected dev_name=iPhone15 dev_uuid="***401E" type=dev_connect INFO[0000] Process start - ivf binary=bin/ivf_pull outSpec=" TCP ://127.0.0.1:7879" proc= IVF type=proc_start uuid="***401E" INFO[0000] Process start - video_enabler binary=bin/video_enabler proc=video_enabler type=proc_start uuid="***401E" INFO[0000] Process start - ios_video_stream binary=bin/ios_video_stream port=8000 proc=ios_video_stream PullSpec =" TCP ://127.0.0.1:7879" tunName=en0 type=proc_start uuid="***401E" WARN[0000] Process end-IVF proc= IVF Type =proc_end uuid="***401E" INFO[0000] Process start-ivf binary=bin/ ivF_pull outSpec=" TCP ://127.0.0.1:7879" proc= IVF type=proc_start uuid="***401E" INFO[0000] Device disconnected dev_name=iPhone15 dev_uuid="***401E" type=dev_disconnect String to parse:{"type":"frame1","width":1170,"height":2532,"clickScale":1000,"uuid":"00008101-001945562001401E"Copy the code

At this point, we see stF_IOS_support, which seems to have found the device, but the device is still not found on the STF web page.

This is because wDA is not started, so there is no device. Why is it not started? Set video.enable to false in stf_ios_support/config.json and try again:

  "video": {
    "enabled": false,},Copy the code

Restart the service stf_IOS_support /run, and be surprised to find:

DezhendeMacBook - Pro. Local proc = stf_ios_provider server_hostname = 192.168.199.191 server_ip = 192.168.199.191 type=proc_start INFO[0001] Device object created dev_ios_port=9240 dev_name=iPhone15 dev_uuid="***401E" type=devd_create  usbmuxd_port=9920 vid_port=8000 vnc_port=5901 wda_port=8100 INFO[0001] Device connected dev_name=iPhone15 dev_uuid="***401E" type=dev_connect trying to get ios version INFO[0003] Process start - wdaproxy --iosDeploy=bin/ios-deploy binary=.. /wdaproxy iosVersion=15.0 proc=wdaproxy type=proc_start uuid="***401E" wdaPort=8100 INFO[0007] WDA Running proc=wdaproxy  type=wda_started uuid="***401E" Status response: {"value":{"build":{"productBundleIdentifier":"com.facebook.WebDriverAgentRunner","time":"Jul 17 2021 14:34:00} ", "device" : {" name ":" whocares ", "udid" : "00008101-001945562001401 - e"}, "ios" : {" IP ":" 169.254.124.5} ", "message" : "WebDr iverAgent is ready to accept Commands, "" OS" : {" name ":" iOS ", "sdkVersion" : "15.0", "testmanagerdVersion" : 28, "version" : "15.0"}, "ready" : "true", "state" : "succ ess"},"status":0} INFO[0010] Fetched WDA session id=AD259002-A2CD-4C72-AF56-B38F42FB60B4 type=wda_session uuid="***401E"  window size response: { "value" : { "width" : 390, "height" : 844 }, "sessionId" : "AD259002-A2CD-4C72-AF56-B38F42FB60B4" } INFO[0010] Fetched device screen dimensions height=844 type=device_dimensions uuid="***401E" width=390 INFO[0010] Process start - stf_device_ios binary=/usr/local/opt/node@12/bin/node ClickHeight =844 clickScale=1000 clickWidth=390 Client_IP =192.168.199.191 Device_name =iPhone15 Frame_server = "ws: / / 192.168.199.191:8000 / echo" node_port = 9240 proc = stf_device_ios server_host = 192.168.199.191 Server_ip =192.168.199.191 stream_height=0 stream_width=0 type=proc_start UUID ="***401E" video_port=8000 vNC_scale =2Copy the code

WDA actually started, and on the browser, we found the device (still with the Android icon, ugly) :

We click on the device, we go in, we see a black screen, no content, no T_T.

When you click the home button and drag and drop the device, the device will also perform the operation, which must be related to the previous “video. Enable” parameter, so change the parameter to true and run stf_ios_support again. Even not on STF (T_T), helpless, I went to go code, log, recompile, repeatedly in stf_ios_support/coordinator/coordiantor. Go file, find a suspected code:

if ! o.config.Video.Enabled || ( o.devd.okVidInterface == true && o.devd.okFirstFrame == true ) || videoMethod == "app" { o.devd.wdaStarted = true time.Sleep( time.Second * 2 ) fmt.Printf("trying to get ios version\n") log.WithFields( log.Fields{ "type": "ios_version", "dev_name": o.devd.name, "dev_uuid": uuid, "ios_version": o.devd.iosVersion, } ).Debug("IOS Version") proc_wdaproxy( o, devEventCh, false ) }Copy the code

Full text only here in the execution of wDA, used! O.config.video. Enable to judge, why? At this point, I don’t want to analyze the author’s intention, so I modify the code as follows:

// if ! o.config.Video.Enabled || // ( o.devd.okVidInterface == true && o.devd.okFirstFrame == true ) || // videoMethod == "app"  { o.devd.wdaStarted = true time.Sleep( time.Second * 2 ) fmt.Printf("trying to get ios version\n") log.WithFields( log.Fields{ "type": "ios_version", "dev_name": o.devd.name, "dev_uuid": uuid, "ios_version": o.devd.iosVersion, } ).Debug("IOS Version") proc_wdaproxy( o, devEventCh, false ) // }Copy the code

Then run the following command:

rm ./bin/coordinator && make && ./run
Copy the code

At this time, we found that the video plug-in was launched together with WDA, but the screen still could not be seen. Here is because the screen of iOS is dependent on video streaming, so the video streaming service needs to be suspended on the device. How to operate it?

To check whether the device is connected, run the following command:

stf_ios_support/bin/ivf_pull list;
Copy the code

If nothing comes out and the device is not linked, go to QuickTime -> New Movie record -> select iPhone and run the command again. We find the following output:

zhudezhen@zhudezhendeMacBook-Pro stf_ios_support % ./bin/ivf_pull list
--Device--
  Name:amp
  UDID:b5457eeaa93eb8a3c6ce3fee90c5c9f75251593b
zhudezhen@zhudezhendeMacBook-Pro stf_ios_support % 
Copy the code

Then, in turn, enter the following commands to trust devices:

// The device trusts the Idevicepair pair // Confirms the pid of the device video pull service./bin/ ios_video_pulp-device-Decimal // Resets the device's video stream./bin/devreset [decimal product ID] 1452Copy the code

Then run stf_ios_support/run again, this time the effect is as follows:

The screen also has, at the same time can also be operated remotely, here, the process is all gone.

FAQ

Docker-compose up installation error

Creating server_rethinkdb_1 ... 
Creating server_auth_1      ... 
Creating server_storage-temp_1 ... error
Creating server_rethinkdb_1    ... done
Creating server_auth_1         ... done

Creating server_triproxy_1     ... done
Creating server_dev-triproxy_1 ... done
Creating server_migrate_1      ... done
Creating server_reaper_1       ... done
Creating server_processor_1    ... done
Creating server_api_1          ... done

ERROR: for storage-temp  Cannot start service storage-temp: OCI runtime create failed: invalid mount {Destination:data Type:bind Source:/var/lib/docker/volumes/af0f503b3f078ce72a2b09a66ca00da675709239e3c4f67bcf42575e44aa511d/_data Options:[rbind]}: mount destination data not absolute: unknown
ERROR: Encountered errors while bringing up the project.
Copy the code

Change the VOLUME value in the target directory server/storage-temp/Dockerfile to an absolute directory, clean up the Docker environment, and run the docker-compose up command again, as follows:

FROM livxtrm/devicefarmer:latest

USER root
RUN mkdir data && chown stf:stf data
USER stf
VOLUME ["/ Users/zhudezhen/Desktop/XXXX XXX/iOS remote real machine control based on STF deployment guide/test/server/storage - temp/data"]
Copy the code

IOS15 device support is not particularly good

After testing, on iOS15 devices, the link is not very stable, often disconnected, wda does not work.

Can QuickTime be turned off?

Once connected, you can turn it off.

Occasionally CTRL + C can not shut down the service, how to do?

Go to Activity Monitor, locate the Coordinator service, and forcibly disable it.