Related documents of this article

Received a storm of Applause. That’s what’s expected of you.

robotjs

This door starts the project

Github.com/Licht-club/…

Branch main update 1. Webpack multi-entry multi-page, multiple rendering threads can be completed with the same Webpack configuration 2. Some basic optimization of WebpackCopy the code

The use of the desktop – capturer

/render-process/control/getDesSource.ts

That’s what’s expected of you. Women, that’s what’s expected of you. That’s what’s expected of you

Two changes have been made:

  1. If (source.name === ‘Electron’

  2. Video :{mandatory:{}} The es. Dom signature does not match and requires @ts-ignore magic annotation

    import {desktopCapturer,} from ‘electron’

    export type handleStream = (stream: MediaStream) => void export type handleError = (err: Error) => void

    export function getDesSource(handleStream: handleStream, handleError ? : handleError) { desktopCapturer.getSources({types: [‘window’, ‘screen’]}).then(async sources => { for (const source of sources) { // if (source.name === ‘Electron’) { } try { const stream = await navigator.mediaDevices.getUserMedia({ audio: false, video: { // @ts-ignore mandatory: { chromeMediaSource: ‘desktop’, chromeMediaSourceId: source.id, minWidth: 1280, maxWidth: 1280, minHeight: 720, maxHeight: 720 } } }) handleStream(stream) } catch (e) { handleError && handleError(e) } return } }) }

render-process/control/DesktopCapturerVideo.tsx

Wrap a video component and call the getDesSource method above

import {desktopCapturer} from 'electron' import React, {useEffect, useMemo, useRef} from "react"; import {getDesSource} from "./getDesSource"; const DesktopCapturerVideo = () => { const videoRef = useRef<HTMLVideoElement>(null); const handleStream = (stream: MediaStream) => { console.log(videoRef.current) if (videoRef.current) { const video = videoRef.current video.srcObject =  stream video.onloadedmetadata = (e) => video.play() } } const handleError = (error:Error) => { Console. log(error,' get desktop flow error ')} useEffect(() => {getDesSource(handleStream, handleError)}, []) return <video ref={videoRef}></video> } export default DesktopCapturerVideoCopy the code

render-process/control/index.tsx

The rendering thread (control page) calls the DesktopCapturerVideo component

import ReactDom from "react-dom"; import React, {useEffect} from "react"; import DesktopCapturerVideo from "./DesktopCapturerVideo"; Function App(){return <div> <span> <span> <DesktopCapturerVideo /> </div>} reactdom.render (<App></App>, document.getElementById('root'))Copy the code

Start the process

The Mac system displays a message asking you to configure the permission

After configuration, you see that the desktop stream was successfully captured

Interlude – Introduces Concurrently

It can be seen that our current project is rather troublesome to start, requiring three services: a React/Webpack rendering service, a TSC real-time compilation service and a electron window service

This condition, which can be used with this condition, can be resolved with Concurrently

The installation

$ npm install -g concurrently
or
$ npm install concurrently --save
Copy the code

I’m not going to install it globally

Configuring Startup Commands

 "start:dev": "concurrently -k -p \\"[{name}]\\" -n \\"React,TypeScript,Electron\\" -c \\"yellow.bold,cyan.bold,green.bold\\" \\"npm run start:react\\" \\"npm run dev:main\\" \\"npm run start:electron\\""
Copy the code

Perform the start: dev, view the command line, you can see the React, TypeScript, Electron three threads are started, but Electron window is blank, This is because the React thread has not finished starting when the Electron thread starts successfully. You can refresh the Electron window or use waIT-on

wait-on

Wait – on the installation
npm install wait-on # local version
OR
npm install -g wait-on # global version
Copy the code
Modify our start:dev command
"start:dev": "concurrently -k -p \"[{name}]\" -n \"React,TypeScript,Electron\" -c \"yellow.bold,cyan.bold,green.bold\" \"npm run start:react\" \"npm run dev:main\" \" wait-on http://localhost:8080/main.html && npm run start:electron\""
Copy the code
Start the

As you can see, our Electron thread is successfully stuck after React

The remote control

Robotjs implements desktop automation and controls the mouse and screen

The installation of robotjs

NPM install robotjs // The installation will take a whileCopy the code

After the installation is successful, restart start:dev and type require(‘robotjs’) in the electron window, an error will be detected

An error figure

Robotjs is written in c++ and needs to be recompiled for different platforms and node versions.

Build robotJS

Ensure that the corresponding dependencies have been installed on the corresponding platform

Windows windows-build-tools npm package (npm install --global --production windows-build-tools from an elevated Mac Xcode Command Line Tools. Linux Python (v2.7 recommended, v3.x.x is not supported). make. A C/C++ compiler like GCC. libxtst-dev and libpng++-dev (sudo apt-get install libxtst-dev libpng++-dev).Copy the code

Begin to compile

$ npm install electron-rebuild --save-dev


$ npx electron-rebuild
Copy the code

Compile successfully as shown in figure!

main-process/index.ts

Need to close the app. AllowRendererProcessReuse

import {app, BrowserWindow} from 'electron' import {create} from './mainWindow' import {ipc} from "./ipc"; App. AllowRendererProcessReuse = false / / default is true, prevent the renderer load of context awareness in the process of the machine module. app.on('ready', () => { process.env['ELECTRON_DISABLE_SECURITY_WARNINGS'] = 'true'; // Disable web security warning ipc() create()})Copy the code

Restart start:dev and try reqire(‘rebotjs’) again. If it succeeds, the installation is successful

The use of robotjs

main-process/onRobot.ts

Robotjs can only be run in the main process, so the robotJS code in the main process via ipc allows the renderer to call the main process to do software control (keyboard and mouse).

import {ipcMain} from 'electron'; import robot from 'robotjs' export interface RobotData { keyCode: number; shift: boolean; meta: boolean; alt: boolean; screen? : { width: number; height: number }, video? : { width:number; height:number; }} export type RobotType = 'mouse' | 'key'/keyboard/mouse const robotHandle = (function () {function mouseHandle (data: RobotData) {console.log(data, 'temporary logic --mouseHandle')} function keyHandle(data: RobotData) {console.log(data, 'temporary logic --keyHandle')} return {mouseHandle, keyHandle } })() export default function onRobot() { ipcMain.on('robot', (e, type: RobotType, data: RobotData) => { if (type === 'mouse') { robotHandle.mouseHandle(data) } else if (type === 'key') { robotHandle.keyHandle(data) } }) }Copy the code

main-process/index.ts

The main thread entry adds the logic encapsulated above

import {app, BrowserWindow} from 'electron' import {create} from './mainWindow' import {ipc} from "./ipc"; import onRobot from "./onRobot"; app.allowRendererProcessReuse = false app.on('ready', () => { process.env['ELECTRON_DISABLE_SECURITY_WARNINGS'] = 'true'; Ipc () create() onRobot()})Copy the code

render-process/control/events.ts

Using nodeJS’s built-in Events module (publish subscribe mode)

import Events from 'events' import {RobotData, RobotType} from ".. /.. /main-process/onRobot"; import {ipcRenderer} from "electron"; const peer = new Events() peer.on('robot', (type: RobotType, data: RobotData) => { if (type === 'mouse') { data.screen = { width: window.screen.width, height: Window.screen. height}} // ipcrenderer. send Ipcrenderer. send('robot', type, data)}) export default peer;Copy the code

render-process/control/setRobot.ts

The render thread listens for mouse and keyboard events

import {ipcRenderer} from 'electron'
import peer from "./events";

const geneRobotKeyData = (e: KeyboardEvent) => {
    return {
        keyCode: e.keyCode,
        shift: e.shiftKey,
        meta: e.metaKey,
        alt: e.altKey
    }
}

const geneRobotMouseData = (e: MouseEvent) => {
    return {
        clientX: e.clientX,
        clientY: e.clientY,
        video: {
            // width:
        }
    }
}

export  function setRobot(videoDom: HTMLVideoElement) {
    window.addEventListener('keydown', (e) => {
        const data = geneRobotKeyData(e)
        peer.emit('robot', 'key', data)
    })

    window.addEventListener('mouseup', (e) => {
        const data = geneRobotMouseData(e)
        data.video = {
            width: videoDom.getBoundingClientRect().width,
            height: videoDom.getBoundingClientRect().height
        }
        peer.emit('robot', 'mouse', data)
    })
}
Copy the code

render-process/control/DesktopCapturerVideo.tsx

The console (video) component imports the above listening logic,

 useEffect(() => {
        getDesSource(handleStream, handleError)
        setRobot(videoRef.current!)
    }, [])
Copy the code

Restart the project, click the mouse or keyboard, and view the command line output

9. The printing is successful

Let’s go through the process

Render thread: listen to mouseup->event.emit Render thread: event.on->ipcRender. Send Main thread: ipcmain.onCopy the code

Improve the Handle logic of the main thread

/utils/vkey.ts

Keyboard mapping, source: github.com/chrisdickin… More than 100 lines of QaQ

'use strict' var ua = typeof window ! == 'undefined' ? window.navigator.userAgent : '' , isOSX = /OS X/.test(ua) , isOpera = /Opera/.test(ua) , maybeFirefox = ! /like Gecko/.test(ua) && ! isOpera let i=0; let output :Record<number, string> = { 0: isOSX ? '<menu>' : '<UNK>' , 1: '<mouse 1>' , 2: '<mouse 2>' , 3: '<break>' , 4: '<mouse 3>' , 5: '<mouse 4>' , 6: '<mouse 5>' , 8: '<backspace>' , 9: '<tab>' , 12: '<clear>' , 13: '<enter>' , 16: '<shift>' , 17: '<control>' , 18: '<alt>' , 19: '<pause>' , 20: '<caps-lock>' , 21: '<ime-hangul>' , 23: '<ime-junja>' , 24: '<ime-final>' , 25: '<ime-kanji>' , 27: '<escape>' , 28: '<ime-convert>' , 29: '<ime-nonconvert>' , 30: '<ime-accept>' , 31: '<ime-mode-change>' , 32: '<space>' , 33: '<page-up>' , 34: '<page-down>' , 35: '<end>' , 36: '<home>' , 37: '<left>' , 38: '<up>' , 39: '<right>' , 40: '<down>' , 41: '<select>' , 42: '<print>' , 43: '<execute>' , 44: '<snapshot>' , 45: '<insert>' , 46: '<delete>' , 47: '<help>' , 91: '<meta>' // meta-left -- no one handles left and right properly, so we coerce into one. , 92: '<meta>' // meta-right , 93: isOSX ? '<meta>' : '<menu>' // chrome,opera,safari all report this for meta-right (osx mbp). , 95: '<sleep>' , 106: '<num-*>' , 107: '<num-+>' , 108: '<num-enter>' , 109: '<num-->' , 110: '<num-.>' , 111: '<num-/>' , 144: '<num-lock>' , 145: '<scroll-lock>' , 160: '<shift-left>' , 161: '<shift-right>' , 162: '<control-left>' , 163: '<control-right>' , 164: '<alt-left>' , 165: '<alt-right>' , 166: '<browser-back>' , 167: '<browser-forward>' , 168: '<browser-refresh>' , 169: '<browser-stop>' , 170: '<browser-search>' , 171: '<browser-favorites>' , 172: '<browser-home>' // ff/osx reports '<volume-mute>' for '-' , 173: isOSX && maybeFirefox ? '-' : '<volume-mute>' , 174: '<volume-down>' , 175: '<volume-up>' , 176: '<next-track>' , 177: '<prev-track>' , 178: '<stop>' , 179: '<play-pause>' , 180: '<launch-mail>' , 181: '<launch-media-select>' , 182: '<launch-app 1>' , 183: '<launch-app 2>' , 186: '; = ', ', 187:188: ', ', 189: '-', 190, '. ', 191: '/', 192: \ ` ', 219: '[', 220: \ \ \ \', 221: '] ', 222: "'" , 223: '<meta>' , 224: '<meta>' // firefox reports meta here. , 226: '<alt-gr>' , 229: '<ime-process>' , 231: isOpera ? '\`' : '<unicode>' , 246: '<attention>' , 247: '<crsel>' , 248: '<exsel>' , 249: '<erase-eof>' , 250: '<play>' , 251: '<zoom>' , 252: '<no-name>' , 253: '<pa-1>' , 254: '<clear>' } for(i = 58; i < 65; ++i) { output[i] = String.fromCharCode(i) } // 0-9 for(i = 48; i < 58; ++i) { output[i] = (i - 48)+'' } // A-Z for(i = 65; i < 91; ++i) { output[i] = String.fromCharCode(i) } // num0-9 for(i = 96; i < 106; ++i) { output[i] = '<num-'+(i - 96)+'>' } // F1-F24 for(i = 112; i < 136; ++i) { output[i] = 'F'+(i-111) } export default outputCopy the code

Found a bug

Dist /main-process/index.js, but the main field in package.json is wrong

Should be “main”: “dist/main-process/index.js”

Continue to

main-process/onRobot.ts

import {ipcMain} from 'electron'; import robot from 'robotjs' import vkey from ".. /utils/vkey"; export interface ScreenVideoInfo { screen: { width: number; height: number }, video: { width: number; height: number } } export type RobotKeyData = { keyCode: number; shift: boolean; meta: boolean; alt: boolean; } export type RobotMouseData = { clientX: number; clientY: number; } & ScreenVideoInfo export type RobotType = 'mouse' | 'key'/keyboard/mouse const robotHandle = (function () {function mouseHandle(data: RobotMouseData) { const {clientX, clientY, video, screen} = data let x = clientX * screen.width / video.width let y = clientY * screen.height / video.height // MoveMouse (x, y) // robot. MouseClick () console.log(\ 'robot click \${x},\${y}\')} function keyHandle(data: RobotKeyData) {const modiFIERS = [] // if (data.meta) {modiFIERS. Push ('meta')} if (data.shift) { modifiers.push('shift') } if (data.alt) { modifiers.push('alt') } let key = vkey[data.keyCode].toLowerCase() if (key[0] ! == '<') {robot. KeyTap (key, modifiers) console.log(\ 'pressing a key \${key}\ ')}} return {mouseHandle, keyHandle } })() export default function onRobot() { ipcMain.on('robot', (e, type: RobotType, data: RobotKeyData | RobotMouseData) => { if (type === 'mouse') { robotHandle.mouseHandle(data as RobotMouseData) } else if (type === 'key') { robotHandle.keyHandle(data as RobotKeyData) } }) }Copy the code

This article code repository: github.com/Licht-club/…

legacy

  • How do we test the ability to control someone else’s desktop
  • How does a desktop stream select the applications to be monitored
  • The keyboard event is executed multiple times for unknown reasons