Related documents of this article
Received a storm of Applause. That’s what’s expected of you.
robotjs
This door starts the project
Github.com/Licht-club/…
Branch main update 1. Webpack multi-entry multi-page, multiple rendering threads can be completed with the same Webpack configuration 2. Some basic optimization of WebpackCopy the code
The use of the desktop – capturer
/render-process/control/getDesSource.ts
That’s what’s expected of you. Women, that’s what’s expected of you. That’s what’s expected of you
Two changes have been made:
-
If (source.name === ‘Electron’
-
Video :{mandatory:{}} The es. Dom signature does not match and requires @ts-ignore magic annotation
import {desktopCapturer,} from ‘electron’
export type handleStream = (stream: MediaStream) => void export type handleError = (err: Error) => void
export function getDesSource(handleStream: handleStream, handleError ? : handleError) { desktopCapturer.getSources({types: [‘window’, ‘screen’]}).then(async sources => { for (const source of sources) { // if (source.name === ‘Electron’) { } try { const stream = await navigator.mediaDevices.getUserMedia({ audio: false, video: { // @ts-ignore mandatory: { chromeMediaSource: ‘desktop’, chromeMediaSourceId: source.id, minWidth: 1280, maxWidth: 1280, minHeight: 720, maxHeight: 720 } } }) handleStream(stream) } catch (e) { handleError && handleError(e) } return } }) }
render-process/control/DesktopCapturerVideo.tsx
Wrap a video component and call the getDesSource method above
import {desktopCapturer} from 'electron' import React, {useEffect, useMemo, useRef} from "react"; import {getDesSource} from "./getDesSource"; const DesktopCapturerVideo = () => { const videoRef = useRef<HTMLVideoElement>(null); const handleStream = (stream: MediaStream) => { console.log(videoRef.current) if (videoRef.current) { const video = videoRef.current video.srcObject = stream video.onloadedmetadata = (e) => video.play() } } const handleError = (error:Error) => { Console. log(error,' get desktop flow error ')} useEffect(() => {getDesSource(handleStream, handleError)}, []) return <video ref={videoRef}></video> } export default DesktopCapturerVideoCopy the code
render-process/control/index.tsx
The rendering thread (control page) calls the DesktopCapturerVideo component
import ReactDom from "react-dom"; import React, {useEffect} from "react"; import DesktopCapturerVideo from "./DesktopCapturerVideo"; Function App(){return <div> <span> <span> <DesktopCapturerVideo /> </div>} reactdom.render (<App></App>, document.getElementById('root'))Copy the code
Start the process
The Mac system displays a message asking you to configure the permission
After configuration, you see that the desktop stream was successfully captured
Interlude – Introduces Concurrently
It can be seen that our current project is rather troublesome to start, requiring three services: a React/Webpack rendering service, a TSC real-time compilation service and a electron window service
This condition, which can be used with this condition, can be resolved with Concurrently
The installation
$ npm install -g concurrently
or
$ npm install concurrently --save
Copy the code
I’m not going to install it globally
Configuring Startup Commands
"start:dev": "concurrently -k -p \\"[{name}]\\" -n \\"React,TypeScript,Electron\\" -c \\"yellow.bold,cyan.bold,green.bold\\" \\"npm run start:react\\" \\"npm run dev:main\\" \\"npm run start:electron\\""
Copy the code
Perform the start: dev, view the command line, you can see the React, TypeScript, Electron three threads are started, but Electron window is blank, This is because the React thread has not finished starting when the Electron thread starts successfully. You can refresh the Electron window or use waIT-on
wait-on
Wait – on the installation
npm install wait-on # local version
OR
npm install -g wait-on # global version
Copy the code
Modify our start:dev command
"start:dev": "concurrently -k -p \"[{name}]\" -n \"React,TypeScript,Electron\" -c \"yellow.bold,cyan.bold,green.bold\" \"npm run start:react\" \"npm run dev:main\" \" wait-on http://localhost:8080/main.html && npm run start:electron\""
Copy the code
Start the
As you can see, our Electron thread is successfully stuck after React
The remote control
Robotjs implements desktop automation and controls the mouse and screen
The installation of robotjs
NPM install robotjs // The installation will take a whileCopy the code
After the installation is successful, restart start:dev and type require(‘robotjs’) in the electron window, an error will be detected
An error figure
Robotjs is written in c++ and needs to be recompiled for different platforms and node versions.
Build robotJS
Ensure that the corresponding dependencies have been installed on the corresponding platform
Windows windows-build-tools npm package (npm install --global --production windows-build-tools from an elevated Mac Xcode Command Line Tools. Linux Python (v2.7 recommended, v3.x.x is not supported). make. A C/C++ compiler like GCC. libxtst-dev and libpng++-dev (sudo apt-get install libxtst-dev libpng++-dev).Copy the code
Begin to compile
$ npm install electron-rebuild --save-dev
$ npx electron-rebuild
Copy the code
Compile successfully as shown in figure!
main-process/index.ts
Need to close the app. AllowRendererProcessReuse
import {app, BrowserWindow} from 'electron' import {create} from './mainWindow' import {ipc} from "./ipc"; App. AllowRendererProcessReuse = false / / default is true, prevent the renderer load of context awareness in the process of the machine module. app.on('ready', () => { process.env['ELECTRON_DISABLE_SECURITY_WARNINGS'] = 'true'; // Disable web security warning ipc() create()})Copy the code
Restart start:dev and try reqire(‘rebotjs’) again. If it succeeds, the installation is successful
The use of robotjs
main-process/onRobot.ts
Robotjs can only be run in the main process, so the robotJS code in the main process via ipc allows the renderer to call the main process to do software control (keyboard and mouse).
import {ipcMain} from 'electron'; import robot from 'robotjs' export interface RobotData { keyCode: number; shift: boolean; meta: boolean; alt: boolean; screen? : { width: number; height: number }, video? : { width:number; height:number; }} export type RobotType = 'mouse' | 'key'/keyboard/mouse const robotHandle = (function () {function mouseHandle (data: RobotData) {console.log(data, 'temporary logic --mouseHandle')} function keyHandle(data: RobotData) {console.log(data, 'temporary logic --keyHandle')} return {mouseHandle, keyHandle } })() export default function onRobot() { ipcMain.on('robot', (e, type: RobotType, data: RobotData) => { if (type === 'mouse') { robotHandle.mouseHandle(data) } else if (type === 'key') { robotHandle.keyHandle(data) } }) }Copy the code
main-process/index.ts
The main thread entry adds the logic encapsulated above
import {app, BrowserWindow} from 'electron' import {create} from './mainWindow' import {ipc} from "./ipc"; import onRobot from "./onRobot"; app.allowRendererProcessReuse = false app.on('ready', () => { process.env['ELECTRON_DISABLE_SECURITY_WARNINGS'] = 'true'; Ipc () create() onRobot()})Copy the code
render-process/control/events.ts
Using nodeJS’s built-in Events module (publish subscribe mode)
import Events from 'events' import {RobotData, RobotType} from ".. /.. /main-process/onRobot"; import {ipcRenderer} from "electron"; const peer = new Events() peer.on('robot', (type: RobotType, data: RobotData) => { if (type === 'mouse') { data.screen = { width: window.screen.width, height: Window.screen. height}} // ipcrenderer. send Ipcrenderer. send('robot', type, data)}) export default peer;Copy the code
render-process/control/setRobot.ts
The render thread listens for mouse and keyboard events
import {ipcRenderer} from 'electron'
import peer from "./events";
const geneRobotKeyData = (e: KeyboardEvent) => {
return {
keyCode: e.keyCode,
shift: e.shiftKey,
meta: e.metaKey,
alt: e.altKey
}
}
const geneRobotMouseData = (e: MouseEvent) => {
return {
clientX: e.clientX,
clientY: e.clientY,
video: {
// width:
}
}
}
export function setRobot(videoDom: HTMLVideoElement) {
window.addEventListener('keydown', (e) => {
const data = geneRobotKeyData(e)
peer.emit('robot', 'key', data)
})
window.addEventListener('mouseup', (e) => {
const data = geneRobotMouseData(e)
data.video = {
width: videoDom.getBoundingClientRect().width,
height: videoDom.getBoundingClientRect().height
}
peer.emit('robot', 'mouse', data)
})
}
Copy the code
render-process/control/DesktopCapturerVideo.tsx
The console (video) component imports the above listening logic,
useEffect(() => {
getDesSource(handleStream, handleError)
setRobot(videoRef.current!)
}, [])
Copy the code
Restart the project, click the mouse or keyboard, and view the command line output
9. The printing is successful
Let’s go through the process
Render thread: listen to mouseup->event.emit Render thread: event.on->ipcRender. Send Main thread: ipcmain.onCopy the code
Improve the Handle logic of the main thread
/utils/vkey.ts
Keyboard mapping, source: github.com/chrisdickin… More than 100 lines of QaQ
'use strict' var ua = typeof window ! == 'undefined' ? window.navigator.userAgent : '' , isOSX = /OS X/.test(ua) , isOpera = /Opera/.test(ua) , maybeFirefox = ! /like Gecko/.test(ua) && ! isOpera let i=0; let output :Record<number, string> = { 0: isOSX ? '<menu>' : '<UNK>' , 1: '<mouse 1>' , 2: '<mouse 2>' , 3: '<break>' , 4: '<mouse 3>' , 5: '<mouse 4>' , 6: '<mouse 5>' , 8: '<backspace>' , 9: '<tab>' , 12: '<clear>' , 13: '<enter>' , 16: '<shift>' , 17: '<control>' , 18: '<alt>' , 19: '<pause>' , 20: '<caps-lock>' , 21: '<ime-hangul>' , 23: '<ime-junja>' , 24: '<ime-final>' , 25: '<ime-kanji>' , 27: '<escape>' , 28: '<ime-convert>' , 29: '<ime-nonconvert>' , 30: '<ime-accept>' , 31: '<ime-mode-change>' , 32: '<space>' , 33: '<page-up>' , 34: '<page-down>' , 35: '<end>' , 36: '<home>' , 37: '<left>' , 38: '<up>' , 39: '<right>' , 40: '<down>' , 41: '<select>' , 42: '<print>' , 43: '<execute>' , 44: '<snapshot>' , 45: '<insert>' , 46: '<delete>' , 47: '<help>' , 91: '<meta>' // meta-left -- no one handles left and right properly, so we coerce into one. , 92: '<meta>' // meta-right , 93: isOSX ? '<meta>' : '<menu>' // chrome,opera,safari all report this for meta-right (osx mbp). , 95: '<sleep>' , 106: '<num-*>' , 107: '<num-+>' , 108: '<num-enter>' , 109: '<num-->' , 110: '<num-.>' , 111: '<num-/>' , 144: '<num-lock>' , 145: '<scroll-lock>' , 160: '<shift-left>' , 161: '<shift-right>' , 162: '<control-left>' , 163: '<control-right>' , 164: '<alt-left>' , 165: '<alt-right>' , 166: '<browser-back>' , 167: '<browser-forward>' , 168: '<browser-refresh>' , 169: '<browser-stop>' , 170: '<browser-search>' , 171: '<browser-favorites>' , 172: '<browser-home>' // ff/osx reports '<volume-mute>' for '-' , 173: isOSX && maybeFirefox ? '-' : '<volume-mute>' , 174: '<volume-down>' , 175: '<volume-up>' , 176: '<next-track>' , 177: '<prev-track>' , 178: '<stop>' , 179: '<play-pause>' , 180: '<launch-mail>' , 181: '<launch-media-select>' , 182: '<launch-app 1>' , 183: '<launch-app 2>' , 186: '; = ', ', 187:188: ', ', 189: '-', 190, '. ', 191: '/', 192: \ ` ', 219: '[', 220: \ \ \ \', 221: '] ', 222: "'" , 223: '<meta>' , 224: '<meta>' // firefox reports meta here. , 226: '<alt-gr>' , 229: '<ime-process>' , 231: isOpera ? '\`' : '<unicode>' , 246: '<attention>' , 247: '<crsel>' , 248: '<exsel>' , 249: '<erase-eof>' , 250: '<play>' , 251: '<zoom>' , 252: '<no-name>' , 253: '<pa-1>' , 254: '<clear>' } for(i = 58; i < 65; ++i) { output[i] = String.fromCharCode(i) } // 0-9 for(i = 48; i < 58; ++i) { output[i] = (i - 48)+'' } // A-Z for(i = 65; i < 91; ++i) { output[i] = String.fromCharCode(i) } // num0-9 for(i = 96; i < 106; ++i) { output[i] = '<num-'+(i - 96)+'>' } // F1-F24 for(i = 112; i < 136; ++i) { output[i] = 'F'+(i-111) } export default outputCopy the code
Found a bug
Dist /main-process/index.js, but the main field in package.json is wrong
Should be “main”: “dist/main-process/index.js”
Continue to
main-process/onRobot.ts
import {ipcMain} from 'electron'; import robot from 'robotjs' import vkey from ".. /utils/vkey"; export interface ScreenVideoInfo { screen: { width: number; height: number }, video: { width: number; height: number } } export type RobotKeyData = { keyCode: number; shift: boolean; meta: boolean; alt: boolean; } export type RobotMouseData = { clientX: number; clientY: number; } & ScreenVideoInfo export type RobotType = 'mouse' | 'key'/keyboard/mouse const robotHandle = (function () {function mouseHandle(data: RobotMouseData) { const {clientX, clientY, video, screen} = data let x = clientX * screen.width / video.width let y = clientY * screen.height / video.height // MoveMouse (x, y) // robot. MouseClick () console.log(\ 'robot click \${x},\${y}\')} function keyHandle(data: RobotKeyData) {const modiFIERS = [] // if (data.meta) {modiFIERS. Push ('meta')} if (data.shift) { modifiers.push('shift') } if (data.alt) { modifiers.push('alt') } let key = vkey[data.keyCode].toLowerCase() if (key[0] ! == '<') {robot. KeyTap (key, modifiers) console.log(\ 'pressing a key \${key}\ ')}} return {mouseHandle, keyHandle } })() export default function onRobot() { ipcMain.on('robot', (e, type: RobotType, data: RobotKeyData | RobotMouseData) => { if (type === 'mouse') { robotHandle.mouseHandle(data as RobotMouseData) } else if (type === 'key') { robotHandle.keyHandle(data as RobotKeyData) } }) }Copy the code
This article code repository: github.com/Licht-club/…
legacy
- How do we test the ability to control someone else’s desktop
- How does a desktop stream select the applications to be monitored
- The keyboard event is executed multiple times for unknown reasons