Whenever we see such cool games live, do we wonder how the function of bullet screen is implemented?
At first, the bullet-screen was implemented on the principle of window hierarchy, with a relatively simple structure. Specific implementation points are as follows:
- Create a separate barrage window
- Window top, set mouse penetration
Based on the above two points, when the full-screen game is windotized, our bullet-screen window will always be displayed on the top layer, so it can meet the needs of anchors to watch the bullet-screen while broadcasting the game. This scheme has drawbacks: the game process needs to be run in the form of Windows, that is, the exclusive display mode cannot be used, and the FPS of the game will be reduced, which may affect the game experience of the host, and thus the viewing experience of the audience.
Realizing the seriousness of the problem, we learned from many cutting-edge live broadcast technology solutions and improved the implementation of our bullet screen!!
First, let’s take a look at the resource consumption and efficiency comparison chart of the new barrage scheme:
| Live from eternity | League of Legends live | Dota2 live | |
|---|---|---|---|
| FPS(fluctuating range) | No rounds: 56-65 Barrage available: 58-61 |
No barrage: 165-179 Artillery barrage: 158-171 |
No barrage: 107-118 Artillery barrage: 102-118 |
It can be seen that there is little impact on the game experience under the situation of bullet screen, so xiaobian will take you to interpret the implementation scheme of the new full-screen bullet screen
New version of the barrage implementation
Compared with the old version, the new version adopts a new technology to achieve the implementation of barrage. Based on the premise of Hook, our drawing module is injected into the game process, so as to achieve the realization of our barrage inside the game, which relies on the following main knowledge points (bold fonts in the following are the main steps to achieve the function of barrage) :
- Use of dear-imgui Open Source library
- The use of hooks
- DirectX basic drawing principle
Dear – imgui is introduced
Dear ImGui is a lightweight user GUI library that outputs our custom GUI to the application’s vertex buffer during application rendering (i.e. textures described in rendering below) for final application rendering. The detailed usage of the interface library can be found in the link :(link). This article will be skipped for the moment
DirectX basic drawing principle
Game development requires the use of graphics rendering apis (DirectX, OpenGL, etc.). The same is true for Windows games. The vast majority of games are based on the official DirectX interface provided by Microsoft. There are currently DirectX 9, DirectX 11.0, DirectX 11.1, and DirectX 12(to a lesser extent). To draw a barrage inside the game, it is necessary to understand the principle of DirectX drawing. The following is a brief introduction of DirectX, which is the key for us to find hook points and draw a barrage in the future:
- Direct3D is a low-level drawing API (Application Programming Interface) that allows us to draw 3D worlds faster with 3D hardware. In essence, Direct3D provides a set of software interfaces through which we can control the drawing hardware. For example, to command the drawing equipment to empty the render target (such as screen), you can call Direct3D ID3D11DeviceContext: : ClearRenderTargetView method to finish the job. The Direct3D layer sits between the application and the drawing hardware so that software developers don’t have to worry about the implementation details of the 3D hardware. As long as the device supports Direct3D 11, the 3D hardware can be controlled through the Direct3D 11 API
- A 2D texture is a matrix of data elements. One of the uses of 2D textures is to store 2D image data, storing a pixel color in each element of the texture. But that’s not the only use of texture; For example, an advanced technique called Normal mapping stores 3D vectors in texture elements instead of colors. For this reason, in a general sense, textures are used to store image data, but in practice they can have a wider use.
- Rendering. In general, to avoid flicker in the animation, it is best to perform all animation frame drawing in an off-screen texture called the back buffer. When we finish drawing a given frame in the background buffer, we can display the background buffer as a complete frame on the screen; With this method, the user is not aware of the frame being drawn, only sees the complete frame. In theory, the time it takes to bring a frame to the screen is less than the screen’s vertical refresh time. The hardware automatically maintains two built-in texture buffers to do this, called the front buffer and the background buffer. The foreground buffer stores the image data currently displayed on the screen, while the next frame of the animation is drawn in the background buffer. When the background buffer is finished drawing, the roles of the two buffers are reversed: the background buffer becomes the foreground buffer, and the foreground buffer becomes the background buffer in preparation for the next frame drawing. This exchange of buffer functions is called presenting. As shown in the figure below
The use of hook and the implementation process of barrage
A Hook is a platform for Windows message processing on which an application program can set subroutines to monitor a message in a specified window that may be created by another process. When the message arrives, process it before the target window handler. The hook mechanism allows applications to intercept processing window messages or specific events. Its application modes mainly include: observation mode, injection mode, replacement mode, centralized mode and so on. Related modes are introduced as follows:
- Observation mode (The observer can only view the information and handle their own business according to the content they care about, but can not change, such as lyrics)
- Injection mode (the addition of new extension code and the original code will be coordinated to handle similar services, such as full screen live screen)
- Replacement mode (old code is replaced by new code)
- Centralized mode (dealing with one type of thing in a unified way, such as keyboard locks)
Here we need to use injection mode, and the Direct3D drawing API process hook, and full screen barrage hook API, to DirectX 11 as an example, we hook is IDXGISwapChain::Present method, the function of the method is to display the texture on the display device.
Livestreaming assistant games are captured in injection mode, HHOOK WINAPI SetWindowsHookEx(__in int idHook, hook type __in HOOKPROC LPFN, callback function address __in HINSTANCE hMod, Instance handle __in DWORD dwThreadId thread ID), implement injection behavior, the key code is as follows:
int inject_library_safe_obf(DWORD thread_id, const wchar_t *dll, const char *set_windows_hook_ex_obf, uint64_t obf1) { HMODULE user32 = GetModuleHandleW(L"USER32"); set_windows_hook_ex_t set_windows_hook_ex; HMODULE lib = LoadLibraryW(dll); LPVOID proc; HHOOK hook; size_t i; if (! lib || ! user32) { return INJECT_ERROR_UNLIKELY_FAIL; } #ifdef _WIN64 proc = GetProcAddress(lib, "dummy_debug_proc"); #else proc = GetProcAddress(lib, "_dummy_debug_proc@12"); #endif if (! proc) { return INJECT_ERROR_UNLIKELY_FAIL; } set_windows_hook_ex = get_obfuscated_func(user32, set_windows_hook_ex_obf, obf1); hook = set_windows_hook_ex(WH_GETMESSAGE, proc, lib, thread_id); if (! hook) { return GetLastError(); } for (i = 0; i < RETRY_COUNT; i++) { Sleep(RETRY_INTERVAL_MS); PostThreadMessage(thread_id, WM_USER + 432, 0, (LPARAM)hook); } return 0; }Copy the code// An explicit call to the SetWindowsHookEx function is blocked and mistakenly reported by security software. Therefore, the shookex function is encrypted and decrypted to avoid security software
static inline int inject_library_safe(DWORD thread_id, const wchar_t *dll)
{
return inject_library_safe_obf(thread_id, dll, "[bs^fbkmwuKfmfOvI",
0xEAD293602FCF9778ULL);
}
Copy the codeWith the above code, we can inject our own DLL (graphics-hook.dll) into the game process. Graphics-hook. DLL works in two main steps:
- Hook IDXGISwapChain: : the Present method
- Draw the barrage interface
The key code is as follows: Through the VirtualProtect system API, change the read and write permission of the process memory page, and change the address of the DXD present method in the memory page to its own present address.
Loading in dxgi. DLL
static inline bool dxgi_init(dxgi_info &info) { HMODULE d3d10_module; d3d10create_t create; create_fac_t create_factory; IDXGIFactory1 *factory; IDXGIAdapter1 *adapter; IUnknown *device; HRESULT hr; info.hwnd = CreateWindowExA(0, DUMMY_WNDCLASS, "d3d10 get-offset window", WS_POPUP, 0, 0, 2, 2, nullptr, nullptr, GetModuleHandleA(nullptr), nullptr); if (! info.hwnd) { return false; } info.module = LoadLibraryA("dxgi.dll"); if (! info.module) { return false; } create_factory = (create_fac_t)GetProcAddress(info.module, "CreateDXGIFactory1"); d3d10_module = LoadLibraryA("d3d10.dll"); if (! d3d10_module) { return false; } create = (d3d10create_t)GetProcAddress(d3d10_module, "D3D10CreateDeviceAndSwapChain"); . return true; }Copy the codeGet DXD interface and address offset
void get_dxgi_offsets(struct dxgi_offsets *offsets) { dxgi_info info = {}; bool success = dxgi_init(info); HRESULT hr; if (success) { offsets->present = vtable_offset(info.module, info.swap, 8); offsets->resize = vtable_offset(info.module, info.swap, 13); IDXGISwapChain1 *swap1; hr = info.swap->QueryInterface(__uuidof(IDXGISwapChain1), (void **)&swap1); if (SUCCEEDED(hr)) { offsets->present1 = vtable_offset(info.module, swap1, 22); swap1->Release(); } } dxgi_free(info); } void hook_init(struct func_hook *hook, void *func_addr, void *hook_addr, const char *name) { memset(hook, 0, sizeof(*hook)); hook->func_addr = (uintptr_t)func_addr; hook->hook_addr = (uintptr_t)hook_addr; hook->name = name; Fix_permissions ((void *)(hook-> func_addr-jmp_32_size), JMP_64_SIZE + JMP_32_SIZE); memcpy(hook->unhook_data, func_addr, JMP_64_SIZE); }Copy the codeInsert function address
static inline void hook_reverse_new(struct func_hook *hook, uint8_t *p) { hook->call_addr = (void *)(hook->func_addr + 2); hook->type = HOOKTYPE_REVERSE_CHAIN; hook->hooked = true; E9 is the jump instruction, and the following four bytes are the jump's offset address P [0] = 0xE9; *((uint32_t *)&p[1]) = (uint32_t)(hook->hook_addr - hook->func_addr); *((uint16_t *)&p[5]) = X86_JMP_NEG_5; }Copy the codeAfter completing the hook of IDXGISwapChain::Present method, then we can draw the bullet-screen interface in our self-defined method, which requires dedear imgui. Of course, we also need to actively call IDXGISwapChain:: present in our present, otherwise the game will not draw properly. The following describes the use of IMGUI through a simple control drawing code
Resource initialization
void init_imgui(){
IMGUI_CHECKVERSION();
ImGui::CreateContext();
ImGuiIO &io = ImGui::GetIO();
StyleColorsYuer(nullptr);
// Setup Platform/Renderer backends
ImGui_ImplWin32_Init(hwnd);
ImGui_ImplDX11_Init(device, context);
}
Copy the codeInterface to draw
void paint() { ImGui_ImplDX11_NewFrame(); ImGui_ImplWin32_NewFrame(); ImGui::NewFrame(); ImGui::End(); {static float f = 0.0f; static int counter = 0; ImGui::Begin("Hello, world!" ); // Create a window called "Hello, world!" and append into it. ImGui::Text("This is some useful text."); // Display some text (you can use a format strings too) ImGui::Checkbox("Demo Window", &show_demo_window); // Edit bools storing our window open/close state ImGui::Checkbox("Another Window", &show_another_window); ImGui: : SliderFloat (" float ", & f, f, 0.0 1.0 f); // Edit 1 float using a slider from 0.0f to 1.0f ImGui::ColorEdit3("clear color", (float*)&clear_color); // Edit 3 floats representing a color if (ImGui::Button("Button")) // Buttons return true when clicked (most widgets return true when edited/activated) counter++; ImGui::SameLine(); ImGui::Text("counter = %d", counter); ImGui::Text("Application Average %.3f ms/frame (%.1f FPS)", 1000.0f/ImGui::GetIO().framerate, ImGui::GetIO().Framerate); ImGui::End(); } //Rendering ImGui::EndFrame(); ImGui::Render(); ImGui_ImplDX11_RenderDrawData(ImGui::GetDrawData()); }Copy the codeRelease resources
void destroy(){
ImGui_ImplDX11_Shutdown();
ImGui_ImplWin32_Shutdown();
ImGui::DestroyContext();
}
Copy the codeconclusion
At this point, the function of barrage is basically realized. Of course, the program also has a lot of space to optimize, this article only introduced the general implementation principle and process, if there are mistakes, please also correct!