The foreword 0.

Recently I have been writing Lua scripts, sometimes there is a problem, I don’t know whether it is the Lua layer problem, or the upstream problem, I don’t know where to start. So I learned a little bit about C/C++ and JNI and read through the logic of the process of executing Lua scripts. After reading through it, I came up with the idea of implementing a Lua script for Android. Hence this blog post. I’m not familiar with either C/C++ or Kotlin, so I’ll write mainly in those two languages (so it’ll be Java Style).

1. Environment construction

First of all, download the source code of Lua from Lua’s official website. I used the 5.3.5 version. Then import the source code into the Project and write CMakeList:

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.Cmake_minimum_required (VERSION 3.4.1 track)# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.
add_definitions(-Wno-deprecated)

add_library( # Sets the name of the library.
        luabridge

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        src/main/jni/lua/lapi.c
        src/main/jni/lua/lauxlib.c
        src/main/jni/lua/lbaselib.c
        src/main/jni/lua/lbitlib.c
        src/main/jni/lua/lcode.c
        src/main/jni/lua/lcorolib.c
        src/main/jni/lua/lctype.c
        src/main/jni/lua/ldblib.c
        src/main/jni/lua/ldebug.c
        src/main/jni/lua/ldo.c
        src/main/jni/lua/ldump.c
        src/main/jni/lua/lfunc.c
        src/main/jni/lua/lgc.c
        src/main/jni/lua/linit.c
        src/main/jni/lua/liolib.c
        src/main/jni/lua/llex.c
        src/main/jni/lua/lmathlib.c
        src/main/jni/lua/lmem.c
        src/main/jni/lua/loadlib.c
        src/main/jni/lua/lobject.c
        src/main/jni/lua/lopcodes.c
        src/main/jni/lua/loslib.c
        src/main/jni/lua/lparser.c
        src/main/jni/lua/lstate.c
        src/main/jni/lua/lstring.c
        src/main/jni/lua/lstrlib.c
        src/main/jni/lua/ltable.c
        src/main/jni/lua/ltablib.c
        src/main/jni/lua/ltm.c
        src/main/jni/lua/lua.c
        #src/main/jni/lua/luac.c
        src/main/jni/lua/lundump.c
        src/main/jni/lua/lutf8lib.c
        src/main/jni/lua/lvm.c
        src/main/jni/lua/lzio.c)

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        luabridge

        # Links the target library to the log library
        # included in the NDK.
        ${log-lib})
Copy the code

I want to run a *. Lua type script, so leave lua.c and delete luac.c, CMakeList also comment out. Also, since I imported the Lua source code as a library, I don’t need the main method anymore. I’ll comment out the main method in Lua.c. Finally, Rebuild the Project.

2. Android calls Lua unidirectionally

Set a small target, the Android layer calls the Lua layer’s function, the Lua layer does an addition and returns the result to the Android layer. Write the Lua script first:

function test(a, b)
	return a + b
end
Copy the code

This Lua script is very simple and returns the sum of a and B that are passed. Now we can start thinking about the implementation of the Native layer. Before considering implementation, you need to understand the Lua virtual stack and several Lua C apis.

2.1. Lua virtual stack

Data exchange between Lua layer and Native layer is accomplished through Lua virtual stack. This virtual stack is slightly different from a normal stack in that it can access a given element through a negative index. As shown in figure:

2.2. The Lua C APIs

Lua provides C APIs to facilitate communication between Native layer and Lua layer. The following Demo will use these C apis.

  • lua_State *luaL_newstate (void);

    Create a new Lua context.

  • int luaL_loadbuffer (lua_State *L, const char *buff, size_t sz, const char *name);

    Compile a Lua Chunk. If the compilation succeeds, it wraps the result into a function and pushes the function onto the stack. Otherwise, the compilation fails and it pushes the error message onto the stack.

    parameter type instructions
    L lua_State* The context of the Lua
    buff const char* Lua script buffer that needs to be loaded
    sz size_t The length of the Lua script buffer
    name const char* The name of this chunk is nullable
  • int lua_pcall (lua_State *L, int nargs, int nresults, int errfunc);

    A function called in safe mode will not crash even if it throws an exception. When an exception is thrown, if errFunc is 0, the Lua virtual stack pushes the error message to the Lua virtual stack. If errFunc is not 0, the error is handled by the Lua virtual stack function whose index is errFunc. When the execution is complete, the Lua virtual machine pops the arguments and the called function off the stack.

    parameter type instructions
    L lua_State* The context of the Lua
    nargs int The number of arguments to the function to be called
    nresults int The number of results returned by the function to be called
    errfunc int The index of the error handler in the Lua virtual stack. If 0, the error message is pushed to the Lua virtual stack
  • void lua_getglobal (lua_State *L, const char *name);

    Get the global variable named name and push it onto the stack.

    parameter type instructions
    L lua_State* The context of the Lua
    name const char* The variable name
  • void lua_pushinteger (lua_State *L, lua_Integer n);

  • Push a lua_Integer onto the stack

    parameter type instructions
    L lua_State* The context of the Lua
    n lua_Integer The number to push in
  • lua_Integer lua_tointeger (lua_State *L, int index);

    Converts the index element on the stack to lua_Integer and returns it

    parameter type instructions
    L lua_State* The context of the Lua
    index int Specifies the index of the element on the stack

In addition to these C apis, see the official website for an introduction and usage.

By understanding the Lua virtual stack and knowing some Lua C apis, we can implement a simple Native layer to call Lua layer functions.

jint startScript(JNIEnv* env, jobject obj, jstring jLuaStr, jint a, Jint b) {// Create a lua context lua_State* luaContext = lua_newState (); // initialize lua lib luaL_openlibs(luaContext); const char* cLuaStr = env->GetStringUTFChars(jLuaStr, NULL); Int loadStatus = luaL_loadbuffer(luaContext, cLuaStr, strlen(cLuaStr), NULL); int loadStatus = luaL_loadbuffer(luaContext, cLuaStr, strlen(cLuaStr), NULL);if(LUA_OK ! = loadStatus) { const char *szError = luaL_checkstring(luaContext, -1); Log_e(LOG_TAG,"%s", szError);
        return- 1; } env->ReleaseStringUTFChars(jLuaStr, cLuaStr); int callStatus = lua_pcall(luaContext, 0, LUA_MULTRET, 0);if(LUA_OK ! = callStatus) { const char *szError = luaL_checkstring(luaContext, -1); Log_e(LOG_TAG,"%s", szError);
        return- 1; } / / accesstestMethods lua_getglobal (luaContext,"test");
    if(LUA_TFUNCTION ! = lua_type(luaContext, -1)) { Log_d(LOG_TAG,"can not found func : %s"."test");
        return false; } // Push the argument lua_pushINTEGER (luaContext, a); lua_pushinteger(luaContext, b); / / executiontestInt callTestStatus = lua_pCall (luaContext, 2, 1, 0);if(LUA_OK == callTestStatus) {
		int ret = lua_tointeger(luaContext, 1)
		return ret;
	} else {
		const char* errMsg = lua_tostring(luaContext, 1)
		Log_e(LOG_TAG, "%s", errMsg);
		return -1;
	}
}
Copy the code

The process is like a comment. During this process, the contents of the Lua virtual stack change as shown below, starting with luaL_loadbuffer:

First, after luaL_loadbuffer passes through luaL_loadbuffer, luaL_loadbuffer takes the *. Lua file’s buffer as a Lua Chunk and compiles it. After compiling, wrap the result into a function and push it into the Lua virtual stack. After lua_pcall, the Lua virtual machine pops the function and its parameters from the Lua virtual stack. Next, lua_getGlobal gets the Lua layer global variable “test”, and lua_getGlobal pushes the value of this variable into the Lua virtual stack. The function is ready, and after lua_PUSHINTEGER (a) and lua_PUSHINTEGER (b), the function and arguments have been pushed in order, and the prerequisites for calling lua_pCall have been met. Next, after calling lua_pCall, the Lua virtual machine pushes the results into the Lua virtual stack based on the nResults that were passed in for the call to lua_pCall. Finally, we just need lua_toINTEGER (index) to get the result of the execution and return it to the Android layer. As you can see from beginning to end, the Lua virtual stack plays an important role as a bridge between data exchanges.

Next, just Register NativeMethods in the Native layer and declare the Native method in the Android layer.

class LuaExecutor {
    init {
        System.loadLibrary("luabridge")
    }

    external fun startScript(luaString: String): Boolean
}
Copy the code

However, the above implementation only has the ability to launch scripts. In practice, you can’t start a script and not have some control over the flow of script execution. Therefore, it is necessary to add a stop script function. How do I stop an executing script? Let’s start with the C API provided by Lua:

  • int luaL_error (lua_State *L, const char *fmt, …) ;

    Throws an exception with the error message FMT.

    parameter type instructions
    L lua_State* The context of the Lua
    fmt const char* The error message
  • int lua_sethook (lua_State *L, lua_Hook f, int mask, int count);

    Set a hook function.

    parameter type instructions
    L lua_State* The context of the Lua
    f lua_Hook The hook function that contains the statement to execute
    mask int Specifies when to call. The value is the bitwise or of the constants LUA_MASKCALL, LUA_MASKRET, LUA_MASKLINE, and LUA_MASKCOUNT.
    The mask value instructions
    LUA_MASKCALL Represents that hook function f is executed after entering any function
    LUA_MASKRET Represents that the hook function is executed before exiting any function
    LUA_MASKLINE Represents that hook function f is executed before executing a line of code within the function
    LUA_MASKCOUNT The hook function f is executed after the lua interpreter executes count instructions

With these two C apis, the stop function of the script can be implemented:

void stopLuaHooker(lua_State *L, lua_Debug *ar) {
    luaL_error(L, "quit Lua");
}

void forceStopLua(lua_State *L) {
    int mask = LUA_MASKCOUNT;
    lua_sethook(L, &stopLuaHooker, mask, 1);
}
Copy the code

When we call forceStopLua, we set up a hook function for Lua script execution. This hook function is executed when the Lua interpreter finishes executing an instruction after lua_sethook has been executed. That is, when we call forceStopLua anywhere in the Lua layer code, the Lua interpreter executes an instruction, then stopLuaHooker, then Lua_error, throws an exception, and the script terminates. Thus, the start and stop functions of the script are already implemented in a class called LuaEngine:

#ifndef ANDROIDLUA_LUAENGINE_H
#define ANDROIDLUA_LUAENGINE_H

#include <cstring>
#include <string>
#include <jni.h>
#include "lua/lua.hpp"

#include "utils/Log.h"
#include "JniManager.h"

#define LOG_TAG "LuaEngine"

class LuaEngine {
public:
    LuaEngine();

    virtual ~LuaEngine();

    lua_State *getScriptContext() {
        return mScriptContext;
    }

    bool startScript(jstring jBuff, const char *functionName);

    bool isScriptRunning() {
        return scriptRunning;
    }

    bool stopScript();

private:
    lua_State *mScriptContext;
    bool scriptRunning;

    bool loadBuff(jstring jBuff);

    bool runLuaFunction(const char *functionName);
};

void quitLuaThread(lua_State *L);

void quitLuaThreadHooker(lua_State *L, lua_Debug *ar);

#endif //ANDROIDLUA_LUAENGINE_H
Copy the code
#include "LuaEngine.h"

LuaEngine::LuaEngine() {
    mScriptContext = luaL_newstate();
    scriptRunning = false;
}

LuaEngine::~LuaEngine() {
    if (isScriptRunning()) {
        stopScript();
    }
    mScriptContext = nullptr;
}

bool LuaEngine::startScript(jstring jBuff, const char *functionName) {
    scriptRunning = true;
    luaL_openlibs(mScriptContext);
    if (this->loadBuff(jBuff)) {
        Log_d(LOG_TAG, "script start running..");
        bool success = this->runLuaFunction(functionName);
        scriptRunning = false;
        return success;
    } else {
        scriptRunning = false;
        return false;
    }
}

bool LuaEngine::stopScript() {
    if (scriptRunning) {
        quitLuaThread(mScriptContext);
        scriptRunning = false;
        return true;
    } else {
        Log_d(LOG_TAG, "script is Not running");
        return false;
    }
}

bool LuaEngine::loadBuff(jstring jBuff) {
    // 读取buff
    JNIEnv *env;
    JniManager::getInstance()->getJvm()->GetEnv((void **) &env, JNI_VERSION_1_6);
    const char *cBuff = env->GetStringUTFChars(jBuff, nullptr);
    if(LUA_OK ! = luaL_loadbuffer(mScriptContext, cBuff, strlen(cBuff), NULL)) { const char *szError = luaL_checkstring(mScriptContext, - 1); Log_e(LOG_TAG,"%s", szError);
        return false; } // Load the buff to memoryif(LUA_OK ! = lua_pcall(mScriptContext, 0, LUA_MULTRET, 0)) { const char *szError = luaL_checkstring(mScriptContext, -1); Log_e(LOG_TAG,"%s", szError);
        return false;
    }
    env->ReleaseStringUTFChars(jBuff, cBuff);
    env->DeleteGlobalRef(jBuff);
    return true;
}

bool LuaEngine::runLuaFunction(const char *functionName) {// Get errorFunc // The error is handled by __TRACKBACK__, which can be used to print error messages, // __TRACKBACK__ function needs to be defined in lua script lua_getGlobal (mScriptContext,"__TRACKBACK__");
    if(lua_type(mScriptContext, -1) ! = LUA_TFUNCTION) { Log_d(LOG_TAG,"can not found errorFunc : __TRACKBACK__");
        return false; } int errfunc = lua_gettop(mScriptContext); // get the specified method lua_getGlobal (mScriptContext,functionName);
    if(lua_type(mScriptContext, -1) ! = LUA_TFUNCTION) { Log_d(LOG_TAG,"can not found func : %s".functionName);
        return false; } // Run the specified methodreturn LUA_OK == lua_pcall(mScriptContext, 0, 0, errfunc);
}

void quitLuaThread(lua_State *L) {
    int mask = LUA_MASKCOUNT;
    lua_sethook(L, &quitLuaThreadHooker, mask, 1);
}

void quitLuaThreadHooker(lua_State *L, lua_Debug *ar) {
    luaL_error(L, "quit Lua");
}
Copy the code

3. Lua calls Android unidirectionally

The previous implementation only allowed the Android layer to call Lua methods, and the Lua layer could not call Android layer methods. Can I call Android layer methods from Lua layer? The answer is yes. One idea is that the Lua layer calls the Native layer’s methods, and the Native layer calls the Android layer’s methods via reflection. Let’s look at how the Lua layer calls Native layer methods. Lua provides a C API: Lua_Register, which is modeled after:

  • void lua_register (lua_State *L, const char *name, lua_CFunction f);

    Register a CFunction.

    parameter type instructions
    L lua_State* The context of the Lua
    name const char* Lua layer global variable name
    f lua_CFunction C functions. The prototype is int functionXXX(lua_State* L); The meaning of its return value represents the number of results returned.

We can use this C API to implement Lua layer call Native layer method:

lua_register(mScriptContext, "getString" , getString);

int getString(lua_State *L) {
    const char *cStr = "String From C Layer";
    lua_pushstring(L, cStr);
    return 1;
}
Copy the code

The above code is very simple. First, register a global variable named getString, pointing to the C function getString. The C function getString declares and allocates a string cStr, pushes the string onto the Lua stack, and returns the number of results. Therefore, in Lua Layer, if you execute getString(), you get the String “String From C Layer”, and Lua Layer can call Native Layer methods.

Then look at how the Native layer calls the Android layer. The code is as follows:

int getString(lua_State *L) {
	JNIEnv* env;
	g_pJvm->GetEnv((void **) &env, JNI_VERSION_1_6);
	
    jclass clazz = env->FindClass("com/zspirytus/androidlua/shell/ShellBridge");
	if(! clazz) { Log_d(LOG_TAG,"class not found!");
        return 0;
    }
	
    jmethodID methodId = env->GetStaticMethodID(clazz, "getStringFromKotlinLayer"."()Ljava/lang/String;");
    if(! methodId) { Log_d(LOG_TAG,"method %s not found!"."getStringFromStaticJavaMethod");
        return 0;
    }
	
    jstring jStr = (jstring) env->CallStaticObjectMethod(clazz, methodId);
	
    const char *cStr = env->GetStringUTFChars(jStr, NULL);
    lua_pushstring(L, cStr);
    env->ReleaseStringUTFChars(jStr, cStr);
    env->DeleteLocalRef(jStr);
    return 1;
}
Copy the code

Jstring = JNI_OnLoad = JNI_OnLoad = JNI_OnLoad = JNI_OnLoad = JNI_OnLoad = JNI_OnLoad A c-style string is pushed into the Lua stack to release resources and return the number of results.

In the Android layer, a method is left to call:

@Keep
object ShellBridge {

    private val TAG = ShellBridge.javaClass.simpleName

    @Keep
    @JvmStatic
    fun getStringFromKotlinLayer(): String {
        return "String From Android Layer"}}Copy the code

At this point, the Interaction between the Android layer and the Lua layer has been implemented.

4. Avoid ANR

However, the above implementation can lead to ANR because Lua script execution can be time consuming. If the Lua script takes more than 5 seconds to execute, ANR is required. One solution is to put the execution of Lua scripts into child threads. Should the child thread be better managed by the Native layer or the Android layer? I personally think it is better to put it in the Native layer, so that the Android layer does not need to create and manage threads to execute Lua scripts, and the code is not too complicated. Even though the logic of Native layer is complicated, so is generally used as a library without touching it. So again, create and manage threads in the Native layer. Pthread_create is a function for creating threads on Unix, Linux, and other systems.

  • int pthread_create(pthread_t *restrict tidp, const pthread_attr_t *restrict attr, void *(*start_rtn)(void *), void *restrict arg);

    parameter type instructions
    tidp pthread_t *restrict Thread ID
    attr const pthread_attr_t*restrict Thread property, default is NULL
    *(*start_rtn)(void *) void A function that runs on a new thread
    *restrict arg void Start_rtn Specifies the required parameter

Therefore, we can move the logic for executing Lua scripts to a new thread:

void startWork() {
    pthread_create(&mThreadId, NULL, &startWorkInner, (void*)this);
}

void stopWork() {
    stopScript();
    mThreadId = 0;
}

void* startWorkInner(void *args) {
    startScript();
    return nullptr;
}
Copy the code

This way, startScript() runs in the new thread without the risk of ANR. We put it in a class called LuaTask, which manages the start and end of a Lua script.

#ifndef ANDROIDLUA_LUATASK_H
#define ANDROIDLUA_LUATASK_H

#include <sys/types.h>
#include <pthread.h>
#include <jni.h>

#include "LuaEngine.h"

class LuaTask {

public:
    LuaTask(jstring jBuff);

    virtual ~LuaTask();

    void startWork();

    void stopWork();

    bool isRunning();

private:
    static void *startWorkInner(void *args);

private:
    jstring mLuaBuff;
    pthread_t mThreadId;
    LuaEngine *mLuaEngine;
};

#endif //ANDROIDLUA_LUATASK_H
Copy the code
#include "LuaTask.h"

LuaTask::LuaTask(jstring jBuff) {
    mLuaBuff = jBuff;
    mLuaEngine = new LuaEngine();
    mThreadId = 0;
}

LuaTask::~LuaTask() {
    delete mLuaEngine;
}

void LuaTask::startWork() {
    pthread_create(&mThreadId, NULL, &LuaTask::startWorkInner, (void*)this);
}

void LuaTask::stopWork() {
    mLuaEngine->stopScript();
    mThreadId = 0;
}

void* LuaTask::startWorkInner(void *args) {
    LuaTask* task = (LuaTask*) args;
    task->mLuaEngine->startScript(task->mLuaBuff, "main");
    return nullptr;
}

bool LuaTask::isRunning() {
    returnmThreadId ! = 0; }Copy the code

However, this is our newly created thread and we haven’t attached it to JavaVM yet. If you don’t attach to JavaVM, you won’t find the JNIEnv, so you have to attach to JavaVM in order to get the JavaVM’S JNI environment variables and call the Android layer methods. So startWorkInner needs to be improved:

void* startWorkInner(void *args) {
    JNIEnv* env = nullptr;
    JavaVMAttachArgs args{JNI_VERSION_1_6, nullptr, nullptr};
    g_pJvm->AttachCurrentThread(&env, &args);
    startScript()
    g_pJvm->DetachCurrentThread();
    return nullptr;
}
Copy the code

Before the thread exits, remember to check with JavaVM detach so that the thread exits properly.

5. Run the script package

At this point, we have finished executing Lua scripts that can start and stop at any time and print stack information if errors occur. In practice, though, you can’t just run a single script, and the script may require some resource files. Therefore, we usually package the script and resource files into a script package. Before running the script, unpack it and parse it before running it. So is the logic for parsing the script in Native or Android? I personally think it’s better on the Android layer. There are two reasons:

  1. The format of the script package is uncertain, and the Native layer cannot accommodate every case, so it is left to the user to parse.
  2. The single responsibility principle, Native layer is responsible for only one function is better. And recompiling an SO file to parse the script package is too much of a hassle, so leave the parsing to the user.

Speaking of script packages, I’ll talk briefly about my implementation. My implementation is to compress lua scripts and resource files into a zip file. In the zip file, there is a config file that contains the relative paths of all Lua scripts. During parsing, first extract the config in memory and read out the relative paths of all Lua scripts. Then extract all Lua script files in memory and spliced them together before handing them to Native layer for running. As for the resource files, decompress them dynamically based on how the script is running. I simply encapsulate it:

private external fun startScript(luaString: String): Boolean external fun stopScript(): Boolean external fun isScriptRunning(): Boolean fun runScriptPkg(scriptPkg: File, configFile: String) { mThreadPool? .execute { val start = System.currentTimeMillis() initScriptPkg(scriptPkg) val zipFile = ZipFile(scriptPkg) val config =  ZipFileUtils.getFileContentFromZipFile(zipFile, configFile) val luaScriptPaths = config.split("\r\n")
        val luaScript = ZipFileUtils.getFilesContentFromZipFile(zipFile, luaScriptPaths)
        Log.d("USE_TIME"."${System.currentTimeMillis() - start} ms") mHandler? .post { startScript(luaScript) } } } object ZipFileUtils { fun getFileContentFromZipFile(zipFile: ZipFile, targetFile: String): String { var ins: InputStream? = null try { val ze = zipFile.getEntry(targetFile)return if(ze ! = null) { ins = zipFile.getInputStream(ze) FileUtils.readInputStream(ins) }else {
                ""} } finally { ins? .close() } } fun getFilesContentFromZipFile(zipFile: ZipFile, targetFiles: List<String>): String { val stringBuilder = StringBuilder() targetFiles.filter { it.isNotEmpty() and it.isNotBlank() }.forEach { val content = getFileContentFromZipFile(zipFile, it) stringBuilder.append(content).append('\n')}return stringBuilder.toString()
    }
}

object FileUtils {

    fun readInputStream(ins: InputStream): String {
        return ins.bufferedReader().use(BufferedReader::readText)
    }
}
Copy the code

At this point, we added the function of running script package on the basis of the original function. The complete code can be seen in the repository.

6. Summary

My feelings 7.

Android running Lua script this process is actually very simple, not the main difficulty. The main stumbling block this time was in the JNI section, because I found that the C syntax I knew was too old to keep up with the current C language. Although MY C language code is not much, and I do not know much about some of the PROGRAMMING specifications of JNI, so I stumbled along the way, but finally wrote. Get familiar with Kotlin and C/C++.