An overview of the
Two years ago, wechat opened the Matrix project and provided APM implementation schemes for Android and ios. For Android, it mainly includes APK Checker, Resource Canary, Trace Canary, SQLite Lint, and IO Canary. This article mainly introduces the source code implementation of APK Checker, and the rest of the source code analysis will follow.
Code framework analysis
The overall code structure is relatively clear, mainly including three parts: ApkJob, Task, Result. ApkJob indicates the overall APK detection Task, Task indicates the subdivided detection Task, and Result indicates the Result of the detection Task. The overall process is as follows: ApkJob reads configuration information and instantiates related tasks. Output the Result to a file (default: MMTaskJsonResult) after the related Task is executed.
Task Task implementation analysis
UnZipTask
Purpose: Decompress THE Apk, parse the Class obfuscating rules and Res obfuscating rules, and output the original size of each entry in the Apk and the compressed size in the ZIP package. Some raw data is stored to prepare for subsequent tasks.
@Override
public TaskResult call(a) throws TaskExecuteException {
try {
/ / the apk
ZipFile zipFile = newZipFile(inputFile); .//Result Output objectTaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config); ./ / apk total size
((TaskJsonResult) taskResult).add("total-size", inputFile.length());
// Read the Class mapping rule and store it in the config object
readMappingTxtFile();
config.setProguardClassMap(proguardClassMap);
// Read the Res mapping rule and store it in the config object
readResMappingTxtFile();
config.setResguardMap(resguardMap);
Enumeration entries = zipFile.entries();
JsonArray jsonArray = new JsonArray();
String outEntryName = "";
while (entries.hasMoreElements()) {
ZipEntry entry = (ZipEntry) entries.nextElement();
outEntryName = writeEntry(zipFile, entry);
if(! Util.isNullOrNil(outEntryName)) { JsonObject fileItem =new JsonObject();
// Print the name and compressed size of each item in Apk
fileItem.addProperty("entry-name", outEntryName);
fileItem.addProperty("entry-size", entry.getCompressedSize());
jsonArray.add(fileItem);
//Map: uncompressed file (relative path) -> (uncompressed Size, compressed Size)
entrySizeMap.put(outEntryName, Pair.of(entry.getSize(), entry.getCompressedSize()));
//Map: Apk file name -> : decompressed file (relative path)entryNameMap.put(entry.getName(), outEntryName); }}// Store to the config object
config.setEntrySizeMap(entrySizeMap);
config.setEntryNameMap(entryNameMap);
// Output to Result
((TaskJsonResult) taskResult).add("entries", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeMapping file parsing rules for Task mapping files:
- Class Mapping
Class Mapping file snippet:
. android.arch.core.executor.ArchTaskExecutorThe $1 -> android.arch.a.a.aThe $1:
42:42:void <init>() -> <init>
45:46:void execute(java.lang.Runnable) -> execute
android.arch.core.executor.ArchTaskExecutor$2 -> android.arch.a.a.a$2:
50:50:void <init>() -> <init>
53:54:void execute(java.lang.Runnable) -> execute
android.arch.core.executor.DefaultTaskExecutor -> android.arch.a.a.b:
java.lang.Object mLock -> a
java.util.concurrent.ExecutorService mDiskIO -> b
android.os.Handler mMainHandler -> c
31:33:void <init>() -> <init>
40:41:void executeOnDiskIO(java.lang.Runnable) -> a
45:54:void postToMainThread(java.lang.Runnable) -> b
58:58:boolean isMainThread() -> b
...
Copy the code* Original class name -> obfuscated class name (top) * Original field name -> obfuscated field name (first Tab reserved) * Original function name -> Obfuscated function name (first Tab reserved)Copy the code- Res Mapping
Excerpt from res Mapping file:
res path mapping:
res/layout-v22 -> r/a
res/drawable -> r/b
res/color-night-v8 -> r/c
res/xml -> r/d
res/layout -> r/e
...
res id mapping:
com.example.app.R.attr.avatar_border_color -> com.example.app.R.attr.a
com.example.app.R.attr.actualImageScaleType -> com.example.app.R.attr.b
com.example.app.R.attr.backgroundImage -> com.example.app.R.attr.c
com.example.app.R.attr.fadeDuration -> com.example.app.R.attr.d
com.example.app.R.attr.failureImage -> com.example.app.R.attr.e
Copy the code* Original resource directory -> Confounded resource directory * Original resource name -> Confounded resource nameCopy the codeManifestAnalyzeTask
Purpose: Parse Manifest files and ARSC files
public TaskResult call(a) throws TaskExecuteException {
try {
ManifestParser manifestParser = null;
// Create the Manifest object
if(! FileUtil.isLegalFile(arscFile)) { manifestParser =new ManifestParser(inputFile);
} else {
manifestParser = new ManifestParser(inputFile, arscFile);
}
TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
if (taskResult == null) {
return null;
}
long startTime = System.currentTimeMillis();
JsonObject jsonObject = manifestParser.parse();
// Output the Manifest result
((TaskJsonResult) taskResult).add("manifest", jsonObject);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeHere’s a look at the ARSC file. Arsc files exist in binary form and store index information of resources. The basic file format is shown as follows (image source network) :
Use binary tools to view the contents of arSC files:
The details of the ARSC file format are not expanded at this time, but refer to the article. Here is a brief analysis of some of the information presented visually in the binary tool.
- ResTable_header part
- ResStringPool part
- ResTablePackage
See article for more information about ARSC file parsing
ShowFileSizeTask
Objective: Collect statistics about the files that exceed the threshold.
public TaskResult call(a) throws TaskExecuteException {...long startTime = System.currentTimeMillis();
// Get the file name recorded in UnZipTask -> (file size after compression, file size before compression) map
Map<String, Pair<Long, Long>> entrySizeMap = config.getEntrySizeMap();
if(! entrySizeMap.isEmpty()) {for (Map.Entry<String, Pair<Long, Long>> entry : entrySizeMap.entrySet()) {
final String suffix = getSuffix(entry.getKey());
Pair<Long, Long> size = entry.getValue();
// Record files that exceed the threshold
if (size.getFirst() >= downLimit * ApkConstants.K1024) {
if(filterSuffix.isEmpty() || filterSuffix.contains(suffix)) { entryList.add(Pair.of(entry.getKey(), size.getFirst())); }}}}.../ / sorting
JsonArray jsonArray = new JsonArray();
for (Pair<String, Long> sortFile : entryList) {
JsonObject fileItem = new JsonObject();
fileItem.addProperty("entry-name", sortFile.getFirst());
fileItem.addProperty("entry-size", sortFile.getSecond());
jsonArray.add(fileItem);
}
// Output to the result
((TaskJsonResult) taskResult).add("files", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeMethodCountTask
Objective: Count the number of methods defined in this DEX file and methods not defined in this DEX file.
public TaskResult call(a) throws TaskExecuteException {
try{...long startTime = System.currentTimeMillis();
JsonArray jsonArray = new JsonArray();
for (int i = 0; i < dexFileList.size(); i++) {
RandomAccessFile dexFile = dexFileList.get(i);
// Calculate method information in dex
countDex(dexFile);
// The defined method can be found in the dex
int totalInternalMethods = sumOfValue(classInternalMethod);
// Cross-dex method
int totalExternalMethods = sumOfValue(classExternalMethod);
JsonObject jsonObject = new JsonObject();
jsonObject.addProperty("dex-file", dexFileNameList.get(i));
// Aggregate by Class dimension
if (JobConstants.GROUP_CLASS.equals(group)) {
List<String> sortList = sortKeyByValue(classInternalMethod);
JsonArray classes = new JsonArray();
for (String className : sortList) {
JsonObject classObj = new JsonObject();
classObj.addProperty("name", className);
classObj.addProperty("methods", classInternalMethod.get(className));
classes.add(classObj);
}
jsonObject.add("internal-classes", classes);
// Aggregate by package dimension
} else if (JobConstants.GROUP_PACKAGE.equals(group)) {
String packageName;
for (Map.Entry<String, Integer> entry : classInternalMethod.entrySet()) {
packageName = ApkUtil.getPackageName(entry.getKey());
if(! Util.isNullOrNil(packageName)) {if(! pkgInternalRefMethod.containsKey(packageName)) { pkgInternalRefMethod.put(packageName, entry.getValue()); }else {
pkgInternalRefMethod.put(packageName, pkgInternalRefMethod.get(packageName) + entry.getValue());
}
}
}
List<String> sortList = sortKeyByValue(pkgInternalRefMethod);
JsonArray packages = new JsonArray();
for (String pkgName : sortList) {
JsonObject pkgObj = new JsonObject();
pkgObj.addProperty("name", pkgName);
pkgObj.addProperty("methods", pkgInternalRefMethod.get(pkgName));
packages.add(pkgObj);
}
jsonObject.add("internal-packages", packages);
}
jsonObject.addProperty("total-internal-classes", classInternalMethod.size());
jsonObject.addProperty("total-internal-methods", totalInternalMethods);
if (JobConstants.GROUP_CLASS.equals(group)) {
List<String> sortList = sortKeyByValue(classExternalMethod);
JsonArray classes = new JsonArray();
for (String className : sortList) {
JsonObject classObj = new JsonObject();
classObj.addProperty("name", className);
classObj.addProperty("methods", classExternalMethod.get(className));
classes.add(classObj);
}
jsonObject.add("external-classes", classes);
} else if (JobConstants.GROUP_PACKAGE.equals(group)) {
String packageName = "";
for (Map.Entry<String, Integer> entry : classExternalMethod.entrySet()) {
packageName = ApkUtil.getPackageName(entry.getKey());
if(! Util.isNullOrNil(packageName)) {if(! pkgExternalMethod.containsKey(packageName)) { pkgExternalMethod.put(packageName, entry.getValue()); }else {
pkgExternalMethod.put(packageName, pkgExternalMethod.get(packageName) + entry.getValue());
}
}
}
List<String> sortList = sortKeyByValue(pkgExternalMethod);
JsonArray packages = new JsonArray();
for (String pkgName : sortList) {
JsonObject pkgObj = new JsonObject();
pkgObj.addProperty("name", pkgName);
pkgObj.addProperty("methods", pkgExternalMethod.get(pkgName));
packages.add(pkgObj);
}
jsonObject.add("external-packages", packages);
}
jsonObject.addProperty("total-external-classes", classExternalMethod.size());
jsonObject.addProperty("total-external-methods", totalExternalMethods);
jsonArray.add(jsonObject);
}
((TaskJsonResult) taskResult).add("dex-files", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeThe focus of this code is how to statically analyze the DEX file
private void countDex(RandomAccessFile dexFile) throws IOException {
classInternalMethod.clear();
classExternalMethod.clear();
pkgInternalRefMethod.clear();
pkgExternalMethod.clear();
DexData dexData = new DexData(dexFile);
// Load dex data
dexData.load();
MethodRef[] methodRefs = dexData.getMethodRefs();
ClassRef[] externalClassRefs = dexData.getExternalReferences();
// Get confused Class maping rules
Map<String, String> proguardClassMap = config.getProguardClassMap();
String className = null;
for (ClassRef classRef : externalClassRefs) {
className = ApkUtil.getNormalClassName(classRef.getName());
if (proguardClassMap.containsKey(className)) {
// The original className before the obfuscation
className = proguardClassMap.get(className);
}
if (className.indexOf('. ') = = -1) {
continue;
}
classExternalMethod.put(className, 0);
}
for (MethodRef methodRef : methodRefs) {
className = ApkUtil.getNormalClassName(methodRef.getDeclClassName());
if (proguardClassMap.containsKey(className)) {
className = proguardClassMap.get(className);
}
if(! Util.isNullOrNil(className)) {if (className.indexOf('. ') = = -1) {
continue;
}
if (classExternalMethod.containsKey(className)) {
classExternalMethod.put(className, classExternalMethod.get(className) + 1);
} else if (classInternalMethod.containsKey(className)) {
classInternalMethod.put(className, classInternalMethod.get(className) + 1);
} else {
classInternalMethod.put(className, 1); }}}//remove 0-method referenced class
Iterator<String> iterator = classExternalMethod.keySet().iterator();
while (iterator.hasNext()) {
if (classExternalMethod.get(iterator.next()) == 0) { iterator.remove(); }}}Copy the codeBefore we understand the code above, let’s look at the dex file format. The DEX file can be divided into Header part, String index table, type index table, method prototype index table, field index table, method index table, class definition and Data Data area.
- The Header section
- String index table
- Type index table
- Method prototype index table
- Field index table
- Method index table
- The class definition
Through binary tools, roughly explained the dex file format. Back in the code, there’s a difference between classInternalMethod and classExternalMethod; A TypeId is first parsed with an internal field indicating whether the type is defined in the dex file.
/** * Holds the contents of a type_id_item. * * This is chiefly a list of indices into the string table. We need * some additional bits of data, such as whether or not the type ID * represents a class defined in this DEX, so we use an object for * each instead of a simple integer. (Could use a parallel array, but * since this is a desktop app it's not essential.) */
static class TypeIdItem {
public int descriptorIdx; // index into string_ids
public boolean internal; // defined within this DEX file?
}
Copy the codeThe internal field assignment is as follows:
/**
* Sets the "internal" flag on type IDs which are defined in the
* DEX file or within the VM (e.g. primitive classes and arrays).
*/
void markInternalClasses() {
for (int i = mClassDefs.length - 1; i >= 0; i--) {
mTypeIds[mClassDefs[i].classIdx].internal = true;
}
for (int i = 0; i < mTypeIds.length; i++) {
String className = mStrings[mTypeIds[i].descriptorIdx];
if (className.length() == 1) {
// primitive class
mTypeIds[i].internal = true;
} else if (className.charAt(0) == '[') {
mTypeIds[i].internal = true;
}
//System.out.println(i + "" +
// (mTypeIds[i].internal ? "INTERNAL" : "external") + "-"+ // mStrings[mTypeIds[i].descriptorIdx]); }}Copy the codeAll types defined in ClassDef are internal, while the converted className type of length 1 (the underlying data type) is interal, and the last array type is internal.
The specific classification rules of classInternalMethod and classExternalMethod are as follows:
private void countDex(RandomAccessFile dexFile) throws IOException {... .for (ClassRef classRef : externalClassRefs) {
className = ApkUtil.getNormalClassName(classRef.getName());
if (proguardClassMap.containsKey(className)) {
className = proguardClassMap.get(className);
}
if (className.indexOf('. ') = = -1) {
continue;
}
// Add class names whose class definitions are not in this dex file to map
classExternalMethod.put(className, 0);
}
for (MethodRef methodRef : methodRefs) {
className = ApkUtil.getNormalClassName(methodRef.getDeclClassName());
if (proguardClassMap.containsKey(className)) {
className = proguardClassMap.get(className);
}
if(! Util.isNullOrNil(className)) {if (className.indexOf('. ') = = -1) {
continue;
}
// Add different categories according to the class name
if (classExternalMethod.containsKey(className)) {
classExternalMethod.put(className, classExternalMethod.get(className) + 1);
} else if (classInternalMethod.containsKey(className)) {
classInternalMethod.put(className, classInternalMethod.get(className) + 1);
} else {
classInternalMethod.put(className, 1); }}}//remove 0-method referenced class
Iterator<String> iterator = classExternalMethod.keySet().iterator();
while (iterator.hasNext()) {
if (classExternalMethod.get(iterator.next()) == 0) { iterator.remove(); }}}Copy the codeResProguardCheckTask
Purpose: Determine whether APK performs resource obfuscation.
@Override
public TaskResult call(a) throws TaskExecuteException {
File resDir = newFile(inputFile, ApkConstants.RESOURCE_DIR_PROGUARD_NAME); .if (resDir.exists() && resDir.isDirectory()) {
Log.d(TAG, "find resource directory " + resDir.getAbsolutePath());
// the folder named r performs support obfuscation
((TaskJsonResult) taskResult).add("hasResProguard".true);
} else {
resDir = new File(inputFile, ApkConstants.RESOURCE_DIR_NAME);
if (resDir.exists() && resDir.isDirectory()) {
File[] dirs = resDir.listFiles();
boolean hasProguard = true;
for (File dir : dirs) {
// If any folder does not comply with the naming rules for resource obfuscation, resource obfuscation is not performed
if(dir.isDirectory() && ! fileNamePattern.matcher(dir.getName()).matches()) { hasProguard =false;
Log.i(TAG, "directory " + dir.getName() + " has a non-proguard name!");
break;
}
}
((TaskJsonResult) taskResult).add("hasResProguard", hasProguard); . }Copy the codeFindNonAlphaPngTask
Purpose: Detect PNG files with no transparency (should be replaced with JPG to take up less space)
private void findNonAlphaPng(File file) throws IOException {
if(file ! =null) {
if (file.isDirectory()) {
File[] files = file.listFiles();
for(File tempFile : files) { findNonAlphaPng(tempFile); }}else if(file.isFile() && file.getName().endsWith(ApkConstants.PNG_FILE_SUFFIX) && ! file.getName().endsWith(ApkConstants.NINE_PNG)) { BufferedImage bufferedImage = ImageIO.read(file);// No alpha information
if(! bufferedImage.getColorModel().hasAlpha()) { String filename = file.getAbsolutePath().substring(inputFile.getAbsolutePath().length() +1);
if (entryNameMap.containsKey(filename)) {
filename = entryNameMap.get(filename);
}
long size = file.length();
if (entrySizeMap.containsKey(filename)) {
size = entrySizeMap.get(filename).getFirst();
}
if (size >= downLimitSize * ApkConstants.K1024) {
nonAlphaPngList.add(Pair.of(filename, file.length()));
}
}
}
}
}
Copy the codeMultiLibCheckTask
Purpose: Check if multiple folders exist in the lib folder.
@Override
public TaskResult call(a) throws TaskExecuteException {
try {
TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
if (taskResult == null) {
return null;
}
long startTime = System.currentTimeMillis();
JsonArray jsonArray = new JsonArray();
if (libDir.exists() && libDir.isDirectory()) {
File[] dirs = libDir.listFiles();
for (File dir : dirs) {
if (dir.isDirectory()) {
jsonArray.add(dir.getName());
}
}
}
((TaskJsonResult) taskResult).add("lib-dirs", jsonArray);
if (jsonArray.size() > 1) {
((TaskJsonResult) taskResult).add("multi-lib".true);
} else {
((TaskJsonResult) taskResult).add("multi-lib".false);
}
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeUncompressedFileTask
Objective: To compare the size of each entry in apK compression package after compression and before compression. If the sizes are the same, the files are not compressed.
@Override
public TaskResult call(a) throws TaskExecuteException {
try{...if(! entrySizeMap.isEmpty()) {//take advantage of the result of UnzipTask.
for (Map.Entry<String, Pair<Long, Long>> entry : entrySizeMap.entrySet()) {
final String suffix = getSuffix(entry.getKey());
Pair<Long, Long> size = entry.getValue();
if (filterSuffix.isEmpty() || filterSuffix.contains(suffix)) {
if(! uncompressSizeMap.containsKey(suffix)) { uncompressSizeMap.put(suffix, size.getFirst()); }else {
uncompressSizeMap.put(suffix, uncompressSizeMap.get(suffix) + size.getFirst());
}
if(! compressSizeMap.containsKey(suffix)) { compressSizeMap.put(suffix, size.getSecond()); }else{ compressSizeMap.put(suffix, compressSizeMap.get(suffix) + size.getSecond()); }}else {
// Log.d(TAG, "file: %s, filter by suffix.", entry.getKey());}}}for (String suffix : uncompressSizeMap.keySet()) {
// Size comparison
if (uncompressSizeMap.get(suffix).equals(compressSizeMap.get(suffix))) {
JsonObject fileItem = new JsonObject();
fileItem.addProperty("suffix", suffix);
fileItem.addProperty("total-size", uncompressSizeMap.get(suffix));
jsonArray.add(fileItem);
}
}
((TaskJsonResult) taskResult).add("files", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeCountRTask
Objective: Count the number of R files.
@Override
public TaskResult call(a) throws TaskExecuteException {
try {
TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
long startTime = System.currentTimeMillis();
Map<String, String> classProguardMap = config.getProguardClassMap();
for (RandomAccessFile dexFile : dexFileList) {
DexData dexData = new DexData(dexFile);
dexData.load();
ClassRef[] defClassRefs = dexData.getInternalReferences();
for (ClassRef classRef : defClassRefs) {
String className = ApkUtil.getNormalClassName(classRef.getName());
if (classProguardMap.containsKey(className)) {
className = classProguardMap.get(className);
}
// Remove the inner class
String pureClassName = getOuterClassName(className);
// Identify R files
if (pureClassName.endsWith(".R") | |"R".equals(pureClassName)) {
if(! classesMap.containsKey(pureClassName)) { classesMap.put(pureClassName, classRef.getFieldArray().length); }else {
classesMap.put(pureClassName, classesMap.get(pureClassName) + classRef.getFieldArray().length);
}
}
}
}
JsonArray jsonArray = new JsonArray();
long totalSize = 0;
Map<String, String> proguardClassMap = config.getProguardClassMap();
for (Map.Entry<String, Integer> entry : classesMap.entrySet()) {
JsonObject jsonObject = new JsonObject();
if (proguardClassMap.containsKey(entry.getKey())) {
jsonObject.addProperty("name", proguardClassMap.get(entry.getKey()));
} else {
jsonObject.addProperty("name", entry.getKey());
}
jsonObject.addProperty("field-count", entry.getValue());
totalSize += entry.getValue();
jsonArray.add(jsonObject);
}
((TaskJsonResult) taskResult).add("R-count", jsonArray.size());
((TaskJsonResult) taskResult).add("Field-counts", totalSize);
((TaskJsonResult) taskResult).add("R-classes", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeDuplicateFileTask
Purpose: Determine whether identical files exist in APK by calculating MD5.
private void computeMD5(File file) throws NoSuchAlgorithmException, IOException {
if(file ! =null) {
if (file.isDirectory()) {
File[] files = file.listFiles();
for(File resFile : files) { computeMD5(resFile); }}else {
MessageDigest msgDigest = MessageDigest.getInstance("MD5");
BufferedInputStream inputStream = new BufferedInputStream(new FileInputStream(file));
byte[] buffer = new byte[512];
int readSize = 0;
long totalRead = 0;
while ((readSize = inputStream.read(buffer)) > 0) {
msgDigest.update(buffer, 0, readSize);
totalRead += readSize;
}
inputStream.close();
if (totalRead > 0) {
final String md5 = Util.byteArrayToHex(msgDigest.digest());
String filename = file.getAbsolutePath().substring(inputFile.getAbsolutePath().length() + 1);
if (entryNameMap.containsKey(filename)) {
filename = entryNameMap.get(filename);
}
if(! md5Map.containsKey(md5)) { md5Map.put(md5,new ArrayList<String>());
if (entrySizeMap.containsKey(filename)) {
fileSizeList.add(Pair.of(md5, entrySizeMap.get(filename).getFirst()));
} else{ fileSizeList.add(Pair.of(md5, totalRead)); }}//md5 the same file listmd5Map.get(md5).add(filename); }}}}Copy the code@Override
public TaskResult call(a) throws TaskExecuteException {... .for (Pair<String, Long> entry : fileSizeList) {
//md5 the same file
if (md5Map.get(entry.getFirst()).size() > 1) {
JsonObject jsonObject = new JsonObject();
jsonObject.addProperty("md5", entry.getFirst());
jsonObject.addProperty("size", entry.getSecond());
JsonArray jsonFiles = new JsonArray();
for (String filename : md5Map.get(entry.getFirst())) {
jsonFiles.add(filename);
}
jsonObject.add("files", jsonFiles);
jsonArray.add(jsonObject);
}
}
((TaskJsonResult) taskResult).add("files", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
} catch (Exception e) {
throw new TaskExecuteException(e.getMessage(), e);
}
return taskResult;
}
Copy the codeMultiSTLCheckTask
Objective: To determine whether SO has multiple copies of STL standard library.
@Override
public TaskResult call(a) throws TaskExecuteException {
try{...for (File libFile : libFiles) {
if (isStlLinked(libFile)) {
Log.d(TAG, "lib: %s has stl link", libFile.getName());
jsonArray.add(libFile.getName());
}
}
((TaskJsonResult) taskResult).add("stl-lib", jsonArray);
if (jsonArray.size() > 1) {
((TaskJsonResult) taskResult).add("multi-stl".true);
} else {
((TaskJsonResult) taskResult).add("multi-stl".false);
}
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeprivate boolean isStlLinked(File libFile) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder(toolnmPath, "-D"."-C", libFile.getAbsolutePath());
Process process = processBuilder.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = reader.readLine();
while(line ! =null) {
String[] columns = line.split("");
// Log.d(TAG, "%s", line);
if (columns.length >= 3 && columns[1].equals("T") && columns[2].startsWith("std::")) {
return true;
}
line = reader.readLine();
}
reader.close();
process.waitFor();
return false;
}
Copy the codeUnusedResourcesTask
Purpose: To detect unreferenced resources in code, resource files.
@Override
public TaskResult call(a) throws TaskExecuteException {
try {
TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
long startTime = System.currentTimeMillis();
readMappingTxtFile();
readResourceTxtFile();
// Add all declared resources
unusedResSet.addAll(resourceDefMap.values());
Log.d(TAG, "find resource declarations %d items.", unusedResSet.size());
// Find all the resources used in the code
decodeCode();
Log.d(TAG, "find resource references in classes: %d items.", resourceRefSet.size());
// Find the resource referenced in all resources
decodeResources();
Log.d(TAG, "find resource references %d items.", resourceRefSet.size());
// Remove the referenced resource
unusedResSet.removeAll(resourceRefSet);
Log.d(TAG, "find unused references %d items", unusedResSet.size());
JsonArray jsonArray = new JsonArray();
for (String name : unusedResSet) {
jsonArray.add(name);
}
((TaskJsonResult) taskResult).add("unused-resources", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeprivate void readMappingTxtFile(a) throws IOException {
// com.tencent.mm.R$string -> com.tencent.mm.R$l:
// int fade_in_property_anim -> aRW
if(mappingTxt ! =null) {
BufferedReader bufferedReader = new BufferedReader(new FileReader(mappingTxt));
String line = bufferedReader.readLine();
boolean readRField = false;
String beforeClass = "", afterClass = "";
try {
while(line ! =null) {
if(! line.startsWith("")) {
String[] pair = line.split("- >");
if (pair.length == 2) {
beforeClass = pair[0].trim();
afterClass = pair[1].trim();
afterClass = afterClass.substring(0, afterClass.length() - 1);
if(! Util.isNullOrNil(beforeClass) && ! Util.isNullOrNil(afterClass) && ApkUtil.isRClassName(ApkUtil.getPureClassName(beforeClass))) {// Log.d(TAG, "before:%s,after:%s", beforeClass, afterClass);
readRField = true;
} else {
readRField = false; }}else {
readRField = false; }}else {
if (readRField) {
String[] entry = line.split("- >");
if (entry.length == 2) {
String key = entry[0].trim();
String value = entry[1].trim();
if(! Util.isNullOrNil(key) && ! Util.isNullOrNil(value)) { String[] field = key.split("");
if (field.length == 2) {
// Log.d(TAG, "%s -> %s", afterClass.replace('$', '.') + "." + value, getPureClassName(beforeClass).replace('$', '.') + "." + field[1]);
// add r.java full path field after confusion -> r.java full path field before confusion
rclassProguardMap.put(afterClass.replace('$'.'. ') + "." + value, ApkUtil.getPureClassName(beforeClass).replace('$'.'. ') + "." + field[1]); } } } } } line = bufferedReader.readLine(); }}finally{ bufferedReader.close(); }}}Copy the codeprivate void readResourceTxtFile(a) throws IOException {
/ / read R.t xt
BufferedReader bufferedReader = new BufferedReader(new FileReader(resourceTxt));
String line = bufferedReader.readLine();
try {
while(line ! =null) {
String[] columns = line.split("");
if (columns.length >= 4) {
final String resourceName = "R." + columns[1] + "." + columns[2];
if(! columns[0].endsWith("[]") && columns[3].startsWith("0x")) {
//int styleable ActionBar_title 27
if (columns[3].startsWith("0x01")) {
Log.d(TAG, "ignore system resource %s", resourceName);
} else {
final String resId = parseResourceId(columns[3]);
if(! Util.isNullOrNil(resId)) {// Resource ID Resource name mappingresourceDefMap.put(resId, resourceName); }}}else {
//int[] styleable ActionMode { 0x7f030034, 0x7f030036, 0x7f030056, 0x7f0300ad, 0x7f030168, 0x7f03019e }
Log.d(TAG, "ignore resource %s", resourceName);
if (columns[0].endsWith("[]") && columns.length > 5) {
Set<String> attrReferences = new HashSet<String>();
for (int i = 4; i < columns.length; i++) {
if (columns[i].endsWith(",")) {
attrReferences.add(columns[i].substring(0, columns[i].length() - 1));
} else{ attrReferences.add(columns[i]); }}/ / style mappingstyleableMap.put(resourceName, attrReferences); } } } line = bufferedReader.readLine(); }}finally{ bufferedReader.close(); }}Copy the codeParse smali code in dex file:
private void decodeCode(a) throws IOException {
for (String dexFileName : dexFileNameList) {
DexBackedDexFile dexFile = DexFileFactory.loadDexFile(new File(inputFile, dexFileName), Opcodes.forApi(15));
BaksmaliOptions options = new BaksmaliOptions();
List<? extends ClassDef> classDefs = Ordering.natural().sortedCopy(dexFile.getClasses());
for (ClassDef classDef : classDefs) {
String[] lines = ApkUtil.disassembleClass(classDef, options);
if(lines ! =null) { readSmaliLines(lines); }}}}Copy the codeprivate void readSmaliLines(String[] lines) {
if (lines == null) {
return;
}
for (String line : lines) {
line = line.trim();
if(! Util.isNullOrNil(line)) {if (line.startsWith("const")) {
String[] columns = line.split(",");
if (columns.length == 2) {
final String resId = parseResourceId(columns[1].trim());
// Get the resource name from id
if(! Util.isNullOrNil(resId) && resourceDefMap.containsKey(resId)) { resourceRefSet.add(resourceDefMap.get(resId)); }}}else if (line.startsWith("sget")) {
String[] columns = line.split("");
if (columns.length == 3) {
// Get the resource name
final String resourceRef = parseResourceNameFromProguard(columns[2]);
if(! Util.isNullOrNil(resourceRef)) {//Log.d(TAG, "find resource reference %s", resourceRef);
if (styleableMap.containsKey(resourceRef)) {
//reference of R.styleable.XXX
for(String attr : styleableMap.get(resourceRef)) { resourceRefSet.add(resourceDefMap.get(attr)); }}else {
resourceRefSet.add(resourceRef);
}
}
}
}
}
}
}
Copy the codeprivate String parseResourceNameFromProguard(String entry) {
if(! Util.isNullOrNil(entry)) {// sget v6, Lcom/tencent/mm/R$string; ->chatting_long_click_menu_revoke_msg:I
// sget v1, Lcom/tencent/mm/libmmui/R$id; ->property_anim:I
// sput-object v0, Lcom/tencent/mm/plugin_welab_api/R$styleable; ->ActionBar:[I
// const v6, 0x7f0c0061
String[] columns = entry.split("- >");
if (columns.length == 2) {
int index = columns[1].indexOf(':');
if (index >= 0) {
final String className = ApkUtil.getNormalClassName(columns[0]);
final String fieldName = columns[1].substring(0, index);
if(! rclassProguardMap.isEmpty()) { String resource = className.replace('$'.'. ') + "." + fieldName;
if (rclassProguardMap.containsKey(resource)) {
return rclassProguardMap.get(resource);
} else {
return ""; }}else {
if (ApkUtil.isRClassName(ApkUtil.getPureClassName(className))) {
return (ApkUtil.getPureClassName(className) + "." + fieldName).replace('$'.'. ');
}
}
}
}
}
return "";
}
Copy the codeUnusedAssetsTask
Purpose: Detect asset resources in APK that are not being used (there are legacies if the code implementation only overwrites string constants).
@Override
public TaskResult call(a) throws TaskExecuteException {
try {
TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
long startTime = System.currentTimeMillis();
File assetDir = new File(inputFile, ApkConstants.ASSETS_DIR_NAME);
// Find all asset files
findAssetsFile(assetDir);
generateAssetsSet(assetDir.getAbsolutePath());
Log.d(TAG, "find all assets count: %d", assetsPathSet.size());
// Parse the asset reference in the code
decodeCode();
Log.d(TAG, "find reference assets count: %d", assetRefSet.size());
// Remove the referenced resource
assetsPathSet.removeAll(assetRefSet);
JsonArray jsonArray = new JsonArray();
for (String name : assetsPathSet) {
jsonArray.add(name);
}
((TaskJsonResult) taskResult).add("unused-assets", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeprivate void generateAssetsSet(String rootPath) {
HashSet<String> relativeAssetsSet = new HashSet<String>();
for (String path : assetsPathSet) {
int index = path.indexOf(rootPath);
if (index >= 0) {
String relativePath = path.substring(index + rootPath.length() + 1);
//Log.d(TAG, "assets %s", relativePath);
relativeAssetsSet.add(relativePath);
if (ignoreAsset(relativePath)) {
Log.d(TAG, "ignore assets %s", relativePath);
// Get the relative path in which the asset is used
assetRefSet.add(relativePath);
}
}
}
assetsPathSet.clear();
assetsPathSet.addAll(relativeAssetsSet);
}
Copy the codeprivate void readSmaliLines(String[] lines) {
if (lines == null) {
return;
}
for (String line : lines) {
line = line.trim();
// invoke-virtual {p0}, Lcom/ss/android/alog/App; ->getAssets()Landroid/content/res/AssetManager;
//move-result-object v1
//const-string v2, "video"
//invoke-virtual {v1, v2}, Landroid/content/res/AssetManager; ->open(Ljava/lang/String;) Ljava/io/InputStream;
//:try_end_13
//.catch Ljava/io/IOException; {:try_start_a .. :try_end_13} :catch_1a
// This const-string judgment is not very good, can only be written to dead values
if(! Util.isNullOrNil(line) && line.startsWith("const-string")) {
String[] columns = line.split(",");
if (columns.length == 2) {
String assetFileName = columns[1].trim();
assetFileName = assetFileName.substring(1, assetFileName.length() - 1);
if(! Util.isNullOrNil(assetFileName)) {// Check if the constant is in the asset file name collection
for (String path : assetsPathSet) {
if (path.endsWith(assetFileName)) {
assetRefSet.add(path);
}
}
}
}
}
}
}
Copy the codeUnStrippedSoCheckTask
Objective: To detect untrimmed SO in APK.
@Override
public TaskResult call(a) throws TaskExecuteException {
try{...if (libDir.exists() && libDir.isDirectory()) {
File[] dirs = libDir.listFiles();
for (File dir : dirs) {
if (dir.isDirectory()) {
File[] libs = dir.listFiles();
for (File libFile : libs) {
if (libFile.isFile() && libFile.getName().endsWith(ApkConstants.DYNAMIC_LIB_FILE_SUFFIX)) {
libFiles.add(libFile);
}
}
}
}
}
for (File libFile : libFiles) {
// Determine whether to crop
if(! isSoStripped(libFile)) { Log.d(TAG,"lib: %s is not stripped", libFile.getName());
jsonArray.add(libFile.getName());
}
}
((TaskJsonResult) taskResult).add("unstripped-lib", jsonArray);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeDetermine whether so is clipped from the command line
private boolean isSoStripped(File libFile) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder(toolnmPath, libFile.getAbsolutePath());
Process process = processBuilder.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = reader.readLine();
if(! Util.isNullOrNil(line)) {//Log.d(TAG, "%s", line);
String[] columns = line.split(":");
if (columns.length == 3 && columns[2].trim().equalsIgnoreCase("no symbols")) {
return true;
}
}
reader.close();
process.waitFor();
return false;
}
Copy the codeCountClassTask
Objective: Count the number of classes.
@Override
public TaskResult call(a) throws TaskExecuteException {
try {
TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
long startTime = System.currentTimeMillis();
Map<String, String> classProguardMap = config.getProguardClassMap();
JsonArray dexFiles = new JsonArray();
for (int i = 0; i < dexFileList.size(); i++) {
RandomAccessFile dexFile = dexFileList.get(i);
DexData dexData = new DexData(dexFile);
dexData.load();
ClassRef[] defClassRefs = dexData.getInternalReferences();
Set<String> classNameSet = new HashSet<>();
for (ClassRef classRef : defClassRefs) {
String className = ApkUtil.getNormalClassName(classRef.getName());
if (classProguardMap.containsKey(className)) {
className = classProguardMap.get(className);
}
if (className.indexOf('. ') = = -1) {
continue;
}
classNameSet.add(className);
}
JsonObject jsonObject = new JsonObject();
jsonObject.addProperty("dex-file", dexFileNameList.get(i));
//Log.d(TAG, "dex %s, classes %s", dexFileNameList.get(i), classNameSet.toString());
Map<String, Set<String>> packageClass = new HashMap<>();
if (JobConstants.GROUP_PACKAGE.equals(group)) {
String packageName = "";
for (String clazzName : classNameSet) {
packageName = ApkUtil.getPackageName(clazzName);
if(! Util.isNullOrNil(packageName)) {if(! packageClass.containsKey(packageName)) { packageClass.put(packageName,new HashSet<String>());
}
// Aggregate as package
packageClass.get(packageName).add(clazzName);
}
}
JsonArray packages = new JsonArray();
for (Map.Entry<String, Set<String>> pkg : packageClass.entrySet()) {
JsonObject pkgObj = new JsonObject();
pkgObj.addProperty("package", pkg.getKey());
JsonArray classArray = new JsonArray();
for (String clazz : pkg.getValue()) {
classArray.add(clazz);
}
// All classes in a single package
pkgObj.add("classes", classArray);
packages.add(pkgObj);
}
jsonObject.add("packages", packages);
}
dexFiles.add(jsonObject);
}
((TaskJsonResult) taskResult).add("dex-files", dexFiles);
taskResult.setStartTime(startTime);
taskResult.setEndTime(System.currentTimeMillis());
return taskResult;
} catch (Exception e) {
throw newTaskExecuteException(e.getMessage(), e); }}Copy the codeconclusion
The code logic of Matrix static APK scanning part is relatively simple. After a preliminary understanding of the DEX file format, arSC file format, code understanding should not be too much of a problem.