An App generally has many scenarios to upload data generated in the App, such as APM, buried statistics, developer custom data, etc. So this article will show you how to design a common, configurable, multi-handle data reporting SDK.
Front that
Because this article and APM are a sister article, so there are some things you don’t know or curious to read this article can lead you to build a set of APM monitoring system.
In addition, I see the following code section, some naming styles, abbreviations, classification, method naming, etc., I simply do a description.
- The data report SDK calls
HermesClient, we specify that the class name is usually abbreviated by the SDK name, which in the current case is abbreviated asHCT - So let’s name a Category, the rule is
Class name + SDK prefix abbreviation lowercase format + underscore + camel name format function description. For example, add a class to NSDate that gets a millisecond timestampNSDate+HCT_TimeStamp - The rule for naming a Category method is
SDK prefix abbreviation lowercase format + underscore + camel name format function description. For example, add a method to NSDate that gets a millisecond timestamp based on the current time+ (long long)HCT_currentTimestamp;
First, define what needs to be done
What we want to do is “a general configurable, multi-handle data reporting SDK”, which means that the SDK has several functions:
- The ability to pull configuration information from the server to control SDK reporting behavior (do you need default behavior?)
- SDK has the multi-handle feature, that is, it has multiple objects, each object has its own control behavior, and the operation and operation between each other are isolated from each other
- APM monitoring exists as a very special capability that also uses data to report to the SDK. Its ability is the guarantee of App quality control, so the data reporting channel for APM needs special processing.
- Whether to save the data according to the configuration, and then how to report the data according to the configuration
To understand what we need to do, the next step is to analyze and design how to do it.
Pull configuration information
1. What configuration information is required
First of all, make clear the following principles:
- Because monitoring data reporting is a special case, monitoring configuration information should also be handled in a special way.
- Monitoring capabilities include many things, such as lag, network, crash, memory, battery, startup time, CPU usage. Each monitoring capability requires configuration information, such as monitoring type, whether to report only in Wi-Fi environment, whether to report in real time, and whether to carry Payload data. (Note: The Payload is gZip compressed, AES-CBC encrypted data)
- Multiple handles, so you need a field to identify each configuration information, which is the concept of a namespace
- Each namespace has its own configuration, For example, the server address after data upload, data report switch, whether to delete the data saved in the previous version after the App upgrade, the maximum volume of data packets uploaded at a time, the maximum number of data records, the maximum daily traffic reported in a non-Wi-Fi environment, the number of data expiration days, and the data report switch
- For APM data configuration, there is also a need to collect switch.
So the data fields are basically as follows
@interface HCTItemModel : NSObject <NSCoding> @property (nonatomic, copy) NSString *type; /< report data type */ @property (nonatomic, assign) BOOL onlyWifi; /< Whether only Wi-Fi is reported */ @property (nonatomic, assign) BOOL isRealtime; /< Whether to report real-time */ @property (nonatomic, assign) BOOL isUploadPayload; /< Payload*/ @end@interface HCTConfigurationModel: NSObject <NSCoding> @property (nonatomic, copy) NSString *url; / / @property (nonatomic, assign) BOOL isUpload; /< global report switch */ @property (nonatomic, assign) BOOL isGather; / / @property (nonatomic, assign) BOOL isUpdateClear; /< Delete data after upgrade */ @property (nonatomic, assign) NSInteger maxBodyMByte; / @property (nonatomic, assign) NSInteger periodicTimerSecond; /< Timer reporting time unit (range: 1-30 seconds)*/ @property (nonatomic, assign) NSInteger maxItem; /< maximum number of bytes (range < 100)*/ @property (nonatomic, assign) NSInteger maxFlowMByte; /< Daily maximum non-Wi-Fi upload traffic unit M (range < 100M)*/ @Property (nonatomic, assign) NSInteger expirationDay; /< data expiration in days (range < 30)*/ @property (nonatomic, copy) NSArray<HCTItemModel *> *monitorList; /< configuration item */ @endCopy the codeBecause data needs to be persisted, the NSCoding protocol needs to be implemented.
A small tip, each attribute to write encode, decode will be very troublesome, you can use macros to achieve fast writing.
#define HCT_DECODE(decoder, dataType, keyName) \ { \ _##keyName = [decoder decode##dataType##ForKey:NSStringFromSelector(@selector(keyName))]; The \}; #define HCT_ENCODE(aCoder, dataType, key) \ { \ [aCoder encode##dataType:_##key forKey:NSStringFromSelector(@selector(key))]; The \}; - (instancetype)initWithCoder:(NSCoder *)aDecoder { if (self = [super init]) { HCT_DECODE(aDecoder, Object, type) HCT_DECODE(aDecoder, Bool, onlyWifi) HCT_DECODE(aDecoder, Bool, isRealtime) HCT_DECODE(aDecoder, Bool, isUploadPayload) } return self; } - (void)encodeWithCoder:(NSCoder *)aCoder { HCT_ENCODE(aCoder, Object, type) HCT_ENCODE(aCoder, Bool, onlyWifi) HCT_ENCODE(aCoder, Bool, isRealtime) HCT_ENCODE(aCoder, Bool, isUploadPayload) }Copy the codeRaises a question: since monitoring is important, don’t configure it, just upload it all.
We consider this problem, the monitoring data are not directly upload, monitoring the SDK’s, duty is to collect monitoring data, and monitoring data very much, during the running of the App requests may be there is n times, App start time, caton, collapse, such as memory may not much, but these data directly upload later expand sex is very bad, For example, according to the APM monitoring market analysis of a monitoring capability temporarily shut down. At this point, there’s nothing to change, and you have to wait for the next SDK release. Monitoring data must be stored first. If crash occurs, the data must be saved and then assembled and uploaded for the next startup. In addition, data is being consumed and new data is constantly being produced. If the upload fails, it also needs to deal with the failed data, so there is a lot of logic, which is not suitable for monitoring the SDK to do this. The answer is obvious, it must be configured (configuration of monitoring switches, configuration of data reporting behavior).
2. Default Settings
Because monitoring is really special, the performance and quality data of the App need to be collected as soon as the App is started, so a default configuration information is required.
/ / initialize a default configuration - (void) setDefaultConfigurationModel {HCTConfigurationModel * configurationModel = [[HCTConfigurationModel alloc] init]; configurationModel.url = @"https://***DomainName.com"; configurationModel.isUpload = YES; configurationModel.isGather = YES; configurationModel.isUpdateClear = YES; configurationModel.periodicTimerSecond = 5; configurationModel.maxBodyMByte = 1; configurationModel.maxItem = 100; configurationModel.maxFlowMByte = 20; configurationModel.expirationDay = 15; / /... configurationModel.monitorList = @[appCrashItem, appLagItem, appBootItem, netItem, netErrorItem]; self.configurationModel = configurationModel; }Copy the codeThe above example is a copy of the default configuration
3. Pull strategy
Network pull uses the mGet capability of the base SDK (non-network SDK) to register network services according to the key. These keys are generally defined within the SDK, such as the unified hop routing table.
The common feature of this type of key is that the App will have a built-in default configuration in the packaging stage. After the App starts, it will pull the latest data and then complete the data cache. The cache will establish a cache folder in NSDocumentDirectory according to the SDK name, App version number, packaging task ID and key assigned on the packaging platform.
In addition, it does not request the network until the App is started to obtain data, which will not affect the startup of the App.
The flow chart is as follows
Here is a snippet of code to compare with the image above.
@synthesize configurationDictionary = _configurationDictionary;
#pragma mark - Initial Methods
+ (instancetype)sharedInstance {
static HCTConfigurationService *_sharedInstance = nil;
static dispatch_once_t onceToken;
dispatch_once(&onceToken, ^{
_sharedInstance = [[self alloc] init];
});
return _sharedInstance;
}
- (instancetype)init {
if (self = [super init]) {
[self setUp];
}
return self;
}
#pragma mark - public Method
- (void)registerAndFetchConfigurationInfo {
__weak typeof(self) weakself = self;
NSDictionary *params = @{@"deviceId": [[HermesClient sharedInstance] getCommon].SYS_DEVICE_ID};
[self.requester fetchUploadConfigurationWithParams:params success:^(NSDictionary * _Nonnull configurationDictionary) {
weakself.configurationDictionary = configurationDictionary;
[NSKeyedArchiver archiveRootObject:configurationDictionary toFile:[self savedFilePath]];
} failure:^(NSError * _Nonnull error) {
}];
}
- (HCTConfigurationModel *)getConfigurationWithNamespace:(NSString *)namespace {
if (!HCT_IS_CLASS(namespace, NSString)) {
NSAssert(HCT_IS_CLASS(namespace, NSString), @"需要根据 namespace 参数获取对应的配置信息,所以必须是 NSString 类型");
return nil;
}
if (namespace.length == 0) {
NSAssert(namespace.length > 0, @"需要根据 namespace 参数获取对应的配置信息,所以必须是非空的 NSString");
return nil;
}
id configurationData = [self.configurationDictionary objectForKey:namespace];
if (!configurationData) {
return nil;
}
if (!HCT_IS_CLASS(configurationData, NSDictionary)) {
return nil;
}
NSDictionary *configurationDictionary = (NSDictionary *)configurationData;
return [HCTConfigurationModel modelWithDictionary:configurationDictionary];
}
#pragma mark - private method
- (void)setUp {
// 创建数据保存的文件夹
[[NSFileManager defaultManager] createDirectoryAtPath:[self configurationDataFilePath] withIntermediateDirectories:YES attributes:nil error:nil];
[self setDefaultConfigurationModel];
[self getConfigurationModelFromLocal];
}
- (NSString *)savedFilePath {
return [NSString stringWithFormat:@"%@/%@", [self configurationDataFilePath], HCT_CONFIGURATION_FILEPATH];
}
// 初始化一份默认配置
- (void)setDefaultConfigurationModel {
HCTConfigurationModel *configurationModel = [[HCTConfigurationModel alloc] init];
configurationModel.url = @"https://.com";
configurationModel.isUpload = YES;
configurationModel.isGather = YES;
configurationModel.isUpdateClear = YES;
configurationModel.periodicTimerSecond = 5;
configurationModel.maxBodyMByte = 1;
configurationModel.maxItem = 100;
configurationModel.maxFlowMByte = 20;
configurationModel.expirationDay = 15;
HCTItemModel *appCrashItem = [[HCTItemModel alloc] init];
appCrashItem.type = @"appCrash";
appCrashItem.onlyWifi = NO;
appCrashItem.isRealtime = YES;
appCrashItem.isUploadPayload = YES;
HCTItemModel *appLagItem = [[HCTItemModel alloc] init];
appLagItem.type = @"appLag";
appLagItem.onlyWifi = NO;
appLagItem.isRealtime = NO;
appLagItem.isUploadPayload = NO;
HCTItemModel *appBootItem = [[HCTItemModel alloc] init];
appBootItem.type = @"appBoot";
appBootItem.onlyWifi = NO;
appBootItem.isRealtime = NO;
appBootItem.isUploadPayload = NO;
HCTItemModel *netItem = [[HCTItemModel alloc] init];
netItem.type = @"net";
netItem.onlyWifi = NO;
netItem.isRealtime = NO;
netItem.isUploadPayload = NO;
HCTItemModel *netErrorItem = [[HCTItemModel alloc] init];
netErrorItem.type = @"netError";
netErrorItem.onlyWifi = NO;
netErrorItem.isRealtime = NO;
netErrorItem.isUploadPayload = NO;
configurationModel.monitorList = @[appCrashItem, appLagItem, appBootItem, netItem, netErrorItem];
self.configurationModel = configurationModel;
}
- (void)getConfigurationModelFromLocal {
id unarchiveObject = [NSKeyedUnarchiver unarchiveObjectWithFile:[self savedFilePath]];
if (unarchiveObject) {
if (HCT_IS_CLASS(unarchiveObject, NSDictionary)) {
self.configurationDictionary = (NSDictionary *)unarchiveObject;
[self.configurationDictionary enumerateKeysAndObjectsUsingBlock:^(id _Nonnull key, id _Nonnull obj, BOOL * _Nonnull stop) {
if ([key isEqualToString:HermesNAMESPACE]) {
if (HCT_IS_CLASS(obj, NSDictionary)) {
NSDictionary *configurationDictionary = (NSDictionary *)obj;
self.configurationModel = [HCTConfigurationModel modelWithDictionary:configurationDictionary];
}
}
}];
}
}
}
#pragma mark - getters and setters
- (NSString *)configurationDataFilePath {
NSString *filePath = [NSString stringWithFormat:@"%@/%@/%@/%@", NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES).firstObject, @"hermes", [CMAppProfile sharedInstance].mAppVersion, [[HermesClient sharedInstance] getCommon].WAX_CANDLE_TASK_ID];
return filePath;
}
- (HCTRequestFactory *)requester {
if (!_requester) {
_requester = [[HCTRequestFactory alloc] init];
}
return _requester;
}
- (void)setConfigurationDictionary:(NSDictionary *)configurationDictionary
{
@synchronized (self) {
_configurationDictionary = configurationDictionary;
}
}
- (NSDictionary *)configurationDictionary
{
@synchronized (self) {
if (_configurationDictionary == nil) {
NSDictionary *hermesDictionary = [self.configurationModel getDictionary];
_configurationDictionary = @{HermesNAMESPACE: hermesDictionary};
}
return _configurationDictionary;
}
}
@end
Copy the codeData storage
1. Data storage technology selection
Remember in the data reporting technology review meeting, Android colleagues said to use WCDB, features are ORM, multi-threaded security, high performance. And then it was questioned. Since the technology used in the last version was based on the system’s built-in SQlite2, introducing an additional tripartite library just for ORM and multithreading issues was not convincing. There are several questions
-
ORM is not the core appeal. The Runtime can be modified on the basis of ORM functionality
-
Thread safety. The implementation of WCDB in thread safety is mainly based on Handle, HandlePool and Database classes. Handle is the SQlite3 pointer that HandlePool uses to Handle connections.
RecyclableHandle HandlePool::flowOut(Error &error) { m_rwlock.lockRead(a); std::shared_ptr<HandleWrap> handleWrap = m_handles.popBack(a);if (handleWrap == nullptr) { if (m_aliveHandleCount < s_maxConcurrency) { handleWrap = generate(error); if (handleWrap) { ++m_aliveHandleCount; if (m_aliveHandleCount > s_hardwareConcurrency) { WCDB::Error::Warning(("The concurrency of database:" + std::to_string(tag.load()) + " with " + std::to_string(m_aliveHandleCount) + " exceeds the concurrency of hardware:" + std::to_string(s_hardwareConcurrency)) .c_str()); }}}else { Error::ReportCore( tag.load(), path, Error::CoreOperation::FlowOut, Error::CoreCode::Exceed, "The concurrency of database exceeds the max concurrency", &error); }}if (handleWrap) { handleWrap->handle->setTag(tag.load()); if (invoke(handleWrap, error)) { return RecyclableHandle( handleWrap, [this](std::shared_ptr<HandleWrap> &handleWrap) { flowBack(handleWrap); }); } } handleWrap = nullptr; m_rwlock.unlockRead(a);return RecyclableHandle(nullptr.nullptr); } void HandlePool::flowBack(const std::shared_ptr<HandleWrap> &handleWrap) { if (handleWrap) { bool inserted = m_handles.pushBack(handleWrap); m_rwlock.unlockRead(a);if(! inserted) { --m_aliveHandleCount; }}}Copy the codeSo the WCDB connection pool keeps threads safe through read/write locks. So the previous version of the place to implement thread safety modifications under the bug can be. Sqlite3 has been added, which seems to be a few megabytes in size, but is deadly to public teams. Every time the developers of the business line access the SDK, they will notice the change of the volume of the App package. It is unacceptable to add several megabytes for data reporting.
-
WCDB’s built-in SQlite3 enables write-Ahead Logging (WAL). When WAL files exceed 1000 pages, SQLite3 writes WAL files to database files. Aka Checkpointing. When a large amount of data is written, it is inefficient to continuously commit files to database transactions. WCDB uses a delay queue to process files when checkpoint is triggered to avoid repeatedly triggering WalCheckpoint calls. Delay the WalCheckpoint merge of the same database to 2 seconds via TimedQueue
{ Database::defaultCheckpointConfigName, [](std::shared_ptr<Handle> &handle, Error &error) -> bool { handle->registerCommittedHook( [](Handle *handle, int pages, void *) { static TimedQueue<std::string> s_timedQueue(2); if (pages > 1000) { s_timedQueue.reQueue(handle->path); } static std::thread s_checkpointThread([] () {pthread_setname_np(("WCDB-" + Database::defaultCheckpointConfigName) .c_str()); while (true) { s_timedQueue.waitUntilExpired( [](const std::string &path) { Database database(path); WCDB::Error innerError; database.exec(StatementPragma().pragma( Pragma::WalCheckpoint), innerError); }); }});static std::once_flag s_flag; std::call_once(s_flag, []() { s_checkpointThread.detach(a); }); },nullptr); return true; }, (Configs::Order) Database::ConfigOrder::Checkpoint, }, Copy the code
Generally speaking, common groups do things, SDK naming, interface name, interface number, parameter number, parameter name, parameter data type are strictly the same, the difference is only language. If the capability cannot be built up, it can be inconsistent, but the reason should be explained in the technical review meeting, and it should be reflected in the release of documents and access documents.
So the final decision was to change from the previous version, which was FMDB.
2. Database maintenance queue
1. FMDB queue
– (void)inTransaction:(__attribute__((noescape)) void (^)(FMDatabase *db, BOOL *rollback) block and – (void)inDatabase:(__attribute__((noescape)) void (^)(FMDatabase *db))block. The implementation of these two methods is as follows
- (void)inDatabase:(__attribute__((noescape)) void (^)(FMDatabase *db))block { #ifndef NDEBUG /* Get the currently executing queue (which should probably be nil, but in theory could be another DB queue * and then check it against self to make sure we're not about to deadlock. */ FMDatabaseQueue *currentSyncQueue = (__bridge id)dispatch_get_specific(kDispatchQueueSpecificKey); assert(currentSyncQueue ! = self && "inDatabase: was called reentrantly on the same queue, which would lead to a deadlock"); #endif FMDBRetain(self); dispatch_sync(_queue, ^() { FMDatabase *db = [self database]; block(db); if ([db hasOpenResultSets]) { NSLog(@"Warning: there is at least one open result set around after performing [FMDatabaseQueue inDatabase:]"); #if defined(DEBUG) && DEBUG NSSet *openSetCopy = FMDBReturnAutoreleased([[db valueForKey:@"_openResultSets"] copy]); for (NSValue *rsInWrappedInATastyValueMeal in openSetCopy) { FMResultSet *rs = (FMResultSet *)[rsInWrappedInATastyValueMeal pointerValue]; NSLog(@"query: '%@'", [rs query]); } #endif } }); FMDBRelease(self); }Copy the code- (void)inTransaction:(__attribute__((noescape)) void (^)(FMDatabase *db, BOOL *rollback))block { [self beginTransaction:FMDBTransactionExclusive withBlock:block]; } - (void)beginTransaction:(FMDBTransaction)transaction withBlock:(void (^)(FMDatabase *db, BOOL *rollback))block { FMDBRetain(self); dispatch_sync(_queue, ^() { BOOL shouldRollback = NO; switch (transaction) { case FMDBTransactionExclusive: [[self database] beginTransaction]; break; case FMDBTransactionDeferred: [[self database] beginDeferredTransaction]; break; case FMDBTransactionImmediate: [[self database] beginImmediateTransaction]; break; } block([self database], &shouldRollback); if (shouldRollback) { [[self database] rollback]; } else { [[self database] commit]; }}); FMDBRelease(self); }Copy the codeThe _queue above is actually a serial queue, _queue = dispatch_queue_create([[NSString stringWithFormat:@” fMDB. %@”, self] UTF8String], NULL); To create. Therefore, the core of FMDB is to submit tasks synchronously to a serial queue to ensure read-write problems for multithreaded operations (much more efficient than per-operation locking). You can execute the next task only after one task is completed.
The previous version of data reporting SDK has a relatively simple function, that is, reporting data monitored by APM, so the amount of data is not very large. The previous human encapsulation is super simple, which only encapsulates one layer of FMDB’s add, delete, change and check operations in the form of transactions. Then there’s a problem. If the SDK is accessed by the business line, the developer of the business line does not know the internal implementation of the data reported to the SDK, and directly calls the interface to write a large amount of data. As a result, the App gets stuck, and you have to feedback that this SDK is super difficult to use.
2. Improvements for FMDB
The change method is also relatively simple, let’s first understand the reason why FMDB is designed this way. The environment of the database operation may be the main thread, the child thread, etc., to modify the data, the main thread, the child thread to read the data, so the creation of a serial queue to perform the actual data add, delete, modify and query.
The goal is to allow different threads to use FMDB without blocking the current thread. Since FMDB maintains a serial queue internally to handle multithreaded data operations, it is easier to create a concurrent queue and asynchronously submit the task to FMDB, which has a serial queue internally to perform the real task.
The following code
Self. dbOperationQueue = dispatch_queue_create(HCT_DATABASE_OPERATION_QUEUE, DISPATCH_QUEUE_CONCURRENT); self.dbQueue = [FMDatabaseQueue databaseQueueWithPath:[HCTDatabase databaseFilePath]]; // Take deleting data as an example. Submit tasks to a concurrent queue in the form of an asynchronous task. Task FMDatabaseQueue internal call to serial perform each task - (void) removeAllLogsInTableType: HCTLogTableType tableType {[self isExistInTable:tableType]; __weak typeof(self) weakself = self; dispatch_async(self.dbOperationQueue, ^{ NSString *tableName = HCTGetTableNameFromType(tableType); [weakself removeAllLogsInTable:tableName]; }); } - (void)removeAllLogsInTable:(NSString *)tableName { NSString *sqlString = [NSString stringWithFormat:@"delete from %@", tableName]; [self.dbQueue inDatabase:^(FMDatabase *_Nonnull db) { [db executeUpdate:sqlString]; }]; }Copy the code
3. Data table design
The SDK saves and reports data. From the perspective of data, data can be divided into APM monitoring data and business data of business lines.
What are the characteristics of the data? APM monitoring data can be generally divided into basic information, exception information and thread information, that is, to restore the data of the thread in question to the maximum extent. The line of business data basically does not have the so-called large number of data, at most the number of data is very large. Given this situation, data tables can be designed as meta tables and payload tables. The Meta table stores APM basic data and service line data, and the payload table stores APM thread stack data.
The design of the data sheet is based on the business situation. So there’s a couple of backgrounds
- APM monitoring data needs to be reported to the alarm (see the APM article for details, and the address is at the beginning). Therefore, data is reported to the SDK and data is analyzed in real time
- The product side such as monitoring the market can be slow, so the symbolic system is asynchronous
- Monitoring data is so large that simultaneous parsing can cause performance bottlenecks due to stress
Therefore, the monitoring data is divided into two parts, namely, the meta table and the payload table. The meta table records index information, which is all the server needs to care about. The payload data is not processed on the server, but is processed by an asynchronous service.
The structure of the META table and payload table is as follows:
create table if not exists ***_meta (id integer NOT NULL primary key autoincrement, report_id text, monitor_type text, is_biz integer NOT NULL, created_time datetime, meta text, namespace text, is_used integer NOT NULL, size integer NOT NULL);
create table if not exists ***_payload (id integer NOT NULL primary key autoincrement, report_id text, monitor_type text, is_biz integer NOT NULL, created_time datetime, meta text, payload blob, namespace text, is_used integer NOT NULL, size integer NOT NULL);
Copy the code4. Database table encapsulation
- (instancetype)init { self = [super init]; self.dateFormatter = [[NSDateFormatter alloc] init]; [self.dateFormatter setDateFormat:@"yyyy-MM-dd HH:mm:ss A Z"]; self.dbOperationQueue = dispatch_queue_create(HCT_DATABASE_OPERATION_QUEUE, DISPATCH_QUEUE_CONCURRENT); self.dbQueue = [FMDatabaseQueue databaseQueueWithPath:[HCTDatabase databaseFilePath]]; [self.dbQueue inDatabase:^(FMDatabase *_Nonnull db) { [self createLogMetaTableIfNotExist:db]; [self createLogPayloadTableIfNotExist:db]; }]; return self; } #pragma mark - public Method - (void)add:(NSArray<HCTLogModel *> *)logs inTableType:(HCTLogTableType)tableType { [self isExistInTable:tableType]; __weak typeof(self) weakself = self; dispatch_async(self.dbOperationQueue, ^{ NSString *tableName = HCTGetTableNameFromType(tableType); [weakself add:logs inTable:tableName]; }); } / /... curd - (void)rebuildDatabaseFileInTableType:(HCTLogTableType)tableType { __weak typeof(self) weakself = self; dispatch_async(self.dbOperationQueue, ^{ NSString *tableName = HCTGetTableNameFromType(tableType); [weakself rebuildDatabaseFileInTable:tableName]; }); } #pragma mark - CMDatabaseDelegate - (void)add:(NSArray<HCTLogModel *> *)logs inTable:(NSString *)tableName { if (logs.count == 0) { return; } __weak typeof(self) weakself = self; [self.dbQueue inTransaction:^(FMDatabase *_Nonnull db, BOOL *_Nonnull rollback) { [db setDateFormat:weakself.dateFormatter]; for (NSInteger index = 0; index < logs.count; Index ++) {id obj = logs[index]; } / /.. curd - (void)rebuildDatabaseFileInTable:(NSString *)tableName { NSString *sqlString = [NSString stringWithFormat:@"vacuum %@", tableName]; [self.dbQueue inDatabase:^(FMDatabase *_Nonnull db) { [db executeUpdate:sqlString]; }]; } #pragma mark - private method + (NSString *)databaseFilePath { // ... HCTLOG(@" report system database file location -> %@", dbPath); return dbPath; } - (void)createLogMetaTableIfNotExist:(FMDatabase *)db { // ... } NS_INLINE NSString *HCTGetTableNameFromType(HCTLogTableType type) { if (type == HCTLogTableTypeMeta) { return HCT_LOG_TABLE_META; } if (type == HCTLogTableTypePayload) { return HCT_LOG_TABLE_PAYLOAD; } return @""; } / /... @endCopy the codeThere is one caveat above, because the data table is often read by type and used very frequently, so it is written as an inline function
NS_INLINE NSString *HCTGetTableNameFromType(HCTLogTableType type) {
if (type == HCTLogTableTypeMeta) {
return HCT_LOG_TABLE_META;
}
if (type == HCTLogTableTypePayload) {
return HCT_LOG_TABLE_PAYLOAD;
}
return @"";
}
Copy the code5. Data storage process
APM monitoring data is special. For example, iOS cannot report a crash after a crash occurs. The only way to report a crash is to save the crash information in a file and read the crash log folder after the App starts next time. After a crash occurs, Android can immediately send the crash information to the data and report it to the SDK due to the different mechanism.
The payload, or stack data, is very large, so there are limits on the reported interfaces and the maximum size of a packet that can be uploaded to the interface at a time.
If you look at the Model information,
@interface HCTItemModel : NSObject <NSCoding> @property (nonatomic, copy) NSString *type; /**< report data type */ @property (nonatomic, assign) BOOL onlyWifi; /**< Whether to report only to Wi-Fi */ @property (nonatomic, assign) BOOL isRealtime; /**< Whether to report real-time */ @property (nonatomic, assign) BOOL isUploadPayload; /**< Payload / @end@interface HCTConfigurationModel: NSObject <NSCoding> @property (nonatomic, copy) NSString *url; /**< Namespace report address */ @property (nonatomic, assign) BOOL isUpload; /**< global report switch */ @property (nonatomic, assign) BOOL isGather; /**< global collection switch */ @property (nonatomic, assign) BOOL isUpdateClear; /**< Delete data after upgrade */ @property (nonatomic, assign) NSInteger maxBodyMByte; */ @property (nonatomic, assign) NSInteger periodicTimerSecond; /**< Timer reporting time unit (range: 1-30 seconds)*/ @property (nonatomic, assign) NSInteger maxItem; /**< maximum number of bytes (range < 100)*/ @property (nonatomic, assign) NSInteger maxFlowMByte; /**< Daily maximum non-Wi-Fi upload traffic unit M (range < 100M)*/ @property (nonatomic, assign) NSInteger expirationDay; /**< data expiration in days (range < 30)*/ @property (nonatomic, copy) NSArray<HCTItemModel *> *monitorList; /**< configuration item */ @endCopy the codeMonitoring data storage process:
-
Check whether the collection function is enabled for each data (monitoring data and service line data) in the namespace where the data resides
-
Determines whether data can be dropped from the database based on whether the type of the data interface matches the type of any item in the monitorList of the reported configuration data
-
Monitoring data is written to the META table and determines whether the data is written to the payload table. The check criterion is whether the payload of the monitoring data exceeds the maxBodyMByte that reports configuration data. Data larger than the size cannot be entered into the database, because this is the upper limit of the payload consumed by the server
-
Basic information (such as App name, App version number, packing task ID, device type, etc.) is added to the monitoring data from the monitoring interface inside the method.
@property (nonatomic, copy) NSString *xxx_APP_NAME; /**<App name (wax)*/ @property (nonatomic, copy) NSString *xxx_APP_VERSION; /**<App version (wax)*/ @property (nonatomic, copy) NSString *xxx_CANDLE_TASK_ID; / / @property (nonatomic, copy) NSString *SYS_SYSTEM_MODEL; /**< system type (Android/iOS) */ @property (nonatomic, copy) NSString *SYS_DEVICE_ID; /**< device id*/ @property (nonatomic, copy) NSString *SYS_BRAND; / / @property (nonatomic, copy) NSString *SYS_PHONE_MODEL; /**< device model */ @property (nonatomic, copy) NSString *SYS_SYSTEM_VERSION; /**< system version */ @property (nonatomic, copy) NSString *APP_PLATFORM; /**< platform */ @property (nonatomic, copy) NSString *APP_VERSION; /**<App version (business version)*/ @property (nonatomic, copy) NSString *APP_SESSION_ID; /**<session id*/ @property (nonatomic, copy) NSString *APP_PACKAGE_NAME; /**< package name */ @property (nonatomic, copy) NSString *APP_MODE; /**<Debug/Release*/ @property (nonatomic, copy) NSString *APP_UID; /**<user id*/ @property (nonatomic, copy) NSString *APP_MC; /**< channel number */ @property (nonatomic, copy) NSString *APP_MONITOR_VERSION; /**< Monitor version number. */ @property (nonatomic, copy) NSString *REPORT_ID; /**< unique ID*/ @property (nonatomic, copy) NSString *CREATE_TIME; /**< time */ @property (nonatomic, assign) BOOL IS_BIZ; /**< Whether to monitor data */Copy the code -
Because the data of crash type submitted to data and reported to SDK this time is the data of last crash, the rule mentioned in point 4 is not applicable, and APM crash type is a special case.
-
Calculate the size of each piece of data. metaSize + payloadSize
-
And write the payload
-
Check whether real-time reporting is triggered, and go to the follow-up procedure.
- (void)sendWithType:(NSString *)type meta:(NSDictionary *)meta payload:(NSData *__nullable)payload { // 1. NSString * Warning = [NSString stringWithFormat:@"%@ cannot be an empty string ", type]; if (! HCT_IS_CLASS(type, NSString)) { NSAssert1(HCT_IS_CLASS(type, NSString), warning, type); return; } if (type.length == 0) { NSAssert1(type.length > 0, warning, type); return; } if (! HCT_IS_CLASS(meta, NSDictionary)) { return; } if (meta.allKeys.count == 0) { return; } // 2. Check whether collection is enabled for the namespace. If (! self.configureModel.isGather) { HCTLOG(@"%@", [NSString stringWithFormat:@"Namespace: Self.namespace]); return ; } // 3. Check whether the data is valid. BOOL isValidate = [self validateLogData:type]; BOOL isValidate = [self validateLogData:type]; if (! isValidate) { return; HCTCommonModel *commonModel = [[HCTCommonModel alloc] init]; [self sendWithType:type namespace:HermesNAMESPACE meta:meta isBiz:NO commonModel:commonModel]; // 4. If the payload does not exist, exit if (! HCT_IS_CLASS(payload, NSData) && ! payload) { return; } / / 5. Add restrictions (more than the size of the data will not be able to put in storage, because this is the service side consume content of a cap) CGFloat payloadSize = [self calculateDataSize: payload]; if (payloadSize > self.configureModel.maxBodyMByte) { NSString *assertString = [NSString stringWithFormat:@"payload The size of the data more than the critical value % zdKB ", the self. The configureModel. MaxBodyMByte]; NSAssert(payloadSize <= self.configureModel.maxBodyMByte, assertString); return; } // 6. Merge meta and Common base data, NSMutableDictionary *metaDictionary = [NSMutableDictionary dictionary]; NSDictionary *commonDictionary = [commonModel getDictionary]; If ([type isEqualToString:@"appCrash"]) {[metaDictionary addEntriesFromDictionary:commonDictionary]; [metaDictionary addEntriesFromDictionary:meta]; } else { [metaDictionary addEntriesFromDictionary:meta]; [metaDictionary addEntriesFromDictionary:commonDictionary]; } NSError *error; NSData *metaData = [NSJSONSerialization dataWithJSONObject:metaDictionary options:0 error:&error]; if (error) { HCTLOG(@"%@", error); return; } NSString *metaContentString = [[NSString alloc] initWithData:metaData encoding:NSUTF8StringEncoding]; NSMutableData *totalData = [NSMutableData data]; [totalData appendData:metaData]; [totalData appendData:payload]; // add the payload to the payload table. [HCT_DATABASE add:@[payloadModel] inTableType:HCTLogTableTypePayload]; / / 9. Determine whether the trigger real-time report [self handleUploadDataWithtype: type]. }Copy the codeThe process of storing service line data is similar to that of storing monitoring data. The difference is that certain fields are marked to distinguish service line data.
4. Data reporting mechanism
1. Data reporting process and mechanism design
The data reporting mechanism should be designed based on the characteristics of the data, which can be divided into APM monitoring data and service line uploading data. First, analyze the characteristics of the two parts of data.
-
Service line data may be reported in real time, and you need to configure data control based on the reported data
-
During the data aggregation and reporting process, you need to configure the data control function based on the data reporting function and trigger the reporting function at intervals
-
The ability to control the reporting of all data (service data and APM monitoring data) needs to be configured
-
Because the data collected by App in a certain version may be invalid for the next version, it is necessary to have the ability to delete the data of the previous version after the reported SDK starts (if the deletion switch in the reported configuration data is turned on).
-
Also, the ability to delete expired data is required (delete data from the number of natural days ago, and also send the reported configuration item).
-
Because APM monitoring data is very large, and the data reported to SDK must be large, the design of a network communication mode will affect the quality of SDK, so traditional key/value transmission is not adopted for network performance. Customize the packet structure
-
Data reporting process can be triggered in three ways: Triggered after App startup (when APM monitors the crash, it writes the data to the local computer, and processes the data of the last crash after startup, which is a special case). Timer trigger; Data invocation Data matches the real-time reporting logic after being reported to the SDK interface
-
After data is dropped into the database, a complete reporting process is triggered
-
The first step in the reporting process is to check whether the type of the data can be reported to the configured type. If the value of the configuration item reported in real time is true, the subsequent data aggregation process is performed immediately. Otherwise interrupt (only drop library, not trigger report)
-
Because the frequency will be relatively high, so need to do throttling logic
Many people don’t know the difference between anti-shake and throttling. To put it in a nutshell: “Function stabilization focuses on events that are triggered consecutivelyfor a certain period of time and are executed only once last, whereas function throttling focuses on executing only once in a period of time.” This is not the focus of this article, those who are interested can check this article out
-
The reporting process will first determine (to save user traffic)
- If the current network is Wi-Fi, the report is reported in real time
- If the current network is unavailable, the system interrupts subsequent operations in real time
- Check whether the current network is a cellular networkWhether the usage flow exceeds the standard within 1 natural dayThe judgment of the
- If T(current timestamp) -t (timestamp saved last time) is greater than 24 hours, the used traffic is cleared and the current timestamp is recorded in the variable of the last time reported
- T(current timestamp) -t (timestamp saved last time) <= 24h, check whether the used traffic in a natural day exceeds the upper limit field in the configured traffic report. If so, exit. Otherwise, follow the procedure
-
Data aggregation is carried out in separate tables, and there are certain rules
- Obtain crash data first
- During a single network report, the total number of data items cannot be limited. The data size cannot exceed the size specified in the data configuration
-
After data is retrieved, the batch of data is marked as dirty
-
Meta table data needs to be compressed using gZip and encrypted using AES 128
-
Payload Table data Packets in a user-defined format. Format is as follows
The Header part:
Data structure (2 bytes, data type unsigned int, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned short, data type unsigned shortCopy the codeHeader + Meta data + Payload dataCopy the code -
Initiates a network request for data report
- Successful callback: remove the flag as
dirtyThe data. If it is a traffic environment, the batch data size is superimposed to a variable of the used traffic size in a natural day. - Failed callback: Update marked as
dirtyIs in normal state. If it is a traffic environment, the batch data size is superimposed to a variable of the used traffic size in a natural day.
- Successful callback: remove the flag as
The whole reporting process is as follows:
2. Potholes that have been stepped on && places that have been done well
-
Previously, the network interface was basically developed on the key/value protocol of the existing protocol. Its advantage is simple to use, but its disadvantage is that the protocol body is too large. When designing the solution, the analysis channel data reporting must be very high-frequency, so we need to design a customized packet protocol. For this part, we can refer to the TCP packet header structure.
-
The server failed to parse the data reported during the interface interconnection with the back-end. Procedure Breakpoint debugging found that the size, number, compression, and encryption of the aggregated data are normal and can be reversed after local Mock. But why can’t it be resolved to the server? After joint investigation, it was found to be a big-endian problem. A brief introduction is as follows. Please check out my article for more details on small and large enendings
Host Byte Order (HBO) : Depends on the CPU type. Big-endian: PowerPC, IBM, Sun. Little Endian: x86, DEC
Network Byte Order (NBO) : The Network Byte Order is big endian by default.
-
One step in the preceding logic is to delete the data marked as dirty after the network reports successfully. However, after a lot of data is deleted, the database file size remains the same, which theoretically needs to free up the memory data size space.
Sqlite uses various-length record storage. When data is deleted, unused disk space is added to an internal “free list” for the next insertion. This is one of the optimizations, and SQLite provides the vacuum command to release it.
This problem is similar to the meaning of file reference count in Linux, although not the same, but raised for reference. Here’s the experiment
-
Let’s take a look at the current size of each mount directory: df -h
-
First we generate a 50M file
-
Write a piece of code to read the file
#include<stdio.h> #include<unistd.h> int main(void) { FILE *fp = NULL; fp = fopen("/boot/test.txt"."rw+"); if(NULL == fp){ perror("open file failed"); return - 1; } while(1) {//do nothing sleep(1); } fclose(fp); return 0; } Copy the code -
Run the rm command to delete files in cli mode
-
Check the file size: df -h, and find that the file is deleted, but the free space in the directory is not increased
Explanation: In fact, unlink deletion is only possible if a file’s reference count is zero (including the number of hard links), as long as it is not zero, it will not be deleted. A delete is simply a deletion of the link between the file name and the inode. As long as new data is not written to the disk, the block is not deleted. Therefore, you can see that some data can be recovered even if the deletor runs out. In other words, when a program opens a file (and gets the file descriptor), its reference count is increased by 1. Rm appears to delete the file but actually just decreases the reference count by 1, but the file is not deleted because the reference count is not zero.
-
-
During data aggregation, obtain crash data first. The total number of data items and the total data size must be smaller than the limit for the reported configuration data. The processing here uses recursion, changing function parameters
- (void)assembleDataInTable:(HCTLogTableType)tableType networkType:(NetworkingManagerStatusType)networkType completion:(void (^)(NSArray<HCTLogModel *> *records))completion { // 1. To get the right type of Crash data [self fetchCrashDataByCount: self. ConfigureModel. MaxFlowMByte inTable: tableType upperBound:self.configureModel.maxBodyMByte completion:^(NSArray<HCTLogModel *> *records) { NSArray<HCTLogModel *> * crashData = records; / / 2. The data needed to compute the number and the rest of the need to the size of the data NSInteger remainingCount = self. ConfigureModel. MaxItem - CrashData. Count; float remainingSize = self. ConfigureModel. MaxBodyMByte - [self calculateDataSize: crashData]; / / 3. Access to divide For data other than Crash type, And you need to conform to the corresponding rules BOOL isWifi = (networkType = = NetworkingManagerStatusReachableViaWiFi); [the self fetchDataExceptCrash:remainingCount inTable:tableType upperBound:remainingSize isWiFI:isWifi completion:^(NSArray<HCTLogModel *> *records) { NSArray<HCTLogModel *> *dataExceptCrash = records; NSMutableArray *dataSource = [NSMutableArray array]; [dataSource addObjectsFromArray:crashData]; [dataSource addObjectsFromArray:dataExceptCrash]; if (completion) { completion([dataSource copy]); } }]; }]; } - (void)fetchDataExceptCrash:(NSInteger)count inTable:(HCTLogTableType)tableType upperBound:(float)size isWiFI:(BOOL)isWifi completion:(void (^)(NSArray<HCTLogModel *> *records))completion { // 1. __block NSMutableArray *conditions = [NSMutableArray array]; [self.configureModel.monitorList enumerateObjectsUsingBlock:^(HCTItemModel * _Nonnull obj, NSUInteger idx, BOOL * _Nonnull stop) { if (isWifi) { if (![obj.type isEqualToString:@"appCrash"]) { [conditions addObject:[NSString stringWithFormat:@"'%@'", obj.type]]; } } else { if (!obj.onlyWifi && ![obj.type isEqualToString:@"appCrash"]) { [conditions addObject:[NSString stringWithFormat:@"'%@'", HCT_SAFE_STRING(obj.type)]]; } } }]; NSString *queryCrashDataCondition = [NSString stringWithFormat:@"monitor_type in (%@) and is_used = 0 and namespace = '%@'", [conditions componentsJoinedByString:@","], self.namespace]; / / 2. According to whether there is Wifi to find the corresponding data [HCT_DATABASE getRecordsByCount: count condtion: queryCrashDataCondition inTableType: tableType Completion :^(NSArray<HCTLogModel *> *_Nonnull records) {// 3. Float dataSize = [self CalculateDataSize: records); / / 4. Greater than the maximum volume is recursive obtain maxItem - 1 the Crash data collection and data size if (size = = 0) {if (completion) { completion(records); } } else if (dataSize > size) { NSInteger currentCount = count - 1; return [self fetchDataExceptCrash:currentCount inTable:tableType upperBound:size isWiFI:isWifi completion:completion]; } else { if (completion) { completion(records); } } }]; }Copy the code -
The Unit Test pass rate of the entire SDK is 100%, and the code branch coverage is 93%. Tests are based on TDD and BDD. Test framework: XCTest, third-party OCMock, Kiwi, Expecta, Specta. The tests use the base class, and each subsequent file designs a class that inherits from the test base class.
Xcode can see test coverage for the entire SDK and for individual files
Slather can also be used. Create the.slather.yml configuration file in the project terminal environment, Slather coverage-s –scheme herm-client-example –workspace herm-client-.xcworkspace herm-client-.xcodeproj
Software testing, one of the most basic and reliable schemes for quality assurance, has some points to pay attention to at each end and also needs to be combined with engineering. I will write a special article to talk about my experience.
5. Interface design and core implementation
1. Interface design
@interface HermesClient : NSObject - (instancetype)init NS_UNAVAILABLE; + (instancetype)new NS_UNAVAILABLE; /** Instantiate a globally unique object. SetUp @return singleton object */ + (instanceType)sharedInstance; /** The current SDK is initialized. Current function: Registers the configuration delivery service. */ - (void)setup; /** data of the payload type @param type monitoring type @param meta metadata @param payload data of the payload type */ - (void)sendWithType:(NSString) *)type meta:(NSDictionary *)meta payload:(NSData *__nullable)payload; /** To report meta data, pass three parameters. Type indicates the type of data; Prefix indicates a prefix, which is reported to the background using prefix+type. Meta is dictionary type metadata @param type data type @param prefix Prefix of data type. It is usually the initials of a line of business. For example, bookkeeping: jz@param meta description metadata */ - (void)sendWithType:(NSString *)type prefix:(NSString *)prefix meta:(NSDictionary *)meta; @return Return basic information */ - (HCTCommonModel *)getCommon; @return reporting switch */ - (BOOL)isGather:(NSString *)namespace; @endCopy the codeThe HermesClient class is the entry point to the entire SDK and the interface provider. – (void)sendWithType:(NSString *)type prefix:(NSString *)prefix meta:(NSDictionary *)meta; The interface is used by the business side.
– (void)sendWithType:(NSString *)type meta:(NSDictionary *)meta payload:(NSData *__nullable)payload; Use for monitoring data.
The setup method enables processing handlers for multiple namespaces.
- (void)setup {// Obtain configuration information about monitoring and service lines. Multiple namespaces are generated parallel to and isolated from each other. [[HCTConfigurationService sharedInstance] registerAndFetchConfigurationInfo]; [self.configutations enumerateObjectsUsingBlock:^(id _Nonnull obj, NSUInteger idx, BOOL * _Nonnull stop) { HCTService *service = [[HCTService alloc] initWithNamespace:obj]; [self.services setObject:service forKey:obj]; }]; HCTService *hermesService = [self.services objectForKey:HermesNAMESPACE]; if (! hermesService) { hermesService = [[HCTService alloc] initWithNamespace:HermesNAMESPACE]; [self.services setObject:hermesService forKey:HermesNAMESPACE]; }}Copy the codeSixth, summary and thinking
1. Technology
Multithreading is powerful, but it can go wrong. Ordinary business with some simple GCD, NSOperation and so on can meet the basic needs, but do SDK is different, you need to consider various scenarios. For example, FMDB designed the FMDatabaseQueue to execute tasks synchronously in serial queues when multithreaded reads and writes. However, if the user inserts data into the database n times in the main thread, ANR will occur, so we also have to maintain a task dispatch queue to maintain the tasks submitted by the business side, which is a concurrent queue, which is submitted to FMDB as an asynchronous task and executed on a serial queue as a synchronous task.
AFNetworking 2.0 uses NSURLConnection and maintains a resident thread to handle network success callbacks. AF has one resident thread. If m of the other N SDKS have resident threads enabled, then your App will have 1+ M resident threads after integration.
AFNetworking 3.0 replaces NSURLConnection with NSURLSession and eliminates resident threads. Why did you change it? 😂 as a last chance, Apple officially issued NSURLSession, so NSURLConnection is not required and resident threads are created for it. Why doesn’t NSURLSession need resident threads? What does it do more than NSURLConnecction, but we’ll talk about that later
The process of creating a thread requires physical memory and CPU time. When you create a thread, the system allocates some memory in the process space as the thread stack. The stack size is a multiple of 4KB. On iOS the main thread stack size is 1MB and the newly created child thread stack size is 512KB. In addition, with more threads being created, the CPU will update the register when switching thread context. When updating the register, it needs to address, and the addressing process is CPU expensive. If the line is too long, there will be a lot of memory and CPU consumption, and ANR will even be killed by force.
If 🌰 is the author of FMDB and AFNetworking, the design of FMDB does not wrap ANR, and AFNetworking must use resident threads. Why? It is because multithreading is too powerful, flexible, developers too much manipulation, so FMDB design is the simplest to ensure the safety of database operation threads, specific use can maintain their own queue to package a layer. Multithreading in AFNetworking is also designed strictly based on system characteristics.
Therefore, it is necessary to take a look at multi-threading. I recommend reading the GCD source code, also known as libdispatch
2. Normative aspects
A lot of development doesn’t do testing, and our company has a strict testing protocol. Writing the basic SDK is even more so. The basic functions of an App must be of stable quality, so testing is one of the guarantee means. Be sure to write Unit Test. In this way, the version is constantly iterated. For UT, the input is constant and the output is constant. In this way, it is not necessary to care about how the internal implementation changes. (For each function on the basis of a single principle also meet UT). Another advantage is that when discussing with others, you can draw a technical flow chart, technical architecture diagram, test case, test input and output clearly, and the listener can see whether the boundary situation is fully considered. Basically, the communication is completed quickly, and the efficiency is high.
When designing the INTERFACE of SDK, the method name, parameter number, parameter type, parameter name, return value name, type and data structure should be consistent between iOS and Android as far as possible. Unless in some special cases, the consistent output cannot be guaranteed. Don’t ask why? There are so many benefits, mature SDKS do this.
For example, a data report to the SDK. I need to consider what data source is, what information the interface I design needs to expose, how data is stored efficiently, how data is verified, and how data is reported efficiently and timely. If the data reporting SDK I made can report APM monitoring data and open its ability to the business line, the business line will write and save the data they are interested in and how to report efficiently without losing it. Because the data real-time report, so need to consider the upload, wi-fi network environment is not the same as the environment and the logic of the 4 g environment, data aggregation of assembled into a custom message and report to upload data needs to be done, a natural days flow limit and so on, the App version upgrade some data might be lost meaning, the data stored there are, of course, timeliness. All of these things need to be considered before development. So the basic platform is basically design thinking time: coding time = 7:3.
Why is that? Suppose you have a requirement and expect 10 days; The preliminary architecture design, class design and Uint Test design are estimated to take 7 days, and the coding development will be completed in 2 days. There are many benefits to doing this, such as:
-
Unless it is very good, the difference will be found in the real development. The discovery of coding is different from the preliminary scheme design. Therefore, it is recommended to design a table with flow chart, UML diagram, technical architecture diagram and UT, so that coding is the work of coding when it is time to translate the diagram into code
-
There is no need to read the code line by line when discussing or communicating with others or conducting code review with CTO. Show him the relevant architecture diagrams, flow diagrams, and UML diagrams. He looks at the UT of some key logic to make sure the input and output are correct, and that’s generally enough
3. Quality assurance
UT is one aspect of quality assurance, another is the MR mechanism. Our team uses the +1 mechanism for MR. Each Merge Request must have at least 3 people on the team +1, one of whom must be a colleague in the same technology stack who is more senior than you, and one of whom must be a colleague working on the same project as you.
When someone comments or has a question, you must answer it clearly, and the change points suggested by others must be corrected or explained clearly, then you can +1. When the +1 number is greater than 3, the branch code is merged.
Joint liability system. When there is a bug in your online code, you have joint and several liability for your MR +1 colleague.
The content is too long, excerpted, please visit the original