Author: Sun Quan

preface

As we all know, the so-called network framework is the encapsulation of a group of network communication capabilities, which provides the technical basis for APP data transmission, and its importance is self-evident. The network framework used by Snowball has not been updated for many years, and there are many problems, such as:

  • Complex access: The business layer needs to write a lot of redundant code

  • Poor performance: Not efficient in areas such as network latency and transmission speed

  • Poor architecture: There are compatibility issues with scenarios such as common parameter extension, common error handling, and MVVM page architecture

  • Weak functions: For example, upload and download operations cannot be supported

  • The development community is not active: according to industry research, it is not the mainstream solution in the industry

  • Fragmentation of invocation methods: Snowball APP integrates many sub-projects such as community, fund and securities in a component-based manner, but the invocation methods are inconsistent, which is extremely unfavorable to the unification of code specifications and affects the development efficiency

Therefore, it is extremely urgent to encapsulate a set of network framework with simple invocation, reasonable architecture, perfect functions and high availability combined with current business requirements, and at the same time smooth migration to the current project.

The following is a comparison diagram to illustrate the goals of this transformation:

To avoid reinventing the wheel, we used OkHttp and Retrofit as the underlying support for the web framework. Every time you approach a technology stack, there are three questions that come to mind:

  • What is it?

  • Why do you use it?

  • How to use it?

Connecting these three questions together with the business pain points mentioned earlier shows why we use it and how we can use it to transform our network framework.

What is it?

OkHttp

It is an HTTP client focused on connection efficiency. OkHttp provides support for HTTP/2 and SPDY, as well as connection pooling, GZIP compression, and HTTP response caching.

Retrofit

It is an open source effort from SquareUp and has been widely welcomed for its easy interface configuration, powerful extension support, and elegant code structure. Retrofit is an encapsulation of a RESTful style HTTP Web request framework. It is not described here as a Web request framework, mainly because the web request work is not done by Retrofit. OkHttp was built into Retrofit2.0, with the former focusing on encapsulation of interfaces and the latter on efficient network requests. OkHttp is an engine, Retrofit is a car shell and parts that not only make the best of the engine, but also make it easy for us to change the car into something we love.

Why do you use it?

There are many open source network base libraries on the market, and I focused on volley, xUtils, Android-Async-HTTP, Afinal, Retrofit and other network frameworks. When making a choice, I mainly consider the following three aspects:

  • Architectural rationality

  • High availability

  • Yes No Maintenance and update

Architectural rationality

The so-called network architecture that normally meets our development needs actually contains two parts: one is the part that helps us splice request packets, initiate request, receive server response and preprocess response packets based on HTTP protocol; And the other part is secondary packaging so that we can use it more flexibly and efficiently. Combined with the RXJava-based MVVM page architecture the team is currently using, OkHttp and Retrofit fit fit these two roles in a way that no other web framework can.

High availability

In terms of high availability, Retrofit is based on OkHttp and inherits Okio, connection pool reuse, data compression and other good features from OkHttp. Very fast compared to other network libraries. Here are the performance comparisons of several network libraries:

It is not difficult to see that OkHttp+Retrofit is far better than Volley and Android-Async-HTTP in the speed performance of establishing 1 connection, 7 connections and 25 connections.

Yes No Maintenance and update

Through investigation, volley, xUtils, Afinal are no longer updated, android-async-HTTP because Android6.0 no longer uses apache-HTTP package, is now also in the abandoned state. Retrofit and OkHttp are still being maintained and have active communities, making them mainstream solutions in the industry.

summary

There is more to why than the above points. For example, file upload: Volley framework is not suitable for file upload, the client wrote a lot of upload related code; For example, when multiple requests need to be chained, the code can be cumbersome. So the advent of OkHttp and Retrofit solves many of the practical problems encountered in development, which is why OkHttp and Retrofit were chosen.

How to use it?

When we understand what it is and convince ourselves to use it, then how does it work? How can we build a network framework suitable for our existing business based on it?

There are many online examples of basic Retrofit uses, but I won’t go through them all here, but I’ll highlight the following:

  • What are the problems encountered during the transformation? How is it solved?

  • How was the web framework encapsulated based on OkHttp and Retrofit in the project?

  • How to migrate the interface of network request?

What are the problems?

In the process of network Retrofit based on OkHttp and Retrofit, various potholes were encountered in order to be compatible with certain business scenarios. Here are the solutions to the following problems:

  • Domain switching

  • Custom annotations

  • Data validation

Domain switching

As we all know, improving the quality of network access is a requirement of almost every mobile project. Many projects introduce HTTP DNS as one of the most fundamental and important optimizations for network access. The core of HTTP DNS is to deliver the optimal IP address corresponding to a domain name in the background. The basic point can be connected to the nearest place, or even deliver the optimal IP address according to the actual speed measurement data of online users. During HTTP access, the client must directly replace the HOST in the URL with the IP address delivered from the background. Advantages of direct IP connection over domain name access are as follows:

  • The DNS resolution step is omitted to reduce time consumption

  • Access nearby or even fast access to reduce time consumption

  • Avoiding DNS hijacking

  • When a terminal has multiple IP access options, the terminal has certain DISASTER recovery capability

Snowball’s domain name is dynamic IP, and IPManager is responsible for unified management of the domain name list, completely independent of the network framework. Using Interceptor to intercept urls and change IP is a better architectural and performance choice than creating a Retrofit and setting BaseUrl for each request. Part of the implementation code snippet is as follows:

fun getUrl(oldHttpUrl: HttpUrl): URL { val path: String? = oldHttpUrl.toUrl().path val host: String val protocol: String var newFullUrl: HttpUrl try {// Obtain the corresponding protocol (HTTP or HTTPS?) from IPManager. Protocol = ipManager.getInstance ().protocol // IPManager = ipManager.getInstance ().matchedDomain(path) NewFullUrl = oldhttPurl.newBuilder ().scheme(protocol) // Replace protocol.host (host) // Replace host .encodedPath(path!!) .build() // Add a public parameter val sign = generateSign(newFullURl.tourl ().toString(), getParams(newFullUrl)) newFullUrl = newFullUrl.newBuilder() .addQueryParameter( URLEncoder.encode(PARAM_KEY_TRACE_ID, UTF_8), URLEncoder.encode(mTraceIdGenerator.traceId(), UTF_8) ) ... return newFullUrl.toUrl() } catch (throwable: Throwable) { } return oldHttpUrl.toUrl() }Copy the code

Note that the requested HOST has been replaced with an IP address, causing the underlying HOST to fail to verify the certificate. OkHttp provides an interface that allows an endpoint to set a certificate HOST verification policy. You only need to disable certificate authentication when IP access is used. Some code snippets are as follows:

override fun verify(hostname: String, session: SSLSession): Boolean {// IP access does not require certificate authentication, If (pattern.ip.matcher (hostname.trim {it <= "}).find()) {return true} return HttpsURLConnection.getDefaultHostnameVerifier().verify(hostname, session) }Copy the code

Custom annotations

In some scenarios, special processing is often required for individual requests, such as personalized processing for general error codes (when a server error occurs, the general logic is a pop-up error message, but in the Snowball transaction module, users need to be directed to other pages for payment Settings). There are also interfaces that support frequent pull-down refresh, such as the homepage Timeline. In order to reduce server pressure and avoid repetition or confusion of list data, reprotection is required for these interfaces. Our interfaces are declared in THE Api. Therefore, in order to reduce the invasiveness, it is the most suitable solution to determine which interfaces need special treatment by adding custom annotations where the interfaces are defined.

To mask general error handling for example, first we need to define a DisableCommonError annotation:

/** * custom annotations, Determine whether shielding general error * / @ Retention (AnnotationRetention. RUNTIME) @ Target (AnnotationTarget. FUNCTION) the annotation of the class DisableCommonErrorCopy the code

How is this annotation used? That’s the next thing to consider. As anyone who has used Retrofit knows, the reason we were able to just define an Api without writing an implementation class to complete a network request is because Retrofit uses a dynamic proxy that loads a proxy Api implementation class at runtime through a ClassLoader. The implementation details are not discussed here, but you can explore Retrofit source code if you are interested. Based on this design idea, we modify the buildApi method in the infrastructure layer, using the idea of a dual dynamic proxy to implement annotations on the fetch method. Some code snippets are as follows:

override fun <T> buildApi(service: Class<T>): T { val Api = mRetrofit!! .create(service) return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) { java.lang.reflect.Proxy.newProxyInstance( NetManager::class.java.classLoader, arrayOf<Class<*>>( service ) ) { _, method, args -> val disableCommonErrorAnnotation = method.getDeclaredAnnotation(DisableCommonError::class.java) val needDisableCommonError = disableCommonErrorAnnotation ! = null var url: String? = null for (annotation in method.declaredAnnotations) { if (annotation is GET) { url = annotation.value } else if (annotation is POST) {url = annotation. Value}} if (needDisableCommonError && url! = null) { DisableCommonErrorSets.add(url) } if (args ! = null) { method.invoke(Api, *args) } else { method.invoke(Api) } } as T } else { Api } }Copy the code

The above code implements a dual proxy to the Api, saving urls that need to be shielded from the generic error interface while calling the request method.

Next, in the response interceptor, make a judgment:

private fun modify(throwable: Throwable, path: (String) {if DisableCommonErrorSets. The contains (path)) {return} Pools. MAIN_WORKER. Schedule {/ / some common mistakes, Can on the Application of unified processing DJApiController. GetTokenEventOutput (). The modify (throwable)}}Copy the code

Here TokenEventOutput is the global semaphore defined to handle common errors. If this URL is detected, the common error needs to be masked and no exception message is sent using this semaphore.

OkHttp interceptors and Retrofit in general are cleverly designed to add all sorts of extra operations.

Data validation

The data provided by the back-end interface to the client is returned as a STRING in Json format in most scenarios, which is parsed and displayed by the client. Therefore, the robustness of the returned Json format is critical. The following problems often occur in actual development:

  • The returned string or array is null. If a null pointer is not added, a null pointer exception may occur

  • The value used for the test is 0, and the resulting object defaults to an int, but it is possible that the real type of the field is float, so later receiving data of type float could result in parsing errors

To solve this problem, we used the addConverterFactory function provided by Retrofit, passing in the Gson object of the custom TypeAdapter.

After reading the source code of Gson, we found that the data parsing of Gson is entrusted to each TypeAdapter for processing. Typeadapters (String, int, long, double, and so on) are preloaded in the constructors of Gson and stored in factories. We can customize the TypeAdapter and put it in factories so that Gson uses the corresponding TypeAdapter when parsing Json, and the TypeAdapter we add manually will take precedence over the default TypeAdapter. Interested can look at Gson related source code, or relatively simple.

Using problem 1 as an example, we define a StringAdapter as follows:

private val StringTypeAdapter = object : TypeAdapter<String>() { @Throws(IOException::class) override fun write(out: JsonWriter, value: String) { out.value(value) } @Throws(IOException::class) override fun read(`in`: JsonReader): String? { if (`in`.peek() == JsonToken.NULL) { `in`.nextNull() return transformer? .transformString(null)? : "" } return try { val result = `in`.nextString() transformer? .transformString(result)? :result } catch (e: Exception) { transformer? .transformString(null)? : ""}}}Copy the code

This automatically returns an empty string when a null node is read. The array part is a bit more troublesome, because the Adapter that Gson parses with arrays is not rewritable, so it needs to be copied out and rewritten as a class, which will not be demonstrated here.

TypeAdapter can be registered directly through the registerTypeAdapter method of GsonBuilder. As follows:

private fun initGson(): Gson {
    val builder = GsonBuilder()
    builder.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
    ...
    builder.registerTypeAdapter(String::class.java, StringTypeAdapter)
    ...
    builder.serializeSpecialFloatingPointValues()
    return builder.create()
}

Copy the code

Finally, where Retrofit is initialized, place the Gson Settings we defined on:

Retrofit.Builder()
    .baseUrl(mBuilder.baseUrl)
    .addConverterFactory(GsonConverterFactory.create(GsonManager.getGson()))
    .addCallAdapterFactory(RxJavaCallAdapterFactory.create())
 ...

Copy the code

Retrofit provides access to custom Json parsing for the access layer through policy patterns, but it has to be said that Retrofit’s decoupling approach is very clever, and many of the routines are difficult to design without years of practical architecture experience.

summary

In addition to the above problems, in the process of network transformation also solved a series of problems such as timeout reconnection, data compression, abnormal monitoring, file download and so on. I’m not going to do it individually because I don’t have enough space. In the process of solving problems, the biggest feeling is that for any stack, there is a huge difference between knowing how to use it and actually applying it to the project. Many potholes are only encountered when they are actually used, and pothole filling itself is a process of growth and evolution.

How to encapsulate the framework?

Considering extensibility and scalability, the Snowball Network framework uses the idea of layers. The following figure shows the layers and their relationships:

As shown above, the overall bottom-up is divided into four layers:

  • Base layer: The underlying support of the network framework

  • Infrastructure layer: Encapsulates the underlying libraries and provides basic functionality while providing customizable interfaces up

  • Middle layer: Initializes network operations and defines interfaces for specified applications, and implements customized operations by invoking interfaces provided by the infrastructure layer

  • Application layer: the actual network request sent by the business side, combined with the existing page architecture, to achieve specific business logic

It should be noted that for the middle layer, there will be differences between different businesses. For example, the differences between snowball and fund businesses cause the inconsistency of the middle layer, but the call method of the upper layer needs to be unified. This design approach of unifying upper-level calls by shielding lower-level differences through mid-level adaptation greatly improves the scalability of the framework. This design is also common in other scenarios, such as WebView, where the kernel used webkit before Android4.4 and then chrome, but has no awareness of the upper level development.

Infrastructure layer

The infrastructure layer encapsulates OkHttp and Retrofit initializations and provides basic interceptors for quick client integration. At the same time, builder can provide some customizable operations to the client to decouple the process of building the network from the internal components. The general class structure is as follows:

Among them, NetManager as the appearance class of the entire network framework, responsible for network initialization, Api creation, etc. NetBuilder as a network initialization configuration to create classes, including timeout, file cache, interceptor and whitelist configuration items Settings; Interceptors, in addition to the basic class related to the request, also includes the response interceptor group, which is used to uniformly parse the response body and distribute it to each response interceptor for processing, avoiding the repeated parsing and data copy of each response interceptor, resulting in performance loss. In addition, it also includes domain names, data parsing and other infrastructure management classes, as well as some utility classes, interfaces, custom annotations, etc., which will not be described here.

Application middle layer

Its functions include initializing network operations for specific applications and defining interfaces. Customized operations can be implemented by invoking interfaces provided by the infrastructure layer. Here, snowball business is taken as an example. The structure is as follows:

ApiControler was responsible for initializing the network SDK, and realized the customized operation combined with snowball business. Interceptor, by setting public parameters interceptor, set cookies, headers and other public parameters; Dynamic domain name replacement through URL interceptor; In addition, for login status check, common error, etc., through the custom response interceptor to achieve, and encapsulated the common network exception, by defining the global semaphore to achieve unified processing; In addition, there are some network request interfaces (apis) that return the RxJava Observable working object in combination with the project’s page framework-related technology stack.

The application layer

The application layer is the interface invocation of the actual network request of the business side, and at the same time combined with the existing page architecture, the concrete business logic is realized.

Here’s how it relates to Snowball’s existing MVVV-based Onion page framework:

The page framework of Snowball Onion is shown in the figure above. For more details, you can see another article in the official account – Snowball Android Client Page Architecture Best Practices [1]. Here is a brief overview of the relationship between the layers of the framework:

  • View: Directly interacts with the user

  • ViewModel: Develop for minimal business requirements

  • Model: A place where specific business logic is implemented, layered by business type

  • Repository: Data source providing layer, including network data, local data, system services, etc

The network framework plays the role of Remote Api. We can directly provide Observable working objects for the Model layer, which greatly simplifies the development of the business layer and makes the code readable and logical layer clearer.

How to migrate?

In order to smooth the migration and eliminate the risks and hidden dangers caused by the coexistence with the old network framework as soon as possible, the following migration schemes are adopted:

  • Abstract interface layer is used to adapt the call mode of network framework

  • The new Api request is invoked using Retrofit standard invocation in conjunction with the Onion framework

The realization idea is shown in the figure:

Retrofit can define a common Api to facilitate the adaptation of the call mode. The code is as follows:

@GET("{path}") fun get( @Path("path", encoded = true) path: String, @QueryMap params: Map<String, String>? , @HeaderMap extraHeader: Map<String, String>? ) : Observable<ResponseBody> @FormUrlEncoded @POST("{path}") fun post( @Path("path", encoded = true) path: String, @FieldMap params: Map<String, String>, @HeaderMap extraHeader: Map<String, String>? ) : Observable<ResponseBody>Copy the code

conclusion

Content Review:

  • Make clear the meaning and goal of network framework transformation

  • Some thoughts on using OkHttp and Retrofit as basic libraries for web frameworks

  • Help you understand Retrofit and OkHttp better by explaining how to solve typical problems such as domain switching, custom annotations, and data validation

  • Introduces how snowball encapsulates the network framework, how the framework is layered, and the relationship among the layers

  • This section describes the migration process of the new network framework

Snowball client team, through the transformation of the network framework, has greatly improved the old network framework caused by poor performance, unreasonable architecture and a series of problems, and is powerful enough to adapt to the increasingly large project and constantly changing requirements. During the transformation process, Retrofit and OkHttp were studied. On the one hand, they praised the exquisite design implementation, and on the other hand, they also improved their abstraction ability and avoided losing themselves in the busy business layer to a great extent.

Of course, the transformation of network framework is a process of continuous evolution. “How to further improve the network success rate?” “, “How to perform weak network optimization? . These questions need continuous thinking, and we will continue to explore and optimize them in the future.

One more thing

Snowball business is developing by leaps and bounds, and the engineer team is looking forward to joining us. If you are interested in “being the premier online wealth management platform for Chinese people”, we hope you can join us. Click “Read the article” to check out the hot jobs.

Hot position: Android/iOS/FE development engineer, Java development engineer, test engineer, operation and maintenance engineer.

The resources

[1]

Snowball Android client page architecture best practices: mp.weixin.qq.com/s/FZ2CXIFtS…