FlatBuffers profile

FlatBuffers is an open-source cross-platform, efficient serialization tool library that provides a C++/Java interface. It was created by Google specifically for game development or other performance-sensitive application needs. Especially suitable for mobile, embedded platforms, where memory size and bandwidth are limited compared to desktop systems, and applications such as games have higher performance requirements. It stores serialized data in a cache that can either be stored in a file or transferred as-is over the network without any parsing overhead. The following is the project address: code hosting home page: github.com/google/flat… Introduced the home page of the project: Google. Making. IO/flatbuffers…

FlatBuffers advantage

Compared to traditional serialization tools such as JSON and Protocol Buffers, FlatBuffers have the following advantages:

  • Access to serialized data without parsing/unpacking: FlatBuffers differs from other libraries in that it uses binary buffers to represent hierarchical data so that they can be accessed directly without parsing and unpacking, while also supporting data structure evolution (forward and backward compatibility).
  • Memory efficiency Fast speed: Only the buffer in memory is required to access data. It requires no extra memory allocation (at least in C++, subject to change in other languages). FlatBuffers are also suitable for use with MMAP or data streams, requiring only a portion of the buffer to be stored in memory. The speed of access is close to that of the original structure, with only a slight delay (a virtual function table called Vtable), to allow format upgrades and optional fields. FlatBuffers are suitable for projects that take a lot of time and space (memory allocation) to access and build serialized data, such as games and other performance-sensitive applications. You can refer to the FlatBuffers benchmark.
  • Flexibility: With optional fields, not only do you have great upgrade and rollback compatibility (especially important for older games, where you don’t have to upgrade all the data for each release), but you also have the freedom to choose what data to store and design data structures.
  • Lightweight code footprint: FlatBuffers requires very little generated code and a small header file that represents minimal dependencies, making it easy to integrate.
  • Strong typing: Instead of having to write your own repetitive error-prone runtime checks when compiling errors, it automatically generates useful code.
  • Ease of use: the generated C++ code allows for streamlined access and build code, as well as optional methods for diagram parsing, json-like runtime string presentation, and more. (The latter is faster and more memory efficient than the JSON parsing library).
  • The code is cross-platform and has no dependencies: C++ code can run on any modern GCC /clang and VS2010, as well as build files (.mk files on Android, cmake files on other platforms) for testing and examples.

VS Protocol Buffers and JSON

Protocol Buffers are indeed similar to FlatBuffers, but the main difference is that FlatBuffers do not need to be parses/unpacks before accessing data, and Protocol Buffers do not have an optional text import/export function. There are also no Schemas feature (such as union).

JSON is a lightweight data interchange format that converts a set of data represented in a JavaScript object into a string that can then be easily passed between functions or from a Web client to a server-side program in an asynchronous application. JSON is handy when used with dynamically typed languages such as JavaScript. However, when serializing data in statically typed languages, JSON not only has the obvious disadvantage of being less efficient, but also makes you write more code to access the data.

How much FlatBuffers have been improved

  • Parsing speed: Parsing a 20KB JSON stream takes 35ms over the UI refresh interval of 16.6ms. If we parse JSON, we will drop frames (visually) as we slide because we have to load the cache from disk.
  • Parser initialization: A JSON parser needs to build the field map before parsing, which takes 100ms to 200ms, significantly slowing down App startup time.
  • Garbage collection creates a lot of small objects when parsing JSON, and in our experiments, parsing a 20KB JSON stream allocated about 100KB of instantaneous storage, putting a lot of pressure on Java memory collection.

FlatBuffers of actual combat

FlatBuffers operation process

First, take a look at what the FlatBuffers project has to offer developers. You can download the source code from the official website. Its directory structure is shown below:

FlatBuffers usage

Just like Parcel and Serializable serialization, FlatBuffers are more complex to use than the most traditional JSON serialization. In the actual above development, in order to reduce the difficulty of development and improve the efficiency of development, we will compile the source code into a third party library that can be implanted. The following uses the Java environment as an example to illustrate a simple use of FlatBuffers. You can download it from the maven repository.

Now, suppose we get a JSON file that looks like this:

{
  "repos": [{"id": 27149168,
      "name": "acai"."full_name": "google/acai"."owner": {
        "login": "google"."id": 1342004,..."type": "Organization"."site_admin": false
      },
      "private": false."html_url": "https://github.com/google/acai"."description": "Testing library for JUnit4 and Guice."."watchers": 21."default_branch": "master"},... ] }Copy the code

Note: You can obtain a more complete JSON object by following the link below

Schema file

We need to prepare a Model file that defines the data structure we want to serialize/deserialize, and this pattern will be used by FLATC to create the Java model and convert from JSON to FlatBuffer binaries.

Now, all we have to do is create three tables: ReposList, Repo, and User, and define root_type. Such as:

table ReposList { repos : [Repo]; } table Repo { id : long; name : string; full_name : string; owner : User; / /... labels_url : string (deprecated); releases_url : string (deprecated); } table User { login : string; id : long; avatar_url : string; gravatar_id : string; / /... site_admin : bool; } root_type ReposList;Copy the code

Note: The full schema file can be obtained by clicking on the link below

FlatBuffers file

Next, all we need to do is repos_json.json to the FlatBuffers binary and produce a Java model that represents our data in a Java-friendly way. Here’s the command for the transformation:

$ ./flatc -j -b repos_schema.fbs repos_json.json
Copy the code

If no error is reported, the following four files will be generated:

Repos_json. bin (to be renamed repos_flat.bin) Repos/ repo.java Repos/ reposlist.java Repos/ user.javaCopy the code

test

Next, we can use the Java libraries provided by FlatBuffers to process the data format directly in Java, using flatbuffers-Java-1.2.0-snapshot.jar.

public class MainActivity extends AppCompatActivity {

    @Bind(R.id.tvFlat)
    TextView tvFlat;
    @Bind(R.id.tvJson)
    TextView tvJson;

    private RawDataReader rawDataReader;

    private ReposListJson reposListJson;
    private ReposList reposListFlat;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        ButterKnife.bind(this);
        rawDataReader = new RawDataReader(this);
    }

    @OnClick(R.id.btnJson)
    public void onJsonClick() { rawDataReader.loadJsonString(R.raw.repos_json).subscribe(new SimpleObserver<String>() { @Override public void onNext(String reposStr) { parseReposListJson(reposStr); }}); } private void parseReposListJson(String reposStr) { long startTime = System.currentTimeMillis(); reposListJson = new Gson().fromJson(reposStr, ReposListJson.class);for (int i = 0; i < reposListJson.repos.size(); i++) {
            RepoJson repo = reposListJson.repos.get(i);
            Log.d("FlatBuffers"."Repo #" + i + ", id: " + repo.id);
        }
        long endTime = System.currentTimeMillis() - startTime;
        tvJson.setText("Elements: " + reposListJson.repos.size() + ": load time: " + endTime + "ms");
    }

    @OnClick(R.id.btnFlatBuffers)
    public void onFlatBuffersClick() { rawDataReader.loadBytes(R.raw.repos_flat).subscribe(new SimpleObserver<byte[]>() { @Override public void onNext(byte[] bytes) { loadFlatBuffer(bytes); }}); } private void loadFlatBuffer(byte[] bytes) { long startTime = System.currentTimeMillis(); ByteBuffer bb = ByteBuffer.wrap(bytes); reposListFlat = frogermcs.io.flatbuffs.model.flat.ReposList.getRootAsReposList(bb);for (int i = 0; i < reposListFlat.reposLength(); i++) {
            Repo repos = reposListFlat.repos(i);
            Log.d("FlatBuffers"."Repo #" + i + ", id: " + repos.id());
        }
        long endTime = System.currentTimeMillis() - startTime;
        tvFlat.setText("Elements: " + reposListFlat.reposLength() + ": load time: " + endTime + "ms"); }}Copy the code

In the example code above, there are two methods that are core and need our attention.

  • ParseReposListJson (String reposStr) : Initializes the Gson parser and converts json strings to Java objects.
  • LoadFlatBuffer (byte[] bytes) converts bytes (which is repos_flat.bin file) to Java objects.

Take the test

Let’s test the time of FlatBuffers and traditional JSON data parsing using a 4MB JSON file as an example.

FlatBuffers Time test

Reference: Using FlatBuffers FlatBuffers in Android official doc