preface

Google has been making a lot of moves in VR/AR in recent years. First, it launched Cardboard as a stepping stone to VR experience, then launched Project Tango as the cornerstone of AR experience, and then launched Mobile VR platform Daydream. Daydream2.0 was released at Google I/O this year, and recently released AR SDK — ARCore based on Android platform, which clearly aims to counter the earlier release of AR SDK — ARKit for iOS platform, which has excited numerous Android developers. The rivalry between the two companies has also spread to the two mobile development forces within the group, with the iOS team next door walking with their noses up in the air for the past few months.

Out of curiosity about VR/AR, I have studied VR and AR at Google (Google I/O ’17) by myself, and also carried out practical operation and analysis of ARCore. Combined with the insights of various professionals, there are inevitable mistakes in this paper. Welcome your guidance and exchange.

VR is different from AR

More formal explanation:

  • Virtual Reality: use computer to create a Virtual space, use Virtual Reality glasses can completely immerse users in a Virtual synthetic environment, use binocular vision principle, Virtual world in glasses is 3D, can not see the real environment.

  • Augmented Reality: The ability to integrate virtual information (objects, pictures, videos, sound, etc.) into the real environment, to enrich the real world and build a more comprehensive fantasy world.

And for small white user, the most intuitive explanation of the earth gas is

  • VR does not require a camera, only a virtual scene, not the real scene in front of you

  • AR needs to open the camera, both virtual scene, but also some real scene.

Google is putting a lot of resources into both

  • On the VR side, the battering RAMS are Cardboard, an experiential solution for entry-level VR devices, and Daydream, an inexpensive and affordable solution for entry-level VR devices. The latter is a consumer VR solution that provides Controller, a handheld remote control device that improves rendering efficiency and optimizes latency. Many app developers have produced a variety of works on Daydream platform, a large number of film and television works, live performances are produced in VR format. The recently launched Daydream 2.0 version proposes Instant Preview, which can be developed on the Unity platform to enable mobile phones and computers to link together and improve development efficiency.

    The update also offers a projection feature that shows what you can see to other friends in the room

  • In terms of AR, the main forces are Project Tango and ARCore. The former utilizes highly configured software and hardware combination scheme to provide environment awareness, location tracking and other services, which are applied to indoor positioning, 3D modeling, robotics and other fields. The products applied to mobile devices are limited.

The latter is a recently launched augmented reality solution for monocular +IMU(Inertial Measurement Unit) on mobile devices. It tries to build the AR platform with the largest number of users. It is still in the preview stage and mainly compares with Apple’s ARKit.

Brief introduction to the working principle of ARCore

There are three key technologies for ARCore to run on mobile devices:

  1. Motion Tracking allows a phone to know and track its location in the real world. Use the camera of the mobile phone to observe the feature points and IMU sensor data in the room, judge the position and direction of the device, and place virtual objects accurately. At the same time, use the Visual Positioning Service (VPS) indoor navigation to locate surrounding objects.

  2. Environmental Understanding allows phones to detect the size and location of flat surfaces like floors or desktops. Virtual objects are usually placed on flat surfaces. ARCore can use data points used in motion tracking to determine horizontal surfaces to ensure that objects can be placed normally and increase the sense of reality.

  3. Light Estimation enables phones to sense lighting conditions in real-world environments, allowing developers to illuminate virtual objects in a way that matches their surroundings. Virtual shadows are automatically adjusted in Light conditions to make virtual objects look more realistic.

ARCore to fit

Official operating equipment

ARCore is designed to support a large number of Android devices, but the device requirements are more stringent, the system requirements are not lower than Android 7.0 Nougat(API 24), support for the following devices:

  • Google Pixel and Pixel XL

  • Samsung Galaxy S8 (SM-G950U, SM-G950N, SM-G950F, SM-G950FD, SM-G950W, SM-G950U1)

In fact, the operating devices are not limited to the above types of mobile phones. I guess that In fact, Google should have developed ARCore in advance. In order to increase the range of use, it spent a lot of time on adapting other types of mobile phones. And directly claimed to be the world’s largest AR platform, not to mention Google, even those of us Android developers are not satisfied. For a variety of reasons, Google presumably released ARCore ahead of time, limiting the number of mobile devices that will work, which will meet stringent testing requirements, and other devices that only support partial functionality and have unknown vulnerabilities that haven’t been announced yet. However, the device restriction can be removed by modifying the code, which is described later.

The development environment

According to the current Google AR documentation, there are four supported development environments: Android Studio, Unity, Unreal, and Web. The following uses Android Studio as an example for development.

First, set up the development environment:

  • Install Android Studio 2.3 or later, running Android 7.0 Nougat(API 24) or later.

  • You need to use the ArCore-capable Android devices mentioned above.

  • You can download the ARCore SDK Preview for Android Studio and unzip it or download the ARCore project directly on Github

ARCore does not support Android Emulator and other virtual devices. In addition to enabling the well-known developer mode, arcore-Preview. Apk should be installed in advance to provide basic ARCore Service. After successful installation, Tango Core will be displayed in the system application, indicating that the installation is successful.

The structure of arcore-preview.apk can also be decomcompiled using Android Studio

According to the so package file name guess, this APK is the core of ARCore, providing jNI interface calls in use. In Android Studio, go to /samples/ javA_arcore_hello_ar and open the HelloAR sample project, relying mainly on the files arcore_client.aar and obj-0.2.1.jar

Compile the project and run it on the device. Open the camera permission and move the shooting position, many stars will appear. After calculation, the plane position will be identified.

Other devices that can use ARCore

When you run This project on an ordinary mobile phone, the prompt “This device does not support AR” will pop up directly, and then it will flash back, and you can’t even see what the app looks like. The officially recommended phones supporting ARCore are high-end phones costing more than 5000 yuan. Github’s arcore-for-all is an easy-to-use way to modify dependency files. It doesn’t solve the problem at all, but it’s still a good way to bypass the phone type judgment step.

The parsing process

  1. Unzip files: Aar (arcore_client.aar); / / libraries (arcore_android_sdK-master); / / Libraries (arcore_client.aar); / / Libraries (arcore_android_sdK-master); / / Libraries (arcore_client.aar); / / Libraries (arcore_client.aar) To com/Google/atap/tangoservice/SupportedDevices class file modification.

  2. Decompiler files: To convert the. Class file to a. Java file, I use the CFR tool to decomcompile the file, and run the java-jar/XXXXXX /cfr.jar supportedDevices. class > supporteddevices. Java command line interface

  3. Modify file: the file name intuitively tells us that this file mainly performs the judgment of the runnable device, the method inside is also very concise: Mainly in the isSupported () in the judgment, and one of the judgment condition deviceCalibrationAvailable () is mainly to Build. The FINGERPRINT (of all kinds of information of equipment patchwork) match the content, so you can directly across the board, Comment out return false in the last judgment branch of isSupported() as follows:

    4. public static boolean isSupported(Context context) { try { ApplicationInfo applicationInfo = context.getPackageManager().getApplicationInfo(context.getPackageName(), 128); int n = applicationInfo.metaData.getInt("com.google.ar.version.major"); int n2 = applicationInfo.metaData.getInt("com.google.ar.version.minor"); int n3 = 0; int n4 = 0; if (n > n3) { Log.e((String)TAG, (String)"Major version of AR Core is too low."); return false; } if (n == n3 && n2 > n4) { Log.e((String)TAG, (String)"Minor version of AR Core is too low."); return false; } } catch (PackageManager.NameNotFoundException nameNotFoundException) { Log.e((String)TAG, (String)"Unexpected error: Packagename of app doing isSupported check should exist."); } if (! SupportedDevices.deviceCalibrationAvailable()) { Log.e((String)TAG, (String)"Device calibration unavailable. "); return false; } return true; } private static boolean deviceCalibrationAvailable() { return Build.FINGERPRINT.contains("sailfish:7") || Build.FINGERPRINT.contains("sailfish:O") || Build.FINGERPRINT.contains("sailfish:8") || Build.FINGERPRINT.contains("marlin:7") || Build.FINGERPRINT.contains("marlin:O") || Build.FINGERPRINT.contains("marlin:8") || Build.FINGERPRINT.contains("walleye:O") || Build.FINGERPRINT.contains("walleye:8") || Build.FINGERPRINT.contains("taimen:O") || Build.FINGERPRINT.contains("taimen:8") || Build.FINGERPRINT.contains("SC-02J/SC-02J:7") || Build.FINGERPRINT.contains("SCV36_jp_kdi/SCV36:7") || Build.FINGERPRINT.contains("dreamqlteue/dreamqlteue:7") || Build.FINGERPRINT.contains("dreamqltesq/dreamqltesq:7") || Build.FINGERPRINT.contains("dreamqlteldusq/dreamqltesq:7") ||  Build.FINGERPRINT.contains("dreamqltezm/dreamqltecmcc:7") || Build.FINGERPRINT.contains("dreamqltevl/dreamqltecan:7") || SupportedDevices.isSupportedExynosDevice(); } public static boolean isSupportedExynosDevice() { return Build.FINGERPRINT.contains("dreamlteldu/dreamlte:7") || Build.FINGERPRINT.contains("dreamlteks/dreamlteks:7") || Build.FINGERPRINT.contains("dreamltexx/dreamlte:7"); }Copy the code

  4. Compiled file: Compile Java files using the Android. jar file in the Android SDK. In the current supporteddevices. Java file directory, run the javac -cp/XXXXX /android.jar -source 1.8-target 1.8 supporteddevices. Java command. Here 1.8 is my local version of Java 8, which varies from person to person.

  5. Package files: File supportedDevices. class into the jar file, run jar CVF classes.jar -c classes. Then run jar CVF arcore_client.aar -c aar_client in the arcore_client.aar directory.

  6. Generated apk: Build -> Clean Project in Android Studio (the new dependency package may not work), and then run it on other devices. After testing on some phones, the Nexus 6P will also work as shown above. The logcat will also show 09-10 19:59:50.478 866-866/? E/SupportedDevices: Device calibration unavailable., enter the decision branch just modified, but compared with Pixel and S8, the recognition speed is slower. On most phones, though they can be found in the app, they just say “Searching for surfaces…” at the bottom. The main display area is also black, and the main functions of ARCore are not available. The Github project also shows a test list of ARCore supported phones, but the only other phones currently supported are the Nexus 6P and OnePlus 5.

ARCore VS Tango

The three core technologies of ARCore mentioned above are all involved in Tango, and the files in ARCore SDK and SO package are named with the word Tango, so it is not difficult to guess that ARCore is actually a functional transformation based on the development of Tango. Tango has to meet certain hardware and software standards to make it work. Here’s some of the components needed for a Tango phone

Tango has a dedicated infrared depth-sensing camera that can create rich 3D textures and directly measure distance. ARCore can be understood as simply detecting flat surfaces and placing virtual objects on them, and it doesn’t just estimate the camera feedback. But demanding of equipment, a large number of OEM producers don’t cooperation and users demand factors such as strong enough makes Tango has been tepid, and will be a key part of the Tango, removed the depth camera features, making it suitable for mobile phones, ARCore is in order to solve these pain points and exist, don’t need special hardware strictly, Mainly from the software side to solve the problem. But it’s still a preview release, and it’s unclear whether Tango will actually achieve what it hasn’t.

ARCore VS ARKit

Let’s start with the basics of ARKit. ARKit is an augmented reality framework for iOS that aims to accurately and realistically immerse virtual content into real-world scenes. ARKit framework provides two AR technologies, one is augmented reality based on 3D scenes (SceneKit) and the other is augmented reality based on 2D scenes (SpriktKit). At its core, ARKit provides support for some basic key functions, including motion tracking, horizontal detection, and ambient light prediction.

  • Motion tracking: Mobile devices offer dedicated motion coprocessors. The function uses VIO(Visual Inertial Odometry) to estimate the 3d positioning and movement of the device based on the data captured by the camera.

  • Horizontal plane detection: the horizontal plane can be identified and the virtual object can be accurately placed in the real physical scene.

  • Ambient light prediction: generate real shadows on virtual objects to match the lighting effect.

The figure below is the comparison of the official demo examples under the two frames. IPhone 7 is on the left and Samsung S8 is on the right. Planes are identified and objects are added

Similarities and differences

  • Light awareness: Both ARKit and ARCore can make simple estimates of ambient light. ARKit provides intensity and color temperature options for developers, while ARCore provides single-pixel intensity values (Android Studio API) and Shader (Unity API).

  • Image construction and positioning: ARCore and ARKit both use discrete point cloud images. The tracker will judge whether there is a preloaded image in advance. If not, it will customize a new model and obtain a 3D range from the camera’s perspective. When you move your device, the camera captures a new image and loads it into the previously created 3D model, adding content. When ARKit builds a graph, it uses “sliding window”, which only saves the latest time and distance data in the graph. The old data will be ignored automatically, while ARCore manages and maintains larger data and maps, and the saved content will be more durable and stable.

  • Market layout: ARKit only supports devices with A9 and above processors, corresponding to iPhone 6s and later models. The high degree of uniformity of mobile phone systems also lays a huge foundation for Apple’s devices. Currently, ARCore preview only applies to a small number of models and systems, and OEM manufacturers operate separately, so it is still lagging behind in popularity in the short term

  • Product Development: ARKit was released a few months in advance, and there are already several useful apps that show the imagination of developers. ARCore, which was released recently, still requires imagination, but Tango has accumulated development tools and technical support over the past 2-3 years. These should provide a good transition and help for developers who have been working on AR for a long time.

In comparison, the core functions of ARCore and ARKit are similar. In demo, THE tracking performance of ARCore on supported devices is very close to the recognition ability of ARKit. ARCore has some advantages in map construction and relocation, and ARKit has certain technical advantages in integration and tracking. The choice depends on the developer’s personal preference.

conclusion

Google and Apple have finally deployed system-level solutions in the AR field and launched actual framework products that can be implemented on mobile devices. Their strong competitive momentum is a blessing for many DEVELOPERS in the AR field, while it may be a heavy blow for third-party independent AR companies. After all, the two giants have a monopoly on mobile hardware and software from OEM manufacturers, and the pressure to survive in the gap can be imagined. ARCore and ARKit love each other, not only improve the experience of AR applications, but also lower the threshold of development, is the spring of AR really coming, let’s wait and see…

The resources

  1. Mp.weixin.qq.com/s/BwjyJwUJK…

  2. https://github.com/tomthecarrot/arcore-for-all

  3. https://www.leiphone.com/news/201708/2WE6jIhArekIryfl.html

  4. http://www.infoq.com/cn/news/2017/09/ARCore-moblie-AR

  5. https://www.leiphone.com/news/201709/hioiZlgdXIj9fFex.html

  6. http://www.infoq.com/cn/news/2017/06/ios-augmented-reality-arkit