I haven’t updated it in a long time, because I’m trying to make a big move, yes, today’s post.

Today I would like to share with you my experience with open source GIS solutions.

Well, given that you’ll probably fall asleep quickly talking about technical solutions alone, I’m going to focus today on my experience of encapsulating map apis, interspersed with some of the open source GIS architectures used at the time.

Article a little long, if you just want to know about the latest open source GIS architecture, can jump directly, go to the fifth edition and the final summary, but I suggest you start from the first edition, because there is no front four version there will be no fifth edition, only see the summary, and read great effect, see when feel reasonable, then forget, Because you can’t empathize.

origin

I have been working in traditional IT industry companies, which are mainly engaged in government projects, commonly known as toG business.

In this company, the application of GIS in which are usually two kinds, one kind of GIS combined with the company’s business is to form a xx geographic information systems, or xx picture system, another is combined with the company’s business encapsulate a set of maps API, other business system map to provide technical support for the company, similar to gold API and baidu map API. There is usually a GIS solution behind a map API

Today we are going to talk about encapsulating the map API.

It’s been seven years, three companies, and five iterations since the first version of the map API I packaged. Among the five versions, ArcGIS is used in the first version and open source GIS is mainly used in the following four versions.

First edition 2014

background

The company is engaged in environmental protection business, and the business system is mainly developed by C#. The part of the business system involving maps is developed by ArcGIS Flex API, and GIS developers are responsible for it. At the time, maps were mostly developed in Flex because Flex felt cool at the time.

As the project grew, many business functions were found to be common in the project. Later, it was discovered that if the business functions and map functions were separated, and the Flex written map functions were packaged into a set of js-oriented interfaces, C# developers could do their own map-related work without the limitations of Flex.

So we are going to encapsulate a JS map API written in Felx for C# developers.

The technical architecture

  1. The front desk uses ArcGIS Flex API to develop map functions, Flex supports interaction with JS, and uses this feature to package the map functions developed by Flex into JS-oriented interfaces.
  2. ArcGIS Server is used for background map and GP service is used for spatial analysis.
  3. The spatial database uses Arc SDE.
  4. Developed a simple help site, provides the foreground JS interface call examples and use instructions.

experience

After the release of map API, I found an interesting phenomenon in the process of technical support. As for the use of map API, junior C# developers who don’t know GIS at all think it is easy to use. The reason is that it can help them solve problems.

Intermediate C# developers who know a little bit about GIS feel it is not good to use it. They would compare our map API with ArcGIS JS API and think the latter is better to use, but ArcGIS JS API’s map is ugly and we do not provide technical support, so they need to study it by themselves, and finally choose our map API.

Senior C# developers who know a little bit about GIS rarely use it, and I was impressed by the reaction of two colleagues who said, “you don’t use what you developed yourself.” The implication is that what you don’t use won’t work. But our idea is to package Flex as a JS map interface because Flex has a threshold to get started. Since we GIS developers know Flex, and the whole geographic information system we develop is developed in Flex, it must be more flexible to use ArcGIS Flex API directly, so it is not necessary.

Another colleague is even better. He went to study Flex directly and asked us if he could not. After getting started, he directly encapsulated a set of map interface for his own use. We have studied the interface encapsulated by him. Although the function is simpler, the starting point of defining the interface is obviously different from ours. We encapsulate the interface from the perspective of functions and try our best to ensure high reuse of the interface, such as adding points, adding lines and buffering. He encapsulates it from the user’s point of view, for example, retrieving a bunch of data from a database, throwing it directly to the interface, and displaying it on a map. In general, his interface has a higher encapsulation degree and is closer to his actual usage habits, while ours is more like translating the Flex interface of ArcGIS into a set of JS.

I also noticed that business developers using our interface often came up to me and asked, why isn’t my map showing up? After much experience, I found that there are usually two reasons. One is that the address of map service is wrong. At that time, map service was issued by ArcGIS Server, and the REST address of ArcGIS Server map service is a web page. Business people do not know which level of address should be copied, often copy wrong, so the map can not be out. Another is that when multiple maps are added, the coordinates between the maps are inconsistent, resulting in the added map is not displayed.

When you think about it, knowledge of map services and map coordinates can be difficult for developers who don’t understand GIS.

Although the first version had some problems, it still improved the overall work efficiency of the department at that time. It’s not just C# developers who can develop maps on their own, it’s also a way for the GIS team to collect and refine the technical work that has been done on various projects.

Second edition 2016

background

The new company does network security business and has massive positioning data. The role of GIS is to extract, analyze and display positioning data, so as to help customers solve business problems.

All systems of the company must be deployed on the customer’s Intranet. The customer’s Intranet cannot access the Internet, and the map uses the Internet map. Therefore, the Internet map tiles need to be downloaded and copied to the customer’s Intranet for publication.

The company has a GIS application system where the business functions strongly associated with GIS can be located. In addition, when other departments have requirements for map functions, they will also find our GIS department, which is very similar to the previous company.

So, without further ado, the second map API is up and running.

The technical architecture

  1. This release starts with open source GIS. The GIS front desk chose OpenLayers, and I had the experience of the first version, which focused on solving the problem of map resources and map coordinates.
    1. Resolve map resource issues. Unified management of base map resources, providing a variety of Internet maps, including Amap, Google Map, Map of heaven and Earth, each map has an ID, the initialization of the map, according to the ID of the use of different base map.
    2. Solve map coordinates. Wgs84 coordinates are uniformly used externally, and the map API is responsible for converting THE COORDINATES of WGS84 into coordinates matching the base map, including COORDINATES of GCJ02 and BD09, etc.
  2. As for the GIS background, because the positioning data are stored in the database of the big data framework, only the Internet offline map tiles need to be published in the background, so Tomcat is directly used to publish tiles, and a function of loading local tiles is written in OpenLayers, which can be considered as the GIS background.
  3. Instead of using spatial database, ArcGIS Engine is used to develop the console program for spatial analysis, and then Java background is used to call the console program.
  4. Interfaces are written with detailed documentation and invocation examples. No packaging, just word document + HTML sample file.

experience

  1. Resolving map resources and coordinates can greatly improve the user experience.
  2. As for map downloaders, although everyone has the ability to write one themselves, it’s really not cost-effective. The best solution is to buy a license at the company’s expense, which is only a few dollars from the company’s point of view.
  3. ArcGIS Engine version, C# version is more stable, Java version is super difficult to use, very unstable, always dead to you, let alone try to deploy it to Linux server, don’t ask me how to know. In order to deploy on Linux, I first developed a version of Java, which crashed several times a day after the deployment, often running out of memory, and became unusable. I had no choice but to write a new version of C# to deploy on Windows server.

Third edition 2017

background

The new company is engaged in pipe network business. Compared with the previous two companies, the business is more closely related to GIS, and the business needs to edit, store, publish and apply the GIS data of pipe network. The company used ArcGIS to develop maps before, and was preparing to switch to open source GIS. Therefore, I started the development of the third version of map API after I joined the company.

Before development, I carefully studied autonavi and Baidu Map APIS and asked myself two questions:

I. Why can non-GIS developers use Autonavi and Baidu map API to solve problems, but not ArcGIS, OpenLayers and Leaflet?

What are the problems non-GIS developers encounter when using the Internet Mapping API?

Here’s what I understand:

The first problem is due to: 1. Non-gis developers don’t need to publish their own map data, maps are official, just use them. 2. The help documentation and examples for the Internet Map API are in Chinese.

The second problem is found after I tried to use it by myself: 1. Offline use of Internet map API is a problem. They can only be used online, and if they cannot access the Internet, they cannot be used. 2. The Internet map API can only use the map coordinates provided by the authorities, and cannot integrate map resources of other coordinates.

From this version, we began to try to learn the advantages of each other and avoid the problems of each other in terms of user experience.

Technology selection

GIS foreground did not continue to use OpenLayers, but turned to the lighter leaflet. At that time, the consideration was as follows:

  1. The Leaflet is very small, the core code structure is simple and easy to understand, and the plasticity is strong, which is suitable for transformation into their own API.
  2. Leaflets are compatible with both Web and mobile terminals.
  3. Leaflet plug-in maintained by ESRI can be better compatible with ArcGIS map service previously released by the company.

Geoserver is selected as the GIS background. Because geoServer is rich in data, it can support both postGIS and SDE spatial databases.

GIS spatial database chooses postGIS, and spatial analysis is mainly realized by postGIS spatial analysis function.

Desktop development continues to use ArcGIS Engine instead of TRYING QGIS. The main reason is that the company has accumulated a large amount of technology on ArcGIS Engine before, which has formed mature products, and the conversion cost will be very high.

The technical architecture

On the basis of Leaflet, a layer of its own interface is encapsulated. The original Leaflet interface is not external. At that time, the consideration is as follows:

  1. Like the previous version, it is easy to solve the problem of Internet map offset after encapsulation. Wgs84 coordinates are used uniformly externally, and data is converted into corresponding coordinates according to different base maps internally.
  2. You can implement a flat interface like JQuery, easy to use.
  3. The midpoint of leaflet is written as [latitude, longitude], while we are usually more used to writing as [longitude, latitude]. This problem can be solved conveniently through encapsulation.

When complex GIS spatial analysis is needed, the extension plug-in of GeoServer is written, which is connected to PostGIS database. By using PostGIS spatial analysis function, the author writes functions to complete spatial analysis.

The rest interface based on GeoServer implements the automatic publishing of map services. In the desktop software developed by ArcGIS Engine, GIS data is first imported into the spatial database, and then the REST interface of Geoserver is called to publish the map. Finally, the effect achieved is the same as the experience of ArcGIS publishing the map.

Build a portal site, including help documents, map resources, update instructions, etc., help documents to the interface as a clue, interface internal interface description and call cases. Map resources provide all maps that can be used, including Internet maps and self-published business maps, each map has an ID, according to which you can easily load the map in the API, do not need to care about how the map service is published.

Change PostGIS, GeoServer, and Tomcat to green versions to facilitate project deployment.

experience

  1. You can use the SLDEditor software to solve the problem that geoServer is not easy to map. The mapping of Geoserver is not easy to use, I tried to map on QGIS and then published it to GeoServer, but I found that many SLD style files generated after mapping on QGIS are not supported by Geoserver, and I don’t know whether the version is not matched or other reasons. Finally, I found an open source tool SLDEditor to edit the style file, which perfectly solved the problem, but the overall use experience was much worse than ArcGIS.
  2. PostGIS spatial analysis function is very easy to use, very powerful. Therefore, we mainly use PostGIS for spatial analysis functions, such as the graphic buffer function shared before.
  3. The GeoServer extension is better suited for developing and generating mapping service-related functionality. The spatial analysis function of GIS was realized by geoServer extension plug-in at the beginning. Later, it was found that the background program of the extension plug-in was mainly used for data transmission, and the most important analysis link was realized in spatial database PostGIS. However, the extended development environment of GeoServer was relatively complex, so it was not as easy to maintain as the Java background written by myself. The geoServer extension has the advantage of being able to invoke geoServer resources and functions directly, making it better suited for developing and generating mapping service-related functions.

Fourth edition 2019

background

After the development of the third version, it has been used in the department for more than a year, and the overall response is good. Some colleagues who know GIS used the ArcGIS JS API before and used the Leaflet encapsulated map API of the last version, their first reaction is that this is very lightweight, much smaller than ArcGIS JS API and fast loading.

After a release, we pay attention to the use of the user habit, basic it is everyone’s habits see sample first, find the sample after, will directly using the code kao, when the sample is not completely meet the requirements, to go through the API specification, the best situation is sample code comments, at a glance can understand, kao come in.

Of course, in the process of use, also slowly found some problems.

The strongest feedback from colleagues who know GIS is whether the native interface can be opened up. There is a colleague, every time he would first find the Leaflet code on the Internet, confirm that it can be realized and then come to us, let us add this function, and even send the information link over, the whole we are quite embarrassed. If the leaflet native interface can be released, he can solve a lot of work by himself.

When I need to study a new function, my first reaction is not to use the map API encapsulated by myself, but to use the native Leaflet to write, because I think the map API encapsulated by myself is not flexible enough to use.

Oh my God!!

It’s the same as the first version, when my colleagues said, “You don’t use what you develop yourself.” If there was Flex in the first version, this third version is written in JS from the inside out. There’s nothing to explain, but it’s just right.

In our daily work, we had other things to do besides encapsulating the map API, and in the last release, it felt like we were so consumed with encapsulating the basic features that we didn’t have time to explore the advanced map features. If the original interface is removed, basic functions can be used directly in the native interface, giving us more time to work on advanced map features.

Can you open up native interfaces?

There are several problems with liberalizing native interfaces:

  1. Map coordinate offset problem. Through encapsulation, a coordinate adaptation layer was constructed between the external interface and the map to solve the coordinate migration problem. If you let go of the native interface, there is no way to use this approach, you need to find another way.
  2. User habits. Will users who do not understand GIS get used to flat interfaces and find native interfaces hard to understand after opening up? The midpoint of the leaflet is written as [latitude, longitude], which is different from [longitude, latitude] normally used. Will some people fail to adapt to it?
  3. Version forward compatibility issues. In the last version, in order to pursue the minimalism of the interface, many data format types were simplified. If the native interface is released, it will produce a lot of work to accommodate these formats, and each subsequent function should consider the compatibility of this kind of data.

Solution:

  1. For the coordinate offset problem.

    There are two schemes, one is to provide users with a coordinate conversion interface, users themselves to convert coordinates, but this is not friendly to users. The second is to rectify the deviation of the Internet map tiles, so that the map unified coordinates, no longer offset, this is the most ideal, but there are technical difficulties. However, we finally overcame the technical difficulties and adopted the second solution. For details, see how to solve the offset problem of Baidu and Autonavi gracefully in Leaflet.

  2. For user habits.

    Why baidu, Autonavi map API does not use flat interface, we do not think they are difficult to use? The conclusion of our research is that the map API will be useful if it solves user problems, has fewer bugs, is rich in examples, and has clear documentation, provided the interface is not particularly complex. It doesn’t really matter if the interface is flat or not. Moreover, the native interface of Leaflet itself is already very simple. Considering simplicity alone, there is no need to encapsulate it.

  3. For version forward compatibility issues. We decided not to be compatible with the previous version, and kept the two versions at the same time for a slow transition. We recommend you to use this version for new projects and new products. We will continue to provide technical support for old projects, but will no longer upgrade functions. So after a year or two, you can slowly switch over. The fact proved that this decision was right. Now it has been more than two years, and most of the systems in the department have been switched to the new version.

The technical architecture

The technical architecture is adjusted as follows on the basis of the previous version:

  1. The leaflet native interface is released, and the interface is no longer encapsulated and functions are expanded in the form of plug-ins.
  2. Integrate various Internet map resources, solve their coordinate deviation problem by correcting the deviation of tiles, and use WGS84 coordinate uniformly externally.
  3. The help documentation has been changed from an interface to an example, with comments to ensure clarity and links to class references for the methods used in the sample code.
  4. Translate leaflet class reference documents and put them into the platform.
  5. When there are new functional requirements, simple functions will not be encapsulated, and examples will be provided directly. Complex functions will be encapsulated in plug-ins.
  6. Map analysis functions that are not relevant to GeoServer will be moved to the Java background, and only mapping functions will remain in GeoServer.

experience

  1. The translation of leaflet class references was not done well. A few pages were not translated in total, but strangely, no one ever complained about this problem. Later, I found that the users hardly looked at the documents, but mostly looked at the examples. If the examples did not exist, they would directly ask us. In fact, only we looked at the documents (covering their faces).
  2. When someone asks a question, try to write it down as an example so that you don’t have to ask it again if you can find it in the example.
  3. At present, the name of the calling sample is in the form of text list, which has the limitation of expression after all. Compared with Autonavi and Baidu Map API, they directly display the final effect of the sample in the way of GIF before the example list, which is more intuitive and easy to understand.
  4. Try to use tools to make the maintenance of the platform easy. If it is too complicated, you will not want to maintain it and the platform will be easily destroyed.

Fifth edition 2021

background

This version has not been completed, just finished the technical pre-research work before the year, here to share the overall idea simply with everyone.

Overall, the last version worked well and there has been very little negative feedback from users. The product has also been used as a technology center in large projects, provided to other companies, the same effect is very good.

However, we still have our own pursuit. Compared with Autonavi and Baidu Map API, we still need to improve in the following points:

  1. Map aesthetics. At present, the base map of the map uses Internet map tiles, which will cover the notes in the base map after superimposing business data, and it is not easy to automatically avoid the notes between business data. In addition, there is no useful map mapping tool, resulting in a chaotic and ugly map when displaying diverse data.
  2. Show performance issues. The map sometimes needs some animation effects, such as the animation effect to express the flow direction and speed of water in the pipe network, and there will be obvious lag in the large amount of data.
  3. Mapping problems. The problem with geoServer illustrations has already been mentioned. Although you can generate color matching files using SLDEditor, you can only generate one layer at a time, so there is no way to preview the whole, which is inefficient and a poor experience. Autonavi, Baidu map custom map mapping tool is very easy to use, artists can directly start, eye for a long time.

The goal of this release is to address the above three issues and continue to improve the user experience.

Technology selection

The foreground is changed to use MapBoxGL instead of Leaflet for two reasons:

  1. Performance. The upper limit of Leaflet is about 100,000 (see how Leaflet loads 100,000 data for details), while MapBoxGL is developed based on WebGL technology. The maximum amount of data depends on the performance of the graphics card and the speed of network transmission. Ideally, it can easily reach millions.
  2. Beautiful and degrees. Mapboxgl supports vector tiles very well. Combined with Maputnik, it is easy to implement the custom map function of Autonavi and Baidu Maps.

Maputnik uses Maputnik for maps and GeoServer for business data. Maputnik does not support vector tiles published by GeoServer, but we have solved this problem

There are two schemes for base map data:

  1. Continue to use the Internet map grid tiles, suitable for the accuracy of the base map data requirements.
  2. Publish THE OSM vector tile map locally. Currently, there is no free vector tile resource that can be directly used on the Internet. You can only download the OSM data locally and publish it yourself. The data quality of OSM map in China is relatively poor. If your business has low requirements on the accuracy of base map data and high requirements on style, such as large screen display system, you can choose this scheme. For details, see: How to implement local publishing and custom mapping of OSM maps

The map visualization effect is realized by deck.gl and L7.

experience

  1. Using Maputnik to load geoServer-published business layers and locally published OSM base maps at the same time, you can achieve deep integration of business data and base maps. For example, you can put rivers in business data under the base map road layer and realize automatic avoidance function of labels. Such experience is very cool.
  2. The compliance of OSM map is doubtful, so it is suggested to calibrate China’s national boundaries again.

Follow-up prospects

  1. Solve the problem of small amount of OSM map data.

    One of the imperfections of the fifth edition solution is that OSM map has a small amount of data in the country, which is why I wanted to automatically extract building contours through machine learning in Flag earlier this year.

  2. Research 3d GIS.

    I have studied CESium, ArcGIS JS API 4.x and three.js, etc., and used them in the project. However, my overall understanding of 3D GIS is still not as transparent as that of 2D GIS. Can’t be as confident as 2d GIS in coming up with a satisfactory open source solution, so we need to keep working on this area.

conclusion

About map API

In toG business companies, if they want to create an API comparable to Autonavi and Baidu Map in ease of use through open source GIS, they need to pay attention to the following points:

  1. Resolve map resource issues. Sort out the map resources on the Internet and the business map resources on the project, so that users can use them directly.
  2. Solve the map coordinate system problem. To solve the offset problem of the Internet map, and the conversion problem between various map coordinates, external users only use one coordinate.
  3. The basic map functionality is implemented through user education by providing complete invocation examples and documentation. Advanced map functionality is implemented by encapsulating plug-ins to avoid the core API becoming redundant over time.
  4. Scenario-rich, ready-to-use invocation examples are more important than interfaces that are brief and well-documented.
  5. From highest to lowest, users find the map API useful in terms of visual factors: problem solving, bugs, rich examples, technical support, and clear documentation.
  6. Do not impose on others what you yourself do not desire./Do not do to others as you would not have them do to you. Find a real business scenario to use your API, and continue to find problems, solve problems, and improve features until you feel comfortable with them, which forces you to think from the user’s point of view. If they are not willing to use, it is certainly not good, and ultimately will not go long.
  7. Do a good job in technical support. Developing a map API is a long, iterative process, and if you have good user support, it can help you make up for those problems.
  8. Learn to use tools to improve platform maintenance efficiency. Think about what parts of the platform maintenance process can be automated or semi-automated, such as document generation, deployment updates, etc., to save time for further research.

About open source GIS solutions

Here are the two combinations I recommend, both of which are already mentioned.

Lightweight version: Leaflet + GeoServer + postGIS

This combination of information on the network, the software is easy to use, strong universality, can meet the vast majority of people’s demand for TWO-DIMENSIONAL GIS.

Vector tiles: MapBoxGL + Maputnik + GeoServer + postGIS + OpenMapTiles + three.js

This combination can build a set of customized maps similar to Autonavi and Baidu Maps, as well as white model 3D maps. If you value map visualization, you are recommended to use this set.




All right, I think I’ll stop there. If you find it helpful, support us by following and sharing as much as possible.

The original address: gisarmory. Xyz/blog/index…

Check out GIS Armory for more quality GIS articles for the first time.

This article is licensed under the Creative Commons Attribution – Noncommercial Use – Share alike 4.0 International License. You are welcome to reprint, use and republish the article under the name “GIS Armory” (including link: gisarmory. Xyz /blog/). It must not be used for commercial purposes, and all works based on this article must be published under the same license.