The common challenge for IaaS software users is to find a desired resource quickly and accurately; For example, the vm with EIP (16.16.16.16) is found from 10,000 VMS. Most IaaS software addresses this problem through specific query logic in the API. Instead of specific queries, ZStack is equipped with a framework that automatically generates queries for each field of each resource and syndicates queries across multiple resources, helping users manage the vast number of resources in the cloud.

motivation

A medium-sized cloud can manage hundreds of physical hosts and tens of thousands of virtual machines, and because IaaS software rarely has full query apis, finding the desired resources can be a challenge. Most IaaS software only allows users to query resources using a few criteria (such as name, UUID) that are hard-coded into the query API. If users want to do a query using criteria other than hard coding, for example, to query virtual machines by creation date, they may have to eventually list all virtual machines and then use for.. Loop to filter the results. Querying resources using arbitrary fields, let alone syndication, is not fully supported in most IaaS software to date; For example, if a user wants to find a virtual machine whose network card applies specific security group rules, they might have to list all the resources (virtual machines, security groups) and then do twice for.. Loop. On the other hand, most IaaS software has the roughest UI compared to jIRa-like software. Many developers may not realize that the root cause of bad UI is not the UI developer’s lack of CSS/HTML/JavaScript skills, but the software itself does not provide a powerful API to support complex UI; For example, to implement a jIRa-like filter that displays only resources that meet specified criteria, the UI might need to do a lot of things listing all then Filtering by for.. Loop postprocessing jobs that screw together a lot of resources.

IaaS software has suffered for some time; The antidote to this is to provide a mechanism that automatically generates queries for every field of every resource and can handle join queries.

The problem

Most IaaS software uses a relational database (such as MySQL) as the background database, in which resources are usually arranged in separate tables, such as virtual machine tables, host tables, and cloud disk tables. For each resource, there is an API to retrieve a separate entry of that resource, which may be named describe API, List API, or Query API; These apis typically have hard-coded parameters that expose a subset of database table columns, allowing users to query resources with a small number of query criteria; These parameters are carefully chosen and are usually columns that are important to the API designer himself, for example, name, UUID. However, since not all columns are exposed, users often encounter situations where the column they want to query does not exist, so they must retrieve all resources and then use one or more for.. Loop for post-processing. A complex query scenario may require the use of federated queries, which typically span multiple database tables; For example, if you find a VM whose EIP is 16.16.16.16, it may involve the virtual table, NIC table, and EIP table. Some IaaS software addresses this problem using database views, another hard-coded approach that only joins selected tables in a fixed format, whereas in reality tables can be joined in a very complex manner. During a software upgrade, if any of the tables a view points to have changed, the view also needs to perform database migration.

Query API

To avoid manually coding query logic in the API and give users the flexibility to query anything, anywhere, ZStack created a framework that automatically generates queries for all resources without requiring developers to write code to implement the query logic; Furthermore, the framework can generate various join queries, as long as the required tables are already joined by foreign keys. In the following pages, we will use the Zone as an example to illustrate this amazing framework. A zone has the following columns in the database:

A user can query a zone by any field or combination of fields, using regular SQL comparison operators such as ‘=’, ‘! > = = ‘and’ > ‘, ‘ ‘, ‘<‘, ‘< =’, ‘in’, ‘not’ and ‘is null in’, ‘is not null,’ ‘like’, ‘not like’.

Note: In command line tools, some operators have different formats: ‘in'(? =), ‘not in'(! ? =), ‘is null'(=null), ‘is not null'(! =null), ‘like'( =), ‘not like'(! =).

QueryZone name=west-coast-zone
QueryZone name=west-coast-zone state=Enabled
Copy the code

Because the zone is the ancestor of the main resources in the ZStack, many resources are more or less related to it; For example, a running virtual machine is always in a zone. Relationships like this can generate federated queries such as:

QueryZone vmInstance.name=web-vm1
Copy the code

As shown in the table above, a zone does not expose any fields called vmInstance, but one condition starts with ‘vmInstance’ in the above query. This type of query is called extended query in ZStack. Here vmInstance represents the VM table, which has a field named zoneUuid (foreign key) pointing to the zone table, so the query framework can understand their relationship and generate a federated query. The above example can be interpreted as “Finding a zone running a virtual machine named Web-VM1.” The EIP table has foreign keys pointing to the VM table and the EIP table has foreign keys pointing to the VM table. The EIP table can also be used as a condition for zone query.

QueryZone vmInstance. VmNics. Eip. VipIp = 16.16.16.16Copy the code

The query is interpreted as “find an area on which the VIRTUAL machine network card has an EIP of 16.16.16.16”. Now you know the power of the query interface! We can even create some very complex queries:

QueryVolumeSnapshot volume.vmInstance.vmNics.l3Network.l2Network.attachedClusterUuids=13238c8e0591444e9160df4d3636be82
Copy the code

The complex queries the purpose is to find the disk snapshots, the target disk snapshot is founded by the virtual machine disk, while the virtual machine has a card on the L3 network, the L3 father L2 network is added in a cluster, this cluster uuid is 13238 c8e0591444e9160df4d3636be82. Don’t panic, you rarely need such a complex query, but it does demonstrate the framework’s capabilities. In addition, SQL features such as field selection, sorting, counting, and paging are also supported:

QueryL3Network name=L3-SYSTEM-PUBLIC count=true
QueryL3Network l2NetworkUuid=33107835aee84c449ac04c9622892dec limit=10
QueryL3Network l2NetworkUuid=33107835aee84c449ac04c9622892dec start=10 limit=100
QueryL3Network fields=name,uuid l2NetworkUuid=33107835aee84c449ac04c9622892dec
QueryL3Network l2NetworkUuid=33107835aee84c449ac04c9622892dec sortBy=createDate sortDirection=desc
Copy the code

implementation

As powerful as the query API is, the implementation is very clean. When adding a new resource, the developer does not need to write any code about the query logic other than defining the query API and the resource itself. To implement the zone query API, the developer needs to:

1. Use inventory for the query metadata annotation zone

@Inventory(mappingVOClass = ZoneVO.class)
@PythonClassInventory
@ExpandedQueries({
        @ExpandedQuery(expandedField = "vmInstance", inventoryClass = VmInstanceInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid"),
        @ExpandedQuery(expandedField = "cluster", inventoryClass = ClusterInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid"),
        @ExpandedQuery(expandedField = "host", inventoryClass = HostInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid"),
        @ExpandedQuery(expandedField = "primaryStorage", inventoryClass = PrimaryStorageInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid"),
        @ExpandedQuery(expandedField = "l2Network", inventoryClass = L2NetworkInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid"),
        @ExpandedQuery(expandedField = "l3Network", inventoryClass = L3NetworkInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid"),
        @ExpandedQuery(expandedField = "backupStorageRef", inventoryClass = BackupStorageZoneRefInventory.class,
                foreignKey = "uuid", expandedInventoryKey = "zoneUuid", hidden = true),
})
@ExpandedQueryAliases({ @ExpandedQueryAlias(alias = "backupStorage", expandedField = "backupStorageRef.backupStorage") })
public class ZoneInventory implements Serializable{
    private String uuid;
    private String name;
    private String description;
    private String state;
    private String type;
  private Timestamp createDate;
  private Timestamp lastOpDate;
}
Copy the code

The above annotations declare the relationship between zones and other resources, which is the basis for zone extension queries.

2. Define a query API

@AutoQuery(replyClass = APIQueryZoneReply.class, inventoryClass = ZoneInventory.class)
public class APIQueryZoneMsg extends APIQueryMessage {}Copy the code

3. Declare the query API in the API configuration file of the area

      
<service xmlns="http://zstack.org/schema/zstack">
    <id>zone</id>
    <interceptor>ZoneApiInterceptor</interceptor>

    <message>
        <name>org.zstack.header.zone.APIQueryZoneMsg</name>
        <serviceId>query</serviceId>
    </message>
</service>
Copy the code

The API APIQueryZoneMsg is routed to the query service by specifying the service ID query. That’s it, the query logic doesn’t need a line of code; The query service does the rest automatically. All ZStack query apis are defined like this, and adding new resource query apis is very easy.

The current limit

The main limitation is that only logical AND is supported in query conditions, AND OR is not. Such as:

QueryZone name=west-coast-zone state=Enabled
Copy the code

The above query statement can be interpreted as “look for the region whose name is West-coast AND whose state is Enabled”. The reason for this is that our analysis of the use of SQL in ZStack source code shows that 99% of the combined query conditions are based on AND logic. On the other hand, if the logic OR is introduced without creating a DSL, it is very difficult to keep the code clean. However, in many cases, OR can be used with the comparison operation in(? =) implementation:

QueryZone name=west-coast-zone state? =Enabled,DisabledCopy the code

The above example says “find a region named West-coast and its status is Enabled or Disabled”. In the future, we will introduce dSL-style query languages such as:

QueryZone name=west-coast-zone AND (state=Enabled OR state=Disabled)
Copy the code

conclusion

In this article, we demonstrated ZStack’s query API. With this powerful tool, users can query any resource in a similar way to a relational database. In the future, ZStack will build an advanced UI that can use the query API to create a variety of views (filters) that, for example, show all virtual machines running on the same L3 network, revolutionizing the IaaS UI user experience.