What is the

Due to historical reasons, Nova think resources are all provided by the compute nodes, so the report some resources, when using different computing nodes in Nova only by querying the database data, simple accumulative usage is calculated, and available resources, it must be not rigorous scientific approach, so in the N version, Nova introduced the Placement API, a separate RESTful API and data model for managing and querying resource providers’ inventory, usage, allocation records, and more to provide better and more accurate resource tracking, scheduling, and allocation capabilities.

What is the

The code directory

Nova Placement API is a separate RESTful API that has its own Endpoint and is started on a different port from Nova API services. Therefore, Nova Placement API is relatively independent from Nova API services in terms of code directory. Its code implements all in/nova/API/it/placement, so it seems I have a look at the nova placement API code directory structure:

F:nova en.f $tree -C API /openstack/ Placement API /openstack/ Placement ├─ __init__.py ├── Hand │ ├── Exercises. Py │ ├── Exercises. Py │ ├── Exercises The inventory. Py │ ├ ─ ─ resource_class. Py │ ├ ─ ─ resource_provider. Py │ ├ ─ ─ root. Py │ ├ ─ ─ trait. Py │ └ ─ ─ the usage. Py ├ ─ ─ lib. Py Heavy Metal Metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal metal Py │ ├── Heavy Exercises. Py │ ├─ Heavy Exercises. Py │ ├─ Heavy Exercises Trait. Py │ └ ─ ─ the usage. Py ├ ─ ─ util. Py ├ ─ ─ wsgi. Py └ ─ ─ wsgi_wrapper. PyCopy the code

Among them, the API/it/placement/schemas directory, you can see the basic schema of data model, But the resource privoder schema definition in the API/it/placement/handlers/resource_provider py. Let’s look at some of these concepts in terms of schema.

Some concepts in the Nova Placement API

Resource Provider

The schema contains only UUID and RP (Resource Provider abbreviation, the same below) basic information, such as name:

GET_RPS_SCHEMA_1_0 = {
    "type": "object"."properties": {
        "name": {
            "type": "string"
        },
        "uuid": {
            "type": "string"."format": "uuid"}},"additionalProperties": False,}Copy the code

The Resource provider may be a compute node, a shared storage pool, or an IP allocation pool. Different RPS provide a variety of resources. Therefore, the concept of Resource Class is introduced.

Resource Class

Resource types, such as CPU, memory, PCI devices, local temporary disks, and so on. Each resource consumed is annotated and tracked by category.

This concept is introduced to solve the hard-coded resource type extensibility problems in Nova. For example, CPU resources may be recorded in the vcpus field of Instance objects. If a new resource type is added later, the data table needs to be modified, which will stop maintenance. There is a lot of downtime in the system and that is not acceptable.

The Placement API provides some standard resource categories, such as:

  • VCPU
  • MEMORY_MB
  • DISK_GB
  • PCI_DEVICE
  • NUMA_SOCKET
  • NUMA_CORE
  • NUMA_THREAD
  • IPV4_ADDRESS
  • .

Note: Data from BP:Introduce Resource classes

In addition to the above standard Resource categories, the Placement API adds the ability to customize Resource classes for RPS in version O, such as automatic FPGA, bare-metal scheduling, and so on.

Inventory

Inventory, stock. It is used to record information such as overratio, total resource, stock, step_size, minimum and maximum unit.

BASE_INVENTORY_SCHEMA = {
    "type": "object"."properties": {
        "resource_provider_generation": {
            "type": "integer"
        },
        "total": {
            "type": "integer"."maximum": db.MAX_INT,
            "minimum": 1,},"reserved": {
            "type": "integer"."maximum": db.MAX_INT,
            "minimum": 0,},"min_unit": {
            "type": "integer"."maximum": db.MAX_INT,
            "minimum": 1
        },
        "max_unit": {
            "type": "integer"."maximum": db.MAX_INT,
            "minimum": 1
        },
        "step_size": {
            "type": "integer"."maximum": db.MAX_INT,
            "minimum": 1
        },
        "allocation_ratio": {
            "type": "number"."maximum": db.SQL_SP_FLOAT_MAX
        },
    },
    "required": [
        "total"."resource_provider_generation"]."additionalProperties": False
}
Copy the code

The resource_PROVIDer_generation field is the flag bit of a consistency view. The generation function is the same when retrieving the RP list, which is CAS (Compare and swap). Optimistic locking — when multiple threads try to update the same variable using CAS at the same time, only one thread can update the value of the variable, and all other threads fail, the failed thread is not suspended, but is informed that it has lost the race and can try again.

Usage

That is, dosage, usage. You can view the usage of an RP or the resource usage of a user in a project.

Aggregate

In the Ocata release, the community began to integrate the Nova-Scheduler service with the Placement API, and some modifications were made in Scheduler to use the Placement API for computing node filtering that meets some basic resource request criteria. Aggregates are added to provide the grouping mechanism for the Resource Provider.

Allocation

That is, allocated quantity, the resources allocated by a RP to a resource consumer (that is, an instance).

Allocation-candidate

For example, the user says, I need 1 VCPU, 512MB of memory, 1 gb of disk resources, Placement you can help me find if there are any suitable resources. Placement then does a variety of processing, giving feedback to the user as to which candidate resource providers can be assigned.

Trait

Literal, characteristic, characteristic. The ResourceProvider and Allocation can control and manage the boot virtual machine requests quantitatively. However, we also need to distinguish resources qualitatively. The most classic example is when we create a virtual machine and need to request disk resources from different RPS. But it is also possible to request an 80GB SSD. That’s what traits are all about.

Databases and data tables

In the Pike version of the Packstack environment I have installed so far, I see a NovA_Placement database, but no tables (perhaps the community wants to put placement related tables in this database?). , the database corresponding to the Placement still uses novA_API:

MariaDB [nova_placement]> use nova_api;
Reading table information forcompletion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [nova_api]> show tables; +------------------------------+ | Tables_in_nova_api | +------------------------------+ | aggregate_hosts | | aggregate_metadata | | aggregates | | allocations | | build_requests | | cell_mappings | | consumers | | flavor_extra_specs | | flavor_projects | | flavors | | host_mappings | | instance_group_member | | instance_group_policy  | | instance_groups | | instance_mappings | | inventories | | key_pairs | | migrate_version | | placement_aggregates | | project_user_quotas | | projects | | quota_classes | | quota_usages | | quotas | | request_specs | | reservations | | resource_classes | | resource_provider_aggregates | | resource_provider_traits | | resource_providers | | traits | | users | +------------------------------+Copy the code

You can see from the presentation that those are Placement related tables, so I won’t expand them here.

Initialization and loading methods

We mentioned earlier that the Nova Placement API is a separate RESTful API, so how is it initialized? With that in mind, let’s first look at nova setup.cfg, which has wsGI_scripts configured as follows:

wsgi_scripts =
    nova-placement-api = nova.api.openstack.placement.wsgi:init_application
    nova-api-wsgi = nova.api.openstack.compute.wsgi:init_application
    nova-metadata-wsgi = nova.api.metadata.wsgi:init_application
Copy the code

One can see the nova – placement – API initialization from nova. API. It. Placement. Wsgi. Init_application, code is as follows:

def init_application(a):
    # initialize the config system
    conffile = _get_config_file()
    config.parse_args([], default_config_files=[conffile])

    # initialize the logging system
    setup_logging(conf.CONF)

    # dump conf if we're at debug
    if conf.CONF.debug:
        conf.CONF.log_opt_values(
            logging.getLogger(__name__),
            logging.DEBUG)

    # build and return our WSGI app
    return deploy.loadapp(conf.CONF)
Copy the code

Where the WSGI app is constructed and returned at the end, that is, deploy.loadapp(conf.conf) is called:

def loadapp(config, project_name=NAME):
    application = deploy(config, project_name)
    return application
def deploy(conf, project_name):
    """Assemble the middleware pipeline leading to the placement app.""". application = handler.PlacementHandler() ...for middleware in (microversion_middleware,
                       fault_wrap,
                       request_log,
                       context_middleware,
                       auth_middleware,
                       cors_middleware,
                       req_id_middleware,
                       ):
        if middleware:
            application = middleware(application)

    return application
Copy the code

And here handler.placementhandler () is our Placement API entry:

class PlacementHandler(object):
    """Serve Placement API. Dispatch to handlers defined in ROUTE_DECLARATIONS. """

    def __init__(self, **local_config):
        # NOTE(cdent): Local config currently unused.
        self._map = make_map(ROUTE_DECLARATIONS)

    def __call__(self, environ, start_response):
        # All requests but '/' require admin.
        if environ['PATH_INFO'] != '/':...Copy the code

As you can see, the PlacementHandler constructs the map from the route definition in __init__ and dispatches the request in __call__. This is a typical WSGI application:

WSGI application is a callable object (a function, method, class, or an instance with a __call__ method) that accepts two positional arguments: WSGI environment variables and a callable with two required positional arguments which starts the response;

Once initialization is found, how is Placement API loading and startup implemented?

First, nova-placement API is a separate script, launched in HTTPD, similar to Keystone (wsGized in 12, see >> Portal), which can be seen through systemctl status HTTPD:

[root@f-packstack ~(keystone_admin)]# systemctl status httpdLow HTTPD. Service - The Apache HTTP Server The Loaded: The Loaded (/ usr/lib/systemd/system/HTTPD. Service; enabled; vendor preset: disabled) Active: active (running) since Fri 2018-02-02 09:17:51 CST; 1 weeks 0 days ago Docs: man:httpd(8) man:apachectl(8) Process: 4087 ExecReload=/usr/sbin/httpd$OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 1309 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"CGroup: / system. Slice/HTTPD service ├ ─ 1309 / usr/sbin/HTTPD - DFOREGROUND ├ ─ 4108 keystone - admin - DFOREGROUND del ├ ─ 4109 Keystone-admin -DFOREGROUND Exercises 4110 Keystone-admin -DFOREGROUND Exercises 4111 Keystone-admin -DFOREGROUND Exercises 4110 Keystone-Admin -DFOREGROUND Exercises 4111 Keystone-Admin -DFOREGROUND Exercises 4112 Keystone - the main - DFOREGROUND ├ ─ 4113 keystone - the main - DFOREGROUND ├ ─ 4114 keystone - the main - DFOREGROUND ├ ─ 4115 keystone - main -Foreground Exercise-exercise exercises - 482 Placement_wsGI-Exercise Exercises - Foreground Exercise-Exercise exercises - 482 Placement_wsgi-Exercise Exercises - Foreground Exercise-Exercise exercises ├ ─ 4119 placement_wsgi - DFOREGROUND ├ ─ 4121 / usr/sbin/HTTPD - DFOREGROUND ├ ─ 4122 / usr/sbin/HTTPD - DFOREGROUND...Copy the code

Knowing that it is started at HTTPD, we go to the configuration file directory:

[root@f-packstack ~(keystone_admin)]# ll /etc/httpd/conf.d/ total 36 -rw-r-----. 1 root root 136 Jan 12 17:46 00-nova-placement-api.conf -rw-r--r--. 1 root root 943 Jan 12 17:48 10-keystone_wsgi_admin.conf -rw-r--r--. 1 root root 938 Jan 12 17:48 10-keystone_wsgi_main.conf -rw-r--r--. 1 root root 941 Jan 12 17:49 10-placement_wsgi.conf -rw-r--r--. 1 root root 697 Jan 12 17:48 15-default.conf -rw-r--r--. 1 root root 2926 Oct 20 04:39 autoindex.conf -rw-r--r--. 1 root  root 366 Oct 20 04:39 README -rw-r--r--. 1 root root 1252 Oct 20 00:44 userdir.conf -rw-r--r--. 1 root root 824 Oct 20 00:44 welcome.confCopy the code

WSGIScriptAllias is defined in 10-placement_wsgi.conf:

. WSGIProcessGroup placement-api WSGIScriptAlias /placement"/ var/WWW/cgi - bin/nova/nova - placement - API"...Copy the code

That is, a request with the URL /placement/ XXX will cause the HTTPD service to run the WSGI application defined in /var/www/cgi-bin/nova/nova-placement- API. In this file, we can see:

The from nova. API. It. Placement. Wsgi import init_application if __name__ = = "__main__" : Import argparse import socket import sys import wsgiref.simple_server as WSS... server = wss.make_server(args.host, args.port, init_application()) ...Copy the code

Also corresponds to the aforementioned PlacementHandler in nova. API. It. Placement. Wsgi. Init_application, at this point, we can understand the nova placement API initialization and loading way.

API Routing Definition

PlacementHandler is a section in the initialization time, according to the route definition structure map map, we will look at the file API/it/placement/handler APIROUTE_DECLARATIONS in py:

# URLs and Handlers
# NOTE(cdent): When adding URLs here, do not use regex patterns in
# the path parameters (e.g. {uuid:[0-9a-zA-Z-]+}) as that will lead
# to 404s that are controlled outside of the individual resources
# and thus do not include specific information on the why of the 404.
ROUTE_DECLARATIONS = {
    '/': {
        'GET': root.home,
    },
    # NOTE(cdent): This allows '/placement/' and '/placement' to
    # both work as the root of the service, which we probably want
    # for those situations where the service is mounted under a
    # prefix (as it is in devstack). While weird, an empty string is
    # a legit key in a dictionary and matches as desired in Routes.
    ' ': {
        'GET': root.home,
    },
    '/resource_classes': {
        'GET': resource_class.list_resource_classes,
        'POST': resource_class.create_resource_class
    },
    '/resource_classes/{name}': {
        'GET': resource_class.get_resource_class,
        'PUT': resource_class.update_resource_class,
        'DELETE': resource_class.delete_resource_class,
    },
    '/resource_providers': {
        'GET': resource_provider.list_resource_providers,
        'POST': resource_provider.create_resource_provider
    },
    '/resource_providers/{uuid}': {
        'GET': resource_provider.get_resource_provider,
        'DELETE': resource_provider.delete_resource_provider,
        'PUT': resource_provider.update_resource_provider
    },
    '/resource_providers/{uuid}/inventories': {
        'GET': inventory.get_inventories,
        'POST': inventory.create_inventory,
        'PUT': inventory.set_inventories,
        'DELETE': inventory.delete_inventories
    },
    '/resource_providers/{uuid}/inventories/{resource_class}': {
        'GET': inventory.get_inventory,
        'PUT': inventory.update_inventory,
        'DELETE': inventory.delete_inventory
    },
    '/resource_providers/{uuid}/usages': {
        'GET': usage.list_usages
    },
    '/resource_providers/{uuid}/aggregates': {
        'GET': aggregate.get_aggregates,
        'PUT': aggregate.set_aggregates
    },
    '/resource_providers/{uuid}/allocations': {
        'GET': allocation.list_for_resource_provider,
    },
    '/allocations': {
        'POST': allocation.set_allocations,
    },
    '/allocations/{consumer_uuid}': {
        'GET': allocation.list_for_consumer,
        'PUT': allocation.set_allocations_for_consumer,
        'DELETE': allocation.delete_allocations,
    },
    '/allocation_candidates': {
        'GET': allocation_candidate.list_allocation_candidates,
    },
    '/traits': {
        'GET': trait.list_traits,
    },
    '/traits/{name}': {
        'GET': trait.get_trait,
        'PUT': trait.put_trait,
        'DELETE': trait.delete_trait,
    },
    '/resource_providers/{uuid}/traits': {
        'GET': trait.list_traits_for_resource_provider,
        'PUT': trait.update_traits_for_resource_provider,
        'DELETE': trait.delete_traits_for_resource_provider
    },
    '/usages': {
        'GET': usage.get_total_usages,
    },
}
Copy the code

How to use

How to deploy

In the official documentation, it is mentioned that the Placement API service must be deployed after the upgrade to 14.0.0 (version N) and before the upgrade to 15.0.0 (version O). The Resource tracker in the Nova-Compute service needs to obtain placement resource provider inventory and allocation information, which will be used by nova-Scheduler in version O.

  1. The Deployment API service – Placement API is currently being developed in Nova, but is designed to be relatively independent so that it can be separated into a separate project in the future. As a separate WSGI application, API services can be deployed using EITHER Apahce2 or Nginx.
  2. Synchronize the database – Perform this operation manually when upgrading version Nnova-manage api_db syncCommand to synchronize the database so that Placement related tables are created
  3. Create a Placement service user with the admin role in Keystone, update the service directory, and configure a separate endpoint.
  4. Configure the [placement] part of nova.conf and restart the nova-compute service. However, for our P version, after O version of a series of functions to complement, especially in O version, if innova.confNot in the configuration[placement]Part of the content cannot be launchednova-computeService.

The nova-compute service will fail to start in Ocata unless the [placement] section of nova.conf on the compute is configured.

For more information on deployment, see the official documentation, >> Portal.

OSC Placement Plugin

CURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL cURL First we get the token:

Auth token curl -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "1234 qwer"}}} '\ -h "content-type: application/json" \ http://localhost:5000/v2.0/tokens... {" issued_AT ": "2018-02-07T07:40:07.000000z "," Expires ": "2018-02-07T08:40:07.000000z "," ID ": "gAAAAABaeq1XrNDoU_F_iRk8uC0lOxYpyzLMW_YRs_ggJHuF1OpGHBN-pymQut-Bp2Er-J4XkYfQkMdJbRlBIBhq4wfhZMHZvag1itnL6Q-TSWhOn7uZpdQ sYqqJDmwgtzCm-hcpg17IwN5FZSanCbcy6S96YZ0Zci5STWNka40861Mn8UQ2yRE", "tenant": { "description": "admin tenant", "enabled": true, "id": "6387fc88b3064149a12eb5b58669e0b2", "name": "Admin"}} # access token, you can also use OSC command: it token issue | grep 'id' | awk '{print $4}'... After receiving the token, construct the request and check the resource providers list.  curl -X GET / -H 'x-auth-token:gAAAAABaeq1XrNDoU_F_iRk8uC0lOxYpyzLMW_YRs_ggJHuF1OpGHBN-pymQut-Bp2Er-J4XkYfQkMdJbRlBIBhq4wfhZMHZvag1itn17I WN5FZSanCbcy6S96YZ0Zci5STWNka40861Mn8UQ2yRE/http://192.168.122.105:8778/placement/resource_providers # 'resources providers list { "resource_providers": [ { "generation": 30, "uuid": "4cae2ef8-30eb-4571-80c3-3289e86bd65c", "links": [ { "href": "/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c", "rel": "self" }, { "href": "/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c/inventories", "rel": "inventories" }, { "href": "/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c/usages", "rel": "usages" } ], "name": "f-packstack" } ] }Copy the code

The generation field, which is the flag bit of a consistent view, is the same as the resource_PROVIDer_generation function in the allocation of the RP, in fact, counts as optimistic lock technology, i.e., CAS, Compare and swap, When multiple threads try to update the same variable using CAS at the same time, only one thread can update the value of the variable, and all other threads fail. The thread that fails is not suspended, but is informed that it has lost the race and can try again.

To obtain aggregate and aggregate, note that the aggregate API was implemented in Version 1.1, so specify OpenStack -apI-version: placement 1.1 in the request header:

curl -g -i -X GET http://192.168.122.105:8778/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c/aggregates \
-H "User-Agent: python-novaclient" \
-H "Accept: application/json" \
-H "X-Auth-Token: gAAAAABaf5nafUZyFTl_pztozfB65wkP0c26HQqrxRgAiJGsxY8g743LxFOZEI3bF_l37xh0UajbF5nQ1kLYGAonOGphV4AivXgYMUOJ84uGrHjpC60NlmNzzQ3lJGVJb-pNxQw74WsMOc9I0D2B5Mzmf2OgDeictae5f0UFgTR9DFb_vaWCWQ4" \
-H "OpenStack-API-Version: placement 1.1"
HTTP/1.1 200 OK
Date: Fri, 15 Sep 2017 09:35:21 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 18
Content-Type: application/json
OpenStack-API-Version: placement 1.1
vary: OpenStack-API-Version
x-openstack-request-id: req-ab28194f-8389-40a1-9a2b-a94dbc792573
Connection: close

{"aggregates": []}


curl -g -i -X GET http://192.168.122.105:8778/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c/inventories \
-H "User-Agent: python-novaclient" \
-H "Accept: application/json" \
-H 'x-auth-token:gAAAAABae6lX26bp4PEVHCac0cjFnNl18W8DjeQKXDYvuKP4drRJ8t6DC-9uzcCm4E9Xf7NjqSqkRX6WGsE3qHmpAt7GmIu1SrLCtyEOVM2IQP5XLNrwMekGGrzQ_ADOaSTc9XpPpCYyYwzT-zCAvWG-T9T6Ip4l3zHWLwNBBPrm35gBZVZeslQ' \

{
    "resource_provider_generation": 30,
    "inventories": {
        "VCPU": {
            "allocation_ratio": 16,
            "total": 4,
            "reserved": 0,
            "step_size": 1,
            "min_unit": 1,
            "max_unit": 128
        },
        "MEMORY_MB": {
            "allocation_ratio": 1.5,
            "total": 8095,
            "reserved": 512,
            "step_size": 1,
            "min_unit": 1,
            "max_unit": 8095
        },
        "DISK_GB": {
            "allocation_ratio": 1,
            "total": 49,
            "reserved": 0,
            "step_size": 1,
            "min_unit": 1,
            "max_unit": 49
        }
    }
}
Copy the code

The thoughtful community figured out how to make it easy for users to operate the Placement API, so we developed an OpenStackClient Plugin, namely OSC-Placement, which we need to manually install and use:

$ pip install osc-placement
Copy the code

With OSC placement commands, we no longer need to use curl commands to simulate HTTP requests and can do so with ease:

[root@f-packstack ~(keystone_admin)]# openstack --debug resource provider list ... http://192.168.122.105:8778 "GET/placement/resource_providers HTTP / 1.1" 200 185 RESP: [200] the Date: Thu, 08 Feb 2018 05:59:56 GMT Server: Apache/2.4.6 (CentOS) OpenStack- API-version: Placement 1.0 vary: OpenStack-API-Version,Accept-Encoding x-openstack-request-id: req-c6077c19-ca05-4cab-95fa-6129ff989400 Content-Encoding: gzip Content-Length: 185 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"resource_providers": [{"generation": 30, "uuid": "4cae2ef8-30eb-4571-80c3-3289e86bd65c", "links": [{"href": "/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c", "rel": "self"}, {"href": "/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c/inventories", "rel": "inventories"}, {"href": "/placement/resource_providers/4cae2ef8-30eb-4571-80c3-3289e86bd65c/usages", "rel": "usages"}], "name": "F - packstack"}}] the GET call to placement for http://192.168.122.105:8778/placement/resource_providers, informs the request id req-c6077c19-ca05-4cab-95fa-6129ff989400 +--------------------------------------+-------------+------------+ | uuid | name | generation | +--------------------------------------+-------------+------------+ | 4cae2ef8-30eb-4571-80c3-3289e86bd65c | f-packstack | 30 | +--------------------------------------+-------------+------------+ clean_up ListResourceProvider: END return value: 0Copy the code

Of course, there are many other commands that you can try out if you are interested.

Highlights: Nova scheduling combined with Placement API

First, let’s take a look at the invocation/scheduling relationships between services during the creation of a virtual machine in version P:

  1. Get allocation candidates
  2. Below, we describe the scheduling process in detail with codes.

Get allocation candidates

At the time of dispatch, Nova – conductor at nova.conductor.manager.Com puteTaskManager# _schedule_instances nova. Call the methods in the scheduler. Client. SchedulerClient# Select_destinations:

    @utils.retry_select_destinations
    def select_destinations(self, context, spec_obj, instance_uuids, return_objects=False, return_alternates=False):
        return self.queryclient.select_destinations(context, spec_obj,
                instance_uuids, return_objects, return_alternates)
Copy the code

Including SchedulerClient again call SchedulerQueryClient, is called the nova. Scheduler. Client. Query. SchedulerQueryClient# select_destinations method:

    def select_destinations(self, context, spec_obj, instance_uuids, return_objects=False, return_alternates=False):
        return self.scheduler_rpcapi.select_destinations(context, spec_obj,
                instance_uuids, return_objects, return_alternates)
Copy the code

Initiate RPC calls in this method, call the nova. The scheduler. Manager. SchedulerManager# select_destinations method:

    @messaging.expected_exceptions(exception.NoValidHost)
    def select_destinations(self, ctxt, request_spec=None, filter_properties=None, spec_obj=_sentinel, instance_uuids=None, return_objects=False, return_alternates=False):
        LOG.debug("Starting to schedule for instances: %s", instance_uuids)
        ...
        The USES_ALLOCATION_CANDIDATES default to True.
        # indicates that the Nova Placement API is used to select resource allocation candidates
        if self.driver.USES_ALLOCATION_CANDIDATES:
            res = self.placement_client.get_allocation_candidates(ctxt,
            if res is None:
                alloc_reqs, provider_summaries, allocation_request_version = (
                        None.None.None)
            else:
                (alloc_reqs, provider_summaries,
                            allocation_request_version) = res
            if not alloc_reqs:
                LOG.debug("Got no allocation candidates from the Placement "
                          "API. This may be a temporary occurrence as compute "
                          "nodes start up and begin reporting inventory to "
                          "the Placement service.")
                raise exception.NoValidHost(reason="")
            else:
                # Build a dict of lists of allocation requests, keyed by
                # provider UUID, so that when we attempt to claim resources for
                # a host, we can grab an allocation request easily
                alloc_reqs_by_rp_uuid = collections.defaultdict(list)
                for ar in alloc_reqs:
                    for rp_uuid in ar['allocations']:
                        alloc_reqs_by_rp_uuid[rp_uuid].append(ar)

        # Only return alternates if both return_objects and return_alternates
        # are True.
        return_alternates = return_alternates and return_objects
        In this case, we configure to use the FilterScheduler,
        # namely again called nova. Scheduler. Filter_scheduler. FilterScheduler# select_destinations
        # We'll get to that later
        selections = self.driver.select_destinations(ctxt, spec_obj,
                instance_uuids, alloc_reqs_by_rp_uuid, provider_summaries,
                allocation_request_version, return_alternates)
        # If `return_objects` is False, we need to convert the selections to
        # the older format, which is a list of host state dicts.
        if not return_objects:
            selection_dicts = [sel[0].to_dict() for sel in selections]
            return jsonutils.to_primitive(selection_dicts)
        return selections

Copy the code

Let’s start with the Placement API called here, which makes a GET request to GET the Allocation Candidates.

Note: the OSC command corresponding to this API was not found, so we use curl to simulate. Additionally, Allocation Candidates API requests are Availiable starting from Version 1.10.

# access token [root @ f - packstack ~ (keystone_admin)] # it token issue | grep 'id' | awk '{print $4}' gAAAAABajn5nIXMCkZQBwcl7LdqeCV8pOuFSN4ltIUa9GcJ_PO4x920rpw5fwz43BZ8rkKIVlWF1OHfDNs1GRhqhoUHPNkEU6SRNK8G1BFKoHKD4nDJESGhS Delete from MrGwDGTIsYeaANqM2D_48tUo_pY0eqCD8iEcRDHi- qCH-c_t_m44so0chvlxtde Resources =DISK_GB:1,MEMORY_MB:512,VCPU:1 curl -g-i-x GET http://192.168.122.105:8778/placement/allocation_candidates? resources=DISK_GB:1,MEMORY_MB:512,VCPU:1 \ -H "User-Agent: python-novaclient" \ -H "Accept: application/json" \ -H "X-Auth-Token: gAAAAABajn5nIXMCkZQBwcl7LdqeCV8pOuFSN4ltIUa9GcJ_PO4x920rpw5fwz43BZ8rkKIVlWF1OHfDNs1GRhqhoUHPNkEU6SRNK8G1BFKoHKD4nDJESGhS MrGwDGTIsYeaANqM2D_48tUo_pY0eqCD8iEcRDHi-QCH-c_t_m44So0cHvlXtdE" \ -H "OpenStack-API-Version: Placement 1.10" HTTP/1.1 200 OK Date: Thu, 22 Feb 2018 08:55:27 GMT Server: Apache/2.4.6 (CentOS) OpenStack- API-version: Placement 1.10 vary: OpenStack-API-Version,Accept-Encoding x-openstack-request-id: req-234db1eb-1386-4e89-99bd-c9269270c603 Content-Length: 381 Content-Type: application/json { "provider_summaries": { "4cae2ef8-30eb-4571-80c3-3289e86bd65c": { "resources": { "VCPU": { "used": 2, "capacity": 64 }, "MEMORY_MB": { "used": 1024, "capacity": 11374 }, "DISK_GB": { "used": 2, "capacity": 49 } } } }, "allocation_requests": [ { "allocations": [ { "resource_provider": { "uuid": "4cae2ef8-30eb-4571-80c3-3289e86bd65c" }, "resources": { "VCPU": 1, "MEMORY_MB": 512, "DISK_GB": 1 } } ] } ] }Copy the code

Placement returned some information after a series of queries, in which allocation_requests is our request parameter, that is, we need these resources, please Placement to see if there is a suitable RP? Placement then helps us find RP’s whose UUID is 4CAe2EF8-30EB-4571-80C3-3289E86BD65C, and is thoughtful enough to list the resources currently used by the RP and their total summaries in provider_summaries. In fact, these two queries correspond to the following two SQL statements:

-- 1. Query the Resource Provider that meets the requirements
SELECT rp.id
FROM resource_providers AS rp
    -- VCPU information join
    -- Total vCPU memory
    INNER JOIN inventories AS inv_vcpu
        ON inv_vcpu.resource_provider_id = rp.id
        AND inv_vcpu.resource_class_id = %(resource_class_id_1)s
    -- VCPU usage information
    LEFT OUTER JOIN (
        SELECT allocations.resource_provider_id AS resource_provider_id,
        sum(allocations.used) AS used
        FROM allocations
        WHERE allocations.resource_class_id = %(resource_class_id_2)s
        GROUP BY allocations.resource_provider_id
    ) AS usage_vcpu
        ON inv_vcpu.resource_provider_id = usage_vcpu.resource_provider_id
    -- Memory information join
    -- Memory Total memory information
    INNER JOIN inventories AS inv_memory_mb
        ON inv_memory_mb.resource_provider_id = rp.id
        AND inv_memory_mb.resource_class_id = %(resource_class_id_3)s
    -- Memory usage information
    LEFT OUTER JOIN (
        SELECT allocations.resource_provider_id AS resource_provider_id,
            sum(allocations.used) AS used
        FROM allocations
        WHERE allocations.resource_class_id = %(resource_class_id_4)s
        GROUP BY allocations.resource_provider_id
    ) AS usage_memory_mb
        ON inv_memory_mb.resource_provider_id = usage_memory_mb.resource_provider_id
    -- Disk information join
    -- Total disk capacity information
    INNER JOIN inventories AS inv_disk_gb
        ON inv_disk_gb.resource_provider_id = rp.id
        AND inv_disk_gb.resource_class_id = %(resource_class_id_5)s
    -- Disk Usage information
    LEFT OUTER JOIN (
        SELECT allocations.resource_provider_id
        AS resource_provider_id, sum(allocations.used) AS used
        FROM allocations
        WHERE allocations.resource_class_id = %(resource_class_id_6)s
        GROUP BY allocations.resource_provider_id
        ) AS usage_disk_gb
            ON inv_disk_gb.resource_provider_id = usage_disk_gb.resource_provider_id
WHERE
-- The vCPU meets the upper limit, lower limit, and step size conditions
coalesce(usage_vcpu.used, %(coalesce_1)s) + %(coalesce_2)s <= (
inv_vcpu.total - inv_vcpu.reserved) * inv_vcpu.allocation_ratio AND
inv_vcpu.min_unit <= %(min_unit_1)s AND
inv_vcpu.max_unit >= %(max_unit_1)s AND
%(step_size_1)s % inv_vcpu.step_size = %(param_1)s AND
-- Memory meets the upper limit, lower limit, and step size conditions
coalesce(usage_memory_mb.used, %(coalesce_3)s) + %(coalesce_4)s <= (
inv_memory_mb.total - inv_memory_mb.reserved) * inv_memory_mb.allocation_ratio AND
inv_memory_mb.min_unit <= %(min_unit_2)s AND
inv_memory_mb.max_unit >= %(max_unit_2)s AND
%(step_size_2)s % inv_memory_mb.step_size = %(param_2)s AND
-- Disk Meets the upper limit, lower limit, and step size conditions
coalesce(usage_disk_gb.used, %(coalesce_5)s) + %(coalesce_6)s <= (
inv_disk_gb.total - inv_disk_gb.reserved) * inv_disk_gb.allocation_ratio AND
inv_disk_gb.min_unit <= %(min_unit_3)s AND
inv_disk_gb.max_unit >= %(max_unit_3)s AND
%(step_size_3)s % inv_disk_gb.step_size = %(param_3)s

-- 2. Query the usage and storage of the Resource Provider
SELECT rp.id AS resource_provider_id, rp.uuid AS resource_provider_uuid,
    inv.resource_class_id, inv.total, inv.reserved, inv.allocation_ratio,
    `usage`.used
FROM resource_providers AS rp
    -- Inventory information, the total amount per RP
    INNER JOIN inventories AS inv
        ON rp.id = inv.resource_provider_id
    - allocation information
    LEFT OUTER JOIN (
        -- Usage per RP and class
        SELECT allocations.resource_provider_id AS resource_provider_id,
        allocations.resource_class_id AS resource_class_id,
        sum(allocations.used) AS used
        FROM allocations
        WHERE allocations.resource_provider_id IN (%(resource_provider_id_1)s) AND
            allocations.resource_class_id IN (
                %(resource_class_id_1)s,
                %(resource_class_id_2)s,
                %(resource_class_id_3)s
            )
        -- Group rp_id and rP_class_id
        GROUP BY allocations.resource_provider_id, allocations.resource_class_id
    ) AS `usage`
        ON `usage`.resource_provider_id = inv.resource_provider_id AND
        `usage`.resource_class_id = inv.resource_class_id
-- Query resources of the specified id and class
WHERE rp.id IN (%(id_1)s) AND
    inv.resource_class_id IN (
        %(resource_class_id_4)s,
        %(resource_class_id_5)s,
        %(resource_class_id_6)s
    )

Copy the code

Schedule by fitlers

After nova-Scheduler obtains the allocation candidates, FilterScheduler is also used to calculate and filter the selected host (candidate) nodes based on the enabled filters and weights.

The following schedulers are implemented in Nova:

  1. FilterScheduler: A scheduler loaded by default that selects the best node based on the specified filter criteria and weight
  2. CachingScheduler: Similar to FilterScheduler, CachingScheduler caches host resource information to local memory for higher scheduling performance. The current master code is labeled as[DEPRECATED]
  3. ChanceScheduler: Random choice, true. But it’s also labeled in the master code[DEPRECATED]
  4. FakeScheduler: Used for testing, no practical function

But how does filter scheduler work?

Let’s start with the code again. Here’s a sequence diagram:

In the swimlanes of FilterScheduler, it can be seen that there are roughly three steps:

  1. Scheduler cache refresh and status update: Passnova.scheduler.host_manager.HostStateTo maintain a host state in memory and return visible compute node information
  2. Filtering: The utility configuration file specifies various filters to filter out hosts that do not meet the criteria. There are two configurations in the configuration fileavailale_filtersandenabled_filters, the former is used to specify all available filters, configured toavailable_filters=nova.scheduler.filters.all_filters; The latter indicates which of the available filters nova-Scheduler uses, as shown in the configurationenabled_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilterAnd so on. Nova supports up to 27 filters in version O, all of which are locatednova/scheduler/filtersUnder the directory, the system can process various information, such as the available resources of the host, parameters of the startup request (such as image information and request retry times), and affinity and anti-affinity of the virtual machine (whether the virtual machine is on the same host node as other virtual machines)
  3. Weighing and ranking all qualified hosts to determine the best host node. All Weigher implementations are locatednova/scheduler/weightsDirectories, such as DiskWeigher:
class DiskWeigher(weights.BaseHostWeigher):
    The maxval and minval attributes can be set to specify the maximum and minimum weights
    minval = 0
    When sorting, multiply each Weigher's weight by its corresponding Weigher
    The disk_weight_multiplier only makes sense if there are multiple weighers
    The default configuration file is 1.0
    def weight_multiplier(self):
        return CONF.filter_scheduler.disk_weight_multiplier
    Free_disk_mb = free_disk_MB = free_disk_MB
    def _weigh_object(self, host_state, weight_properties):
        """Higher weights win. We want spreading to be the default."""
        return host_state.free_disk_mb
Copy the code

Claim Resources

As mentioned above, after obtaining Allocation Candidates (candidate hosts that can be used for resource Allocation), filtering and weight calculation, Nova-Scheduler attempts to Claim resources. Before creating a virtual machine, test whether the available resources of the specified host can meet the requirements of creating a virtual machine. Let us look at the nova together. The scheduler. Utils. Claim_resources code:

def claim_resources(ctx, client, spec_obj, instance_uuid, alloc_req, allocation_request_version=None):.return client.claim_resources(ctx, instance_uuid, alloc_req, project_id,
            user_id, allocation_request_version=allocation_request_version)
Copy the code

In this method, the final call or incoming client claim_resources () method, namely the nova. Scheduler. Client. The report. The SchedulerReportClient# claim_resources:

    @safe_connect
    @retries
    def claim_resources(self, context, consumer_uuid, alloc_request, project_id, user_id, allocation_request_version=None):
        """Creates allocation records for the supplied instance UUID against the supplied resource providers. Param context: The security context: param consumer_uuid: The instance's UUID. :param alloc_request: The JSON body of the request to make to the placement's PUT /allocations API :param project_id: The project_id associated with the allocations. :param user_id: The user_id associated with the allocations. :param allocation_request_version: The microversion used to request the allocations. :returns: True if the allocations were created, False otherwise. """
        ar = copy.deepcopy(alloc_request)

        # If the allocation_request_version less than 1.12, then convert the
        # allocation array format to the dict format. This conversion can be
        # remove in Rocky release.
        if versionutils.convert_version_to_tuple(
                allocation_request_version) < (1.12):
            ar = {
                'allocations': {
                    alloc['resource_provider'] ['uuid'] : {'resources': alloc['resources']}for alloc in ar['allocations']
                }
            }
            allocation_request_version = '1.12'

        url = '/allocations/%s' % consumer_uuid

        payload = ar

        # We first need to determine if this is a move operation and if so
        # create the "doubled-up" allocation that exists for the duration of
        # the move operation against both the source and destination hosts
        r = self.get(url, global_request_id=context.global_id)
        if r.status_code == 200:
            current_allocs = r.json()['allocations']
            if current_allocs:
                payload = _move_operation_alloc_request(current_allocs, ar)

        payload['project_id'] = project_id
        payload['user_id'] = user_id
        r = self.put(url, payload, version=allocation_request_version,
                     global_request_id=context.global_id)
        ifr.status_code ! =204:
            # NOTE(jaypipes): Yes, it sucks doing string comparison like this
            # but we have no error codes, only error messages.
            if 'concurrently updated' in r.text:
                reason = ('another process changed the resource providers '
                          'involved in our attempt to put allocations for '
                          'consumer %s' % consumer_uuid)
                raise Retry('claim_resources', reason)
            else:
                LOG.warning(
                    'Unable to submit allocation for instance '
                    '%(uuid)s (%(code)i %(text)s)',
                    {'uuid': consumer_uuid,
                     'code': r.status_code,
                     'text': r.text})
        return r.status_code == 204
Copy the code

In this case, a PUT request is made to try to declare the required resource for consumer_id first and determine whether the resource was successfully declared based on the HTTP status code returned. Once you can successfully declare the required resources, you can figure out which host node to schedule the virtual machine to, and you can continue the process of creating the actual resources. The Placement API is done here. But for schedulers, there are resources to go to the consumer host to update information in memory, such as host state, and so on.

Current community Placement development

By subscribing to openstack-dev or attending nova weekly meeting, you can get timely updates on community trends and development progress. So for Nova Schedule Team, Jiang Yikun from Huawei has made a detailed record and arrangement of the progress of the current two months:

  • Nova Scheduler Team Meeting Tracking (January)
  • Nova Scheduler Team Meeting Tracking (February)

At this point, it looks like the scheduling team is still working hard to improve Placement features and make the Rocky version work.

What are the shortcomings?

At present, the deficiencies mainly focus on the bugs in use and the functions to be improved. For example, the Nested Resource Providers that are still being developed; Add a limit for obtaining Allocation candidates that controls the number of resource candidates to be fetched each time. For example, the host migration failure causes the occupation of both RP’s and so on. Things like separate Placement, that’s something the community is interested in doing.

reference

[1].Placement API

[2].Placement API Reference

[3].Yikun’s blog