Xml-rpc is a remote procedure call method that uses XML passed over HTTP as a carrier. With it, a client can call a server method with parameters (the server is named after a URI) on a remote server and get structured data. Python has its own XMLRPC implementation. Learning XMLRPC can help us quickly understand the implementation and principle of RPC. This article includes the following parts:

  • XMLRPC demo
  • xmlrpc-API
  • XMLRPC – server implementation
  • XMLRPC – client implementation
  • XMLRPC serialization/deserialization
  • summary

XMLRPC demo

XMLRPC can be run directly to start the RPC service:

# python3 -m xmlrpc.server Serving XML-RPC on localhost port 8000 It is advisable to run this example server within a secure, 127.0.0.1 - - [05/May/2021 18:03:16] "POST /RPC2 HTTP/1.1" 200-127.0.0.1 - - [05/May/2021 18:03:16] "POST /RPC2 HTTP/1.1" 200 -Copy the code

Start the RPC client:

# python3 -m xmlrpc.client
20210505T18:03:16
42
512
3
Copy the code

You can see two HTTP requests from the server. Let’s look at the first HTTP request packet captured:

METHOD: POST URL: http://localhost:8000/RPC2 HEADERS accept-encoding: gzip content-length: 120 content-type: Text/XML host: localhost:8000 user-agent: python-xmlrpc /3.8Copy the code

The requested data is:

<? The XML version = '1.0'? > <methodCall> <methodName> currentTime.getCurrentTime </methodName> <params> </params> </methodCall>Copy the code

HTTP response packet:

STATUS: 200 OK HEADERS content-length: 163 content-type: text/xml date: Wed, 05 May 2021 09:40:57 GMT server: Python / 3.8.5 BaseHTTP / 0.6Copy the code

The response data is:

<? The XML version = '1.0'? > <methodResponse> <params> <param> <value> <dateTime.iso8601> 20210505T18:03:16 </dateTime.iso8601> </value> </param> </params> </methodResponse>Copy the code

I won’t post the data packet for the second request, but here is the request XML:

<? The XML version = '1.0'? > <methodCall> <methodName> system.multicall </methodName> <params> <param> <value> <array> <data> <value> <struct> <member> <name> methodName </name> <value> <string> getData </string> </value> </member> <member> <name> params </name> <value> <array> <data> </data> </array> </value> </member> </struct> </value> <value> <struct> <member> <name> methodName </name> <value> <string> pow </string> </value> </member> <member> <name> params </name> <value> <array> <data> <value> <int> 2 </int> </value> <value> <int> 9 </int> </value> </data> </array> </value> </member> </struct> </value> <value> <struct> <member> <name> methodName </name> <value> <string> add </string> </value> </member> <member> <name> params </name> <value> <array> <data> <value> <int> 1 </int> </value> <value> <int> 2 </int> </value> </data> </array> </value> </member> </struct> </value> </data> </array> </value> </param> </params> </methodCall>Copy the code

Here is the response XML:

<? The XML version = '1.0'? > <methodResponse> <params> <param> <value> <array> <data> <value> <array> <data> <value> <string> 42 </string> </value>  </data> </array> </value> <value> <array> <data> <value> <int> 512 </int> </value> </data> </array> </value> <value> <array> <data> <value> <int> 3 </int> </value> </data> </array> </value> </data> </array> </value> </param> </params> </methodResponse>Copy the code

Note: In order to present the XMLRPC data in its entirety, I have posted the full XML, which is a bit long.

From the demo, you can see the following features of XMLRPC:

  • Data is transmitted using HTTP. usePOSTMethod, the URL isRPC2, is the content-typetext/xml.
  • Request/response is encoded using XML. Request to usemethodCallTag, used in responsemethodResponseThe label.
  • Nesting XML data with layers of redundancy can seem cumbersome (which is probably one reason XMLRPC didn’t catch on).

xmlrpc-API

Continuing with the XMLRPC API usage, the server code:

class ExampleService: def getData(self): return '42' class currentTime: @staticmethod def getCurrentTime(): return datetime.datetime.now() with SimpleXMLRPCServer(("localhost", 8000)) as server: server.register_function(pow) server.register_function(lambda x,y: x+y, 'add') server.register_instance(ExampleService(), allow_dotted_names=True) server.register_multicall_functions() print('Serving XML-RPC on localhost port 8000') print('It  is advisable to run this example server within a secure, closed network.') try: server.serve_forever() except KeyboardInterrupt: print("\nKeyboard interrupt received, exiting.") sys.exit(0)Copy the code

The server does the following:

  • Create an instance server of SimpleXMLRPCServer on port 8000
  • Registered to the serverpowAnd calledaddLambda function interface
  • Register a service instance(app is more appropriate) with the server. Instance takes two functions:getDataandcurrentTime.getCurrentTime
  • Register the System. Multicall implementation with the server, which supports the consolidation of multiple RPC requests using one HTTP request
  • Start the server

The client is used like this:

server = ServerProxy("http://localhost:8000") print(server.currentTime.getCurrentTime()) multi = MultiCall(server) Multi-.getdata () multi-.pow (2,9) multi-.add (1,2) try: for response in multi(): print(response) except Error as v: print("ERROR", v)Copy the code
  • Create a service proxy (rPC-client)
  • Invoke the implementation of the servercurrentTime.getCurrentTime
  • Called with MultiCallgetData.pow 和 addThree RPC interfaces
  • Send the multicall request and loop out the result of the service call

The difference between an XMLRPC service and an HTTP service is obvious:

  • RPC service interfaces are generic functions such as POW, getData, and getCurrentTime; These interfaces are isolated from HTTP request and Response
  • The client requires an additional implementation, not a direct HTTP request

At the same time, we should have an intuitive understanding of remote Procedure Call (RPC), a simple explanation is remote function call. By remote: remote across machines, and in our case, remote across processes. As for how to implement remote function calls, that is the function of each RPC framework, today we will first look at the implementation of XMLRPC.

XMLRPC – the realization of the server

Server HTTP protocol implementation

XMLRPC HTTP protocol by SimpleXMLRPCServer and SimpleXMLRPCRequestHandler implementation:

class SimpleXMLRPCServer(socketserver.TCPServer,
                         SimpleXMLRPCDispatcher):

    allow_reuse_address = True

    _send_traceback_header = False

    def __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler,
                 logRequests=True, allow_none=False, encoding=None,
                 bind_and_activate=True, use_builtin_types=False):
        self.logRequests = logRequests

        SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding, use_builtin_types)
        socketserver.TCPServer.__init__(self, addr, requestHandler, bind_and_activate)
Copy the code

SimpleXMLRPCServer’s parent class, TCPServer, described in a previous blog post, provides an implementation of TCP services. SimpleXMLRPCRequestHandler was responsible for part of the HTTP protocol implementation, and XMLRPC code must use a POST request to the/RPC2 focus on do_POST method:

class SimpleXMLRPCRequestHandler(BaseHTTPRequestHandler):
    # rpc-url
    rpc_paths = ('/', '/RPC2')
    
    def do_POST(self):
        ...
        max_chunk_size = 10*1024*1024
        size_remaining = int(self.headers["content-length"])
        L = []
        while size_remaining:
            chunk_size = min(size_remaining, max_chunk_size)
            chunk = self.rfile.read(chunk_size)
            if not chunk:
                break
            L.append(chunk)
            size_remaining -= len(L[-1])
        data = b''.join(L)
        ...
        response = self.server._marshaled_dispatch(
                    data, getattr(self, '_dispatch', None), self.path
                )
        ...
        self.send_response(200)
        self.send_header("Content-type", "text/xml")
        self.send_header("Content-length", str(len(response)))
        self.end_headers()
        self.wfile.write(response)
Copy the code

The do_POST method has three sections:

  • Reads request data from an HTTP request. The length of the data is determined by Content-Length
  • Call the RPC interface using the server’s _marshaled_dispatch method
  • Wrap the interface return value as an HTTP response return

Server RPC protocol implementation

Another parent of SimpleXMLRPCServer, SimpleXMLRPCDispatcher, provides an implementation of the RPC protocol:

class SimpleXMLRPCDispatcher: def __init__(self, allow_none=False, encoding=None, use_builtin_types=False): Self. funcs = {} # Service instance (app) self.instance = None self.allow_none = allow_none self.encoding = encoding or 'utf-8' self.use_builtin_types = use_builtin_types def register_instance(self, instance, allow_dotted_names=False): Self. instance = instance self.allow_dotted_names = allow_dotted_names def register_function(self.instance = instance self.allow_dotted_names = allow_dotted_names Function =None, name=None): name = function.__name__ self.funcs[name] = function return function def register_multicall_functions(self): Registers the XML-rpc multicall method in the system namespace. Registers the xmL-rpc multicall method in the system namespace. self.system_multicall})Copy the code

Instace and function are relatively easy to register, so we can skip the slightly more complicated implementation of multical and see how the registered interface is called in _marshaled_dispatch:

def _marshaled_dispatch(self, data, dispatch_method = None, path = None): try: Params, method = loads(data, loads) use_builtin_types=self.use_builtin_types) # generate response response = self._dispatch(method, Params) # wrap response in a singleton tuple response = (response,) # generate XML response = dumps(response, methodresponse=1, allow_none=self.allow_none, encoding=self.encoding) except Fault as fault: ... except: ... return response.encode(self.encoding, 'xmlcharrefreplace') def _dispatch(self, method, params): try: Func = self.funcs[method] except KeyError: pass else: If func is not None: return func(*params)... if self.instance is not None: if hasattr(self.instance, '_dispatch'): # call the `_dispatch` method on the instance return self.instance._dispatch(method, params) # call the instance's method directly try: func = resolve_dotted_attribute( self.instance, method, self.allow_dotted_names ) except AttributeError: pass else: if func is not None: return func(*params) ...Copy the code

The code is quite long, and it mainly does two things:

  • Parse params and Method from the request
  • Call a method from func or instance according to method and return it

Multi-call implementation on the server

Once you understand single-Call, it’s easy to look back at the implementation of multi-Call. Registration interface:

def register_multicall_functions(self):
    self.funcs.update({'system.multicall' : self.system_multicall})
Copy the code

The funcs dictionary adds a call to system.multicall and handles system_multicall:

def system_multicall(self, call_list): """system.multicall([{'methodName': 'add', 'params': [2, 2]}, ...] ) => [[4],...  Allows the caller to package multiple XML-RPC calls into a single request. See "" http://www.xmlrpc.com/discuss/msgReader$1208" results = [] # order execute multiple call for call in call_list: method_name = call['methodName'] params = call['params'] ... results.append([self._dispatch(method_name, params)]) ... return resultsCopy the code

System_multicall, like the annotation, takes multiple requests from a request and calls them one by one. Example of call data for system.multicall:

<methodName>
  system.multicall
</methodName>
<params>
    ...
              <member>
                <name>
                  methodName
                </name>
                <value>
                  <string>
                    getData
                  </string>
                </value>
              </member>
    ...
<params>
Copy the code

XMLRPC – client implementation

Client HTTP protocol implementation

Clients also need to implement HTTP protocol, mainly in ServerProxy and Transport (SafeTransport implements HTTPS). ServerProxy package Transport object:

class ServerProxy: def __init__(self, uri, transport=None, encoding=None, verbose=False, allow_none=False, use_datetime=False, use_builtin_types=False, *, headers=(), context=None): # get the url type, uri = urllib.parse._splittype(uri) self.__host, self.__handler = urllib.parse._splithost(uri) .. handler = Transport extra_kwargs = {} transport = handler(use_datetime=use_datetime, use_builtin_types=use_builtin_types, headers=headers, **extra_kwargs) self.__transport = transport ... def __request(self, methodname, params): Request = dumps(params, methodName, encoding=self.__encoding, allow_none=self.__allow_none).encode(self.__encoding, 'xmlcharrefreplace') response = self.__transport.request( self.__host, self.__handler, request, verbose=self.__verbose ) return responseCopy the code

Transport implements HTTP details:

class Transport:
    """Handles an HTTP transaction to an XML-RPC server."""
    def __init__(self, use_datetime=False, use_builtin_types=False,
                 *, headers=()):
        self._use_datetime = use_datetime
        self._use_builtin_types = use_builtin_types
        self._connection = (None, None)
        self._headers = list(headers)
        self._extra_headers = []
    
    def request(self, host, handler, request_body, verbose=False):
        http_conn = self.send_request(host, handler, request_body, verbose)
        resp = http_conn.getresponse()
        if resp.status == 200:
            self.verbose = verbose
            return self.parse_response(resp)
    
    def send_request(self, host, handler, request_body, debug):
        connection = self.make_connection(host)
        headers = self._headers + self._extra_headers
        ...
        connection.putrequest("POST", handler)
        headers.append(("Content-Type", "text/xml"))
        headers.append(("User-Agent", self.user_agent))
        self.send_headers(connection, headers)
        self.send_content(connection, request_body)
        return connection
    
    def make_connection(self, host):
        if self._connection and host == self._connection[0]:
            return self._connection[1]
        # create a HTTP connection object from a host descriptor
        chost, self._extra_headers, x509 = self.get_host_info(host)
        self._connection = host, http.client.HTTPConnection(chost)
        return self._connection[1]
    
    def parse_response(self, response):
        stream = response
        p, u = self.getparser()
        while 1:
            data = stream.read(1024)
            if not data:
                break
            if self.verbose:
                print("body:", repr(data))
            p.feed(data)

        if stream is not response:
            stream.close()
        p.close()

        return u.close()
Copy the code
  • Use http.client to create an HTTP connection
  • Send the HTTP request using send_request
  • Parse HTTP requests using parse_response

Client RPC protocol implementation

Wrapping requests with _Method over the HTTP protocol:

class ServerProxy:
    def __getattr__(self, name):
        # magic method dispatcher
        return _Method(self.__request, name)
        
class _Method:
    # some magic to bind an XML-RPC method to an RPC server.
    # supports "nested" methods (e.g. examples.getStateName)
    def __init__(self, send, name):
        self.__send = send
        self.__name = name
    def __getattr__(self, name):
        return _Method(self.__send, "%s.%s" % (self.__name, name))
    def __call__(self, *args):
        return self.__send(self.__name, args)
Copy the code

You can use the server. The currentTime. GetCurrentTime () sends the request, this is a chain calls. Server. currentTime calls serverProxy. __getattr__ to geta _Method object; __getAttr__ returns a _Method object, and finally uses getCurrentTime() to execute the call method of the method object, The ServerProxy call method is used to send the request.

Client multi-call implementation

Once you understand the client’s single-call implementation, move on to multi-Call, which covers the following three classes:

class _MultiCallMethod: def __init__(self, call_list, name): self.__call_list = call_list self.__name = name def __getattr__(self, name): return _MultiCallMethod(self.__call_list, "%s.%s" % (self.__name, name)) def __call__(self, *args): Self.__call_list.append ((self.__name, args)) # Add a call class to MultiCallIterator: def __init__(self, results): self.results = results def __getitem__(self, i): item = self.results[i] if type(item) == type({}): raise Fault(item['faultCode'], item['faultString']) elif type(item) == type([]): return item[0] else: raise ValueError("unexpected type in multicall result") class MultiCall: def __init__(self, server): self.__server = server self.__call_list = [] def __getattr__(self, name): return _MultiCallMethod(self.__call_list, name) def __call__(self): marshalled_list = [] for name, args in self.__call_list: marshalled_list.append({'methodName' : name, 'params' : Multical return MultiCallIterator(self.__server.system.multicall(marshalled_list))Copy the code

Call, getattr, and getitem; call, getattr, and getitem;

Multi = MultiCall(server) multi-.getdata () multi-.pow (2,9) multi-.add (1,2) for response in multi(): print(response)Copy the code

XMLRPC serialization/deserialization

RPC services need to be transferred across the network, and data between the server and client needs to be serialized/deserialized. Mainly implemented by Marshaller and Unmarshaller classes:

# client.py

class Marshaller:
    ...

class Unmarshaller:
    ...
Copy the code

XMLRPC supports the following nine data types:

  • array
  • base64
  • boolean
  • date/time
  • double
  • integer
  • string
  • struct
  • nil

Some data types, such as double and nil, do not exist in Python. The encoding/decoding of these two types of data is as follows:

class Marshaller:
    
    def dump_double(self, value, write):
        write("<value><double>")
        write(repr(value))
        write("</double></value>\n")
    dispatch[float] = dump_double
    
    def dump_nil (self, value, write):
        if not self.allow_none:
            raise TypeError("cannot marshal None unless allow_none is enabled")
        write("<value><nil/></value>")
    dispatch[type(None)] = dump_nil
    

class Unmarshaller:
    
    def end_double(self, data):
        self.append(float(data)) # float
        self._value = 0
    dispatch["double"] = end_double
    dispatch["float"] = end_double
    
    def end_nil (self, data):
        self.append(None) # None
        self._value = 0
    dispatch["nil"] = end_nil
Copy the code

summary

In the absence of TCP, XMLRPC is primarily a two-tier model with HTTP at the bottom and XMLRPC at the top. HTTP protocol is responsible for network transmission; The XMLRPC protocol is responsible for converting RPC requests into XML data and then deserializing them into request execution.

Refer to the link

  • En.wikipedia.org/wiki/XML-RP…