An overview of the

We are learning Kubernetes in addition to understand its related concepts, we also need to in-depth understanding of the entire Kubernetes implementation mechanism is how, if you can also understand its source code, that basically we are very familiar with Kubernetes. I will explain the entire implementation mechanism of Kubernetes in terms of how Kubernetes generates a Deployment resource, along with a source code interpretation.

Previous article

  • Kubernetes apply command execution process source code parsing – overview
  • Kubectl perform apply command source code parsing

This article will explain how kubectl executes a command and sends it to Apiserver. How apiserver is going to handle that.

Apiserver component description

Kube-apiserver as the whole Kubernetes cluster operation etCD only entrance, responsible for Kubernetes resources authentication & authentication, verification and CRUD operations, provide RESTful APIs for other components to call. The call topology is as follows:

The interface is roughly composed as shown in the following figure:

Apiserver a Deployment resource interface for each field represents the meaning as shown below:

Processing flow

Before we understand how apiserver is called, we still need to have a brief understanding of the startup process of Apiserver.

Start the process

  1. Set parameters.
  2. Check whether the configuration is valid
  3. Execute the final Run method. Essentially configuring routing, access rights, and interaction with the database (ETCD)

The main logic of the Run method is as follows:

  1. CreateServerChain is called to establish a service invocation chain and determine whether to start an insecure HTTP server. The HTTP Server chain contains three servers to be started by Apiserver and routes to the corresponding resources for each server are registered.

  2. Prepare for service run by calling Server.Preparerun, which does health checks, survival checks, and registration of OpenAPI routes;

  3. Call prepared.Run to start the HTTPS server.

Handle the request

Here is a complete flowchart for the request:

  1. Request access control, authentication, authentication, access control, etc
  2. Routing distribution
  3. Database operations

The data structure

  • Hanler, the object that the actual Web processes the request
  • Route, the logical object for actual Route distribution
  • A collection of Web Services and routes
  • Container: a collection of WebServices
  • RESTStorage, which converts the reset request into an abstraction of the storage method

Rely on the module

Many of the methods rely on k8s apiserver. IO/apiserver/PKG/apis/apiserver modules.

The core Web-server module needs to rely on the Go-Restfule module.

GenericAPIServer

Go – restful components

Route

There are two types of routes: RouterJSR311, an implementation of the standard JSR311 interface specification, and CurlyRouter. CurlyRouter supports regular expressions and dynamic parameters, and is lighter than RouterJSR311. Apiserver uses this route. The setting of a Route includes: Http Method, URL Path, processing Method and optional content-Type, Accept and so on.

WebService

WebService is a logical collection of routes, and it is mainly used to uniformly set some common attributes for a group of routes, including root path and data type of request and response. Note that the WebService must be added to the Container to take effect.

Container

A Container is a logical collection of Web Services and can implement multi-terminal functions. For example, the following code creates two Containers that provide services on different ports.

Filter

The Filter is used to dynamically intercept requests and responses. It is similar to the hook placed in front of the corresponding component to capture requests or responses before the function of the corresponding component is run. It is mainly used for logging, authentication, redirection and other functions. There are three types of Go-restful filters

There are three main types of filters

  • Container Filter
  • WebService Filter
  • Route Filter

Source code analysis

First a brief understanding of the apiserver startup process source code analysis.

Start the process

Kubernetes. CMD. Apiserver. Apiserver. Go give priority to function, the whole call link is as follows:

app.NewAPIServerCommand –> Run –> CreateServerChain

The key here is to construct a server.

func CreateServerChain(completedOptions completedServerRunOptions, stopCh <-chan struct{}) (*aggregatorapiserver.APIAggregator, error){...// Initialize kubeapiserver configuration
  kubeAPIServerConfig, serviceResolver, pluginInitializer, err := CreateKubeAPIServerConfig(completedOptions, nodeTunneler, proxyTransport)
	...
	// Register the CustomResourceDefinition (CRD) apiResources and apiVersions, and handle CRD and CustomResource (CR) REST requests
	apiExtensionsServer, err := createAPIExtensionsServer(apiExtensionsConfig, genericapiserver.NewEmptyDelegate())
	iferr ! =nil {
		return nil, err
	}
  // Management of core interface services, responsible for some common processing of requests
	kubeAPIServer, err := CreateKubeAPIServer(kubeAPIServerConfig, apiExtensionsServer.GenericAPIServer)
	iferr ! =nil {
		return nil, err
	}
	...
  // Process APIService resource requests under the 'apiregistration.k8s. IO' group, and aggregated requests from users to the aggregated Server (AA)
	aggregatorServer, err := createAggregatorServer(aggregatorConfig, kubeAPIServer.GenericAPIServer, apiExtensionsServer.Informers)
	iferr ! =nil {
		// we don't need special handling for innerStopCh because the aggregator server doesn't create any go routines
		return nil, err
	}

	return aggregatorServer, nil
}

Copy the code

Kube-apiserver contains three types of Apiserver:

  • AggregatorServer: Processes APIService resource requests under the apiregistration.k8s. IO group, and aggregated requests from users are forwarded to aggregated Server (AA). The following controlller services are also triggered:

    • apiserviceRegistrationController: Responsible for resource registration and deletion in APIServices;
    • availableConditionController: Maintain the availability of APIServices, including the availability of its reference Service.
    • autoRegistrationController: a specific set of APIServices used to maintain an API presence;
    • crdRegistrationController: Responsible for automatically registering CRD GroupVersions with APIServices;
    • openAPIAggregationController: Synchronize changes to APIServices resources to the provided OpenAPI documentation;

  • KubeAPIServer: Responsible for some general handling of requests, including authentication, authentication, and REST services for various built-in resources (POD, Deployment, Service and etc.)

  • ApiExtensionsServer: Register the CustomResourceDefinition (CRD) apiResources and apiVersions, handle CRD and CustomResource (CR) REST requests (404 will be returned if CR cannot be processed), And the last part of Apiserver Delegation

    When kubectl apply -f deploy-nginx.yaml is executed and the request resource object is deployment, which Server will process the request? Let’s move on.

The request processing

The code of ApiServer is relatively complex, so I will focus on sorting out the whole link of the request processing process. The whole link is analyzed in three stages: access control, route distribution and data processing. Container > WebService > Route > Handler is the basic approach when analyzing Web services. Each request processing core logic resides in handler. Based on this idea, we go to see how the source code is implemented.

Access control

Before processing with Apiserver, a series of filtering operations are performed through access control.

There are three mechanisms related to permissions in apiServer, namely, common authentication, authentication, and access control. Apiserver mainly provides reST-style interfaces, so all kinds of permissions are ultimately concentrated on the authority judgment of the interface. In the most core kubeAPIServerConfig, for example, in CreateServerChain method, call the CreateKubeAPIServerConfig method, this method is the main role is to create kubeAPIServer configuration. We go into this method, we call buildGenericConfig to create some generic configuration, and under NewConfig, we return DefaultBuildHandlerChain, which is basically used to chain the Apiserver REST interface, Commonly known as the filter operation.

The invocation path for createAggregatorConfig – > genericapiserver. BuildHandlerChainWithStorageVersionPrecondition – > DefaultBuildHandlerCh ain

// This is the core method for requesting access control
func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
	handler := filterlatency.TrackCompleted(apiHandler)
	handler = genericapifilters.WithAuthorization(handler, c.Authorization.Authorizer, c.Serializer)
	handler = filterlatency.TrackStarted(handler, "authorization")

	ifc.FlowControl ! =nil {
		handler = filterlatency.TrackCompleted(handler)
		handler = genericfilters.WithPriorityAndFairness(handler, c.LongRunningFunc, c.FlowControl)
		handler = filterlatency.TrackStarted(handler, "priorityandfairness")}else {
		handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc)
	}

	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
	handler = filterlatency.TrackStarted(handler, "impersonation")

	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
	handler = filterlatency.TrackStarted(handler, "audit")

	failedHandler := genericapifilters.Unauthorized(c.Serializer)
	failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)

	failedHandler = filterlatency.TrackCompleted(failedHandler)
	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
	handler = filterlatency.TrackStarted(handler, "authentication")

	handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil.nil.nil."true")

	// WithTimeoutForNonLongRunningRequests will call the rest of the request handling in a go-routine with the
	// context with deadline. The go-routine can keep running, while the timeout logic will return a timeout to the client.
	handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc)

	handler = genericapifilters.WithRequestDeadline(handler, c.AuditBackend, c.AuditPolicyChecker,
		c.LongRunningFunc, c.Serializer, c.RequestTimeout)
	handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
	handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
	ifc.SecureServing ! =nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 {
		handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance)
	}
	handler = genericapifilters.WithAuditAnnotations(handler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithWarningRecorder(handler)
	handler = genericapifilters.WithCacheControl(handler)
	handler = genericfilters.WithHSTS(handler, c.HSTSDirectives)
	handler = genericfilters.WithHTTPLogging(handler)
	handler = genericapifilters.WithRequestReceivedTimestamp(handler)
	handler = genericfilters.WithPanicRecovery(handler, c.RequestInfoResolver)
	handler = genericapifilters.WithAuditID(handler)
	return handler
}
Copy the code
RBAC

Perhaps the most important of Kubernetes is RBAC. Within the DefaultBuildHandlerChain method, by calling the genericapifilters. WithAuthorization method, realizes the permissions for each interface filter operation. The WithAuthorization method is as follows

// WithAuthorizationCheck passes all authorized requests on to handler, and returns a forbidden error otherwise.
func WithAuthorization(handler http.Handler, a authorizer.Authorizer, s runtime.NegotiatedSerializer) http.Handler {
   if a == nil {
      klog.Warning("Authorization is disabled")
      return handler
   }
   return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
      ctx := req.Context()
      ae := request.AuditEventFrom(ctx)

      attributes, err := GetAuthorizerAttributes(ctx)
      iferr ! =nil {
         responsewriters.InternalError(w, req, err)
         return
      }
      authorized, reason, err := a.Authorize(ctx, attributes)
      // an authorizer like RBAC could encounter evaluation errors and still allow the request, so authorizer decision is checked before error here.
      if authorized == authorizer.DecisionAllow {
         audit.LogAnnotation(ae, decisionAnnotationKey, decisionAllow)
         audit.LogAnnotation(ae, reasonAnnotationKey, reason)
         handler.ServeHTTP(w, req)
         return
      }
      iferr ! =nil {
         audit.LogAnnotation(ae, reasonAnnotationKey, reasonError)
         responsewriters.InternalError(w, req, err)
         return
      }

      klog.V(4).InfoS("Forbidden"."URI", req.RequestURI, "Reason", reason)
      audit.LogAnnotation(ae, decisionAnnotationKey, decisionForbid)
      audit.LogAnnotation(ae, reasonAnnotationKey, reason)
      responsewriters.Forbidden(ctx, attributes, w, req, reason, s)
   })
}
Copy the code
  1. Call the GetAuthorizerAttributes method to get the values for the various attributes of the configuration
  2. Invoke the Authorize method to determine whether the authorization is passed. Different authorization interfaces are realized to complete the authentication task.
  3. If the authentication succeeds, the handler.ServeHTTP method is invoked to continue the filter operation. Otherwise, an error message is returned.

In RBAC, for example, the Authorize judging method call VisitRulesFor method finally realize permissions, method in kubernetes/PKG/registry/RBAC/validation/rule. Go inside the file.

Routing distribution

If Apiserver distributes routes, it is essentially to figure out how Apiserver maintains the mapping between route and handler. With this in mind, we can look at it from a different Angle, and first look at how routes are registered. Based on the previous server knowledge, it is assumed that the resources in Deployment should be handled by KubeApiServer. Let’s continue to analyze how the route to the Server is registered. The entire function call chain is as follows:

CreateKubeAPIServer --> kubeAPIServerConfig.Complete().New(delegateAPIServer) -->  m.InstallAPIs --> m.GenericAPIServer.InstallAPIGroups -->  s.installAPIResources --> apiGroupVersion.InstallREST --> installer.Install --> a.registerResourceHandlers
Copy the code
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Instance, error){...// install legacy rest storage
	if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
		legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
			StorageFactory:              c.ExtraConfig.StorageFactory,
			ProxyTransport:              c.ExtraConfig.ProxyTransport,
			KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
			EventTTL:                    c.ExtraConfig.EventTTL,
			ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
			SecondaryServiceIPRange:     c.ExtraConfig.SecondaryServiceIPRange,
			ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
			LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
			ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
			ExtendExpiration:            c.ExtraConfig.ExtendExpiration,
			ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
			APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
		}
    // Add the core API Resources to the route. In apiserver, Resources start with/API
		iferr := m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider); err ! =nil {
			return nil, err
		}
	}
  ...
  // Add the extended API Resources to the route. In apiserver, Resources start with /apis;
	iferr := m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...) ; err ! =nil {
		return nil, err
	}
  ...
}
Copy the code

From what we learned earlier, deploymen’s request starts with /apis, so let’s focus on InstallAPIs, as follows:

// InstallAPIs will install the APIs for the restStorageProviders if they are enabled.
func (m *Instance) InstallAPIs(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter, restStorageProviders ... RESTStorageProvider) error{...for _, restStorageBuilder := range restStorageProviders {
		...
		apiGroupInfo, enabled, err := restStorageBuilder.NewRESTStorage(apiResourceConfigSource, restOptionsGetter)
		...
		apiGroupsInfo = append(apiGroupsInfo, &apiGroupInfo)
	}
  // Service GenericAPIServer configuration is constructed by apiGroupsInfo
	iferr := m.GenericAPIServer.InstallAPIGroups(apiGroupsInfo...) ; err ! =nil {
		return fmt.Errorf("error in registering group versions: %v", err)
	}
	return nil
}
Copy the code

Continue to look at the InstallAPIGroups method

// Exposes given api groups in the API.
func (s *GenericAPIServer) InstallAPIGroups(apiGroupInfos ... *APIGroupInfo) error{...for _, apiGroupInfo := range apiGroupInfos {
   // The key function to construct container
   	iferr := s.installAPIResources(APIGroupPrefix, apiGroupInfo, openAPIModels); err ! =nil {
   		return fmt.Errorf("unable to install api resources: %v", err)
   	}
   	// setup discovery
   	// Install the version handler.
   	// Add a handler at /apis/<groupName> to enumerate all versions supported by this group.
   	apiVersionsForDiscovery := []metav1.GroupVersionForDiscovery{}
   	for _, groupVersion := range apiGroupInfo.PrioritizedVersions {
   		// Check the config to make sure that we elide versions that don't have any resources
   		if len(apiGroupInfo.VersionedResourcesStorageMap[groupVersion.Version]) == 0 {
   			continue
   		}
   		apiVersionsForDiscovery = append(apiVersionsForDiscovery, metav1.GroupVersionForDiscovery{
   			GroupVersion: groupVersion.String(),
   			Version:      groupVersion.Version,
   		})
   	}
   	preferredVersionForDiscovery := metav1.GroupVersionForDiscovery{
   		GroupVersion: apiGroupInfo.PrioritizedVersions[0].String(),
   		Version:      apiGroupInfo.PrioritizedVersions[0].Version,
   	}
   	apiGroup := metav1.APIGroup{
   		Name:             apiGroupInfo.PrioritizedVersions[0].Group,
   		Versions:         apiVersionsForDiscovery,
   		PreferredVersion: preferredVersionForDiscovery,
   	}
   	s.DiscoveryGroupManager.AddGroup(apiGroup)
   // Add webService configuration to GenericAPIServer
   	s.Handler.GoRestfulContainer.Add(discovery.NewAPIGroupHandler(s.Serializer, apiGroup).WebService())
   }
   return nil
}
Copy the code
/ installAPIResources is a private method for installing the REST storage backing each api groupversionresource
func (s *GenericAPIServer) installAPIResources(apiPrefix string, apiGroupInfo *APIGroupInfo, openAPIModels openapiproto.Models) error {
	var resourceInfos []*storageversion.ResourceInfo
	for _, groupVersion := range apiGroupInfo.PrioritizedVersions {
	  ...
    // This will be constructed again
		r, err := apiGroupVersion.InstallREST(s.Handler.GoRestfulContainer)
		iferr ! =nil {
			return fmt.Errorf("unable to setup API %v: %v", apiGroupInfo, err)
		}
		resourceInfos = append(resourceInfos, r...) }...return nil
}
Copy the code
// InstallREST registers the REST handlers (storage, watch, proxy and redirect) into a restful Container.
// It is expected that the provided path root prefix will serve all operations. Root MUST NOT end
// in a slash.
func (g *APIGroupVersion) InstallREST(container *restful.Container) ([]*storageversion.ResourceInfo, error) {
	prefix := path.Join(g.Root, g.GroupVersion.Group, g.GroupVersion.Version)
	installer := &APIInstaller{
		group:             g,
		prefix:            prefix,
		minRequestTimeout: g.MinRequestTimeout,
	}
	// Build API resources are generated here
	apiResources, resourceInfos, ws, registrationErrors := installer.Install()
	versionDiscoveryHandler := discovery.NewAPIVersionHandler(g.Serializer, g.GroupVersion, staticLister{apiResources})
	versionDiscoveryHandler.AddToWebService(ws)
  // Add the resource to the container
	container.Add(ws)
	return removeNonPersistedResources(resourceInfos), utilerrors.NewAggregate(registrationErrors)
}
Copy the code
// Install handlers for API resources.
func (a *APIInstaller) Install(a) ([]metav1.APIResource, []*storageversion.ResourceInfo, *restful.WebService, []error) {
	var apiResources []metav1.APIResource
	var resourceInfos []*storageversion.ResourceInfo
	var errors []error
  // Generate a WebService
	ws := a.newWebService()
	// Register the paths in a deterministic (sorted) order to get a deterministic swagger spec.
	paths := make([]string.len(a.group.Storage))
	var i int = 0
	for path := range a.group.Storage {
		paths[i] = path
		i++
	}
	sort.Strings(paths)
	for _, path := range paths {
    // Register path with webService
		apiResource, resourceInfo, err := a.registerResourceHandlers(path, a.group.Storage[path], ws)
		iferr ! =nil {
			errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err))
		}
		ifapiResource ! =nil {
			apiResources = append(apiResources, *apiResource)
		}
		ifresourceInfo ! =nil {
			resourceInfos = append(resourceInfos, resourceInfo)
		}
	}
	return apiResources, resourceInfos, ws, errors
}
Copy the code

Because of the complexity of the registerResourceHandlers, we will only look at the POST request resource deployment.

Code address: k8s. IO/kubernetes/PKG/endpoints/installer. Go

		case "POST": // Create a resource.
			var handler restful.RouteFunction
			if isNamedCreater {
        // This will eventually construct a handler,admit is the admission control, and this will eventually go to createHandle
				handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
			} else {
				handler = restfulCreateResource(creater, reqScope, admit)
			}
			handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, deprecated, removedRelease, handler)
			handler = utilwarning.AddWarningsHandler(handler, warnings)
			article := GetArticleForNoun(kind, "")
			doc := "create" + article + kind
			if isSubresource {
				doc = "create " + subresource + " of" + article + kind
			}
      // Create a route
			route := ws.POST(action.Path).To(handler).
				Doc(doc).
				Param(ws.QueryParameter("pretty"."If 'true', then the output is pretty printed.")).
				Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
				Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...) ...). . Returns(http.StatusOK,"OK", producedObject).
				// TODO: in some cases, the API may return a v1.Status instead of the versioned object
				// but currently go-restful can't handle multiple different objects being returned.
				Returns(http.StatusCreated, "Created", producedObject).
				Returns(http.StatusAccepted, "Accepted", producedObject).
				Reads(defaultVersionedObject).
				Writes(producedObject)
			iferr := AddObjectParams(ws, route, versionedCreateOptions); err ! =nil {
				return nil.nil, err
			}
			addParams(route, action.Params)
			routes = append(routes, route)
Copy the code

The above c. genericConfig. New and installer.Install constitute a request link. When a request comes, the whole Apiserver request link is as follows:

filters(DefaultBuildHandlerChain) => installAPI(/|/metrics|/debug|/version) | GenericAPIServer.installAPIResources(/api/v1) => APIGroupVersion.InstallREST(/api/v1/namespaces/{namespace}/pods/{name})
Copy the code

The entire construction process is viewed from the point of view of the code call chain, which shows that to construct a GenericAPIServer service instance, the container, WebService, route will be constructed based on the information of the GenericAPIServer step by step. And path information by RESTStorageProvider converted, the handler is restfulCreateNamedResource function structure, will exist GenericAPIServer handler instance

The data processing

We all know that ApiServer’s interaction with a database generally refers to its interaction with ETCD. All components of Kubernetes do not interact with ETCD directly, but request Apiserver, and APiserver interacts with ETCD to complete the final drop of data. As mentioned in the previous routing implementation, apiserver’s final implementation of the handler’s back-end data is stored as a Store structure. How to generate the routing data for this request if post a Deployment. We can actually see a NewRESTStorag method in the InstallAPIs method code above. This is used to create Storage. The providers of these storages are RestStorageProviders. Deployment of storage is PKG/registry/apps/deployment/storage/storage. Go

The deployment is storagerest. RESTStorageProvider {}.

pkd/registry/apps/rest/storage_apps.go

func (p StorageProvider) NewRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (genericapiserver.APIGroupInfo, bool, error) {
	apiGroupInfo := genericapiserver.NewDefaultAPIGroupInfo(apps.GroupName, legacyscheme.Scheme, legacyscheme.ParameterCodec, legacyscheme.Codecs)
	// If you add a version here, be sure to add an entry in `k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go with specific priorities.
	// TODO refactor the plumbing to provide the information in the APIGroupInfo

	if apiResourceConfigSource.VersionEnabled(appsapiv1.SchemeGroupVersion) {
    // Generate storagemap, including generate Deployment,daemonset, etc
		storageMap, err := p.v1Storage(apiResourceConfigSource, restOptionsGetter)
		iferr ! =nil {
			return genericapiserver.APIGroupInfo{}, false, err
		}
		apiGroupInfo.VersionedResourcesStorageMap[appsapiv1.SchemeGroupVersion.Version] = storageMap
	}

	return apiGroupInfo, true.nil
}
Copy the code
func (p StorageProvider) v1Storage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (map[string]rest.Storage, error) {
	storage := map[string]rest.Storage{}

	// deployments
  // Finally found the deployment resource
	deploymentStorage, err := deploymentstore.NewStorage(restOptionsGetter)
	iferr ! =nil {
		return storage, err
	}
	storage["deployments"] = deploymentStorage.Deployment
	storage["deployments/status"] = deploymentStorage.Status
	storage["deployments/scale"] = deploymentStorage.Scale

	...
	return storage, nil
}
Copy the code
func NewREST(optsGetter generic.RESTOptionsGetter) (*REST, *StatusREST, *RollbackREST, error) {
	store := &genericregistry.Store{
		NewFunc:                  func(a) runtime.Object { return &apps.Deployment{} },
		NewListFunc:              func(a) runtime.Object { return &apps.DeploymentList{} },
		DefaultQualifiedResource: apps.Resource("deployments"),

		CreateStrategy:      deployment.Strategy,
		UpdateStrategy:      deployment.Strategy,
		DeleteStrategy:      deployment.Strategy,
		ResetFieldsStrategy: deployment.Strategy,

		TableConvertor: printerstorage.TableConvertor{TableGenerator: printers.NewTableGenerator().With(printersinternal.AddHandlers)},
	}
	options := &generic.StoreOptions{RESTOptions: optsGetter}
  // All information will be saved
	iferr := store.CompleteWithOptions(options); err ! =nil {
		return nil.nil.nil. err } statusStore := *store statusStore.UpdateStrategy = deployment.StatusStrategy statusStore.ResetFieldsStrategy = deployment.StatusStrategyreturn &REST{store, []string{"all"}}, &StatusREST{store: &statusStore}, &RollbackREST{store: store}, nil
}

Copy the code

Store.Com pleteWithOptions (options) will call options. RESTOptions. GetRESTOptions initialization configuration. Options. RESTOptions is an interface, and to find the implementation of its GetRESTOptions method, you must know the corresponding instance when options.RESTOptions is initialized. The initialization is in CreateKubeAPIServerConfig — > buildGenericConfig – > s.E TCD. The initialized ApplyWithStorageFactoryTo method, For StorageFactoryRestOptionsFactory RESTOptions corresponding instance. One is that RestStorage is an abstraction between apiserver and database delivery. In addition, we use the ETCD database by default.

RESTStorage

KubeAPIServer creates a RESTStorage for each API resource. The purpose of the RESTStorage is to match the access path of each resource with the operations of the back-end Storage. The RESTStorage interface is constructed to determine which operations the resource can perform (for example: Each operation corresponds to a standard REST method. For example, create corresponds to THE REST Method POST, and Update corresponds to the REST Method PUT. Finally, a handler is added for each operation according to the actions array, and the handler is registered with the Route to provide RESTful apis.

Etcd interaction details

k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server/options/etcd.go:253

  • Read the request content: body, err: = limitedReadBody (the req, scope. MaxRequestBodyBytes)
  • Decode: obj, GVK, err := decoder. decode (body, &defaultGVK, original)
  • The body OBj after decode is admitted
  • Execute the requestFunc (r.state) function, also known as store.create, which checks the validity of obj
  • Finally, call the e.storage. Create function in store. Create to perform the e.storage. Create operation

Code module cleanup

  • Apiserver overall startup logic
    • k8s.io/kubernetes/cmd/kube-apiserver
  • Apiserver Bootstrap-Controller creates & runs logic
    • k8s.io/kubernetes/pkg/master
  • The API Resource is created for the back-end RESTStorage(based on GenericRegistry.Store)
    • k8s.io/kubernetes/pkg/registry
  • Aggregated -apiserver creates & processes logic
    • k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator
  • Extensions -apiserver create & process logic
    • k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver
  • Apiserver creates & runs
    • k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server
  • Register the API Resource section
    • handler(InstallREST&Install&registerResourceHandlers)
    • k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints
  • Deployment, deamonset statefulset storage implementation of the related resources
    • pkd/registry/apps/rest/storage_apps.go
  • Creating a Storage back end (etcDV3)
    • k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage
  • Genericregistry.Store.Com pleteWithOptions initialization
    • k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry

summary

The RESTFul API implementation process in the startup process:

Kube-apiserver contains three servers, namely KubeAPIServer, APIExtensionsServer and AggregatorServer. The three servers are connected together through the delegate mode. The initialization process is similar. First create a config for each server and then initialize the HTTP Server. First initialize GoRestfulContainer and then install the API contained in the SERVER. When installing the API, create a back-end storage named RESTStorage for each API Resource, add a corresponding Handler to verbs supported by each API Resource, and register the handler with route. Finally, register the route with the WebService.

Kubectl apply -f nginx-deploy.yaml’s request is processed at this point, apiserver just stores the request to etCD. We’ll see what Kubernetes does with etCD data in the next article.

conclusion

In kubernetes Paoding Niu series, the source code of Apiserver is still more difficult to read. Because apiserver needs to deal with more API, the design of routing distribution is still more complex, which requires patience. There are bound to be some loose parts in the article, and I hope you will bear with me and absorb the essence (if any) and discard the dregs. If you are interested, you can follow me. My wechat account is Lcomedy2021

Reference documentation

Core principle: blog.csdn.net/huwh_/artic…

Call link analysis: github.com/duyanghao/k…

Source code analysis: blog.csdn.net/cbmljs/arti…

Kubernetes source analysis: www.bookstack.cn/read/source…