How to delete a Harbor image quickly

Wechat official account: Operation and maintenance development story, author: Hua Zai

background

During a recent inspection, the harbor storage space usage reached 80%. So, look at the number of mirror labels under each category. It was found that there were hundreds of mirrored labels under individual projects. Upon closer inquiry, I learned that the project is currently in the debugging phase, and it is being debugged many times a day. Since storage space is running low, delete the previous image tags on Harbor and keep the most recent ones. In the process of manual deletion, hundreds were found, and only 10 were displayed per page. I had to sort by push time, then delete page by page. I thought I’d go through this once and never do it again. Later, think carefully, this is also not easy to control, each inspection found to manually delete too much trouble. So I’m going to write a script, each time through the script to delete the image label, keep the most recent few good. I happen to be learning Golang recently, so just write with it. Embarrassingly, after I finished writing the script and tested it well, I found that the new version of Harbor could already set the retention policy on the UI. Comfort yourself and think of it as an exercise, a trial run!

Github address: github.com/sunsharing-…

The target

The command line enables you to query all current projects, whether public or not, and the number of warehouses
On the cli, you can query the warehouse name, image name, and pull times of a project
On the cli, you can delete mirror labels by specifying labels and the number of reserved labels
The number of labels that can obtain the mirror
After being deleted, garbage cannot be removed immediately. You need to manually remove the garbage (considering that the image cannot be pushed or pulled during garbage removal).

The statement

This script is a personal exercise and does not constitute any advice
I just learned Golang to achieve the goal. The code quality is very poor. Please understand
The Harbor in use is V2.3.1
Please go to Github for all code

implementation

Get all items in Harbor, API available via Harbor’s Swagger

// Define the data structure to obtain according to harbor Swagger test results
type MetaData struct {
	Public string `json:"public"`
}
type ProjectData struct {
	MetaData MetaData `json:"metadata"`
	ProjectId int `json:"project_id"`
	Name string `json:"name"`
	RepoCount int `json:"repo_count"`
}

type PData []ProjectData
// Provide harbor address to get project
func GetProject(url string) []map[string]string {
	/ / define the url
	url = url + "/ API/v2.0 projects"
	//url = url + "/api/projects"
    // Construct the request
	request, _ := http.NewRequest(http.MethodGet, url,nil)
    // Unvalidate
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true}},// Define the client
	client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
	//client := &http.Client{Timeout: 10 * time.Second}
	request.Header.Set("accept"."application/json")
    // Set the user and password
	request.SetBasicAuth("admin"."Harbor12345")
	response, err := client.Do(request)

	iferr ! =nil {
		fmt.Println("excute failed")
		fmt.Println(err)
	}
	/ / for the body
	body, _ := ioutil.ReadAll(response.Body)
	defer response.Body.Close()
	ret := PData{}
	json.Unmarshal([]byte(string(body)), &ret)
	var ps = []map[string]string{}
    // Get the returned data
	for i := 0; i < len(ret); i++ {
		RData := make(map[string]string)
		RData["name"] = (ret[i].Name)
		RData["project_id"] = strconv.Itoa(ret[i].ProjectId)
		RData["repo_count"] =strconv.Itoa(ret[i].RepoCount)
		RData["public"] = ret[i].MetaData.Public

		ps = append(ps, RData)
	}
	return ps
}
Copy the code

Gets the REPO under the project

// Define the data structure to fetch
type ReposiData struct {
	Id int `json:"id"`
	Name string `json:"name"`
	ProjectId int `json:"project_id"`
	PullCount int `json:"pull_count"`
}

type RepoData []ReposiData
// Get the repO under the project by providing the Harbor address and the corresponding project
func GetRepoData(url string, proj string)  []map[string]string {
	/ / / API/v2.0 / projects/goharbor/repositories
	url = url + "/ API/v2.0 / projects/" + proj +  "/repositories"
    // Construct the request
	request, _ := http.NewRequest(http.MethodGet, url,nil)
    // Ignore authentication
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
	request.Header.Set("accept"."application/json")
    // Set the user name and password
	request.SetBasicAuth("admin"."Harbor12345")
	response, err := client.Do(request)

	iferr ! =nil {
		fmt.Println("excute failed")
		fmt.Println(err)
	}
	/ / for the body
	body, _ := ioutil.ReadAll(response.Body)
	defer response.Body.Close()
	ret := RepoData{}
	json.Unmarshal([]byte(string(body)), &ret)
	var ps = []map[string]string{}
    // Get the returned data
	for i := 0; i < len(ret); i++ {
		RData := make(map[string]string)
		RData["name"] = (ret[i].Name)
		pId := strconv.Itoa(ret[i].ProjectId)
		RData["project_id"] = pId
		RData["id"] =(strconv.Itoa(ret[i].Id))
		RData["pullCount"] = (strconv.Itoa(ret[i].PullCount))
		ps = append(ps, RData)
	}
	return ps
}
Copy the code

Mirror Tag operation

// Define the tag data structure to fetch
type Tag struct {
	ArtifactId int 	`json:"artifact_id"`
	Id int `json:"id"`
	Name string `json:"name"`
	RepositoryId int `json:"repository_id"`
	PushTimte string `json:"push_time"`
}

type Tag2 struct {
	ArtifactId string 	`json:"artifact_id"`
	Id string `json:"id"`
	Name string `json:"name"`
	RepositoryId string `json:"repository_id"`
	PushTimte string `json:"push_time"`
}

type Tag2s []Tag2
// Delete tag by specified count
func DeleTagsByCount(tags []map[string]string ,count int) []string {
	var re []string
	
	tt := tags[0] ["tags"]
	ss := Tag2s{}
	json.Unmarshal([]byte(tt), &ss)

	// have a sort
	for i := 0; i < len(ss); i++ {
		for j := i + 1; j < len(ss); j++ {
            // Sort by pushTime
			if ss[i].PushTimte > ss[j].PushTimte {
				ss[i], ss[j] = ss[j], ss[i]
			}
		}
	}
	// get all tags
	for i := 0; i < len(ss); i++ {
		re = append(re, ss[i].Name)
	}
	// Return count of tags to be removed,
	return re[0:count]
}

// Delete tag by specified tag Delete the specified tag
func DelTags(url string, project string, repo string, tag string) (int.map[string]interface{})  {
	url = url + "/ API/v2.0 / projects/" + project + "/repositories/" + repo + "/artifacts/" + tag + "/tags/" + tag
	request, _ := http.NewRequest(http.MethodDelete, url,nil)
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
	request.Header.Set("accept"."application/json")
	request.SetBasicAuth("admin"."Pwd123456")
    // Delete the tag
	response,_ := client.Do(request)
	defer response.Body.Close()

	var result map[string]interface{}
	bd, err := ioutil.ReadAll(response.Body)
	if err == nil {
		err = json.Unmarshal(bd, &result)
	}
	return response.StatusCode,result

}




// Define the tag data structure to fetch
type ArtiData struct {
	Id int `json:"id"`
	ProjectId int `json:"project_id"`
	RepositoryId int `json:"repository_id"`
	//Digest string `json:"digest"`
	Tags []Tag `json:"tags"`
}

type AData []ArtiData
// Retrieve tag data from harbor address, project, and repO
func GetTags(url string, project string, repo string) []map[string]string {
	url = url + "/ API/v2.0 / projects/" + project + "/repositories/" + repo + "/artifacts"
	request, _ := http.NewRequest(http.MethodGet, url,nil)
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
	request.Header.Set("accept"."application/json")
	request.Header.Set("X-Accept-Vulnerabilities"."application/vnd.scanner.adapter.vuln.report.harbor+json; Version = 1.0")
	request.SetBasicAuth("admin"."Harbor12345")
    / / get the tag
	response, err := client.Do(request)

	iferr ! =nil {
		fmt.Println("excute failed")
		fmt.Println(err)
	}

	body, _ := ioutil.ReadAll(response.Body)
	defer response.Body.Close()
	ret := AData{}
	json.Unmarshal([]byte(string(body)),&ret)
	var ps = []map[string]string{}
	sum := 0
	RData := make(map[string]string)
	RData["name"] = repo
    // Get the returned data
	for i := 0; i < len(ret); i++ {

		RData["id"] = (strconv.Itoa(ret[i].Id))
		RData["project_id"] = (strconv.Itoa(ret[i].ProjectId))
		RData["repository_id"] =(strconv.Itoa(ret[i].RepositoryId))
		//RData["digest"] = ret[i].Digest
		var tdata = []map[string]string{}
		sum = len((ret[i].Tags))
        / / get the tag
		for j := 0; j < len((ret[i].Tags)); j++ {
			TagData := make(map[string]string)
			TagData["artifact_id"] = strconv.Itoa((ret[i].Tags)[j].ArtifactId)
			TagData["id"] = strconv.Itoa((ret[i].Tags)[j].Id)
			TagData["name"] = (ret[i].Tags)[j].Name
			TagData["repository_id"] = strconv.Itoa((ret[i].Tags)[j].RepositoryId)
			TagData["push_time"] = (ret[i].Tags)[j].PushTimte
			tdata = append(tdata, TagData)
		}
		RData["count"] = strconv.Itoa(sum)
		ss, err := json.Marshal(tdata)
		iferr ! =nil {
			fmt.Println("failed")
			os.Exit(2)
		}
		RData["tags"] = string(ss)
		ps = append(ps, RData)

	}
	return ps
}
Copy the code

Gets the user’s command line input to list all the items in Harbor

// Define the command operations associated with obtaining project in harbor
var projectCmd = &cobra.Command{
	Use: "project",
	Short: "to operator project",
	Run: func(cmd *cobra.Command, args []string) {

		output, err := ExecuteCommand("harbor"."project", args...)
		iferr ! =nil {
			Error(cmd,args, err)
		}
		fmt.Fprint(os.Stdout, output)
	},
}
// project list
var projectLsCmd = &cobra.Command{
	Use: "ls",
	Short: "list all project",
	Run: func(cmd *cobra.Command, args []string) {
		url, _ := cmd.Flags().GetString("url")
		if len(url) == 0 {
			fmt.Println("url is null,please specified the harbor url first !!!!")
			os.Exit(2)}// Get all the projects
		output := harbor.GetProject(url)
		fmt.Println("Number of item name access level warehouses")
		for i := 0; i < len(output); i++ {

			fmt.Println(output[i]["name"], output[i]["public"], output[i]["repo_count"])}}}// init
func init(a) {
    // ./harbor project ls -u https://
	rootCmd.AddCommand(projectCmd)
	projectCmd.AddCommand(projectLsCmd)
	projectLsCmd.Flags().StringP("url"."u".""."defaults: [https://127.0.0.1]")}Copy the code

Gets the LIST of REPOs

// repo command
var repoCmd = &cobra.Command{
	Use: "repo",
	Short: "to operator repository",
	Run: func(cmd *cobra.Command, args []string) {

		output, err := ExecuteCommand("harbor"."repo", args...)
		iferr ! =nil {
			Error(cmd,args, err)
		}
		fmt.Fprint(os.Stdout, output)
	},
}
// repo list
var repoLsCmd = &cobra.Command{
	Use: "ls",
	Short: "list project's repository",
	Run: func(cmd *cobra.Command, args []string) {
		url, _ := cmd.Flags().GetString("url")
		project, _ := cmd.Flags().GetString("project")
		if len(project) == 0 {
			fmt.Println("sorry, you must specified the project which you want to show repository !!!")
			os.Exit(2)}// get all repo 
		output := harbor.GetRepoData(url, project)
        // Display data
		fmt.Println("Warehouse name ---------- pull times")
		for i := 0; i < len(output); i++ {
			fmt.Println(output[i]["name"],output[i]["pullCount"])}}}func init(a) {
    // ./harbor repo ls -u https:// -p xxx
	rootCmd.AddCommand(repoCmd)
	repoCmd.AddCommand(repoLsCmd)
	repoLsCmd.Flags().StringP("url"."u".""."defaults: [https://127.0.0.1]")
	repoLsCmd.Flags().StringP("project"."p".""."the project")}Copy the code

The tag operation

// tag command
var tagCmd = &cobra.Command{
	Use: "tag",
	Short: "to operator image",
	Run: func(cmd *cobra.Command, args []string) {

		output, err := ExecuteCommand("harbor"."tag", args...)
		iferr ! =nil {
			Error(cmd,args, err)
		}
		fmt.Fprint(os.Stdout, output)
	},
}
// tag ls
var tagLsCmd = &cobra.Command{
	Use: "ls",
	Short: "list all tags of the repository you have specified which you should specified project at the same time",
	Run: func(cmd *cobra.Command, args []string) {
		url, _ := cmd.Flags().GetString("url")
		project, _ := cmd.Flags().GetString("project")
		repo, _ := cmd.Flags().GetString("repo")
        // get all tags 
		ss := harbor.GetTags(url, project, repo)
		for i := 0; i < len(ss); i++ {
			count, _ := strconv.Atoi((ss[i])["count"])
			fmt.Printf("the repo %s has %d images\n", repo, count)
		}

	},
}

// tag del by tag or the number of image you want to save
var tagDelCmd = &cobra.Command{
	Use: "del",
	Short: "delete the tags of the repository you have specified which you should specified project at the same time",
	Run: func(cmd *cobra.Command, args []string) {
        // Get user input and format it
		url, _ := cmd.Flags().GetString("url")
		project, _ := cmd.Flags().GetString("project")
		repo, _ := cmd.Flags().GetString("repo")
		tag,_ := cmd.Flags().GetString("tag")
		count,_ := cmd.Flags().GetString("count")
		ret,_ := strconv.Atoi(count)
		if len(tag) ! =0&& ret ! =0 {
				fmt.Println("You can't choose both between count and tag")
				os.Exit(2)}else if len(tag) == 0&& ret ! =0 {
            // get all tags
			retu := harbor.GetTags(url, project, repo)
            //delete tag by you hsve specied the number of the images you want to save
			rTagCount, _ := strconv.Atoi((retu[0[])"count"])
            // Determine whether to delete the tag based on the count entered by the user and the actual number of tags
			if ret == rTagCount {
				fmt.Printf("the repository %s of the project %s only have %d tags, so you can't delete tags and we will do nothing!! \n", repo, project,ret)
			} else if ret > rTagCount {
				fmt.Printf("the repository %s of the project %s only have %d tags, but you want to delete %d tags, so we suggest you to have a rest and we will do nothing!! \n", repo, project,rTagCount, ret)
			} else {
                // Delete the tag
				fmt.Printf("we will save the latest %d tags  and delete other %d tags !!!\n", ret, (rTagCount - ret))
				tags := harbor.GetTags(url, project, repo)
				retu := harbor.DeleTagsByCount(tags, (rTagCount - ret))
				for i := 0 ; i < len(retu); i++ {
                    // to delete tag
					code, msg := harbor.DelTags(url, project, repo, retu[i])
					fmt.Printf("the tag %s is deleted,status code is %d, msg is %s\n", retu[i], code, msg)
				}
			}
		} else {
            // delete tag by you specied tag
			code, msg := harbor.DelTags(url, project, repo, tag)
			fmt.Println(code, msg["errors"])}}}func init(a) {
    // ./harbor tag ls -u -p -r
	rootCmd.AddCommand(tagCmd)
	tagCmd.AddCommand(tagLsCmd)
	tagLsCmd.Flags().StringP("url"."u".""."defaults: [https://127.0.0.1]")
	tagLsCmd.Flags().StringP("project"."p".""."the project")
	tagLsCmd.Flags().StringP("repo"."r".""."the repository")
	// ./harbor tag del -u -p -r [-t | -c]
	tagCmd.AddCommand(tagDelCmd)
	tagDelCmd.Flags().StringP("url"."u".""."defaults: [https://127.0.0.1]")
	tagDelCmd.Flags().StringP("project"."p".""."the project which you should specified if you want to delete the tag of any repository ")
	tagDelCmd.Flags().StringP("repo"."r".""."the repository which you should specified if you want to delete the tag")
	tagDelCmd.Flags().StringP("tag"."t".""."the tag, You can't choose it with tag together")
	tagDelCmd.Flags().StringP("count"."c".""."the total number you want to save.for example: you set --count=10, we will save the 10 latest tags by use push_time to sort,can't choose it with tag together")}Copy the code

test

/ / get help harbor %. / harbor - h https://harbor.zaizai.com Usage: harbor (flags) harbor (command) the Available Commands: completion generate the autocompletion script for the specified shell help Help about any command project to operator project repo to operator repository tag to operator image Flags: -h, --help help for harbor Use "harbor [command] --help" for more information about a command. // List all project harbor % . / harbor project ls -u project name https://harbor.zaizai.com access level Warehouse number goharbor false 3 library public true true 0 1 / / lists all the repo Harbor %. / harbor repo ls -u -p https://harbor.zaizai.com goharbor warehouse name -- -- -- -- -- -- -- -- -- -- pull goharbor/harbor portal - 0 Goharbor/harbor - 1 goharbor/prepare 0 db / / list tags harbor. % / harbor tag ls -u https://harbor.zaizai.com - p goharbor - r Harbor - db the repo harbor - db has 9 images / / by retaining recent 20 images to remove the tag harbor. % / harbor tag del -u -p at https://harbor.zaizai.com  goharbor -r harbor-db -c 20 the repository harbor-db of the project goharbor only have 9 tags, but you want to delete 20 tags, so we suggest you to have a rest and we will do nothing!! / / by retaining recent 10 images to remove the tag harbor. % / harbor tag del -u -p https://harbor.zaizai.com goharbor -r 10 the harbor - db - c repository harbor-db of the project goharbor only have 9 tags, but you want to delete 10 tags, so we suggest you to have a rest and we will do nothing!! / / by retaining recent five images to remove the tag harbor. % / harbor tag del -u https://harbor.zaizai.com - p goharbor - r harbor - db - c 5 we will save the latest 5 tags and delete other 4 tags !!! The tag V2.3.9 is deleted,status code is 200, MSG is map[] The tag V2.3.10 is deleted,status code is 200, MSG is map[] the tag v2.3.8 is deleted,status code is 200, MSG is map[] the tag V2.3.7 is deleted,status code is 200, MSG is a map [] / / the specified tag to delete caicloud @ MacBook Pro - 2 harbor %. / harbor tag del -u https://harbor.zaizai.com - p goharbor - r Harbor-db-t v2.3.6 200 <nil> ！！！！ Finally, you need to manually go to harbor UI for garbage collection!!Copy the code

reference

github.com/spf13/cobra
harbor swagger

How to delete a Harbor image quickly

background

The target

The statement

implementation

test

reference

Related Posts

Redis course notes (4) — Hash

RocketMQ Principle – Message sending process

Spark2.0 new features