What is the GC

Garbage Collection stands for Garbage Collection. In RGW, GC usually refers to some asynchronous disk space reclamation operations. GC usually occurs in the following three situations. \1. When the client deletes an Object, the disk space occupied by the Object is submitted to the background GC for processing. \2. The client performs the Object overwrite operation, and the space related to the old Object needs to be released. \3. During the upload operation (block upload), some shadow files will be generated. The temporary data generated during the upload process will also be collected by GC.

GC Startup process

GC is started when the RGW service is started in thread mode. The corresponding call process is as follows.

When each GCworker is started, it will be identified by a digital ID, which is actually the ID name of the Object in the GC pool. The GC task list will be stored in the OMAP of the Object in each GC pool. If the GC queue explodes or becomes congested, The SIZE of the OMAP will also explode, which will consume disk performance while making scrub at the bottom, and may cause client requests to block, resulting in 504 errors that will affect service availability. It is worth noting that the ID of GCworker at startup is selected by a random algorithm. If other Gcworkers with the same ID are already running, the lock fails to be obtained and the random selection algorithm needs to be run again. GCworekr starts as follows

The random selection algorithm of GCworker is as follows, where max_objs represents the total number of GC workers.

int RGWGC::process()
{
  int max_secs = cct->_conf->rgw_gc_processor_max_time;

  unsigned start;
  int ret = get_random_bytes((char *)&start, sizeof(start));
  if (ret < 0)
    return ret;

  for (int i = 0; i < max_objs; i++) {
    int index = (i + start) % max_objs;
    ret = process(index, max_secs);
    if (ret < 0)
      return ret;
  }

  return 0;
}
Copy the code

Objects to be reclaimed will be thrown into the corresponding GC queue for processing after meeting the specified conditions, as shown in the following figure

The data structure of the GC queue is as follows, with each China identified by a unique tag name.

Note that each GC record generates two types of index records. The index records starting with 0 are identified by name and the index records starting with 1 are identified by timestamp. The index records are inserted into the LevelDB record of omAP on OSD where GC Object resides.

/*
 * We hold the garbage collection chain data under two different indexes: the first 'name' index
 * keeps them under a unique tag that represents the chains, and a second 'time' index keeps
 * them by their expiration timestamp
 */
#define GC_OBJ_NAME_INDEX 0
#define GC_OBJ_TIME_INDEX 1

static string gc_index_prefixes[] = { "0_",
                                      "1_" };
Copy the code

GC process overview diagram

The final figure summarizes the entire GC process, which involves several parameters directly related to the GC efficiency, you can tune according to their own line conditions.

Some tips on parameter tuning

  • The number of GC workers is controlled by rgw_gC_MAX_objs. When setting this parameter, consider whether your online business will have a large number of GC operations.
  • Rgw_gc_processor_max_time controls the maximum amount of time that a GC task can take. This can be scaled down for high speed storage devices, but may need to be scaled up for slow and high load underlying storage devices to allow for additional GC performance costs.
  • Rgw_gc_obj_min_wait Controls how long it takes to reclaim the underlying data after the data is deleted. The default value is 2 hours. If the online space usage is sensitive, it can be shortened appropriately.
  • Rgw_gc_processor_period Specifies how long it takes for GCworker to start the next GC. If the number of list items required for a single GC is small, shorten this parameter appropriately.