JDFS: A Distributed File Management System, Part 5

A preface

So far, although not perfect, THE JDFS has a complete set of distributed file management features, including redundant file storage, file meta-information query, file download, file deletion, and so on. This article will provide an overview of the JDFS architecture, flow charts, and how to install, deploy, and run the JDFS. Of course, as I mentioned in the previous blog posts, JDFS is not perfect, and there are potentially hard-to-find bugs that occasionally pop up, which will be resolved through continuous testing and debugging. If you are reading the JDFS blog series for the first time, I recommend reading the other blogs in the series, which are linked below:

• JDFS: a distributed file management utility (thread pool, epoll, upload, download) click me
• JDFS: a distributed file management utility. Click on me
• JDFS: A distributed file management system, Part 3 (Streaming cloud Storage) Click on me
• JDFS: a distributed file management system, chapter 4 (streaming cloud storage sequel) Click on me

JDFS code has been uploaded to Github, please click me for the address

PS: This blog is the original work of blog park user “CS primary school”, please indicate the original author and original link, thank you.

Overview of THE JDFS architecture

In the first four blogs, I covered the technical details of JDFS in text and snippets of code, each of which was specific to a particular feature. Now that the JDFS as a whole is tentatively complete, we’ll take a look at what the JDFS looks like in general, what components it consists of, and the path it takes from client submission to completion. The following three sections will describe the ARCHITECTURE of the JDFS server, the flow chart of redundant file streaming storage, and the execution flow of file downloading.

1 JDFS server architecture

Let’s take a look at the JDFS server architecture:

The diagram above shows the architecture of the Server side. In JDFS, the servers running on name node and Data node are all using the architecture in the figure. As can be seen from the figure, the server side mainly consists of two large parts: modules listening for client events (figure left), thread pool & job queue (figure right). Monitoring module to monitor the client’s event, if it is connected, the server to accept it, don’t need the participation of the thread pool, and if the epoll listening to read the event (that) the client sends data, put it into a task, to join in the job queue, threads in the pool will always staring at the job queue, If a task exists, it tries to fetch the task from the queue and execute it. In this case, whether the server adds the task to the job queue or the thread in the thread pool takes the task out of the task queue, it needs to execute the task mutually exclusive under the protection of synchronous lock to ensure the correct access of data.

The previous paragraph described the general process of server side execution. Now we introduce the functions supported by the Server-epoll side and the thread pool respectively, and then describe the details of server side execution in combination with the execution flow and functions implemented in the figure.

The server-epoll side implements the following functions:

1 int Http_server_bind_and_listen(char *ip, int port, int *server_listen_fd);
2 int Http_server_body(char *ip, int port, int *server_listen_fd, threadpool *pool);
3 void *Http_server_callback(void *arg);Copy the code

The thread pool & job queue side implements the following functions:

1 int threadpool_create(threadpool *pool, int num_of_thread, int max_num_of_jobs_of_one_task_queue);
2 int threadpool_add_jobs_to_taskqueue(threadpool *pool, void * (*call_back_func)(void *arg), void *arg);
3 int threadpool_fetch_jobs_from_taskqueue(threadpool *pool, job **job_fetched);
4 void *thread_func(void *arg);
5 int destory_threadpool(threadpool *pool);Copy the code

Threadpool_add_jobs_to_taskqueue () is the interface exposed by the threadpool to the server-epoll server for adding tasks to the job queue. The following is a detailed description of the server startup process with the function interface.

In C, we obviously need to write a main() function to start the server. We need to create the thread pool first and then start the server.

1 threadpool *pool=(threadpool *)malloc(sizeof(threadpool));
2 threadpool_create(pool, 6.20);
3 Http_server_body(ip_str,port,&server_listen_fd,pool);
4 destory_threadpool(pool);Copy the code

We use threadpool_create() to create a threadpool with six threads and a maximum of 20 tasks in the queue. The rectangle represents the threadpool and the while(1) on the right side of the box means that the threadpool is an infinite loop. The loop continuously fetches tasks from the task queue, executes them, and after executing them, continues to fetch the next task mutually exclusive in the while loop. Of course, there are no tasks in the queue at the moment, so all threads in the pool are stuck in a variable of type pthread_cond_t. If a task is added to the queue, the stuck thread will wake up and compete for tasks in the queue.

Now let’s move from the thread pool to the server-epoll side. The top left rectangle with the word “server-epoll” indicates that Http_server_body() has been called and the server has started. Http_server_body() first calls Http_server_bind_and_listen() to listen for < IP,port> on the server side, and subsequent client requests are sent to the server represented by this IP-port. After the listener port is set up, we need to create an epoll and add the listener socket FD to the epoll. Then we need to call epoll_wait() repeatedly in an infinite loop while(1) to check for events. In the figure above, there are three red dotted arrows under the server-epoll rectangle box, and a while(1) next to it represents what we call: monitoring the occurrence of events in an infinite loop. As can be seen from the logic indicated by the dotted red arrows in the figure, there are two types of requests sent from the client: one is a connection request from the client, and the other is a read event detected on a socket FD (the client sends data).

The client connection request will be handled by the server “personally”. As can be seen from the figure, the main thread of the server needs to call accept() function to accept the client connection request first. If the connection request is accepted successfully, a socket-fd will be obtained, representing the socket connection between a client and the server. We then set the socket-fd to non-blocking, edge-triggered, epolloneshot, epollin. The reason for the use of edge trigger and epolloneshot is described in detail in the previous blog. Readers can find these connections in the preface and read them by themselves. Epollin means to listen for read events on the socket-fd. Once these attributes are set, the socket-fd needs to be added to the epoll created earlier, using the epoll_ctl() interface. After all this is done, if the client sends data, ePoll will monitor and report it. As shown in the figure, three small rectangular boxes represent the three read events detected. At this time, the main thread on the server side needs to wrap the read events into a task. Add the job to the taskqueue by calling threadpool_add_jobs_to_taskqueue(), which is thread-safe. A task contains the address of the callback function, socket-fd, and epollfd corresponding to the read event. The callback function is logically integrated with the server-epoll end, but the caller calling the callback function is not the server end, but the thread pool end is called by A pointer. In short, the function at A is called by A function pointer at B.

When the pool is created, all threads are stuck in a variable of type pthread_cond_t, as shown in the figure. After running for some time, the main thread on the server has added five tasks to the queue. If the current task queue is empty, the pthread_cond_broadcast() will be used to wake up all threads that are stuck, as shown in the figure below. Let’s assume that thread T2 successfully competed for task1, so thread T2 gets the address and parameters of the callback function from Task1 and calls it. At this point the callback starts executing, represented by the rectangle with the words “Server Call back function”. The callback first receives a header using recv(). After receiving the header, The callback() function parses the header to see what type of request it is and calls a specific interface function to execute the request, as shown in the figure. For a Data node, there are different interface functions such as query, upload, and download.

In JDFS, there are two kinds of computing nodes, one is data node and the other is name node. The server running on it all works according to the process described in this section.

2 redundant storage execution streams

Let’s take a look at the corresponding execution flow chart:

The diagram above shows how we stored the clRS-en.pdf streaming redundancy into three data nodes. The numbers in the figure indicate the order of execution, and some of the lines are in a particular color, such as a black dotted line, to indicate that the steps are logically connected. Now let’s just follow the numbers 1,2,3…. Take a look at the process in detail. At number 1, user User1 sends a request to the Client to store clRs-en.pdf. The Client provides several user-facing interfaces that the user can directly invoke to complete corresponding functions. The Client will first send an HTTP request to the Name node, as shown by the arrow marked with the number 2, to query the data nodes that are still active in the virtual cluster. < IP,port> < IP,port> < IP,port> < IP,port> The name node sends an active request to the service marked with < IP,port>. This process is represented by the yellow dotted line marked with the number 4. The dotted line is a two-way arrow, and the active data node receives the query request from the Name node. Sends data along the same socket-fd(yellow dotted line in the figure) to the Name node to tell it that I am currently active and available to provide services.

For the next logic, we focus on the black dotted line marked with the number 5,6,7, and 8. The dotted line marked with the number 5 indicates that the Name node will send the IP address, port, etc. of the active data node queried in the previous step to the Client. The two dotted green lines with the number 6 indicate that the Client splits clRS-en.pdf into three blocks according to a certain strategy, and then blocks the clRs-en.pdf based on < active node list, local file partitioning information >. The Client selects several data nodes for each block according to certain policies to store the block redundantly. The Client then sends this information to the Name node along the black dotted line marked 8, telling the Name node “I plan to split crls-en.pdf” into three blocks to store in the virtual cluster. The Name node creates a local file with the same Name to store the file’s meta information. As shown in the figure, the meta information records the following information: the file name, the state of the file (0 indicates intended storage, but not yet completed), the number of blocks in the file, and then the specific information for each block: At which node the block is stored, 0 indicates that it is not intended to be stored in the corresponding data node, and 1 indicates that it should be stored in the corresponding data node but has not been stored successfully. In the JDFS implementation, 10010010 means that the third block should be redundantly stored on nodes 1,4, and 7.

The following four dotted arrows, marked by figure 9, represent the process of streaming redundant storage. As shown in the figure, each block is further divided into 6 pieces before transmission. The client iterates through each piece of each block and transmits it to Datanode3 in sequence. Datanode3 Sends a confirmation message to the Client every time a piece is successfully received. In this way, the Client knows that the piece is successfully transmitted. Otherwise, the Client needs to retransmit the piece.

Next comes the tricky part, where potential bugs can easily emerge: streaming. After receiving the piece from the previous data node, each data node needs to pass the piece to the next data node. While continuing to receive the next piece from the previous data node, this process is like a byte stream, so it is called streaming. The author will find a blog on the Internet to introduce HDFS, which introduces HDFS is to use the way of streaming transmission to transmit data fragments, this paper is to learn from HDFS to achieve this processing, in this thanks to HDFS. Datanode3 –>datanode2– >datanode1 –>datanode1 It is possible that the second piece of the first block arrives before the first piece, and the receive order is already out of order and parallel. Therefore, the problem is that all threads in datanodex receive block pieces at the same time, and who should create the file is a problem. In addition, the author decided to use edge trigger and epolloneshot after a lot of bug debugging. The reasons for these problems can be found in the previous several blogs. I will not repeat it here.

For the Client, after all data fragments were successfully sent to Datanode3, all that remains is to wait file to be completed, marked by the number 11. However, the streaming between datanode3-datanode2-datanode1 is probably not complete, so you need to wait for the data transfer to complete here. The Client will intermittently send requests to name node to check whether the file has been saved. The Name node will read the meta information of the corresponding file and check the meta information of each block. If all the numbers previously marked as 1 change to 2, the file has been saved. The name node marks the file as being saved and tells the Client that the file has been saved. The Client returns when it receives a successful message.

So how did these numbers that were labeled 1 become 2? Let’s move our eyes to the black solid line marked by the number 10 in the figure. After each data node successfully receives a complete block, it will send a message to name node. At this time, name node will update the meta information corresponding to the block, that is, the corresponding 1 should be 2. The block is successfully stored on the datanode. For data node2, 3, the shards received are out of order. It is possible to receive the last shard of a block, but the other shards of the block have not been successfully transmitted. In JDFS we let the thread receiving the last shard of a block take care of this. The thread will wait for all other shards to arrive and then send a message to the Name node to say that the block has been received. For details, see github code.

Clrs-en.pdf is segmented and stored in a virtual cluster by streaming redundant data. Here we list the function interfaces implemented by name node, data node and client respectively.

Name node implementation code:

1 void *Http_server_callback_query_nodelist(void *arg);3 void *Http_server_callback_query_ip_from_node_serial(void *arg);
4 void *Http_server_callback__cloudstr_meta(void *arg);
5 void *Http_server_callback_update_meta_info(void *arg);
6 void *Http_server_callback_wait_meta_complete(void *arg);
7 void *Http_server_callback_delete_meta_file(void *arg);Copy the code

Data node implementation code:

1 void *Http_server_callback_query(void *arg);
2 void *Http_server_callback_upload(void *arg);
3 void *Http_server_callback_download(void *arg);
4 void *Http_server_callback_query_node_alive(void *arg);
5 int   Http_stream_transform_file(callback_arg_upload *cb_arg_upload,  char *namenode_ip, int namenode_port);
6 void *Http_server_callback_delete_file(void *arg);Copy the code

Client implementation code, including the subsequent will be introduced to download, delete, display and other functions of the code:

 1 int JDFS_cloud_query_node_list( char *server_ip,  int server_port, node_list *nli);
 2 int JDFS_cloud_store_file(char *file_name, int flag);
 3 int JDFS_cloud_store_one_block(char *file_name, int block_num,char nodes_8_bits, node_list *nli, int total_num_of_blocks);
 4 int Split_file_into_several_parts(char *file_name, int part_size);
 5 int Extract_part_file_and_store(FILE *fp,char *new_file_name,int range_begin,int range_end);
 6 int JDFS_wait_file_transform_completed(char *file_name,  char *ip_str,  int port);
 7 int JDFS_cloud_query_file(char *file_name, JDFS_file_info *jfi,int flag);//flag, 0: query for reading ,1 :query for deleting
 8 int JDFS_cloud_query_ip_from_node_serial(int serial_num, char *ip_str);
 9 int JDFS_cloud_fetch_file(char *file_name);
10 int JDFS_cloud_merge_file(char *file_name, int num_of_blocks);
11 int JDFS_cloud_delete_file(char *file_name);
12 int JDFS_cloud_delete_file_internal(char *file_name, char *ip_str, int port);
13 int JDFS_cloud_delete_meta_file(char *file_name, char *ip_str, int port);
14 int JDFS_cloud_display_file_info(char *file_name);
16 int JDFS_select_optimal_node_for_one_block(each_block_info *ebi);//return the optimal node num
17 int JDFS_cloud_fetch_one_block(char *file_name, int block_num,int node_num, char *destination_file_path);Copy the code

3 Download the file from the cloud to the local PC

As shown in the figure above, CLRS-en.pdf has been split into blocks, each of which is redundantly stored in a different data node. Here we will download all the blocks that make up the file from the cloud to the local, then merge and restore the original file. At the arrow of the number 1 flag, user user1 issues a fetch file request to the Client and passes the file name clrs-en.pdf to the Client. As shown by the arrows marked with 2 and 3, the Client sends a request to Name node to query the clrs-en. PDF meta information. The Name node then finds the clrs-en. PDF meta information file in the local configuration file, reads it, and sends the information to the Client. Clrs-en. PDF is composed of three blocks. Block 1 is stored redundantly in data node 2 and 5. Block2 redundancy is stored on data nodes 3,4,6,7,8, and block3 redundancy is stored on data nodes 1,4. (Due to space constraints, only Data nodes 1,2, and 3 are listed). To get the CLRs-en.pdf, the client needs to download all three blocks that make up the file. The client decides to download block1 from Data Node 2, block2 from data Node 3, and block3 from data Node 1 based on specific policies. At this time, the Client does not know the IP addresses of Data node 1,2, and 3, so the Client queries the IP addresses of Data node 1,2, and 3 from Name node. The following numbers 8,9, and 10 indicate that the Client downloads the corresponding blocks from different data nodes. At 11, the Client merges the three blocks into original files and returns them to the user. This is the end of the file retrieval process.

In addition, the logic of file display and deletion is more intuitive, which will not be introduced in this article. Readers can read the code on Github or refer to the blog listed in the preface.

Three conclusion

To this JDFS overall end is over, had originally planned to introduce in this blog, how to install, deploy, run JDFS under but write here, feel a little tired, add length is too long to read easily fatigue, I would like to introduce or on the next blog inside, in the meantime the author just JDFS can continue to test, Work out a potential bug or something. Contact: https://github.com/junhuster/