Data crawler and data server resources of aifei Dog background are deployed on K8S and built by Rancher. Try to choose the lowest configuration machine without compromising too much performance. Use swap files instead of (swap) when memory is low. The general structure of the cluster is as follows:
| use | The machine type | The price |
|---|---|---|
| The crawler 1 | 1G 1CPU | $5 |
| The crawler 2 | 1G 1CPU | $5 |
| The crawler 3 | 1G 1CPU | $5 |
| monitoring | 1G 1CPU | $5 |
| etcd | 2G 1CPU | Ten dollars |
| The rancher host | 2G 1CPU | Ten dollars |
| Aiflygo server | 4G 2CPU | Twenty dollars |
It costs around $60 a month. Analysis of the cluster above shows that the two nodes, Rancher and ETCD, are a waste of money, with an overhead of $20 per month. DigitalOcean provides a managed cluster for K8S to save on this overhead. But Droplet, which manages clusters, cannot be customized to use swap partitions and BBR, causing performance bottlenecks. In addition, the minimum requirement for hosted Droplet is also 2 gigabytes of memory, resulting in unnecessary overhead.
One of the disadvantages of K8S is that the lowest machine requirements are relatively high. The worker node with 1G memory is completely lower than the recommended configuration. If the worker node is deployed on it, the direct memory consumption will be about 300M, and the remaining memory space is not much, so swap partition must be used. Etcd nodes have used 1 GB of memory before, but often use swap partitions to cause performance problems and end up crashing the cluster, so 2 GB is needed anyway.
Rancher recently launched the K3S, which focuses on ease of deployment and polar machine consumption. This is very important in terms of cost savings. I have tried the k3S server only takes up about 200M memory, and agent only takes up tens of megabytes of memory, which is very saving. K3s can also be fully managed using Kubectl, configuration files and K8S keep the same, very convenient.
I saved $15 by removing the ETCD node and Rancher host and replacing it with a 1GB 1CPU machine ($5), and then downsized the Aiflygo server to 2GB 2CPU ($15) for a total savings of $20. Thanks to more available memory, crawler performance is now better than before, and the overall cluster performance is also very high.
As for HA, since it is poor enough to use K3S to reduce overhead, it is not a consideration for small clusters and non-critical systems like mine.