preface

In the process of our work, if we encounter the content related to number operation, there will often be some puzzling problems. In this article, I will simply describe some problems in the process of number operation and processing

The decimal

The problem

We typed 0.1 + 0.2 into the browser console and expected to get 0.3, but that wasn’t the case:

The main reason for this is that the code will convert decimals to binary when it processes them. Due to conversion rules, many decimals are converted to binary in a wireless loop.

For decimals, conversion to binary is a round by two operation. Converted into binary: such as 0.1 0.1 = 0.0001100110011001100110011001100… (Loop with 1100)

The number of digits stored by these binary decimals in the computer is limited, so the real accurate floating point numbers are not processed in the process of storage and calculation, which will naturally cause the deviation of the calculation results

To solve

  • One of the first ways that comes to mind is by multiplying the number to be processed by 1,000… To convert it to an integer, then divide it back
  • Reference existing class libraries: mah.js, Decimal. Js, etc

Large number of

In addition to the operation of large numbers, we get some large numbers in JS, often encounter the following situation:

This kind of number is not what we want when we display it, mainly because the number of bits that can be stored in memory after being converted to binary is finite.

To solve

  • Convert to string
  • Using bigInt