In JavaScript floating point arithmetic often appears 0.1+0.2=0.30000000000000004 such a problem, in addition to a large number crisis can not be ignored (large number processing accuracy loss) problem.
This question was also discussed in the group more than once before, and was also discussed in another “Nodejs technology stack – communication group” over the weekend. At that time, we briefly discussed it together in the group, and the atmosphere of communication and learning was good. Here is what we discussed in the group over the weekend.
I’ve shared this before, and I’m just going to sort it out and share it with you, but the front end also works, because we’re all in the same language, JavaScript.
JavaScript maximum safe integer
Before starting this section, hopefully you’ve learned a little bit about JavaScript floating point numbers. In the previous article, the mystery of JavaScript Floating Point Numbers: Why 0.1 + 0.2 Doesn’t Equal 0.3? In a very good introduction to the floating point number storage principle, why the precision loss (recommended to read in advance).
IEEE 754 Double precision floating-point numbers (Double 64 Bits) The mantisse part is used to store the significant Bits of an integer, which is 52 Bits. 1 can be saved and omit a actual values for [- (253-1), 253] [- (2 ^ {53} – 1), 2 ^ {53}] [- (253-1), 253].
Math.pow(2.53) / / 9007199254740992
Number.MAX_SAFE_INTEGER // The maximum safe integer is 9007199254740991
Number.MIN_SAFE_INTEGER // Minimum safe integer -9007199254740991
Copy the codeIt is safe as long as it does not exceed the maximum and minimum safe integer ranges in JavaScript.
The problem of precision loss in large number processing is repeated
Patients with a
When you execute the following code in Chrome’s console or in the Node.js runtime environment, you get the following result: What? Why 200000436035958034 I defined was escaped to 200000436035958050? After understanding JavaScript floating-point number storage, it should be clear that JavaScript’s maximum safe integer range is triggered at this point.
const num = 200000436035958034;
console.log(num); / / 200000436035958050
Copy the codeExample 2
The following example reads the data passed through the stream and saves it in a string data. Because the data passed is an Application/JSON protocol, we need to deserialize the data to an OBJ for business processing.
const http = require('http');
http.createServer((req, res) = > {
if (req.method === 'POST') {
let data = ' ';
req.on('data'.chunk= > {
data += chunk;
});
req.on('end'.() = > {
console.log('No JSON deserialization case:', data);
try {
// Deserialize it into an obj object to handle business
const obj = JSON.parse(data);
console.log('After deserialization with JSON:', obj);
res.setHeader("Content-Type"."application/json");
res.end(data);
} catch(e) {
console.error(e);
res.statusCode = 400;
res.end("Invalid JSON"); }}); }else {
res.end('OK');
}
}).listen(3000)
Copy the codeAfter running the above program, POSTMAN calls 200000436035958034, which is a large number.
After the program executes json.parse (), the accuracy problem occurs again. Why? What’s the catch between JSON conversions and high numerical accuracy?
Deserialization without JSON: {"id": 200000436035958034} : {id: 200000436035958050}Copy the codeThis problem was also encountered in practice. The way it happened was that the third-party interface was called to get a parameter with a large value, and similar problems appeared after the result was JSON. Let’s do the analysis below.
What’s wrong with JSON serialization for parsing large numbers?
The Internet Engineering Task Force 7159 (IETF 7159) is a lightweight, text-independent, and language independent data interaction format derived from the ECMAScript programming language standard.
www.rfc-editor.org/rfc/rfc7159… Access this address to view the protocol.
What we need to focus on in this section is “What is a Value of JSON?” Object, array, number, or string can also be false, null, or true.
JSON is converted to other types of encoding by default during parsing. Large values in our example are coded as type number by default, which is the real reason for the accuracy loss.
The solution of large number operation
1. Common methods turn a string
This is a common scenario in front and back interactions. For example, the order number is stored as a numeric type of long in Java with a maximum value of 2 to the power of 64, MAX_SAFE_INTEGER (math.pow (2, 53) -1) = number.max_safe_INTEGER (math.pow (2, 53) -1) = number.max_safe_INTEGER (math.pow (2, 53) -1); This is a real pit in the docking process.
2. New hope BigInt
Bigint is a new data type in JavaScript that can be used to manipulate integers that are outside the maximum safe range of Number.
Create BigInt method 1
One way is to add the number n after the number
200000436035958034n; // 200000436035958034n
Copy the codeCreate BigInt method 2
Another way to do this is to use the constructor BigInt(). Note that it is better to use a string when using BigInt, otherwise you will still have precision problems. They’re called incurable diseases
BigInt('200000436035958034') // 200000436035958034n
// Use a string or it will be escaped
BigInt(200000436035958034) // 200000436035958048n This is not a correct result
Copy the codeDetection of type
BigInt is a new data type, so it is not exactly equal to Number. For example, 1n will not all be equal to 1.
typeof 200000436035958034n // bigint
1n= = =1 // false
Copy the codeoperation
BitInt supports common operators, but never mix them with Number, always be consistent.
/ / right
200000436035958034n + 1n // 200000436035958035n
/ / error
200000436035958034n + 1
^
TypeError: Cannot mix BigInt and other types, use explicit conversions
Copy the codeConvert BigInt to a string
String(200000436035958034n) / / 200000436035958034
// Or the following
(200000436035958034n).toString() / / 200000436035958034
Copy the codeConflict with JSON
Parse with json. parse(‘{“id”: 200000436035958034}’) causes accuracy loss. Now that a BigInt is present, does the following parse work?
JSON.parse('{"id": 200000436035958034n}');
Copy the codeAfter running the above program, we get a SyntaxError: Unexpected token n in JSON at position 25 error: Unexpected token n in JSON at position 25 Error: Unexpected token n in JSON at position 25 Error: Unexpected token n in JSON at position 25
In the TC39 proposal- Bigint warehouse, there are also people who have raised this issue. As of now, the proposal has not been added to JSON, as this would break the JSON format and probably render it unparsed.
BigInt support
The BigInt proposal is currently at Stage 4 and has been released in Chrome, Node, Firefox, Babel, with version 12+ supported in Node.js.
BigInt summary
We can use BigInt to do some operations, but when we interact with a third party interface, if we serialize JSON strings and encounter some large number problems, precision loss will occur. Obviously, this is caused by the conflict with JSON. The third scheme is given below.
3. Third-party libraries
Some third-party libraries can also be used to solve this problem, but you might be wondering why go through all the twists and turns? We are not happy to turn into a string, but, sometimes you need to connect to the third party interface, the data to contain this large number of cases, and encounter the kind of refusal to change, the business will be completed! Here is a third implementation.
We also take the above large number processing accuracy loss problem recurrence of the second example to explain, through jSON-Bigint this library to solve.
Knowing that the JSON specification is in conflict with JavaScript, instead of using json.parse () directly, parse the data stream as a string, using the json-Bigint library. A value greater than 2 to the power of 53 will be automatically converted to a BigInt. Setting storeAsString: true will automatically convert BigInt to a string.
const http = require('http');
const JSONbig = require('json-bigint') ({'storeAsString': true});
http.createServer((req, res) = > {
if (req.method === 'POST') {
let data = ' ';
req.on('data'.chunk= > {
data += chunk;
});
req.on('end'.() = > {
try {
// Use a third-party library for JSON serialization
const obj = JSONbig.parse(data)
console.log('After deserialization with JSON:', obj);
res.setHeader("Content-Type"."application/json");
res.end(data);
} catch(e) {
console.error(e);
res.statusCode = 400;
res.end("Invalid JSON"); }}); }else {
res.end('OK');
}
}).listen(3000)
Copy the codeCheck again and you will see the following results. This time it is correct and the problem has been solved perfectly!
{id: {id:}}'200000436035958034' }
Copy the codeJson-bigint combined with Request Client
Explain how axios, Nodes-fetch, undici, and undici-fetch request clients handle large numbers in combination with JSON-Bigint.
Mock server
Using BigInt to create a large number of simulated server return data, if the client does not process the request will cause accuracy loss.
const http = require('http');
const JSONbig = require('json-bigint') ({'storeAsString': true});
http.createServer((req, res) = > {
res.end(JSONbig.stringify({
num: BigInt('200000436035958034')
}))
}).listen(3000)
Copy the codeaxios
Create an Axios request instance, request, with the transformResponse property we do some custom processing on the original response data.
const axios = require('axios').default;
const JSONbig = require('json-bigint') ({'storeAsString': true});
const request = axios.create({
baseURL: 'http://localhost:3000'.transformResponse: [function (data) {
return JSONbig.parse(data)
}],
});
request({
url: '/api/test'
}).then(response= > {
/ / 200000436035958034
console.log(response.data.num);
});
Copy the codenode-fetch
Node-fetch is used in node.js. One method is to process the returned text data. Other more convenient methods are not studied in detail.
const fetch = require('node-fetch');
const JSONbig = require('json-bigint') ({'storeAsString': true});
fetch('http://localhost:3000/api/data')
.then(async res => JSONbig.parse(await res.text()))
.then(data= > console.log(data.num));
Copy the codeundici
I’ll leave request as a deprecated Client and recommend a noteworthy Node.js request Client undici, which I’ve written about in the previous section.
const undici = require('undici');
const JSONbig = require('json-bigint') ({'storeAsString': true});
const client = new undici.Client('http://localhost:3000');
(async() = > {const { body } = await client.request({
path: '/api'.method: 'GET'}); body.setEncoding('utf8');
let str = ' ';
for await (const chunk of body) {
str += chunk;
}
console.log(JSONbig.parse(str)); / / 200000436035958034
console.log(JSON.parse(str)); // 200000436035958050 Accuracy lost}) ();Copy the codeundici-fetch
Undici-fetch is a WHATWG FETCH implementation built on top of Undici, using much the same as Node-fetch.
const fetch = require('undici-fetch');
const JSONbig = require('json-bigint') ({'storeAsString': true});
(async() = > {const res = await fetch('http://localhost:3000');
const json = JSONbig.parse(await res.text());
console.log(json.num); / / 200000436035958034}) ();Copy the codeconclusion
In this paper, some causes of large number precision loss are proposed, and several solutions are given. Is recommended in the system design to follow the specification double-precision floating-point number to do, in the process of finding problems, see some use regular to match, personal or not recommended, it is a regular itself is a time-consuming operation, also may find some match two operation rule, carelessly may return results of all the values are converted to strings, It’s not feasible.
This article is available at github.com/qufei1993/N… Welcome to follow.
Reference
- V8. Dev/features/bi…
- Github.com/tc39/propos…
- En.wikipedia.org/wiki/Double…