This is the 17th day of my participation in the Gwen Challenge in November. Check out the details: The last Gwen Challenge in 2021″

conclusion

From the analysis to the solution, it is not difficult to find: if the idempotence needs to be strictly guaranteed, we need to combine the idea of pessimistic lock and optimistic lock to design.

Pessimistic locks and optimistic locks are not appropriate descriptions.

  • Pessimistic locks:

    • The execution process is locked by distributed locking
    • A globally unique ID is used as the unique key of the lock

  • “Optimistic” lock:

    • This is called optimistic lock, but it is not completely correct. Here we use two fields to check:

      • Globally unique ID, this can be the unique key of the pessimistic lock above, it can be something else, it can be some combination of fields, but it has to be unique!

      • Non-rollback status: This status identifies the current processing progress and the status of the processing, using the above unique key and the non-rollback status, can be used to make a pre-decision through CAS.

        Can this state value be removed?

        A: If we have only two results: initial creation and completion, then this field can be replaced by whether or not the record was created.

        But if we are likely to fail, then we need to let subsequent requests try to reprocess the business, then there is one more failed state;

        And because we need to allow subsequent requests to be retried, we also need a state to indicate that it is currently being processed, letting subsequent requests know that it is not a failure, but that another request is being executed.

        So this state, in general, cannot be removed.

        Graph LR New State --> Processing --> Finishing Processing --> Failed Failed --> Processing

So back to the question above:

  • Which field do we design idempotent for?
    • Put it in an order: Which field do we want to be idempotent?
    • For example, in the case of payment, we expect payment to be idempotent, so our idempotent is guaranteed by the unique key in the optimistic lock above, which is the globally unique ID plus the user id, to record whether or not we want to make the payment idempotent, so which field do we want to make idempotent? It’s pretty obvious: idempotent fields must represent states.
    • How to design idempotent fields? So it is also simple, global unique ID, unique key, and then pull down not distributed ID ah UUID snowflake algorithm and so on?

And you can see that at the heart of idempotent guarantee is uniqueness, and global uniqueness, which is how we lock and determine whether the request is successful.

In fact, by idempotent design, the interviewer asked another question:

  • In general, idempotent design involves MQ, so how do YOU ensure the reliability of MQ messages?

In fact, after thinking about it, according to this idempotence, it is easy to understand:

  • We record the sending status of messages at both the consumer and production ends for subsequent comparison and resending (of course, this is the bottom part).

  • If the idempotent processing is finished on the consumer side, it is also put back into MQ, allowing the original production side to update the processing of message sending.

    • So what if something goes wrong again? How do you guarantee that?

    This is the infinite nesting doll problem, and this involves:

    • How to deal with kafka message loss?
    • How to troubleshoot database faults?
    • How to rectify a server fault?
      • Is the server stuck?
        • If jammed, how to check?
      • Or is the server down?
        • Server down is not to do disaster recovery ah?
        • How to roll back quickly?
        • Do you want to make a circuit breaker?
    • How to handle the redis fault?

    Therefore, most design problems are actually more than one point, generally speaking, from point to surface. We use a lot of middleware for temporal superiority

    For example, in order to save the cost of space, we will use a special database: storage such as mySQL, cache such as Redis. For decoupling, peak clipping, and so on, we use Kafka. For development convenience, we will use Spring, Mybatis and so on.

    These things greatly simplify the cost of getting our ideas off the ground. However, the introduction of more middleware inevitably requires more effort to maintain these components, so a mature idea is always to take these boundary conditions into account.