Use Redis functionality to generate a reactive data stream

Redis is one of the most powerful and versatile technologies I’ve come across. Sadly, most people only know it because it’s a good caching solution.

We need to solve this problem.

In particular, I want to tell you that you can create a reactive architecture with Redis as the main component. This is a huge advantage, especially if you already have it as part of your infrastructure due to other requirements (i.e. good caching).

How you use Redis to interact with the features I’ve described here is up to you, and to be honest, any choice is valid at this point. I tend to use Node.js, but that’s me, and you’re free to use whatever works best for you.

Build a reactive architecture

The first thing to understand is what is a reactive architecture, and why do we build a reactive architecture instead of a more traditional approach?

Simply put, a reactive architecture is where every logic is executed when all the prerequisites are met — I guess I should put quotation marks around the word “simple”.

Let me rephrase this: When you need your logic to be triggered after a particular event, you have two options.

  • Periodically check some kind of sign until it is turned on, indicating that an event has occurred.
  • Sit and wait until something else notifies your service that the event is triggered.

The second part is the key to the observer pattern in object-oriented programming. The object being observed lets all interested in its internal state know, in fact, that it has been updated.

What we want to do here is to derive the same OOP pattern into architecture-level design. So, I’m not talking about having a little logic in our program, but the right event to trigger the service’s functionality.

This is the most efficient way to distribute and scale the platform due to the fact that.

  • You don’t have to waste time and network traffic polling for a particular flag of a data source (or something you feel you should poll for). In addition, if you are on a pay-per-use infrastructure, unnecessary polling may result in additional overhead, unnecessary work on the target service, and if more than one event occurs while your code is waiting for polling, you may end up having to aggregate events.
  • You can expand the processing power of services by adding new services, working in parallel, and capturing events as quickly as possible.
  • The platform is more stable. By doing reactive work, you can be sure that your service will work at optimum speed without fear of crashing due to client data overload.

Reactive platforms are asynchronous in nature, so any client application that tries to work with them needs to adapt to the same paradigm. The external API may be REST via HTTP, but that doesn’t mean you’ll get your answer, instead, you’ll get a 200 OK response, meaning your request was received. In order for your application to get actual results, it must subscribe to the specific event that contains the response.

Keep this in mind, otherwise, you’ll spend a lot of time debugging why you’re not getting the response you want.

So what do we need?

Having said that, what do we need to make our platform/architecture a reactive one? It’s not ReactJS, that’s for sure. We need a message broker, something that can centrally distribute messages across multiple services.

With something to act as a broker, we need to make sure that our code is written to subscribe to certain events, letting the broker know where it is and what type of event it needs.

After that, a notification will be sent to our service and our logic will be triggered.

Sounds simple, right? That’s because it is!

So how does Redis work here?

Redis is more than just a key-value memory store; in fact, it has three features THAT I like that allow me to create reactive architectures based on different expected behaviors.

The three functions are.

  • The Pub/Sub. Redis has an internal message queue that allows us to send messages and distribute them to each subscribed process. This is a fire-and-forget contract type, which means that if there is no active listener, the message will be lost. So keep this in mind when using this channel.
  • Key space notification. Probably my favorite Redis feature. These bad boys are events created by Redis itself and distributed to every process that decides to subscribe to them. They’re about changes in key space, that is, anything that happens to the data you store in it. For example, when you delete or update a key, or automatically delete it when its TTL counter reaches 0. Have you ever had to trigger a logic 3 days after “something” happens? That’s the way.
  • Redis flow. This is a mixture of Redis data types, mixing key space notifications and pub/sub, all put together and it works well. Streams attempt to mimic the behavior of the tail -f command on your terminal. If you’ve never seen this command before, it’s a *nix command that displays the last line of a file and keeps listening for changes to that file, so when you add a new line, it lists it immediately. The same thing happens with streams. They are very powerful and useful when used correctly. You can read more about them here.

All of these features allow you to communicate with your process in one way or another, and depending on the type of behavior you are pursuing, you may want to solve one or all of these problems.

Let’s take a quick look at some examples to give you an idea of when to use them.

Event-based classical messaging

The simplest example is that every microservice is waiting for something to happen. An event is fired, and the event may come from outside, a user or client of the system.

Taking a look at the image above, consider that the red tube in the center is Redis Pub/Sub or Blocking list, which is a more reliable custom implementation of Pub/Sub.

The process starts on the 1st with the client application submitting the request and ends on the 9th with the client application being notified of the response. What else? I don’t care, and neither should the client application.

This is one of the benefits of this model, the architecture becomes a black box for the customer. A request can trigger hundreds of events, or just one, and the behavior is the same: once the response is ready, it is delivered to the client. Rather than letting the client know how long it takes or how often it needs to check if it’s ready. None of that matters here.

Please remember the following precautions.

  • A message is posted to a “channel” by its subscribers. If you want to post different types of topics, it is recommended that you have different channels. In addition, if you need additional granularity to distinguish which consumers need to process a particular message, then the details need to be part of the message. This is because all subscribers to a channel get the same message, so if you have multiple processes listening and getting the same message, you may end up taking the same action again. A flag can be implemented in Redis with the ID of the message (for example) to ensure that the process that first created the message will be responsible for handling the event and that other processes can ignore it. This is a reliable approach because setting a key in Redis is an atomic process, so concurrency does not affect this factor.
  • If no subscriber listens for a particular channel, the published message is lost. This happens if you use Pub/Sub mode because it works under the “Fire & Forget” mechanism. If you want to make sure your information stays there until it’s processed, you can use a block list. This solution involves creating a list (that is, a normal list of values) directly on the key space of Redis and having the process subscribe to get key space notifications around that key. Then they can decide what to do with the inserted data (i.e. if they want to ignore it, process it and delete it, etc.).
  • If you are sending a complex message, such as JSON, it needs to be serialized. This is because for blocking lists and Pub/Sub, the only thing you can send is a string. That said, if you need to send complex types over wires without serialization, you might consider using Redis Streams, which they allow. The limitation here, of course, is that the only type allowed is Redis, not the type of the language you’re using to code your solution.

Now let’s look at what happens if your event trigger depends on certain times.

Time-based triggering

Another common behavior of reactive architectures is the ability to trigger events after a predefined time has passed. For example: trigger an alarm 10 minutes after a data problem is detected. Or wait 30 minutes and trigger an alarm that iot devices stop sending data.

These are usually behaviors related to real-world limitations that take a little time to work out, or can even be addressed by “waiting a little bit” and restarting the countdown in case they need it (such as iot devices that have an unreliable connection).

In this case, the architecture remains the same, the only difference being that the central communications hub definitely uses Redis’ key space notification.

You see, there are two main features about Redis that you need to know to do this.

  1. When setting up a key-value pair, you can optionally define a TTL in seconds. This becomes a countdown, and once it reaches zero, the key will destroy itself.
  2. When you subscribe to a key space (this also applies to pub/ Sub, but we don’t use it here), you can subscribe using a pattern. In other words, instead of subscribing to “last_connection_TIME_of_device100002”, you can subscribe to “last_connection_TIME_of_device *”. Then, each key created that fits that pattern will notify you when it changes.

With these two points in mind, you can create services that subscribe to these specific keys and react to them once they are removed (that is, when events are triggered). At the same time, you have the producer constantly update these keys, which also resets the TTL timer. So if you want to track when a device last sent a heartbeat, you can set up a key for each device, as I showed above, and update that key every time a new heartbeat is received. Once the TTL passes, it means that you did not receive a new heartbeat for the configured time. Your subscribing process will only receive the key name, so if all you need is the device ID, you can construct your key as I’ve shown and parse the name to capture the information you need.

Shadow key technology

On the other hand, if you have a complex structure inside that key and you need it, you have to change this approach. This is because when the TTL expires, the key is deleted, and therefore the data inside is also deleted, so you can’t really retrieve it. In this case, you can use a technique called shadow key.

The shadow key is essentially a key that is used to trigger events, but it is actually a shadow of the actual key that contains the data you need. So going back to our example, consider that the producer updates 2 keys every time he receives a heartbeat.

  • “Last_connection_time_of_device100002” is the Unix timestamp from which the payload was last received from the device.
  • “Device_data_id100002” carries additional information about the device.

Of the two keys, only the first will have TTL; the second will not. Therefore, when you receive an expiration notification, you get the ID (last_Connection_TIME_of_DEVICe100002) from the expired key and use it to read the contents of the second key. Then, if you want, you can go ahead and delete this other key, or leave it there, regardless of your usage.

The only thing to consider here is that if you configure Redis in cluster mode, keyspace notifications are not propagated throughout the cluster. This means that you have to make sure your consumers are connected to every node. Some notifications get lost because no one receives them. This is the only downside to this technique, but it’s important to understand it (been there, done there) before you spend days debugging your asynchronous logic.

As you can see, the complexity in both cases is reduced to just making sure you subscribe to the right events or distribution channels. If you try to do this with a normal SQL database, you have to create a lot of logic around your code to effectively determine when new data is coming in, or when a piece of information is being deleted.

Instead, all the complexity is abstracted by Redis, and all you need to worry about is writing your business logic. To me, that’s gold.

Do you use Redis for any other scenarios besides caching? Please share your experiences with others in the comments, I’d love to know how you use one of my favorite technologies.

Build and share separate JS components with bits

Bitbits is an extensible tool that lets you create truly modular applications with components written, versioned, and maintained independently.

Use it to build modular applications and design systems, write and deliver miniature front ends, or simply share components between applications.

A separate source control and shared “card” component (right → its dependency diagram, automatically generated by Bit).

Bits: platform for modular networks

To learn more

  • Will you be replaced by code-generated ARTIFICIAL intelligence?
  • Component-driven development using bits
  • Independent components. New building blocks for the network

Building an Reactive Architecture around Redis was originally published on Medium’s Bits and Pieces, where people continue the conversation by highlighting and responding to the story.