In the previous article “Logstash Translate Filter Introduction,” I covered in detail how to use the Translate filter to enrich our data. In the article “Enriching data with Elasticsearch filters”, I also showed you how to enrich data with Elasticsearch filters. Although Elasticsearch and Translate filters are good for small workloads, we can improve this situation by providing an extensible scaling layer that does not remain state on a single Logstash node.

For example, using the memcached Filter plug-in, you can do a very fast, non-blocking lookup of what you want to match. Examples are malware requests, threat data (known wrong IP), and even static resources (server IP for host look-ups). Lookup of the memory cache is a simple key/value lookup. Another benefit of memcache is that it does not block updates. Because of our heavy read workload, we don’t want to prevent lookups when updating the extended key. Because the memory cache is volatile and does not persist after a restart, you should take this into account when refreshing the data and ensure that this information is stored in persistent storage so that the memory cache can be repopulated as needed.

With memory caches and memory cache pools, we can scale and push as much traffic as needed. With proper configuration, a single memory cache instance can conservatively handle 100,000 queries per second. For each in-memory cached document, they suggested 200K +, but with appropriate adjustments, we see even higher numbers. In a pool configuration, this scales linearly.

The new Memcached Filter Plugin supports the following:

  • Hash Memcache pool through consistency
  • The namespace
  • Multi- Get/ Multi-Set

 

Below, we’ll use a concrete example to show how to use it.

 

The installation

Let’s start by installing memcached. I did it on an Ubuntu 20.04 machine. The Memcached package is included in the Ubuntu default repository and the installation process is simple… Simply run the following command to install it along with the support tools…

sudo apt update
sudo apt install memcached libmemcached-tools
Copy the code

After you run the above command, the Memcached server should be installed and available to use… To check its status, run the following command:

sudo systemctl status memcached
Copy the code

If there are no errors, you should see output similar to the following:

$sudo systemctl status memcached [sudo] password for liuxg: loaded (/lib/systemd/system/memcached.service; enabled; vendor pre> Active: active (running) since Tue 2020-06-23 09:17:06 CST; 17min ago Docs: man:memcached(1) Main PID: 905 (memcached) Tasks: 10 (limit: 18985) Memory: 2.5m CGroup: / system. Slice/memcached service └ ─ 905 / usr/bin/memcached -m 64-11211 - u p memcache -l 127.0.0.1 June 23 09:17:06 - > liuxgu systemd[1]: Started memcached daemon.Copy the code

This is how you install Memcached on Ubuntu… The server should be running and should respond to requests… The following commands can be used to stop, start, and enable Memcached:

sudo systemctl stop memcached.service
sudo systemctl start memcached.service
sudo systemctl enable memcached.service
Copy the code

Configure the Memcached

Now that the server is installed, you can find its configuration file in /etc/memcached.conf.

The default Settings in the file are sufficient for most environments and applications… However, for more advanced Settings, open the file and make changes to apply to your environment…

For example, Memcached listens for the local IP address of the server (127.0.0.1)… If you only want it to listen on other IP, edit the lines in the file to look like the following:

sudo vi /etc/memcached.conf
Copy the code

Then replace the local server IP with the IP address you want to use. You can also change its default port number…

# Default connection port is 11211
-p 11211

# Run the daemon as root. The start-memcached will default to running as root if no
# -u command is present in this config file
-u memcache
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1
Copy the code

Save the file and exit… Then restart the Memcached service to apply the changes…

Use Memcached

Developers of Memcached is not very well, you can refer to the link www.tutorialspoint.com/memcached/m… .

The Memcached set command is used to set a new value to a new or existing key. Grammar:

set key flags exptime bytes [noreply] 
value 
Copy the code

The keywords in the syntax are described below –

  • Key – This is the name of the key that stores and retrieves data from Memcached.
  • Flags – This is the 32-bit unsigned integer that the server stores along with user-supplied data and will return with the data when the item is retrieved.
  • Exptime – This is the expiration time, in seconds. 0 indicates no delay. If expTime exceeds 30 days, Memcached will use it as a UNIX timestamp expiration.
  • Bytes – This is the number of bytes that need to be stored in the data block. This is the length of data that needs to be stored in Memcached.
  • Noreply (optional) – This is a parameter that informs the server not to send any replies.
  • Value – It is the data that needs to be stored. After executing the command with the options above, you need to pass data in a new line.

In the following example, we use 1.1.1.1 as the key and set the value to botip for 900 seconds.

$Telnet localhost 11211 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Set 1.1.1.1 0 900 5 botip STORED get 1.1.1.1 VALUE 1.1.1.1 0 5 botip ENDCopy the code

Run Logstash

Here is a complete example with the logstash configuration file, inputs and outputs:

logstash_memcache.conf

input { stdin { codec => json } } filter { memcached { hosts => ["localhost:11211"] get => { "%{ip}" => "threat_src" } }  } output { stdout { codec => rubydebug } }Copy the code

Above we used the memcached filter to enrich our data. When the data matches the memcahed data, assign the rich data to the Threat_src field. Run our Logstash:

./bin/logstash -f logstash_memcache.conf 
Copy the code

We enter json in the Logstash console as follows:

{" IP ": "1.1.1.1"}Copy the code

We can see the following output:

Above, we can see that the value of thread_srCE is assigned to botip. The botip value comes from Memcached.

When we enter json like this:

{" IP ": "2.2.2.2"}Copy the code

The output of Logstash is:

Since there is no matching IP 2.2.2.2 in Memcached, there is no rich data.

Let’s add 2.2.2.2 in the same way as above.

Above, we added the corresponding value of 2.2.2.2 as liuxG 5 letters. So let’s type again:

{" IP ": "2.2.2.2"}Copy the code

This time, we see a richer set of data.

Reference:

【 1 】 websiteforstudents.com/install-mem…

(2) www.tutorialspoint.com/memcached/m…

【 3 】 www.securitydistractions.com/2019/05/17/…