将所有的memcached密钥转储到文件中最简单的方法是什么?

This is from just a single memcached server with around 20M keys (no expiry) and around 2G of data.

这只是一个拥有大约20M密钥(没有过期)和大约2G数据的memcached服务器。

What's the easiest way to get a dump of all the key/value pairs into a flat file? I first looked at the java net.spy.memcached.MemcachedClient, but this client does not support getting all keys (I think). If I had a list of all keys (which I don't), I could easily use this client to get all of the values.

将所有键/值对转储到平面文件中最简单的方法是什么?我首先查看了java net.spy.memcached。MemcachedClient，但是这个客户机不支持获取所有的键(我认为)。如果我有一个所有键的列表(我没有)，我可以很容易地使用这个客户端来获取所有的值。

I know I can get all keys using some telnet commands (e.g., telnet localhost 11211; stats items; stats cachedump ), but it's not clear to me how to automate this robustly.

我知道我可以使用一些telnet命令(例如，telnet localhost 11211;统计项目;stats cachedump)，但是我不清楚如何将其自动化。

EDIT: Here's what I did to get this working on a toy memcached server on my machine. It seems to work but I only put two keys in memcached, so hopefully this method will scale ok:

编辑:这是我在我的机器上的一个玩具memcached服务器上做的工作。它似乎有用，但我只在memcached内放了两个键，所以希望这个方法可以缩放:

shell commands:

shell命令:

sudo yum install memcached
sudo /etc/init.d/memcached restart # maybe unnecessary
sudo yum install php
sudo yum install php-pecl-memcache
sudo service httpd reload

php script, based on this:

php脚本，基于此:

<?php
$memcache = new Memcache();
$memcache->connect('127.0.0.1', 11211) or die ("Could not connect");
$list = array();
$allSlabs = $memcache->getExtendedStats('slabs');
$items = $memcache->getExtendedStats('items');
foreach($allSlabs as $server => $slabs) {
    foreach($slabs AS $slabId => $slabMeta) {
        if (!is_int($slabId)) {
            continue;
        }
        $cdump = $memcache->getExtendedStats('cachedump', (int) $slabId, 100000000);
        foreach($cdump AS $server => $entries) {
            if ($entries) {
                foreach($entries AS $eName => $eData) {
                    print_r($eName);
                    print_r(":");
                    $val = $memcache->get($eName);
                    print_r($val);
                    print_r("\n");
                }
            }
        }
    }
}
?>

EDIT2: The above script does not seem to return all of the mappings. If I insert the line count($entries), it only returns a little over 50k, even with the limit parameter set to 100M, but executing stats items from telnet shows over 5M entries. Does anyone know why this might be the case?

上面的脚本似乎没有返回所有的映射。如果我插入行计数($entries)，它只返回稍微超过50k的值，即使将限制参数设置为100M，但是执行telnet的stats条目将显示超过500万个条目。有人知道为什么会这样吗?

EDIT3: This link suggests that cachedump doesn't get all keys from memcached. I've hit a limit of around 50k keys that are returned either by cachedump, this PHP script, or a perl script similar to one in the link provided by Zach Bonham. Is there any way around this?

这个链接表明cachedump并不能从缓存的memedump中获得所有的键。我已经达到了50k左右键的限制，这些键可以由cachedump、这个PHP脚本或者与Zach Bonham提供的链接中类似的perl脚本返回。有办法解决这个问题吗?

4 个解决方案

#1

disclaimer: I don't know what I"m doing, just sounded like an interesting problem.

免责声明:我不知道我在做什么，只是听起来是个有趣的问题。

Did you see this article? "How to Dump Keys from Memcache" by Lars Windolf.

你看到这篇文章了吗?《如何从Memcache中转储密钥》，作者Lars Windolf。

From the article:

从这篇文章:

Memcache itself provide the means to peak into the data. The protocol provides commands to peak into the data that is organized by slabs (categories of data of a given size range. There are some significant limitations though:

Memcache本身提供了进入数据峰值的方法。该协议提供了一些命令，可以将其峰值化到由slabs组织的数据(给定大小范围内的数据类别)。尽管有一些明显的限制:

• You can only dump keys per slab class (keys with roughly the same content size)
• 您只能在每个板类中转储密钥(具有大致相同内容大小的键)
• You can only dump one page per slab class (1MB of data)
• 您只能向每个平板类转储一个页面(1MB的数据)
• This is an unofficial feature that might be removed anytime.
• 这是一个非官方的特性，随时可能被删除。

Effectively, it requires some knowledge of how memcache stores data in memory (which I don't). You need to find each 'slab', then you can dump the keys for that slab, and then ultimately, the values for those keys.

实际上，它需要了解memcache如何在内存中存储数据(我没有)。你需要找到每一个“板子”，然后你可以转储板子的键，最后，这些键的值。

There is a tools section in the article which uses various languages to dump at least the keys, but only the perl script dumps both keys and values.

本文中有一个工具部分，它使用各种语言至少转储键，但只有perl脚本同时转储键和值。

#2

memccat

Here is the script which I'm using to dump all the objects into corresponding files:

下面是我用来将所有对象转储到相应文件中的脚本:

while read -r key; do
    [ -f "$key" ] || echo "get $key" | nc localhost 11211 > "$key.dump";
done < <(memcdump --server localhost)

It uses memcdump command which should be part of memcached utils.

它使用memcdump命令，该命令应该是memcached utils的一部分。

For compressed objects, see: How to dump a compressed object for given key from Memcache?

对于压缩对象，请参见:如何从Memcache中转储给定键的压缩对象?

memcdump

To dump a list of keys from a server, use memcdump/memdump tool, e.g.

要从服务器转储密钥列表，请使用memcdump/memdump工具，例如。

memcdump --servers=localhost | tee my_keys.lst

To print the value of one item, use netcat:

若要打印一项的值，请使用netcat:

echo "get 13456_-cache-some_object" | nc localhost 11211

To dump all objects into the screen via memcdump/memdump and netcat:

通过memcdump/memdump和netcat将所有对象转储到屏幕:

memcdump --servers=localhost | xargs -L1 -I% sh -c 'echo "get %" | nc localhost 11211'

Bash

Using Bash and save into the file:

使用Bash并保存到文件中:

exec {memcache}<>/dev/tcp/localhost/11211
printf "stats items\nquit\n" >&${memcache}
cat <&${memcache} > myfile.txt

Related: Writing a Redis client in pure bash (it's Redis, but very similar approach)

相关:在纯bash中编写一个Redis客户机(它是Redis，但非常类似)

memcached-tool

In the recent version of memcached there is also memcached-tool command, e.g.

在memcached最近的版本中，还有一个memcached-tool命令，例如。

memcached-tool localhost:11211 dump | less # dumps keys and values

#3

There is a hardcoded limit of 2MB for the dump of a slab. Unless you rewrite the do_item_cachedump, you will not be able to get all keys out.

对于平板的转储，硬编码的限制是2MB。除非您重写do_item_cachedump，否则您将无法获得所有的密钥。

#4

I used this bash script

我使用了这个bash脚本

#!/bin/sh
MESSAGE=`memdump --servers="127.0.0.1"`
while read -r line; do
    echo $line
    VALUE=`echo "get $line" | nc 127.0.0.1 11211`
    echo $VALUE
done <<< "$MESSAGE"

just replace IP / port if necessary

如果需要，只需替换IP /端口