如何删除Google App Engine中的所有数据存储区？

Does anyone know how to delete all datastore in Google App Engine?

有谁知道如何删除Google App Engine中的所有数据存储区?

27 个解决方案

#1

If you're talking about the live datastore, open the dashboard for your app (login on appengine) then datastore --> dataviewer, select all the rows for the table you want to delete and hit the delete button (you'll have to do this for all your tables). You can do the same programmatically through the remote_api (but I never used it).

如果您正在谈论实时数据存储,请打开应用程序的仪表板(登录appengine)然后数据存储 - > dataviewer,选择要删除的表的所有行并点击删除按钮(您必须为你的所有表格做这个)。您可以通过remote_api以编程方式执行相同操作(但我从未使用过它)。

If you're talking about the development datastore, you'll just have to delete the following file: "./WEB-INF/appengine-generated/local_db.bin". The file will be generated for you again next time you run the development server and you'll have a clear db.

如果您正在讨论开发数据存储区,则只需删除以下文件:“./ WWE-INF / appengine-Generated / local_db.bin”。下次运行开发服务器时,将再次为您生成该文件,并且您将拥有一个明确的数据库。

Make sure to clean your project afterwards.

确保事后清理项目。

This is one of the little gotchas that come in handy when you start playing with the Google Application Engine. You'll find yourself persisting objects into the datastore then changing the JDO object model for your persistable entities ending up with obsolete data that'll make your app crash all over the place.

这是您开始使用Google Application Engine时派上用场的小问题之一。您会发现自己将对象持久存储到数据存储区中,然后更改可持久化实体的JDO对象模型,最终使用过时的数据,这会使您的应用程序在整个地方崩溃。

#2

The best approach is the remote API method as suggested by Nick, he's an App Engine engineer from Google, so trust him.

最好的方法是Nick建议的远程API方法,他是Google的App Engine工程师,所以相信他。

It's not that difficult to do, and the latest 1.2.5 SDK provides the remote_shell_api.py out of the shelf. So go to download the new SDK. Then follow the steps:

这并不困难,最新的1.2.5 SDK提供了现成的remote_shell_api.py。所以去下载新的SDK。然后按照步骤操作:

• connect remote server in your commandline: remote_shell_api.py yourapp /remote_api The shell will ask for your login info, and if authorized, will make a Python shell for you. You need setup url handler for /remote_api in your app.yaml

  在命令行中连接远程服务器:remote_shell_api.py yourapp / remote_api shell将询问您的登录信息,如果获得授权,将为您创建一个Python shell。您需要在app.yaml中为/ remote_api设置url处理程序

• fetch the entities you'd like to delete, the code looks something like:

  获取您要删除的实体,代码如下所示:

    from models import Entry
    query = Entry.all(keys_only=True)
    entries =query.fetch(1000)
    db.delete(entries)
    \# This could bulk delete 1000 entities a time

Update 2013-10-28:

• remote_shell_api.py has been replaced by remote_api_shell.py, and you should connect with remote_api_shell.py -s your_app_id.appspot.com, according to the documentation.

  remote_shell_api.py已被remote_api_shell.py取代,您应该根据文档连接remote_api_shell.py -s your_app_id.appspot.com。

• There is a new experimental feature Datastore Admin, after enabling it in app settings, you can bulk delete as well as backup your datastore through the web ui.

  有一个新的实验性功能Datastore Admin,在应用设置中启用后,您可以批量删除以及通过web ui备份数据存储。

#3

The fastest and efficient way to handle bulk delete on Datastore is by using the new mapper API announced on the latest Google I/O.

在数据存储上处理批量删除的最快速有效的方法是使用最新的Google I / O上公布的新映射器API。

If your language of choice is Python, you just have to register your mapper in a mapreduce.yaml file and define a function like this:

如果您选择的语言是Python,则只需在mapreduce.yaml文件中注册mapper并定义如下函数:

from mapreduce import operation as op
def process(entity):
 yield op.db.Delete(entity)

On Java you should have a look to this article that suggests a function like this:

在Java上你应该看看这篇文章,它建议这样的函数:

@Override
public void map(Key key, Entity value, Context context) {
    log.info("Adding key to deletion pool: " + key);
    DatastoreMutationPool mutationPool = this.getAppEngineContext(context)
            .getMutationPool();
    mutationPool.delete(value.getKey());
}

EDIT:
Since SDK 1.3.8, there's a Datastore admin feature for this purpose

编辑:自SDK 1.3.8以来,为此目的有一个数据存储管理功能

#4

You can clear the development server datastore when you run the server:

您可以在运行服务器时清除开发服务器数据存储:

/path/to/dev_appserver.py --clear_datastore=yes myapp

/path/to/dev_appserver.py --clear_datastore = yes myapp

You can also abbreviate --clear_datastore with -c.

您也可以使用-c缩写--clear_datastore。

#5

If you have a significant amount of data, you need to use a script to delete it. You can use remote_api to clear the datastore from the client side in a straightforward manner, though.

如果您有大量数据,则需要使用脚本将其删除。但是,您可以使用remote_api以直接的方式从客户端清除数据存储区。

#6

Here you go: Go to Datastore Admin, and then select the Entity type you want to delete and click Delete. Mapreduce will take care of deleting!

在这里:转到数据存储区管理员,然后选择要删除的实体类型,然后单击“删除”。 Mapreduce将负责删除!

#7

There are several ways you can use to remove entries from App Engine's Datastore:

您可以使用多种方法从App Engine的数据存储区中删除条目:

1. First, think whether you really need to remove entries. This is expensive and it might be cheaper to not remove them.

   首先,想一想你是否真的需要删除条目。这很昂贵,不去除它们可能更便宜。

2. You can delete all entries by hand using the Datastore Admin.

   您可以使用数据存储管理员手动删除所有条目。

3. You can use the Remote API and remove entries interactively.

   您可以使用远程API并以交互方式删除条目。

4. You can remove the entries programmatically using a couple lines of code.

   您可以使用几行代码以编程方式删除条目。

5. You can remove them in bulk using Task Queues and Cursors.

   您可以使用任务队列和游标批量删除它们。

6. Or you can use Mapreduce to get something more robust and fancier.

   或者你可以使用Mapreduce来获得更强大和更漂亮的东西。

Each one of these methods is explained in the following blog post: http://www.shiftedup.com/2015/03/28/how-to-bulk-delete-entries-in-app-engine-datastore

以下博客文章中解释了这些方法中的每一种:http://www.shiftedup.com/2015/03/28/how-to-bulk-delete-entries-in-app-engine-datastore

Hope it helps!

希望能帮助到你!

#8

The zero-setup way to do this is to send an execute-arbitrary-code HTTP request to the admin service that your running app already, automatically, has:

执行此操作的零设置方法是向管理服务发送执行任意代码HTTP请求,您的运行应用程序已经自动执行:

import urllib
import urllib2

urllib2.urlopen('http://localhost:8080/_ah/admin/interactive/execute',
    data = urllib.urlencode({'code' : 'from google.appengine.ext import db\n' +
                                      'db.delete(db.Query())'}))

#9

Source

I got this from http://code.google.com/appengine/articles/remote_api.html.

我从http://code.google.com/appengine/articles/remote_api.html获得了此信息。

Create the Interactive Console

First, you need to define an interactive appenginge console. So, create a file called appengine_console.py and enter this:

首先,您需要定义一个交互式的appenginge控制台。因此,创建一个名为appengine_console.py的文件并输入:

#!/usr/bin/python
import code
import getpass
import sys

# These are for my OSX installation. Change it to match your google_appengine paths. sys.path.append("/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine")
sys.path.append("/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/yaml/lib")

from google.appengine.ext.remote_api import remote_api_stub
from google.appengine.ext import db

def auth_func():
  return raw_input('Username:'), getpass.getpass('Password:')

if len(sys.argv) < 2:
  print "Usage: %s app_id [host]" % (sys.argv[0],)
app_id = sys.argv[1]
if len(sys.argv) > 2:
  host = sys.argv[2]
else:
  host = '%s.appspot.com' % app_id

remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api', auth_func, host)

code.interact('App Engine interactive console for %s' % (app_id,), None, locals())

Create the Mapper base class

Once that's in place, create this Mapper class. I just created a new file called utils.py and threw this:

一旦到位,创建此Mapper类。我刚刚创建了一个名为utils.py的新文件并将其抛出:

class Mapper(object):
  # Subclasses should replace this with a model class (eg, model.Person).
  KIND = None

  # Subclasses can replace this with a list of (property, value) tuples to filter by.
  FILTERS = []

  def map(self, entity):
    """Updates a single entity.

    Implementers should return a tuple containing two iterables (to_update, to_delete).
    """
    return ([], [])

  def get_query(self):
    """Returns a query over the specified kind, with any appropriate filters applied."""
    q = self.KIND.all()
    for prop, value in self.FILTERS:
      q.filter("%s =" % prop, value)
    q.order("__key__")
    return q

  def run(self, batch_size=100):
    """Executes the map procedure over all matching entities."""
    q = self.get_query()
    entities = q.fetch(batch_size)
    while entities:
      to_put = []
      to_delete = []
      for entity in entities:
        map_updates, map_deletes = self.map(entity)
        to_put.extend(map_updates)
        to_delete.extend(map_deletes)
      if to_put:
        db.put(to_put)
      if to_delete:
        db.delete(to_delete)
      q = self.get_query()
      q.filter("__key__ >", entities[-1].key())
      entities = q.fetch(batch_size)

Mapper is supposed to be just an abstract class that allows you to iterate over every entity of a given kind, be it to extract their data, or to modify them and store the updated entities back to the datastore.

Mapper应该只是一个抽象类,它允许您迭代给定类型的每个实体,无论是提取数据,还是修改它们并将更新的实体存储回数据存储区。

Run with it!

Now, start your appengine interactive console:

现在,启动您的appengine交互式控制台:

$python appengine_console.py <app_id_here>

That should start the interactive console. In it create a subclass of Model:

这应该启动交互式控制台。在其中创建Model的子类:

from utils import Mapper
# import your model class here 
class MyModelDeleter(Mapper):
    KIND = <model_name_here>

    def map(self, entity):
        return ([], [entity])

And, finally, run it (from you interactive console): mapper = MyModelDeleter() mapper.run()

最后,运行它(从您的交互式控制台):mapper = MyModelDeleter()mapper.run()

That's it!

#10

You can do it using the web interface. Login into your account, navigate with links on the left hand side. In Data Store management you have options to modify and delete data. Use respective options.

您可以使用Web界面执行此操作。登录您的帐户,使用左侧的链接进行导航。在Data Store管理中,您可以选择修改和删除数据。使用相应的选项。

#11

I've created an add-in panel that can be used with your deployed App Engine apps. It lists the kinds that are present in the datastore in a dropdown, and you can click a button to schedule "tasks" that delete all entities of a specific kind or simply everything. You can download it here:
http://code.google.com/p/jobfeed/wiki/Nuke

我创建了一个可以与已部署的App Engine应用程序一起使用的加载项面板。它列出了下拉列表中数据存储区中存在的种类,您可以单击按钮来安排删除特定类型的所有实体或仅删除所有实体的“任务”。您可以在此处下载:http://code.google.com/p/jobfeed/wiki/Nuke

#12

For Python, 1.3.8 includes an experimental admin built-in for this. They say: "enable the following builtin in your app.yaml file:"

对于Python,1.3.8包含一个内置的实验管理员。他们说:“在app.yaml文件中启用以下内置功能:”

builtins:
- datastore_admin: on

"Datastore delete is currently available only with the Python runtime. Java applications, however, can still take advantage of this feature by creating a non-default Python application version that enables Datastore Admin in the app.yaml. Native support for Java will be included in an upcoming release."

“数据存储删除目前仅适用于Python运行时。但是,Java应用程序仍然可以通过创建在app.yaml中启用数据存储管理的非默认Python应用程序版本来利用此功能。将包含对Java的本机支持在即将发布的版本中。“

#13

Open "Datastore Admin" for your application and enable Admin. Then all of your entities will be listed with check boxes. You can simply select the unwanted entites and delete them.

为您的应用程序打开“Datastore Admin”并启用Admin。然后将列出所有实体的复选框。您只需选择不需要的entites并删除它们即可。

#14

This is what you're looking for...

这就是你要找的......

db.delete(Entry.all(keys_only=True))

Running a keys-only query is much faster than a full fetch, and your quota will take a smaller hit because keys-only queries are considered small ops.

运行仅密钥查询比完全提取要快得多,并且您的配额将受到较小的影响,因为仅密钥查询被视为小操作。

Here's a link to an answer from Nick Johnson describing it further.

这是Nick Johnson对答案的一个链接。

Below is an end-to-end REST API solution to truncating a table...

下面是截断表格的端到端REST API解决方案......

I setup a REST API to handle database transactions where routes are directly mapped through to the proper model/action. This can be called by entering the right url (example.com/inventory/truncate) and logging in.

我设置了一个REST API来处理数据库事务,其中路由直接映射到正确的模型/操作。这可以通过输入正确的URL(example.com/inventory/truncate)并登录来调用。

Here's the route:

这是路线:

Route('/inventory/truncate', DataHandler, defaults={'_model':'Inventory', '_action':'truncate'})

Here's the handler:

这是处理程序:

class DataHandler(webapp2.RequestHandler):
  @basic_auth
  def delete(self, **defaults):
    model = defaults.get('_model')
    action = defaults.get('_action')
    module = __import__('api.models', fromlist=[model])
    model_instance = getattr(module, model)()
    result = getattr(model_instance, action)()

It starts by loading the model dynamically (ie Inventory found under api.models), then calls the correct method (Inventory.truncate()) as specified in the action parameter.

它首先动态加载模型(即在api.models下找到的Inventory),然后调用action参数中指定的正确方法(Inventory.truncate())。

The @basic_auth is a decorator/wrapper that provides authentication for sensitive operations (ie POST/DELETE). There's also an oAuth decorator available if you're concerned about security.

@basic_auth是一个装饰器/包装器,为敏感操作(即POST / DELETE)提供身份验证。如果您担心安全问题,还可以使用oAuth装饰器。

Finally, the action is called:

最后,该动作被称为:

def truncate(self):
  db.delete(Inventory.all(keys_only=True))

It looks like magic but it's actually very straightforward. The best part is, delete() can be re-used to handle deleting one-or-many results by adding another action to the model.

它看起来像魔术,但它实际上非常简单。最好的部分是,delete()可以通过向模型添加另一个动作来重复使用来处理删除一个或多个结果。

#15

You can Delete All Datastore by deleting all Kinds One by One. with google appengine dash board. Please follow these Steps.

您可以通过逐个删除所有种类来删除所有数据存储。与谷歌appengine仪表板。请按照以下步骤操作。

1. Login to https://console.cloud.google.com/datastore/settings

登录https://console.cloud.google.com/datastore/settings

2. Click Open Datastore Admin. (Enable it if not enabled.)

单击“打开数据存储管理”。 (如果未启用,则启用它。)

3. Select all Entities and press delete.(This Step run a map reduce job for deleting all selected Kinds.)

选择所有实体并按删除。(此步骤运行地图减少作业以删除所有选定的种类。)

for more information see This image http://storage.googleapis.com/bnifsc/Screenshot%20from%202015-01-31%2023%3A58%3A41.png

有关更多信息,请参阅此图像http://storage.googleapis.com/bnifsc/Screenshot%20from%202015-01-31%2023%3A58%3A41.png

#16

If you have a lot of data, using the web interface could be time consuming. The App Engine Launcher utility lets you delete everything in one go with the 'Clear datastore on launch' checkbox. This utility is now available for both Windows and Mac (Python framework).

如果您有大量数据,使用Web界面可能非常耗时。 App Engine Launcher实用程序允许您使用“启动时清除数据存储”复选框一次性删除所有内容。此实用程序现在可用于Windows和Mac(Python框架)。

#17

For the development server, instead of running the server through the google app engine launcher, you can run it from the terminal like:

对于开发服务器,您可以从终端运行它,而不是通过谷歌应用引擎启动器运行服务器,如:

dev_appserver.py --port=[portnumber] --clear_datastore=yes [nameofapplication]

dev_appserver.py --port = [portnumber] --clear_datastore = yes [nameofapplication]

ex: my application "reader" runs on port 15080. After modify the code and restart the server, I just run "dev_appserver.py --port=15080 --clear_datastore=yes reader".

例如:我的应用程序“reader”在端口15080上运行。修改代码并重新启动服务器后,我只运行“dev_appserver.py --port = 15080 --clear_datastore = yes reader”。

It's good for me.

这对我有好处。

#18

Adding answer about recent developments.

增加关于最近发展的答案。

Google recently added datastore admin feature. You can backup, delete or copy your entities to another app using this console.

Google最近添加了数据存储管理功能。您可以使用此控制台将实体备份,删除或复制到另一个应用程序。

https://developers.google.com/appengine/docs/adminconsole/datastoreadmin#Deleting_Entities_in_Bulk

#19

I often don't want to delete all the data store so I pull a clean copy of /war/WEB-INF/local_db.bin out source control. It may just be me but it seems even with the Dev Mode stopped I have to physically remove the file before pulling it. This is on Windows using the subversion plugin for Eclipse.

我经常不想删除所有数据存储,所以我将/war/WEB-INF/local_db.bin的干净副本拉出源代码控制。它可能只是我,但似乎即使启动模式停止我必须在拉动之前物理删除文件。这是在Windows上使用Eclipse的subversion插件。

#20

PHP variation:

import com.google.appengine.api.datastore.Query;
import com.google.appengine.api.datastore.DatastoreServiceFactory;

define('DATASTORE_SERVICE', DatastoreServiceFactory::getDatastoreService());

function get_all($kind) {
    $query = new Query($kind);
    $prepared = DATASTORE_SERVICE->prepare($query);
    return $prepared->asIterable();
}

function delete_all($kind, $amount = 0) {
    if ($entities = get_all($kind)) {
        $r = $t = 0;
        $delete = array();
        foreach ($entities as $entity) {
            if ($r < 500) {
                $delete[] = $entity->getKey();
            } else {
                DATASTORE_SERVICE->delete($delete);
                $delete = array();
                $r = -1;
            }
            $r++; $t++;
            if ($amount && $amount < $t) break;
        }
        if ($delete) {
            DATASTORE_SERVICE->delete($delete);
        }
    }
}

Yes it will take time and 30 sec. is a limit. I'm thinking to put an ajax app sample to automate beyond 30 sec.

是的,需要时间和30秒。是一个限制。我想把一个ajax app样本自动化超过30秒。

#21

for amodel in db.Model.__subclasses__():
                dela=[]
                print amodel
                try:
                    m = amodel()
                    mq = m.all()
                    print mq.count()
                    for mw in mq:
                        dela.append(mw)
                    db.delete(dela)
            #~ print len(dela)

                except:
                    pass

#22

If you're using ndb, the method that worked for me for clearing the datastore:

如果您正在使用ndb,那么清除数据存储区的方法对我有用:

ndb.delete_multi(ndb.Query(default_options=ndb.QueryOptions(keys_only=True)))

#23

For any datastore that's on app engine, rather than local, you can use the new Datastore API. Here's a primer for how to get started.

对于应用引擎上的任何数据存储,而不是本地数据存储,您可以使用新的数据存储API。这是如何入门的入门读物。

I wrote a script that deletes all non-built in entities. The API is changing pretty rapidly, so for reference, I cloned it at commit 990ab5c7f2063e8147bcc56ee222836fd3d6e15b

我写了一个删除所有非内置实体的脚本。 API正在快速变化,所以为了参考,我在提交时克隆了它990ab5c7f2063e8147bcc56ee222836fd3d6e15b

from gcloud import datastore
from gcloud.datastore import SCOPE
from gcloud.datastore.connection import Connection
from gcloud.datastore import query

from oauth2client import client

def get_connection():
  client_email = '[e-mail]'
  private_key_string = open('/path/to/yourfile.p12', 'rb').read()

  svc_account_credentials = client.SignedJwtAssertionCredentials(
    service_account_name=client_email,
    private_key=private_key_string,
    scope=SCOPE)

  return Connection(credentials=svc_account_credentials)


def connect_to_dataset(dataset_id):
  connection = get_connection()
  datastore.set_default_connection(connection)
  datastore.set_default_dataset_id(dataset_id)

if __name__ == "__main__":
  connect_to_dataset(DATASET_NAME)
  gae_entity_query = query.Query()
  gae_entity_query.keys_only()
  for entity in gae_entity_query.fetch():
    if entity.kind[0] != '_':
      print entity.kind
      entity.key.delete()

#24

• continuing the idea of svpino it is wisdom to reuse records marked as delete. (his idea was not to remove, but mark as "deleted" unused records). little bit of cache/memcache to handle working copy and write only difference of states (before and after desired task) to datastore will make it better. for big tasks it is possible to write itermediate difference chunks to datastore to avoid data loss if memcache disappeared. to make it loss-proof it is possible to check integrity/existence of memcached results and restart task (or required part) to repeat missing computations. when data difference is written to datastore, required computations are discarded in queue.

  继续svpino的想法,重用标记为删除的记录是明智之举。 (他的想法不是删除,而是标记为“已删除”未使用的记录)。处理工作副本的一点点缓存/内存缓存和只写状态(在所需任务之前和之后)到数据存储区的差异将使其更好。对于大任务,可以将中间差异块写入数据存储区,以避免数据丢失,如果memcache消失。为了使其防止损失,可以检查memcached结果的完整性/存在性并重新启动任务(或必需部分)以重复丢失的计算。当数据差异写入数据存储区时,队列中将丢弃所需的计算。

• other idea similar to map reduced is to shard entity kind to several different entity kinds, so it will be collected together and visible as single entity kind to final user. entries are only marked as "deleted". when "deleted" entries amount per shard overcomes some limit, "alive" entries are distributed between other shards, and this shard is closed forever and then deleted manually from dev console (guess at less cost) upd: seems no drop table at console, only delete record-by-record at regular price.

  类似于地图缩减的其他想法是将实体种类分解为几种不同的实体类型,因此它将被收集在一起并作为单个实体类可见给最终用户。条目仅标记为“已删除”。当每个分片的“已删除”条目数量超过某个限制时,“活动”条目将在其他分片之间分配,并且此分片将永久关闭,然后从开发控制台手动删除(猜测成本较低)upd:在控制台上似乎没有丢弃表,只能以正常价格删除记录。

• it is possible to delete by query by chunks large set of records without gae failing (at least works locally) with possibility to continue in next attempt when time is over:

  有可能通过查询大块的记录删除没有gae失败的记录(至少在本地工作),并有可能在时间结束时继续下一次尝试:

    qdelete.getFetchPlan().setFetchSize(100);

    while (true)
    {
        long result = qdelete.deletePersistentAll(candidates);
        LOG.log(Level.INFO, String.format("deleted: %d", result));
        if (result <= 0)
            break;
    }

• also sometimes it useful to make additional field in primary table instead of putting candidates (related records) into separate table. and yes, field may be unindexed/serialized array with little computation cost.

有时,在主表中创建附加字段而不是将候选(相关记录)放入单独的表中也很有用。是的,字段可以是无索引/序列化的数组,只需很少的计算成本。

#25

For all people that need a quick solution for the dev server (as time of writing in Feb. 2016):

对于需要开发服务器快速解决方案的所有人(截至2016年2月撰写时):

1. Stop the dev server.

停止开发服务器。

2. Delete the target directory.

删除目标目录。

3. Rebuild the project.

重建项目。

This will wipe all data from the datastore.

这将擦除数据存储区中的所有数据。

#26

I was so frustrated about existing solutions for deleting all data in the live datastore that I created a small GAE app that can delete quite some amount of data within its 30 seconds.

我对现有的删除实时数据存储中的所有数据的解决方案感到非常沮丧,因此我创建了一个小型GAE应用程序,可以在30秒内删除相当多的数据。

How to install etc: https://github.com/xamde/xydra

如何安装等:https://github.com/xamde/xydra

#27

For java

DatastoreService db = DatastoreServiceFactory.getDatastoreService();
List<Key> keys = new ArrayList<Key>();
for(Entity e : db.prepare(new Query().setKeysOnly()).asIterable())
    keys.add(e.getKey());
db.delete(keys);

Works well in Development Server

在Development Server中运行良好