使用app.yaml在GAE中安全地存储环境变量

I need to store API keys and other sensitive information in app.yaml as environment variables for deployment on GAE. The issue with this is that if I push app.yaml to GitHub, this information becomes public (not good). I don't want to store the info in a datastore as it does not suit the project. Rather, I'd like to swap out the values from a file that is listed in .gitignore on each deployment of the app.

我需要在app.yaml中存储API密钥和其他敏感信息作为环境变量，以便在GAE上进行部署。这个问题是，如果我将app.yaml推送到GitHub，这个信息就会公开（不好）。我不想将信息存储在数据存储区中，因为它不适合项目。相反，我想在应用程序的每个部署中交换.gitignore中列出的文件中的值。

Here is my app.yaml file:

这是我的app.yaml文件：

application: myapp
version: 3 
runtime: python27
api_version: 1
threadsafe: true

libraries:
- name: webapp2
  version: latest
- name: jinja2
  version: latest

handlers:
- url: /static
  static_dir: static

- url: /.*
  script: main.application  
  login: required
  secure: always
# auth_fail_action: unauthorized

env_variables:
  CLIENT_ID: ${CLIENT_ID}
  CLIENT_SECRET: ${CLIENT_SECRET}
  ORG: ${ORG}
  ACCESS_TOKEN: ${ACCESS_TOKEN}
  SESSION_SECRET: ${SESSION_SECRET}

Any ideas?

有任何想法吗？

7 个解决方案

#1

Best way to do it, is store the keys in a client_secrets.json file, and exclude that from being uploaded to git by listing it in your .gitignore file. If you have different keys for different environments, you can use app_identity api to determine what the app id is, and load appropriately.

最好的方法是将密钥存储在client_secrets.json文件中，并通过将其列在.gitignore文件中将其排除在上传到git之外。如果您有不同环境的不同密钥，则可以使用app_identity api确定应用程序ID是什么，并正确加载。

There is a fairly comprehensive example here -> https://developers.google.com/api-client-library/python/guide/aaa_client_secrets.

这里有一个相当全面的例子 - > https://developers.google.com/api-client-library/python/guide/aaa_client_secrets。

Here's some example code:

这是一些示例代码：

# declare your app ids as globals ...
APPID_LIVE = 'awesomeapp'
APPID_DEV = 'awesomeapp-dev'
APPID_PILOT = 'awesomeapp-pilot'

# create a dictionary mapping the app_ids to the filepaths ...
client_secrets_map = {APPID_LIVE:'client_secrets_live.json',
                      APPID_DEV:'client_secrets_dev.json',
                      APPID_PILOT:'client_secrets_pilot.json'}

# get the filename based on the current app_id ...
client_secrets_filename = client_secrets_map.get(
    app_identity.get_application_id(),
    APPID_DEV # fall back to dev
    )

# use the filename to construct the flow ...
flow = flow_from_clientsecrets(filename=client_secrets_filename,
                               scope=scope,
                               redirect_uri=redirect_uri)

# or, you could load up the json file manually if you need more control ...
f = open(client_secrets_filename, 'r')
client_secrets = json.loads(f.read())
f.close()

#2

My approach is to store client secrets only within the App Engine app itself. The client secrets are neither in source control nor on any local computers. This has the benefit that any App Engine collaborator can deploy code changes without having to worry about the client secrets.

我的方法是仅在App Engine应用程序本身中存储客户端机密。客户端机密既不在源代码控制中，也不在任何本地计算机上。这样做的好处是，任何App Engine协作者都可以部署代码更改，而无需担心客户端机密。

I store client secrets directly in Datastore and use Memcache for improved latency accessing the secrets. The Datastore entities only need to be created once and will persist across future deploys. of course the App Engine console can be used to update these entities at any time.

我将客户端机密直接存储在数据存储区中，并使用Memcache来改进访问机密的延迟。数据存储区实体只需创建一次，并将在未来的部署中保留。当然，App Engine控制台可以随时用于更新这些实体。

There are two options to perform the one-time entity creation:

执行一次性实体创建有两个选项：

Use the App Engine Remote API interactive shell to create the entities.
使用App Engine Remote API交互式shell创建实体。
Create an Admin only handler that will initialize the entities with dummy values. Manually invoke this admin handler, then use the App Engine console to update the entities with the production client secrets.
创建一个Admin only处理程序，用于初始化具有虚拟值的实体。手动调用此管理处理程序，然后使用App Engine控制台更新具有生产客户机密钥的实体。

#3

If it's sensitive data, you should not store it in source code as it will be checked into source control. The wrong people (inside or outside your organization) may find it there. Also, your development environment probably uses different config values from your production environment. If these values are stored in code, you will have to run different code in development and production, which is messy and bad practice.

如果它是敏感数据，则不应将其存储在源代码中，因为它将被检入源代码管理中。错误的人（组织内部或外部）可能会在那里找到它。此外，您的开发环境可能使用生产环境中的不同配置值。如果这些值存储在代码中，您将不得不在开发和生产中运行不同的代码，这是一个混乱和不好的做法。

In my projects, I put config data in the datastore using this class:

在我的项目中，我使用此类将配置数据放在数据存储区中：

from google.appengine.ext import ndb

class Settings(ndb.Model):
  name = ndb.StringProperty()
  value = ndb.StringProperty()

  @staticmethod
  def get(name):
    NOT_SET_VALUE = "NOT SET"
    retval = Settings.query(Settings.name == name).get()
    if not retval:
      retval = Settings()
      retval.name = name
      retval.value = NOT_SET_VALUE
      retval.put()
    if retval.value == NOT_SET_VALUE:
      raise Exception(('Setting %s not found in the database. A placeholder ' +
        'record has been created. Go to the Developers Console for your app ' +
        'in App Engine, look up the Settings record with name=%s and enter ' +
        'its value in that record\'s value field.') % (name, name))
    return retval.value

Your application would do this to get a value:

您的应用程序将执行此操作以获取值：

API_KEY = Settings.get('API_KEY')

If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.

如果数据存储区中存在该键的值，您将获得该值。如果没有，将创建占位符记录并抛出异常。该异常将提醒您转到Developers Console并更新占位符记录。

I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!

我发现这需要猜测设置配置值。如果您不确定要设置的配置值，只需运行代码即可告诉您！

The code above uses the ndb library which uses memcache and the datastore under the hood, so it's fast.

上面的代码使用了ndb库，它使用了内存中的memcache和数据存储区，所以速度很快。

Update:

更新：

jelder asked for how to find the Datastore values in the App Engine console and set them. Here is how:

jelder询问如何在App Engine控制台中找到数据存储区值并进行设置。方法如下：

Go to https://console.cloud.google.com/datastore/

转到https://console.cloud.google.com/datastore/
Select your project at the top of the page if it's not already selected.

如果尚未选择项目，请在页面顶部选择项目。
In the Kind dropdown box, select Settings.

在种类下拉框中，选择设置。
If you ran the code above, your keys will show up. They will all have the value NOT SET. Click each one and set its value.

如果您运行上面的代码，您的密钥将显示出来。它们都具有NOT SET值。单击每个并设置其值。

Hope this helps!

希望这可以帮助！

#4

You can use the -E command line option of appcfg.py to setup the environment variables when you deploy your app to GAE (appcfg.py update)

将应用程序部署到GAE时，可以使用appcfg.py的-E命令行选项设置环境变量（appcfg.py update）

$ appcfg.py
...
-E NAME:VALUE, --env_variable=NAME:VALUE
                    Set an environment variable, potentially overriding an
                    env_variable value from app.yaml file (flag may be
                    repeated to set multiple variables).
...

#5

It sounds like you can do a few approaches. We have a similar issue and do the following (adapted to your use-case):

听起来你可以采取一些方法。我们有类似的问题，并执行以下操作（适应您的用例）：

Create a file that stores any dynamic app.yaml values and place it on a secure server in your build environment. If you are really paranoid, you can asymmetrically encrypt the values. You can even keep this in a private repo if you need version control/dynamic pulling, or just use a shells script to copy it/pull it from the appropriate place.
创建一个存储任何动态app.yaml值的文件，并将其放在构建环境中的安全服务器上。如果你真的是偏执狂，你可以非对称地加密这些值。如果您需要版本控制/动态拉动，或者只是使用shell脚本将其复制/从适当的位置拉出来，您甚至可以将其保存在私人仓库中。
Pull from git during the deployment script
在部署脚本期间从git中提取
After the git pull, modify the app.yaml by reading and writing it in pure python using a yaml library
在git pull之后，通过使用yaml库在纯python中读取和编写来修改app.yaml

The easiest way to do this is to use a continuous integration server such as Hudson, Bamboo, or Jenkins. Simply add some plug-in, script step, or workflow that does all the above items I mentioned. You can pass in environment variables that are configured in Bamboo itself for example.

最简单的方法是使用连续集成服务器，如Hudson，Bamboo或Jenkins。只需添加一些插件，脚本步骤或工作流程，完成我提到的所有上述项目。例如，您可以传入在Bamboo中配置的环境变量。

In summary, just push in the values during your build process in an environment you only have access to. If you aren't already automating your builds, you should be.

总之，只需在构建过程中在您只能访问的环境中推送值。如果您尚未自动化构建，那么您应该是。

Another option option is what you said, put it in the database. If your reason for not doing that is that things are too slow, simply push the values into memcache as a 2nd layer cache, and pin the values to the instances as a first-layer cache. If the values can change and you need to update the instances without rebooting them, just keep a hash you can check to know when they change or trigger it somehow when something you do changes the values. That should be it.

另一个选项是你说的，把它放在数据库中。如果您不这样做的原因是事情太慢，只需将值作为第二层缓存推送到memcache中，并将值作为第一层缓存固定到实例。如果值可以更改并且您需要更新实例而不重新启动它们，只需保留一个哈希值，您可以检查它们何时更改或以某种方式触发它时更改值。那应该是它。

#6

Just wanted to note how I solved this problem in javascript/nodejs. For local development I used the 'dotenv' npm package which loads environment variables from a .env file into process.env. When I started using GAE I learned that environment variables need to be set in a 'app.yaml' file. Well, I didn't want to use 'dotenv' for local development and 'app.yaml' for GAE (and duplicate my environment variables between the two files), so I wrote a little script that loads app.yaml environment variables into process.env, for local development. Hope this helps someone:

只想注意我是如何在javascript / nodejs中解决这个问题的。对于本地开发，我使用了'dotenv'npm包，它将.env文件中的环境变量加载到process.env中。当我开始使用GAE时，我了解到环境变量需要在'app.yaml'文件中设置。好吧，我不想使用'dotenv'进行本地开发，'app.yaml'用于GAE（并在两个文件之间复制我的环境变量），所以我写了一个小脚本，将app.yaml环境变量加载到进程中.env，用于本地开发。希望这有助于某人：

yaml_env.js:

yaml_env.js：

(function () {
    const yaml = require('js-yaml');
    const fs = require('fs');
    const isObject = require('lodash.isobject')

    var doc = yaml.safeLoad(
        fs.readFileSync('app.yaml', 'utf8'), 
        { json: true }
    );

    // The .env file will take precedence over the settings the app.yaml file
    // which allows me to override stuff in app.yaml (the database connection string (DATABASE_URL), for example)
    // This is optional of course. If you don't use dotenv then remove this line:
    require('dotenv/config');

    if(isObject(doc) && isObject(doc.env_variables)) {
        Object.keys(doc.env_variables).forEach(function (key) {
            // Dont set environment with the yaml file value if it's already set
            process.env[key] = process.env[key] || doc.env_variables[key]
        })
    }
})()

Now include this file as early as possible in your code, and you're done:

现在尽早在您的代码中包含此文件，您就完成了：

require('../yaml_env')

#7

Extending Martin's answer

扩展马丁的答案

from google.appengine.ext import ndb

class Settings(ndb.Model):
    """
    Get sensitive data setting from DataStore.

    key:String -> value:String
    key:String -> Exception

    Thanks to: Martin Omander @ *
    https://*.com/a/35261091/1463812
    """
    name = ndb.StringProperty()
    value = ndb.StringProperty()

    @staticmethod
    def get(name):
        retval = Settings.query(Settings.name == name).get()
        if not retval:
            raise Exception(('Setting %s not found in the database. A placeholder ' +
                             'record has been created. Go to the Developers Console for your app ' +
                             'in App Engine, look up the Settings record with name=%s and enter ' +
                             'its value in that record\'s value field.') % (name, name))
        return retval.value

    @staticmethod
    def set(name, value):
        exists = Settings.query(Settings.name == name).get()
        if not exists:
            s = Settings(name=name, value=value)
            s.put()
        else:
            exists.value = value
            exists.put()

        return True

#1