隐藏spark属性，使其不显示在spark web UI中，而无需实现安全过滤器

The application web UI at http://:4040 lists Spark properties in the “Environment” tab. All values explicitly specified through spark-defaults.conf, SparkConf, or the command line will appear. However, for security reasons, I do not want my Cassandra password to display in the web UI. Is there some sort of switch to ensure that certain spark properties are not displayed??

位于http：//：4040的应用程序Web UI在“环境”选项卡中列出了Spark属性。将显示通过spark-defaults.conf，SparkConf或命令行显式指定的所有值。但是，出于安全原因，我不希望我的Cassandra密码显示在Web UI中。是否有某种开关来确保不显示某些火花属性？

Please note, I see some solutions that suggest implementing a security filter and using spark.ui.filters setting to refer to the class. I am hoping to avoid this complexity.

请注意，我看到一些解决方案建议实现安全过滤器并使用spark.ui.filters设置来引用该类。我希望避免这种复杂性。

1 个解决方案

#1

I think there is no common solution how to hide your custom property from spark WebUI for previous releases.

我认为没有通用的解决方案如何从以前版本的spark WebUI隐藏自定义属性。

I assume you are using spark 2.0 or below (i have not seen feature described below in 2.0) because 2.0.1 supports passwords preprocessing to "*****".

我假设您正在使用spark 2.0或更低版本（我没有看到2.0中描述的功能），因为2.0.1支持将密码预处理为“*****”。

Check issue SPARK-16796 Visible passwords on Spark environment page

检查问题SPARK-16796 Spark环境页面上的可见密码

When we take a look into apache spark source code and do some investigation we can see some processing how to "hide" property in spark web ui.

当我们看一下apache spark源代码并做一些调查时，我们可以看到一些处理如何在spark web ui中“隐藏”属性。

SparkUI by default the Environment page is attached within initialization attachTab(new EnvironmentTab(this)) [line 71]

SparkUI默认情况下，环境页面附加在初始化attachTab（new EnvironmentTab（this））中[第71行]

EnvironmentPage renders properties to EnvironmentPage as tab in web gui as:

EnvironmentPage将属性作为web gui中的选项卡呈现给EnvironmentPage：

def render(request: HttpServletRequest): Seq[Node] = {
    val runtimeInformationTable = UIUtils.listingTable(
      propertyHeader, jvmRow, listener.jvmInformation, fixedWidth = true)
    val sparkPropertiesTable = UIUtils.listingTable(
      propertyHeader, propertyRow, listener.sparkProperties.map(removePass), fixedWidth = true)
    val systemPropertiesTable = UIUtils.listingTable(
      propertyHeader, propertyRow, listener.systemProperties, fixedWidth = true)
    val classpathEntriesTable = UIUtils.listingTable(
      classPathHeaders, classPathRow, listener.classpathEntries, fixedWidth = true)
    val content =
      <span>
        <h4>Runtime Information</h4> {runtimeInformationTable}
        <h4>Spark Properties</h4> {sparkPropertiesTable}
        <h4>System Properties</h4> {systemPropertiesTable}
        <h4>Classpath Entries</h4> {classpathEntriesTable}
      </span>

    UIUtils.headerSparkPage("Environment", content, parent)
  }

all properties are rendered without some kind of hiding preprocessing except sparkProperties - with functionality provided in removePass.

除了sparkProperties之外，所有属性都在没有某种隐藏预处理的情况下呈现 - 在removePass中提供了功能。

private def removePass(kv: (String, String)): (String, String) = {
    if (kv._1.toLowerCase.contains("password")) (kv._1, "******") else kv
}

as we can see every key that contains "password" (BTW: in the master branch they also filtering keys with keyword "secret" check if u are interested in)

因为我们可以看到包含“密码”的每个密钥（顺便说一句：在主分支中，他们还使用关键字“秘密”过滤密钥，检查您是否感兴趣）

I cannot tested now but u can try to update spark. so eg. SparkSubmitArguments.scala in mergeDefaultSparkProperties() will consider spark.cassandra.auth.password as spark and populate as sparkProperties (with removePass preprocessing).

我现在无法测试，但你可以尝试更新火花。所以，例如。 mergeDefaultSparkProperties（）中的SparkSubmitArguments.scala将spark.cassandra.auth.password视为spark并填充为sparkProperties（使用removePass预处理）。

And at the end of the day in EnvironmentTab in web gui this property should be visible as ****.

在web gui的EnvironmentTab中，这个属性应该在****中显示。

#1