使用apache beam sdk时无法连接到Redis服务器

So I have a dataflow job doing

所以我有一个数据流工作

p.apply(RedisIO.read()
    .withEndpoint(<public endpoint>, 6379)
    .withAuth(<password>)
    .withTimeout(60000)
    .withKeyPattern("UID*"))
 .apply(ParDo.of(new Format()))
 .apply(TextIO.write().to(options.getOutput()));

The redis endpoint is public authenticated with a password with no firewall settings. When I run the above, I get the following error.

redis端点使用没有防火墙设置的密码进行公共身份验证。当我运行上面的操作时，我收到以下错误。

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven- 
plugin:1.6.0:java (default-cli) on project word-count-beam: An 
exception occured while executing the Java class. 
org.apache.beam.sdk.util.UserCodeException: 
redis.clients.jedis.exceptions.JedisConnectionException: 
java.net.ConnectException: Connection refused -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to 
execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default- 
cli) on project word-count-beam: An exception occured while executing 
the Java class. org.apache.beam.sdk.util.UserCodeException: 
redis.clients.jedis.exceptions.JedisConnectionException: 
java.net.ConnectException: Connection refused
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception 
occured while executing the Java class. 
org.apache.beam.sdk.util.UserCodeException: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused
at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:339)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent 
.UncheckedExecutionException: 
org.apache.beam.sdk.util.UserCodeException: 
redis.clients.jedis.exceptions.JedisConnectionException: 
java.net.ConnectException: Connection refused

So connection is not getting established with public redis endpoint. I am getting the same error when I am running through DirectRunner. Am i missing something here?

因此，使用公共redis端点无法建立连接。我在运行DirectRunner时收到同样的错误。我错过了什么吗？

1 个解决方案

#1

There is a known error in the Apache Beam source code for RedisIO where withEndpoint ignores the input host and will attempt to use localhost instead. Attempting to connect to a redis server on localhost when there is none will give the error you are seeing.

RedisIO的Apache Beam源代码中存在一个已知错误，其中withEndpoint忽略输入主机并将尝试使用localhost。当没有redis服务器时，尝试连接到localhost上的redis服务器会出现您看到的错误。

You can read more about the issue here, and see a pull request with a fix here.

您可以在此处阅读有关此问题的更多信息，并在此处查看带有修复的拉取请求。

Until that pull request gets merged you should be able to resolve the problem by implementing the change yourself by copying RedisIO.java into your project and changing

在拉取请求合并之前，您应该能够通过将RedisIO.java复制到项目中并自行更改来自行实现更改来解决问题

.setConnectionConfiguration(connectionConfiguration().withHost(host))
.setConnectionConfiguration(connectionConfiguration().withPort(port))

至

.setConnectionConfiguration(connectionConfiguration().withHost(host).withPort(port))

Note this same error occurs 3 times in RedisIO, once each for Read (line 168), ReadAll (line 233), and Write (line 365).

请注意，同样的错误在RedisIO中发生3次，每次发生一次读取（第168行），ReadAll（第233行）和写入（第365行）。

#1