使用talend进行红移的CSV文件。

Windows 8.1 talend version:5.6

Windows 8.1 talend版本:5.6

JOb design:

作业设计:

tFileinputDelimited >> tredshiftoutput

tFileinputDelimited > > tredshiftoutput

I am loading 1 millon data from csv file to redshift. After loading of 5 lakshs data I am getting these errors:

我从csv文件加载了1米的数据到红移。在加载了5个lakshs数据后，我得到了这些错误:

Exception in component tRedshiftOutput_1 org.postgresql.util.PSQLException: ERROR: /rds/bin/padb.1.0.867/data/exec/58/0: failed to map segment from shared object: Cannot allocate memory Detail:

异常在组件tRedshiftOutput_1 org.postgresql.util。PSQLException: ERROR: /rds/bin/padb.1.0.867/data/exec/58/0:未能从共享对象映射片段:无法分配内存细节:

error: /rds/bin/padb.1.0.867/data/exec/58/0: failed to map segment from shared object: Cannot allocate memory code: 1015 context: dlopen(/rds/bin/padb.1.0.867/data/exec/58/0,RTLD_LAZY) query: 4234372 location: exec_plan.cpp:2213 process: padbmaster [pid=15630]

错误:/rds/bin/padb.1.0.867/数据/exec/58/0:未能从共享对象中映射片段:无法分配内存代码:1015 context: dlopen(/rds/bin/padb.1.0.867/data/exec/58/0,RTLD_LAZY)查询:4234372位置:exec_plan。cpp:2213过程:padbmaster(pid = 15630)

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2096)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1829)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332)
at project_1.red_mysqltest_0_1.red_mysqltest.tFileInputDelimited_1Process(red_mysqltest.java:1056)
at project_1.red_mysqltest_0_1.red_mysqltest.runJobInTOS(red_mysqltest.java:1802)
at project_1.red_mysqltest_0_1.red_mysqltest.main(red_mysqltest.java:1646)

[statistics] disconnected

(统计)断开连接

How to resolve these errors??

如何解决这些错误?

1 个解决方案

#1

To use COPY command you need to copy all the csv file in S3.

要使用COPY命令，您需要复制S3中的所有csv文件。

To copy files in S3 you can use tSystem component with below command -

要在S3中复制文件，可以使用下面命令的tSystem组件。

 "aws s3 cp /home/filename.csv  s3://data/"

then you can use tRedshiftRow to use COPY command which copy the data from S3 to table. If you are not using S3 then just direct pass path of the file.

然后，您可以使用tRedshiftRow来使用复制命令，该命令将数据从S3复制到表中。如果您没有使用S3，那么只需直接通过文件路径。

 "COPY tablename
  FROM 's3://location/filename.csv'
  credentials 
          'aws_access_key_id= enter_access_key;
           aws_secret_access_key=enter_aws_secret_access_ket' 

          CSV DELIMITER ';' IGNOREHEADER 1  BLANKSASNULL EMPTYASNULL MAXERROR 10;"

For more detail see this COPY

有关更多细节，请参阅本副本。

#1