使用jq合并s3清单文件

I have multiple s3 manifest files each corresponding to a date for a given date range. I am looking to merge all of the manifest files to generate a single manifest file, thus allowing me to perform a single Redshift copy.

我有多个s3清单文件，每个文件对应于给定日期范围的日期。我希望合并所有清单文件以生成单个清单文件，从而允许我执行单个Redshift副本。

manifest file 1:

清单文件1：

{
    "entries": [
        {
            "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
        }
    ]
}

manifest file 2:

清单文件2：

{
    "entries": [
        {
            "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
        }
    ]
}

I am looking for an output like:-

我正在寻找一个输出： -

{
    "entries": [
         {
            "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
         },
         {
            "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
         }
    ]
}

I did try

我确实试过了

jq -s '.[]' "manifest_file1.json" "manifest_file2.json"

and other suggestions posted in * but couldn't make it work.

和*中发布的其他建议，但无法使其工作。

2 个解决方案

#1

Or, without resorting to reduce:

或者，不采取减少：

$ jq -n '{entries: [inputs.entries[]]}' manifest_file_{1,2}.json
{
  "entries": [
    {
      "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
    },
    {
      "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
    }
 ]
}

Note that inputs was introduced in jq version 1.5. If your jq does not have inputs, you can use jq -s as follows:

请注意，输入是在jq 1.5版中引入的。如果您的jq没有输入，您可以使用jq -s，如下所示：

$ jq -s '{entries: [.[].entries[]]}' manifest_file_{1,2}.json

#2

So if by "merge" you mean to combine the "entries" arrays into a single array by concatenating them, you could do this:

因此，如果通过“合并”意味着通过连接它们将“条目”数组组合成单个数组，则可以执行以下操作：

$ jq 'reduce inputs as $i (.; .entries += $i.entries)' manifest_file{1,2}.json

Which yields:

产量：

{
  "entries": [
    {
      "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
    },
    {
      "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
    }
  ]
}

#1