嵌套类型Array[struct]保存至Hive

时间:2021-04-17 15:41:52

以豆瓣电影:霸王别姬为例

嵌套类型Array[struct]保存至Hive

其主演存在多个,同时每个演员都有自己的id及name属性,而类型仅仅有name属性不包含id属性,
因此主演对应为Array[struct]类型,可表示为

[{"id":1003494,"name":"张国荣"},
{"id":1050265,"name":"张丰毅"},
{"id":1035641,"name":"巩俐"},
{"id":1000905,"name":"葛优"},
{"id":1011479,"name":"英达"}]

类型对应为Array类型,可表示为

["剧情","爱情","同性"]

生成数据文件,其内容为

1291546|霸王别姬|1003494$张国荣^1050265$张丰毅^1035641$巩俐^1000905$葛优^1011479$英达|剧情^爱情^同性

定义Hive表结构为

CREATE  TABLE o_douban(
doubanid bigint,
title string,
starring array<struct<id:bigint,name:string>>,
genre array<string>)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
COLLECTION ITEMS TERMINATED BY '^'
MAP KEYS TERMINATED BY '$'

将数据文件load进hive表

LOAD DATA LOCAL INPATH '/home/admin/test/douban.txt' OVERWRITE INTO TABLE douban 

查询结果为

hive> select doubanid, title, editor,  genre  from o_douban 
> where doubanid = 1291546;
1291546|
霸王别姬|
[{"id":1003494,"name":"张国荣"},
{"id":1050265,"name":"张丰毅"},
{"id":1035641,"name":"巩俐"},
{"id":1000905,"name":"葛优"},
{"id":1011479,"name":"英达"}]|
剧情^爱情^同性