文件名称：Training and Analyzing Deep Recurrent Neural Networks
文件大小：333KB
文件格式：PDF
更新时间：2021-08-11 02:58:32
Recurrent Neural Networks
Abstract Time series often have a temporal hierarchy, with information that is spread out over multiple time scales. Common recurrent neural networks, however, do not explicitly accommodate such a hierarchy, and most research on them has been focusing on training algorithms rather than on their basic architecture. In this pa- per we study the effect of a hierarchy of recurrent neural networks on processing time series. Here, each layer is a recurrent network which receives the hidden state of the previous layer as input. This architecture allows us to perform hi- erarchical processing on difficult temporal tasks, and more naturally capture the structure of time series. We show that they reach state-of-the-art performance for recurrent networks in character-level language modeling when trained with sim- ple stochastic gradient descent. We also offer an analysis of the different emergent time scales.