为什么这个简单的回归（keras）ANN失败如此糟糕？

I am trying to do a non-linear regression on a very simple data. When running the following code i got really bad results. Almost every time the result is a simple linear regression. When i check the weights of my model most (if not all) neurons are 'dead'. They all have negative weights with negative biases making the ReLu function to return 0 for all inputs (since all inputs are in the range [0,1]).

我试图对非常简单的数据进行非线性回归。运行以下代码时,我得到了非常糟糕的结果。几乎每次结果都是简单的线性回归。当我检查我的模型的权重时,大多数(如果不是全部)神经元都“死”。它们都具有负偏差和负偏差,使得ReLu函数对所有输入都返回0(因为所有输入都在[0,1]范围内)。

As far as i can tell this is a problem with the optimizer. I also tried using a very low and a very high learning rate, no luck. The optimizer seems to be getting stuck in a 'very' sub optimal local minima.

据我所知,这是优化器的问题。我也试过使用非常低和非常高的学习率,没有运气。优化器似乎陷入了“非常”次优的局部最小值。

I also tried to set the initial weights to be all positive [0,0.1], the optimizer 'cheats' its way into a linear regression by setting all biases roughly at the same value.

我还尝试将初始权重设置为全正[0,0.1],优化器通过将所有偏差设置为大致相同的值来欺骗其进入线性回归。

Any can help me? what i am doing wrong? Is this really the best a state of the art ANN can achieve on a simple regression problem?

有人可以帮帮我吗?我做错了什么?这真的是ANN在简单的回归问题上可以达到的最佳状态吗?

library(keras)
fun <- function(x) 0.2+0.4*x^2+0.3*x*sin(15*x)+0.05*cos(50*x).

x_test <- seq(0,1,0.01)
y_test <- fun(x_test)
plot(x_test, y_test, type = 'l')

x_train <- runif(50)
y_train <- fun(x_train)
points(x_train, y_train)

model <- keras_model_sequential() %>%
  layer_dense(10, 'relu', input_shape = 1) %>%
  layer_dense(1)

model %>% compile(
  optimizer = 'sgd',
  loss = "mse"
)

history <- model %>% 
  fit(x = x_train, y = y_train,
      epochs = 100,
      batch_size = 10,
      validation_data = list(x_test, y_test)
  )

y_pred <- model %>% predict(x_test)

plot(x_test, y_test, type = 'l')
points(x_train, y_train)
lines(x_test, y_pred, col = 'red')

predicted outputs versus actual ones.

预测产出与实际产出。

1 个解决方案

#1

Change sigmoid with relu activation and fix your ) type error in the end of sgd.

使用relu激活更改sigmoid并在sgd结尾处修复您的类型错误。

EDIT

Also add a second dense layer and train for much more epochs, like this:

还要添加第二个密集层并训练更多的纪元,如下所示:

model <- keras_model_sequential() %>%
  layer_dense(10, 'relu', input_shape = 1) %>%
  layer_dense(10, 'relu') %>%
  layer_dense(1)

model %>% compile(
  optimizer = 'sgd',
loss = "mse"
)

history <- model %>% 
  fit(x = x_train, y = y_train,
      epochs = 2000,
      batch_size = 10,
      validation_data = list(x_test, y_test)
  )

#1