I am trying to do a non-linear regression on a very simple data. When running the following code i got really bad results. Almost every time the result is a simple linear regression. When i check the weights of my model most (if not all) neurons are 'dead'. They all have negative weights with negative biases making the ReLu function to return 0 for all inputs (since all inputs are in the range [0,1]).
我试图对非常简单的数据进行非线性回归。运行以下代码时,我得到了非常糟糕的结果。几乎每次结果都是简单的线性回归。当我检查我的模型的权重时,大多数(如果不是全部)神经元都“死”。它们都具有负偏差和负偏差,使得ReLu函数对所有输入都返回0(因为所有输入都在[0,1]范围内)。
As far as i can tell this is a problem with the optimizer. I also tried using a very low and a very high learning rate, no luck. The optimizer seems to be getting stuck in a 'very' sub optimal local minima.
据我所知,这是优化器的问题。我也试过使用非常低和非常高的学习率,没有运气。优化器似乎陷入了“非常”次优的局部最小值。
I also tried to set the initial weights to be all positive [0,0.1], the optimizer 'cheats' its way into a linear regression by setting all biases roughly at the same value.
我还尝试将初始权重设置为全正[0,0.1],优化器通过将所有偏差设置为大致相同的值来欺骗其进入线性回归。
Any can help me? what i am doing wrong? Is this really the best a state of the art ANN can achieve on a simple regression problem?
有人可以帮帮我吗?我做错了什么?这真的是ANN在简单的回归问题上可以达到的最佳状态吗?
library(keras)
fun <- function(x) 0.2+0.4*x^2+0.3*x*sin(15*x)+0.05*cos(50*x).
x_test <- seq(0,1,0.01)
y_test <- fun(x_test)
plot(x_test, y_test, type = 'l')
x_train <- runif(50)
y_train <- fun(x_train)
points(x_train, y_train)
model <- keras_model_sequential() %>%
layer_dense(10, 'relu', input_shape = 1) %>%
layer_dense(1)
model %>% compile(
optimizer = 'sgd',
loss = "mse"
)
history <- model %>%
fit(x = x_train, y = y_train,
epochs = 100,
batch_size = 10,
validation_data = list(x_test, y_test)
)
y_pred <- model %>% predict(x_test)
plot(x_test, y_test, type = 'l')
points(x_train, y_train)
lines(x_test, y_pred, col = 'red')
predicted outputs versus actual ones.
预测产出与实际产出。
1 个解决方案
#1
0
Change sigmoid with relu activation and fix your ) type error in the end of sgd.
使用relu激活更改sigmoid并在sgd结尾处修复您的类型错误。
EDIT
Also add a second dense layer and train for much more epochs, like this:
还要添加第二个密集层并训练更多的纪元,如下所示:
model <- keras_model_sequential() %>%
layer_dense(10, 'relu', input_shape = 1) %>%
layer_dense(10, 'relu') %>%
layer_dense(1)
model %>% compile(
optimizer = 'sgd',
loss = "mse"
)
history <- model %>%
fit(x = x_train, y = y_train,
epochs = 2000,
batch_size = 10,
validation_data = list(x_test, y_test)
)
#1
0
Change sigmoid with relu activation and fix your ) type error in the end of sgd.
使用relu激活更改sigmoid并在sgd结尾处修复您的类型错误。
EDIT
Also add a second dense layer and train for much more epochs, like this:
还要添加第二个密集层并训练更多的纪元,如下所示:
model <- keras_model_sequential() %>%
layer_dense(10, 'relu', input_shape = 1) %>%
layer_dense(10, 'relu') %>%
layer_dense(1)
model %>% compile(
optimizer = 'sgd',
loss = "mse"
)
history <- model %>%
fit(x = x_train, y = y_train,
epochs = 2000,
batch_size = 10,
validation_data = list(x_test, y_test)
)