吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

时间:2022-12-14 15:20:17

 

主要内容:

一.初始化参数

二.前向传播

三.计算代价函数

四.反向传播

五.更新参数(梯度下降)

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

 

一.初始化参数

1.由于将参数W初始化为0会导致神经网络失效,因而需要对W进行随机初始化。一般的做法是用np.random.np()生成一个高斯分布的数,然后再乘上一个很小的数比如0.01,以限制它的范围。所以可知W的初始值是一个很小的数(绝对值小),那为什么不能取绝对值较大的数呢?根据sigmoid或者tanh函数的图像可知,当z = wx + b 很大或很小时,曲线的斜率很小,这就会导致梯度下降的速度非常慢,不利于快速达到局部最优值。而当w绝对值很大时,z绝对值也会很大,所以w的绝对值值应该尽量小,以保证在初期有较快的梯度下降速度。

2.由于参数b的初始化对神经网络没有较大的影响,因此可以直接设置为0。

3.对于神经网络而言,由于每一层的结点数都可能不一样,这样也就导致了每一层结点上的参数W的形状不一样,这里也是非容易出错,因而需要理每一层网络的参数的形状。假设输入层X有12288个特征,有209个样本,那么这个深度神经网络的参数具体如下(n[l]表示第l层的结点数):

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

4.代码如下:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程吴恩达深度学习笔记(二)—— 深层神经网络的训练过程
#layer_dims为每一层的结点数,包括输入层
def initialize_parameters_deep(layer_dims):
    """
    Arguments:
    layer_dims -- python array (list) containing the dimensions of each layer in our network
    
    Returns:
    parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])
                    bl -- bias vector of shape (layer_dims[l], 1)
    """
    
    np.random.seed(3)
    parameters = {}         #用字典存储参数W和b
    L = len(layer_dims) - 1           # 神经网络的层数,不考虑输入层

    #初始化第1到第L层结点的参数,其中第0层为输入层X
    for l in range(1, L+1):
        ### START CODE HERE ### (≈ 2 lines of code)
        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l-1]) * 0.01
        parameters['b' + str(l)] = np.zeros((layer_dims[l],1))
        ### END CODE HERE ###
        
        assert(parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l-1]))
        assert(parameters['b' + str(l)].shape == (layer_dims[l], 1))

        
    return parameters
View Code

 

 

二.前向传播

1.前向传播相对简单,对于每一层可分为两个步骤

1)首先计算出线性值:Z[l] = W[l]A[l-1] + b[l]

2)然后将Z[l]带入到激活函数中,作为这一层的输出:A[l] = g(Z[l])

2.在执行这两个步骤时,需要将Z[l]、A[l-1]、W[l]、b[l]存起来,以待反向传播之用。实际上b[l]不是必须的,只不过b[l]可以用来检测db[l]的形状是否正确。

3.对于第1~L-1层,一般使用relu()函数作为激活函数,而对于第L层,一般使用sigmoid()函数,因而sigmoid()函数的输出值是[0,1],可以作为概率。

1)线性计算:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程吴恩达深度学习笔记(二)—— 深层神经网络的训练过程
# GRADED FUNCTION: linear_forward
def linear_forward(A, W, b):
    """
    Implement the linear part of a layer's forward propagation.

    Arguments:
    A -- activations from previous layer (or input data): (size of previous layer, number of examples)
    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)
    b -- bias vector, numpy array of shape (size of the current layer, 1)

    Returns:
    Z -- the input of the activation function, also called pre-activation parameter 
    cache -- a python dictionary containing "A", "W" and "b" ; stored for computing the backward pass efficiently
    """
    
    ### START CODE HERE ### (≈ 1 line of code)
    Z = np.dot(W,A) + b
    ### END CODE HERE ###
    
    assert(Z.shape == (W.shape[0], A.shape[1]))
    cache = (A, W, b)
    
    return Z, cache
View Code

2)单层计算:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程吴恩达深度学习笔记(二)—— 深层神经网络的训练过程
# GRADED FUNCTION: linear_activation_forward
def linear_activation_forward(A_prev, W, b, activation):
    """
    Implement the forward propagation for the LINEAR->ACTIVATION layer

    Arguments:
    A_prev -- activations from previous layer (or input data): (size of previous layer, number of examples)
    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)
    b -- bias vector, numpy array of shape (size of the current layer, 1)
    activation -- the activation to be used in this layer, stored as a text string: "sigmoid" or "relu"

    Returns:
    A -- the output of the activation function, also called the post-activation value 
    cache -- a python dictionary containing "linear_cache" and "activation_cache";
             stored for computing the backward pass efficiently
    """
    
    if activation == "sigmoid":
        # Inputs: "A_prev, W, b". Outputs: "A, activation_cache".
        ### START CODE HERE ### (≈ 2 lines of code)
        Z, linear_cache = linear_forward(A_prev,W,b)
        A, activation_cache = sigmoid(Z)
        ### END CODE HERE ###
    
    elif activation == "relu":
        # Inputs: "A_prev, W, b". Outputs: "A, activation_cache".
        ### START CODE HERE ### (≈ 2 lines of code)
        Z, linear_cache = linear_forward(A_prev,W,b)
        A, activation_cache = relu(Z)
        ### END CODE HERE ###
    
    assert (A.shape == (W.shape[0], A_prev.shape[1]))
    cache = (linear_cache, activation_cache)

    return A, cache
View Code

3)for循环执行前向传播:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程吴恩达深度学习笔记(二)—— 深层神经网络的训练过程
# GRADED FUNCTION: L_model_forward
def L_model_forward(X, parameters):
    """
    Implement forward propagation for the [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID computation
    
    Arguments:
    X -- data, numpy array of shape (input size, number of examples)
    parameters -- output of initialize_parameters_deep()
    
    Returns:
    AL -- last post-activation value
    caches -- list of caches containing:
                every cache of linear_relu_forward() (there are L-1 of them, indexed from 0 to L-2)
                the cache of linear_sigmoid_forward() (there is one, indexed L-1)
    """

    caches = []
    A = X
    L = len(parameters) // 2                  # number of layers in the neural network,W和b是成对出现的,所以层数是参数的个数/2
    
    # Implement [LINEAR -> RELU]*(L-1). Add "cache" to the "caches" list.
    for l in range(1, L):   
        ### START CODE HERE ### (≈ 2 lines of code)
        A, cache = linear_activation_forward(A_prev=A,W=parameters["W"+str(l)],b=parameters["b"+str(l)],activation="relu")
        caches.append(cache)
        ### END CODE HERE ###
    
    # Implement LINEAR -> SIGMOID. Add "cache" to the "caches" list.
    ### START CODE HERE ### (≈ 2 lines of code)
    AL, cache = linear_activation_forward(A_prev=A,W=parameters["W"+str(L)],b=parameters["b"+str(L)],activation="sigmoid")
    caches.append(cache)
    ### END CODE HERE ###
    
    assert(AL.shape == (1,X.shape[1]))  
    return AL, caches
View Code

 

 

三.计算代价函数

 由于第L层使用的是sigmoid()函数,所以最终的代价函数为:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

代码如下:

# GRADED FUNCTION: compute_cost

def compute_cost(AL, Y):
    """
    Implement the cost function defined by equation (7).

    Arguments:
    AL -- probability vector corresponding to your label predictions, shape (1, number of examples)
    Y -- true "label" vector (for example: containing 0 if non-cat, 1 if cat), shape (1, number of examples)

    Returns:
    cost -- cross-entropy cost
    """
    
    m = Y.shape[1]

    # Compute loss from aL and y.
    ### START CODE HERE ### (≈ 1 lines of code)
    cost = -1/m*np.sum(Y*np.log(AL)+(1-Y)*np.log(1-AL))
    ### END CODE HERE ###
    
    cost = np.squeeze(cost)      # To make sure your cost's shape is what we expect (e.g. this turns [[17]] into 17).
    assert(cost.shape == ())
    
    return cost

 

 

四.反向传播

1.反向传播最主要的作用就是计算出dW、db这两类倒数,用于梯度下降。

2.反向传播的起始是计算出dAL,根据代价函数,对AL进行求导,可得:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

3.之后的过程就是:对于第l层,输入dA[l] 和 cache(存储A[l-1]、Z[l]、W[l]、b[l],其中b[l]非必须),输出dA[l-1]、dW[l]、db[l]

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

代码如下:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程吴恩达深度学习笔记(二)—— 深层神经网络的训练过程
# GRADED FUNCTION: linear_activation_backward

def linear_activation_backward(dA, cache, activation):
    """
    Implement the backward propagation for the LINEAR->ACTIVATION layer.
    
    Arguments:
    dA -- post-activation gradient for current layer l 
    cache -- tuple of values (linear_cache, activation_cache) we store for computing backward propagation efficiently
    activation -- the activation to be used in this layer, stored as a text string: "sigmoid" or "relu"
    
    Returns:
    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev
    dW -- Gradient of the cost with respect to W (current layer l), same shape as W
    db -- Gradient of the cost with respect to b (current layer l), same shape as b
    """
    linear_cache, activation_cache = cache
    
    if activation == "relu":
        ### START CODE HERE ### (≈ 2 lines of code)
        dZ =  relu_backward(dA, activation_cache)  #dA乘上dA对Z的导数,得到dZ。需要根据不同的激活函数而定,因而在这里封装起来
        dA_prev, dW, db = linear_backward(dZ=dZ,cache=linear_cache)
        ### END CODE HERE ###
        
    elif activation == "sigmoid":
        ### START CODE HERE ### (≈ 2 lines of code)
        dZ = sigmoid_backward(dA, activation_cache)
        dA_prev, dW, db = linear_backward(dZ=dZ,cache=linear_cache)
        ### END CODE HERE ###
    
    return dA_prev, dW, db
View Code

4.整个过程如下:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

代码如下:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程吴恩达深度学习笔记(二)—— 深层神经网络的训练过程
# GRADED FUNCTION: L_model_backward

def L_model_backward(AL, Y, caches):
    """
    Implement the backward propagation for the [LINEAR->RELU] * (L-1) -> LINEAR -> SIGMOID group
    
    Arguments:
    AL -- probability vector, output of the forward propagation (L_model_forward())
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat)
    caches -- list of caches containing:
                every cache of linear_activation_forward() with "relu" (it's caches[l], for l in range(L-1) i.e l = 0...L-2)
                the cache of linear_activation_forward() with "sigmoid" (it's caches[L-1])
    
    Returns:
    grads -- A dictionary with the gradients
             grads["dA" + str(l)] = ...
             grads["dW" + str(l)] = ...
             grads["db" + str(l)] = ...
    """
    grads = {}
    L = len(caches) # the number of layers
    m = AL.shape[1]
    Y = Y.reshape(AL.shape) # after this line, Y is the same shape as AL

    # Initializing the backpropagation
    ### START CODE HERE ### (1 line of code)
    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL))
    ### END CODE HERE ###
    
    # Lth layer (SIGMOID -> LINEAR) gradients. Inputs: "DAL, Y, caches". Outputs: "grads["dAL-1"], grads["dWL"], grads["dbL"]
    ### START CODE HERE ### (approx. 2 lines)
    #计算第L层,分开计算是因为激活函数不同
    current_cache = caches[L-1]   #注意caches的下标从0开始,而网络层从1开始,所以第L层的缓存对应caches[L-1]
    grads["dA" + str(L-1)], grads["dW" + str(L)], grads["db" + str(L)] = linear_activation_backward(dAL, current_cache, activation = 'sigmoid')
    ### END CODE HERE ###

    #从第L-1层计算到第一层:
    for l in reversed(range(1,L)):
        # Inputs: "grads["dA" + str(l)], caches". Outputs: "grads["dA" + str(l - 1)] , grads["dW" + str(l)] , grads["db" + str(l)] 
        ### START CODE HERE ### (approx. 5 lines)
        current_cache = caches[l-1]   #注意caches的下标从0开始,而网络层从1开始,所以第L层的缓存对应caches[l-1]
        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads["dA"+str(l)],current_cache,"relu")
        grads["dA" + str(l-1)] = dA_prev_temp
        grads["dW" + str(l)] = dW_temp
        grads["db" + str(l)] = db_temp
        ### END CODE HERE ###

    return grads
View Code

 

 

五.更新参数(梯度下降)

通过反向传播求出dW、db之后,剩下的工作就相对轻松了,直接进行梯度下降,更新参数:

吴恩达深度学习笔记(二)—— 深层神经网络的训练过程

代码如下(没有进行正则化):

# GRADED FUNCTION: update_parameters

def update_parameters(parameters, grads, learning_rate):
    """
    Update parameters using gradient descent
    
    Arguments:
    parameters -- python dictionary containing your parameters 
    grads -- python dictionary containing your gradients, output of L_model_backward
    
    Returns:
    parameters -- python dictionary containing your updated parameters 
                  parameters["W" + str(l)] = ... 
                  parameters["b" + str(l)] = ...
    """
    
    L = len(parameters) // 2 # number of layers in the neural network

    # Update rule for each parameter. Use a for loop.
    ### START CODE HERE ### (≈ 3 lines of code)
    for l in range(L):
        parameters["W" + str(l+1)] = parameters["W" + str(l+1)] - learning_rate*grads["dW" + str(l+1)]
        parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - learning_rate*grads["db" + str(l+1)]
    ### END CODE HERE ###
        
    return parameters