吴恩达深度学习作业(一)单层神经网络实现

利用两天时间看完了这两周的课程内容,顺手把作业写完了。本周的作业比较简单,这里使用逻辑回归构建的神经网络实现了图片是否为猫的判断,由于神经网络层数不高(单层隐藏层),出现了过拟合的现象,训练集上的准确率接近高达99%,而在测试集中只有70%。在这里贴上Python代码方便自己复习以及交流,本周课程页面:Neural Networks Basics,如有疑问可以留言,抽空回复。

库的加载

这里加载的几个库都是常用于机器学习模型训练的库之一,把功能解释一下:

  • numpy是基础的Python科学计算库,带有很多经过SIMD优化的矩阵计算方法。
  • h5py是一个常见的与使用H5文件存储的数据交互的库。
  • matplotlib是一个知名的数据可视化的库。
  • PILscipy用来加载我们自己的图片以测试我们的模型。

查看问题集合

在这个数据包里面包含了如下的内容:

  • 带有m_train个样本的数据集,并且利用y=1或者y=0来标记这个样本是否是一只猫。
  • 一组同样带有标记的m_test个样本的测试集。
  • 每组图片采用RGB矩阵来存储,其大小是(num_px, num_px, 3),前两个代表分辨率,最后一个分别是代表R、G、B深度的矩阵。

首先加载数据集:

1
2
# Loading the data (cat/non-cat)
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()

用orig来命名是因为在后面还要对RGB图像矩阵进行线性向量化的操作。
可以用下面的命令来查看数据代表的图片:

1
2
3
4
# Example of a picture
index = 25
plt.imshow(train_set_x_orig[index])
print ("y = " + str(train_set_y[:, index]) + ", it's a '" + classes[np.squeeze(train_set_y[:, index])].decode("utf-8")

输出如下:

对于向量化,一定要了解每个维度代表什么,对于你不清楚数据集具体情况的情况下最好shape属性查看一下各个矩阵的维度:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
### START CODE HERE ### (≈ 3 lines of code)
m_train = train_set_x_orig.shape[0]
m_test = test_set_x_orig.shape[0]
num_px = train_set_x_orig.shape[1]
### END CODE HERE ###
print ("Number of training examples: m_train = " + str(m_train))
print ("Number of testing examples: m_test = " + str(m_test))
print ("Height/Width of each image: num_px = " + str(num_px))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_set_x shape: " + str(train_set_x_orig.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x shape: " + str(test_set_x_orig.shape))
print ("test_set_y shape: " + str(test_set_y.shape))

确定之后我们使用reshape来吧各个数据集线性化,这里利用了一个优雅小技巧省略了乘法的计算,直接把(a,b,c,d)矩阵转换为(bcd,a)的矩阵:

1
2
3
4
5
6
7
8
9
10
11
12
# Reshape the training and test examples
### START CODE HERE ### (≈ 2 lines of code)
train_set_x_flatten = train_set_x_orig.reshape(train_set_x_orig.shape[0],-1).T
test_set_x_flatten = test_set_x_orig.reshape(test_set_x_orig.shape[0],-1).T
### END CODE HERE ###
print ("train_set_x_flatten shape: " + str(train_set_x_flatten.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x_flatten shape: " + str(test_set_x_flatten.shape))
print ("test_set_y shape: " + str(test_set_y.shape))
print ("sanity check after reshaping: " + str(train_set_x_flatten[0:5,0]))

接下来把训练集的数据标准化到0-1这个区间中:

1
2
train_set_x = train_set_x_flatten/255.
test_set_x = test_set_x_flatten/255.

神经网络结构

本次作业要实现的是一个单层(隐藏层)神经网络,相关公式在本文中省略,其基本结构如下:

构造算法部分函数代码

代码均采用向量化实现,保证算法的高效运行。

sigmoid函数

首先定义sigmoid函数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
def sigmoid(z):
"""
Compute the sigmoid of z
Arguments:
z -- A scalar or numpy array of any size.
Return:
s -- sigmoid(z)
"""
### START CODE HERE ### (≈ 1 line of code)
s = 1/(1+np.exp(-z))
### END CODE HERE ###
return s

初始化函数

然后是对参数进行初始化的函数,由于可以b是偏置量,所以处处都是相等的,利用广播特性,将其赋为常数即可:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
def initialize_with_zeros(dim):
"""
This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.
Argument:
dim -- size of the w vector we want (or number of parameters in this case)
Returns:
w -- initialized vector of shape (dim, 1)
b -- initialized scalar (corresponds to the bias)
"""
### START CODE HERE ### (≈ 1 line of code)
w = np.zeros((dim, 1))
b = 0
### END CODE HERE ###
assert(w.shape == (dim, 1))
assert(isinstance(b, float) or isinstance(b, int))
return w, b

向前传播FP算法以及反向传播BP算法

通过先前传播计算代价的值,然后利用反向传播算法计算各个参数的导数值(w和b分离),这里定义的是一次向前推进以及一次向后推进的操作:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
def propagate(w, b, X, Y):
"""
Implement the cost function and its gradient for the propagation explained above
Arguments:
w -- weights, a numpy array of size (num_px * num_px * 3, 1)
b -- bias, a scalar
X -- data of size (num_px * num_px * 3, number of examples)
Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)
Return:
cost -- negative log-likelihood cost for logistic regression
dw -- gradient of the loss with respect to w, thus same shape as w
db -- gradient of the loss with respect to b, thus same shape as b
Tips:
- Write your code step by step for the propagation. np.log(), np.dot()
"""
m = X.shape[1]
# FORWARD PROPAGATION (FROM X TO COST)
### START CODE HERE ### (≈ 2 lines of code)
A = sigmoid(np.dot(w.T,X)+b) # compute activation
cost = (np.dot(-Y , np.log(A.T)) - np.dot((1-Y) , np.log((1-A).T)))/m # compute cost
### END CODE HERE ###
# BACKWARD PROPAGATION (TO FIND GRAD)
### START CODE HERE ### (≈ 2 lines of code)
dw = np.dot(X, (A-Y).T) / m
db = np.sum(A-Y) / m
### END CODE HERE ###
assert(dw.shape == w.shape)
assert(db.dtype == float)
cost = np.squeeze(cost)
assert(cost.shape == ())
grads = {"dw": dw,
"db": db}
return grads, cost

对每次操作进行循环迭代得到最优化算法:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# GRADED FUNCTION: optimize
def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
"""
This function optimizes w and b by running a gradient descent algorithm
Arguments:
w -- weights, a numpy array of size (num_px * num_px * 3, 1)
b -- bias, a scalar
X -- data of shape (num_px * num_px * 3, number of examples)
Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
num_iterations -- number of iterations of the optimization loop
learning_rate -- learning rate of the gradient descent update rule
print_cost -- True to print the loss every 100 steps
Returns:
params -- dictionary containing the weights w and bias b
grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
Tips:
You basically need to write down two steps and iterate through them:
1) Calculate the cost and the gradient for the current parameters. Use propagate().
2) Update the parameters using gradient descent rule for w and b.
"""
costs = []
for i in range(num_iterations):
# Cost and gradient calculation (≈ 1-4 lines of code)
### START CODE HERE ###
grads, cost = propagate(w, b, X, Y)
### END CODE HERE ###
# Retrieve derivatives from grads
dw = grads["dw"]
db = grads["db"]
# update rule (≈ 2 lines of code)
### START CODE HERE ###
w = w - learning_rate * dw
b = b - learning_rate * db
### END CODE HERE ###
# Record the costs
if i % 100 == 0:
costs.append(cost)
# Print the cost every 100 training examples
if print_cost and i % 100 == 0:
print ("Cost after iteration %i: %f" %(i, cost))
params = {"w": w,
"b": b}
grads = {"dw": dw,
"db": db}
return params, grads, costs

在迭代结束之后就得到了我们要的参数了,利用参数构造预测函数如下:
def predict(w, b, X):
‘’’
Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)

Arguments:
w -- weights, a numpy array of size (num_px * num_px * 3, 1)
b -- bias, a scalar
X -- data of size (num_px * num_px * 3, number of examples)

Returns:
Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
'''

m = X.shape[1]
Y_prediction = np.zeros((1,m))
w = w.reshape(X.shape[0], 1)

# Compute vector "A" predicting the probabilities of a cat being present in the picture
### START CODE HERE ### (≈ 1 line of code)
# 在本课程中,全部向量都按列组织,得到横向预测值,Python矩阵从0开始计数,和Octave不一样
A = sigmoid(np.dot(w.T,X)+b)
### END CODE HERE ###

for i in range(A.shape[1]):

    # Convert probabilities A[0,i] to actual predictions p[0,i]
    ### START CODE HERE ### (≈ 4 lines of code)
    if(A[0][i] > 0.5):
        Y_prediction[0][i] = 1
    else:
        Y_prediction[0][i] = 0
    ### END CODE HERE ###

assert(Y_prediction.shape == (1, m))

return Y_prediction

综合所有部分得到模型

把前面的所有算法综合起来就是我们的单层神经网络模型了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
def model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False):
"""
Builds the logistic regression model by calling the function you've implemented previously
Arguments:
X_train -- training set represented by a numpy array of shape (num_px * num_px * 3, m_train)
Y_train -- training labels represented by a numpy array (vector) of shape (1, m_train)
X_test -- test set represented by a numpy array of shape (num_px * num_px * 3, m_test)
Y_test -- test labels represented by a numpy array (vector) of shape (1, m_test)
num_iterations -- hyperparameter representing the number of iterations to optimize the parameters
learning_rate -- hyperparameter representing the learning rate used in the update rule of optimize()
print_cost -- Set to true to print the cost every 100 iterations
Returns:
d -- dictionary containing information about the model.
"""
### START CODE HERE ###
# initialize parameters with zeros (≈ 1 line of code)
w, b = initialize_with_zeros(X_train.shape[0])
# Gradient descent (≈ 1 line of code)
parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost)
# Retrieve parameters w and b from dictionary "parameters"
w = parameters["w"]
b = parameters["b"]
# Predict test/train set examples (≈ 2 lines of code)
Y_prediction_test = predict(w, b, X_test)
Y_prediction_train = predict(w, b, X_train)
### END CODE HERE ###
# Print train/test Errors
print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))
print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))
d = {"costs": costs,
"Y_prediction_test": Y_prediction_test,
"Y_prediction_train" : Y_prediction_train,
"w" : w,
"b" : b,
"learning_rate" : learning_rate,
"num_iterations": num_iterations}
return d

其训练集上的准确率为99%,测试集上的准确率仅为70%,显然出现了过拟合。

模型分析

可以通过查看不同学习速率的图形来确定学习速率的选取(当然是用高级优化函数的时候不用考虑这个问题,这次作业并没有采用):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
learning_rates = [0.01, 0.001, 0.0001]
models = {}
for i in learning_rates:
print ("learning rate is: " + str(i))
models[str(i)] = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 1500, learning_rate = i, print_cost = False)
print ('\n' + "-------------------------------------------------------" + '\n')
for i in learning_rates:
plt.plot(np.squeeze(models[str(i)]["costs"]), label= str(models[str(i)]["learning_rate"]))
plt.ylabel('cost')
plt.xlabel('iterations')
legend = plt.legend(loc='upper center', shadow=True)
frame = legend.get_frame()
frame.set_facecolor('0.90')
plt.show()

可以看到如下图,显然选取0.01是最好的:

我们还可以选取其他图片来进行测试,其代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
## START CODE HERE ## (PUT YOUR IMAGE NAME)
my_image = "tiger.jpg" # change this to the name of your image file
## END CODE HERE ##
# We preprocess the image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))
my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_predicted_image = predict(d["w"], d["b"], my_image)
plt.imshow(image)
print("y = " + str(np.squeeze(my_predicted_image)) + ", your algorithm predicts a \"" + classes[int(np.squeeze(my_predicted_image)),].decode("utf-8") + "\" picture.")

在网上随便找了张RGB格式的老虎的图片进行测试:

大概是因为都是猫科动物?模型判断图中的老虎为猫..