Coursera Machine Learning Week 5 - Neural Networks Learning

'Talk is cheap. Show me the code.'
— Linus Torvalds.

Foreword

This is a set of presentation slides from my first-year graduate education group study in SJTU.

Starting from Assignment

That's what we care about most!

Neural Networks Notations

NN Notations

Neural Networks Overview

NN Overview

NN Propagation

Learning Algorithm Details

a1 = X';
a1 = [ones(1, m); a1];

z2 = Theta1 * a1;
a2 = sigmoid(z2);
a2 = [ones(1, m); a2];

z3 = Theta2 * a2;
a3 = sigmoid(z3);

NN Cost Function

without Regularization

NN Cost Function

Jc = zeros(num_labels, 1);
for c= 1:num_labels
  Jc(c) = (1 / m) * ( -log(a3(c, :)) * (y==c) - log(1 - a3(c, :)) * (1 - (y==c)));
end
J = sum(Jc);

with Regularization

NN Cost Function

Theta1R = Theta1(:, 2:end);
Theta2R = Theta2(:, 2:end);

nn_params_R = [Theta1R(:) ; Theta2R(:)];

J = J + (lambda / (2 * m)) * nn_params_R' * nn_params_R;

general Regularization

NN Cost Function

NN Gredient Calculation

Method

Backpropagation Calculation

Detailed Backpropagation Calculations

Error

Error Calculation for Output Layer

Error Calculation for Hidden Layer

for t = 1:m
  %delta3 = a3(:, t) - !([1:num_labels]' - y(t)); %not work for matlab, works for octave
  delta3 = a3(:, t) - not([1:num_labels]' - y(t));
  delta2 = Theta2R' * delta3 .* sigmoidGradient(z2(:, t));
end
Gredient

Gradient Accumulation

for t = 1:m
  Delta2 = Delta2 + delta3 * a2(:, t)';
  Delta1 = Delta1 + delta2 * a1(:, t)';
end

NN Gradient

NN Gradient Regularization

D2 = (1 / m) * Delta2;
D1 = (1 / m) * Delta1;

%D2(:, 2:end) += (lambda / m) * Theta2(:, 2:end);
%D1(:, 2:end) += (lambda / m) * Theta1(:, 2:end);
D2(:, 2:end) = D2(:, 2:end) + (lambda / m) * Theta2(:, 2:end);
D1(:, 2:end) = D1(:, 2:end) + (lambda / m) * Theta1(:, 2:end);

Theta2_grad = D2;
Theta1_grad = D1;

What's Next?

Learning Algoritm Overview

Learning Algorithm

Learning Algorithm Procedures

Learning Algorithm Procedures

How to check Gredient

Check Gradient

Zero Initialization Problem

Zero Initialization Problem

So, that's All.

In fact, not all!


* cached version, generated at 2018-12-06 05:00:47 UTC.

Subscribe by RSS