High in this video we are going to talk about activation functions.
So let's get started.
This is what we have been discussing in the previous video that there are several activation functions.
We have the step function for percept Thrones and we have sigmoid activation function for the sigmoid
network owns.
So basically a stat function is the sign function.
I'm not sure at all you're familiar with this but this function is going to return a minus on it.
If the X impled is smaller than 0 and it's going to return plus 1 if the X input value is greater than
zero.
So we have to apply one of these activation functions on the Somme.
We have calculated previously.
This is what we have been discussing in the previous lecture that does activation functions are going
to determine the output on the given.
Now we're on.
OK first we have to sum up the incoming cycles then we have to apply the activation function on there
to give us some.
So this is the step function we are able to transform it several ways.
For example if we have the X minus a then the stop function is shifted to the right.
And basically it means that this a is this threshold if that given X import is smaller than a dandy
step function is going to return the minus swan if the axiom put is greater then the a threshold.
The step function is going to return a plus one.
So this a basically is the threshold.
OK.
Usually this threshold is zero but a step function can be defined like this that it's not going to return
a minus one and plus one.
But it's going to return 0 when Paswan.
So even the X is smaller than zero then the result is going to be zero.
And if the input for that stat function is greater than zero then the result will be Paswan.
So that's all about a step function by Daler other activation functions.
For example the sigmoid function it is defined something like this wanted D-y by one plus exponential
function to the power of minus X where X is the input as you can see that it can have the values in
the range 0 and warm if the input.
So if the X is very very big then the output of the sigmoid function is going to be won approximately.
The input.
So the X is very very small as you can see the value of the sigmoid function is going to be close to
zero.
So it's going to be a range 0 and 1.
But what's very important that it can have the well use between 0 and 1 the sigmoid function.
Can we turn the well use 0.1 0.2 0.2 to 5 6 0.4 0.8 and so on.
So instead of a binary output 0 when we want as we have seen for the stat function it can have an output
value in the range 0 when.
Any real number in this range.
But you may have the intuition that OK there's some problem with the sigmoid function and the problem
is that it doesn't have any negative values.
And this is why the so-called tangan hyperbolic function has came to be because we are able to return
values ranging from minor Swane to Swan.
It is defined like this exponential X minus exponential minus X divide by exponential X plus exponential
minus X and sometimes it is a better activation function because it can have negative values as well.
We are not going to have a tank and hyperbolic function as an activation of function in the concrete
implementations because we are going to use the sigmoid function and we are going to ENIC code the output
so that the sigmoid function is going to work just fine.
So that's all about the activation functions.
We have the step function we have the sigmoid function and we have the tongue and hyperbolic function
in order we need negative values as well.
And I don't want to repeat myself over and over again but it's crucial to see that.
Why is it good to use for example a sigmoid function because it can have the output 0 and Paswan as
we have seen for desktop function.
But more important that it can have any values besides one and zero so it can have a well you in the
range 0 and 1.
And this is what we are after if you may recall the problem with the percept wrong is that a small change
in the weights of any single person in the network can sometimes code the output of that percept or
on to completely flip from zero to 1 or 2 1 to zero.
And this is why we like sigmoid.
Now we're on because the input and the outputs can take anywhere between 0 and 1.
Ok this is what we have been discussing.
It can be done.
We hop over the sigmoid activation function has the name sigmoid now wrong.
And why is it good because we would like to make sure that small changes in the edge weights cause small
changes in the output.
We don't want to have Flip's at all.
So this is why it's good to use sigmoid function instead of the stat function because Spurr sections
are using stop functions sigmoid.
Now we're ons are using sigmoid function and this is why if we want to make a good out of network we
have to use sigmoid now roans banks are watching.