Which activation function are the most popular? What is Logistic Function? What is difference between Logic Function and Sigmoid Function? How can I use these functions in my C++ app? Briefly, a Sigmoid Function is one of the most popular activation functions that are used in AI Technologies. A Sigmoid Function is a simple type of Logistic Function aka Logistic Curve. Let’s recap what an activation function is and explain these terms.
Table of Contents
What is an activation function in an artificial neural network (ANN)?
An Activation Function ( phi() ) also called a transfer function, or threshold function determines the activation value ( a = phi(sum) ) from a given value (sum) from the Net Input Function . Net Input Function, here the sum is a sum of signals in their weights, and activation function is a new value of this sum with a given function or conditions. In another term. The activation function is a way to transfer the sum of all weighted signals to a new activation value of that signal. There are different activation functions, mostly Linear (Identity), bipolar and logistic (sigmoid) functions are used. The activation function and its types are explained well here.
In C++ (in general in most Programming Languages) you can create your activation function. Note that sum is the result of Net Input Function which calculates the sum of all weighted signals. We will use some as a result of the input function. Here activation value of an artificial neuron (output value) can be written by the activation function as below,
By using this sum Net Input Function Value and phi() activation functions, let’s see some of activation functions in C++; Now Let’s see how we can use Binary Step Function as in this example formula,
What is a Logistic Function (Logistic Curve) ?
A Logistic function or Logistic Curve is a common S-shaped curve (sigmoid curve) with equation below,
Here,
L is maximum value of curve,
x0 is the value of curves mid point,
k the logistic growth rate or steepness of the curve
In C or C++ programming language, Logistic Function function can be written as below,
|
1 2 3 4 5 6 7 8 9 10 |
const double L = 1; const double k = 0.8; const double x0 = 0; double phi(double sum) { return ( L/(1+ std::exp( -k*(sum-x0))) ); // Logistic Function or Logistic Curve } |
If L, k and x0 are constant as above, they can be directly included in the function as in this example (note that values may vary)
|
1 2 3 4 5 6 |
double phi(double sum) { return ( 1/(1+ std::exp( -0.8*(sum))) ); // Logistic Function or Logistic Curve i } |
What is a Standard Logistic Function (Sigmoid Function)?
The most used Logistic Function is a Standard Logistic Function (Sigmoid Function), where L and k is 1 and x0=0. Thus, our function can be written one of these terms as below,
|
1 2 3 4 5 6 |
double phi(double sum) { return ( 1/(1+ std::exp( -1*sum)) ); // Standard Logistic Function, Sigmoid Function } |
Note that here division costs more CPU usage than a multiplication, as ın function given above we can use the version with tanh() as below,
|
1 2 3 4 5 6 |
double phi(double sum) { return ( 0.5*(1+ std::tanh( 0.5*sum)) ); // Standard Logistic Function, Sigmoid Function } |
As you see, we have only multiplication and addition with tanh() function here. If the network’s sum values are in a range i.e. between (0-10) , to achieve faster approximate results arrayed results can be used. There may be y array with 10000 members and for example y[1001] can hold pre-calculated value for phi(1.0001). This will make your neural network faster but also may cause more errors or hard to achieve desired epoch numbers. It should be tested with the one of the normal sigmoid function versions as above.
An Example of a Sigmoid Function (Standard Logistic Function)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
#include <iostream> #define NN 2 // number of neurons class Tneuron // neuron class { public: bool a; // activity of each neurons float w[NN+1]; // weight of links between each neurons Tneuron() { a=0; for(int i=0; i<=NN; i++) w[i]=-1; // if weight is negative there is no link } // let's define an activation function (or threshold) for the output neuron double phi(double sum) { return ( 0.5*(1+ std::tanh( 0.5*sum)) ); // Standard Logistic Function, Sigmoid Function } }; Tneuron ne[NN+1]; // neuron objects void fire(int nn) { float sum = 0; for ( int j=0; j<=NN; j++ ) { if( ne[j].w[nn]>=0 ) sum += ne[j].a*ne[j].w[nn]; } ne[nn].a = ne[nn].activation_function(sum); } int main() { //let's define activity of two input neurons (a0, a1) and one output neuron (a2) ne[0].a = 0; ne[1].a = 1; ne[2].a = 0; //let's define weights of signals comes from two input neurons to output neuron (0 to 2 and 1 to 2) ne[0].w[2] = 0.3; ne[1].w[2] = 0.2; // Let's fire our artificial neuron activity, output will be fire(2); printf("%d\n", ne[2].a); getchar(); return 0; } |
Logistic functions are generally used in ANN applications to introduce nonlinearity in a development model, or it is used to clamp signals to within a specified interval. ANN element computes a linear combination of its input signals and applies a bounded logistic function as the activation function as an output (activation value). This can be defined as a smooth version of the classical threshold neuron. A most common choice for the activation functions, used to clip for large magnitudes to keep the response of the neural network bounded.
which is a logistic function. There is more information about Logistic Functions in Wikipedia.
Are you ready to try some of this for yourself? Download a free trial of C++ Builder today!
Design. Code. Compile. Deploy.
Start Free Trial
Free C++Builder Community Edition
