tanh Neurons are Bayesian Decision Makers

The hyperbolic tangent (tanh) is a traditional choice for the activation function of the neurons of an artificial neural network. Here, we go through a simple calculation that shows that this modeling choice is linked to Bayesian decision theory. Our brief, tutorial-like discussion is intended as a reference to an observation rarely mentioned in standard textbooks.