Learning in networks either relies on well-behaved statistical properties of the input (Dayan and Abbott, 2001) or requires simplifying assumptions (Hopfield, 1982). Hence, predicting the temporal development of the network's connections when dropping assumptions like, for instance, stationary inputs is still an open question. For instance, current models of network dynamics (e.g. Memmesheimer and Timme (2006)) require a particular configuration of the network's connections. Up to now, those networks have predefined fixed connection strength and it is of interest whether and how those configurations develop in biological neuronal networks. At the same time it would be also possible to infer relevant parameters of the used plasticity rule while the network's behavior is close to the behavior recorded in the brain (Barbour et al., 2007). We developed a method to analytically calculate the temporal weight development for any linear Hebbian plasticity rule (Kolodziejski and Wörgötter, submitted). This includes differential Hebbian plasticity which is the biophysical counterpart to spike-timing-dependent plasticity (Markram et al., 1997). In this work we concentrate on small and presumably simple networks with up to three neurons and analytically investigate the dynamics of the network's connections and, if existing, their fixed points. The results support the notion that the dynamics depend on the particular type of input distribution used. Hence, it shows that in order to infer relevant parameters of biological networks we would additionally need to take the network's input into considerations. As we cannot assume that all connections in the brain are predefined, learning in networks demands a better understanding and the results presented here might serve as a first step towards a more generalized description of learning in large networks with non-stationary inputs.