This work introduces an algebraic graphical language of perceptrons, multilayer perceptrons, recurrent neural networks, and long short-term memory neural networks, via string diagrams of a suitable hypergraph category equipped with a concatenation diagram operation by means of a monoidal endofunctor. Using this language, we introduce a neural network architecture for modeling sequential data in which each sequence is subject to a specific context with a temporal structure, that is, each data point of a sequence is conditioned to a different past, present, and future context than the other points. As proof of concept, this architecture is implemented as a generative model of jazz solo improvisations.