The construction of a model of the background of a scene still remains as a challenging task in video surveillance systems, in particular for moving cameras. This work presents a novel approach for constructing a panoramic background model based on the neural gas network and a subsequent piecewise linear interpolation by Delaunay triangulation. Furthermore, an ensemble model of neural gas networks is also proposed. The approach can handle arbitrary camera directions and zooms for a pan-tilt-zoom camera-based surveillance system. After testing the proposed approach on several indoor sequences, the results demonstrate that the proposed methods are effective and suitable to use for real-time video surveillance applications.