The selection of the best color space is a fundamental task in detecting foreground objects on scenes. In many situations, especially on dynamic backgrounds, neither grayscale nor RGB color spaces represent the best solution to detect foreground objects. Other standard color spaces,
such as YCbCr or HSV, have been proposed for background modeling in the literature; although the best results have been achieved using diverse color spaces according to the application, scene, algorithm, etc. In this work, a color space and color component weighting selection process is proposed to detect foreground objects in video sequences using self-organizing maps. Experimental results are also provided using well known benchmark videos.