Artificial neural networks are often proposed as an alternative approach for formalizing various quantitative and qualitative aspects of complex systems. This paper examines the robustness of using neural networks as a simulation metamodel to estimate manufacturing system performances. Simulation models of a job shop system are developed for various configurations to train neural network metamodels. Extensive computational tests are carried out with the proposed models at various factor levels (study horizon, system load, initial system status, stochasticity, system size and error assessment methods) to see the metamodel accuracy. The results indicate that simulation metamodels with neural networks can be effectively used to estimate the system performances.