Most experts agree that large language models (LLMs), such as those used by Copilot and ChatGPT, are expected to revo-
lutionize the way in which software is developed. Many papers are currently devoted to analyzing the potential advantages
and limitations of these generative AI models for writing code. However, the analysis of the current state of LLMs with
respect to software modeling has received little attention. In this paper, we investigate the current capabilities of ChatGPT to
perform modeling tasks and to assist modelers, while also trying to identify its main shortcomings. Our findings show that,
in contrast to code generation, the performance of the current version of ChatGPT for software modeling is limited, with
various syntactic and semantic deficiencies, lack of consistency in responses and scalability issues. We also outline our views
on how we perceive the role that LLMs can play in the software modeling discipline in the short term, and how the modeling
community can help to improve the current capabilities of ChatGPT and the coming LLMs for software modeling.
