Continuing growth of energy use by commercial buildings has created a need to develop innovative techniques to reduce and optimize building energy use. Recently Building Energy Management Systems (BEMS) have gained popularity because of increasing interest in building energy conservation and savings. In this study, a conceptual framework for real-time weather responsive control systems combined with BEMS is proposed to achieve model simulation based Smart BEMS. The proposed control system is developed using building energy control patterns, which are generated from the combinations of weather data changes. As a result, building energy use can be adjusted by, for example, using daylighting responsive controls for electrical lighting as well as by adjusting the HVAC operational schedule, in response to weather changes. To create control logics for model based Smart systems, BIM and Computational Fluid Dynamic (CFD) simulation are used to obtain material properties and to develop air flow operational algorithms, respectively.