The bus, which at the time was quiet on a busy road, was buried under rubble in Gwangju, about 270 kilometers southwest of the capital Seoul.
Another 25 people were injured, including eight seriously, the ministry added.
The 16 demolition workers on site are safe and have not suffered any injuries when the building collapsed.
About 190 firefighters were deployed for search and rescue, and police participated in the rescue effort, the ministry said.
The cause of the crash was unclear, Reuters reported, citing officials at Gwangju Fire and Safety Headquarters. The Ministry of Land, Infrastructure and Transport will investigate the cause of the collapse.
South Korea has struggled with poor infrastructure security registration – a department store collapse killed more than 500 people in 1995, and a bridge collapse in 1994 killed 49 people. In recent years, the country has tried to improve its security register.