Authorities said they expect the death toll to rise as they sift through the rubble and find further victims.
The factory near Jaar city manufactures ammunition and Kalashnikov automatic rifles. Militants not linked to Al-Qaeda reportedly seized the factory on Sunday.
Yemen is facing anti-government protests a successionist rebellion in the south and a militant Islamist insurgency believed to driven by Al-Qaeda in its south.
Evidence is emerging that the country is becoming an Al-Qaeda stronghold.