The incident happened just after 11:30 a.m. near the intersection of Broadway and Prince in SoHo.
Pictures from the scene show wooden planks all over the street, and FDNY firefighters at the scene.
FDNY officials said "we're absolutely lucky" there aren't more injured in this busy neighborhood. The scaffolding collapsed over a subway entrance, but thankfully no train had let out at the time.
Investigators said strong wind is to blame for the collapse. A piece of plywood "acted like a sail" and blew the whole rig down.
Cellphone video shot moments after the collapse shows bystanders running in to help people trapped:
Two people had to be rescued from under the rubble. They and three others were taken to the hospital to be treated for minor, non-life threatening injuries.