The building is a seven-unit multiple-family dwelling condo building. Officials said the second and third-floor front decks suffered a complete "pancake" collapse, trapping multiple people.
The collapse occurred during an annual firefighters convention in the beach resort town. Firefighters were likely among those hurt or trapped.
According to officials with Cape May Regional Health System, 21 people were being treated for injuries at their facility.
By early Sunday morning, a spokesperson for the hospital said 19 patients, including three children, had been treated and released.
At least one additional person was taken to AtlantiCare Regional Medical Center in Atlantic City.
In an interview with ABC News, Chief Jon Frato of the Branchville Fire Department said he was on the second floor deck when the collapse occurred.
"It was just chaos," said Frato. "We had a four-month-old who was being held by a female, the deck collapsed and she ended up underneath the deck with the baby under her, and we had to get the deck picked up to get her out and the baby out."
Frato also described how a second child was on a portion of the deck that did not immediately collapse and another firefighter who he refers to as "hero" climbed up and brought the child to safety just moments before that part of the deck also collapsed.
Photos and videos from the scene showed more than a dozen firefighters surrounding the debris, which was scattered in front of a home and on the sidewalk.
"I could hear people over there calling for help," said neighbor June Cheeseman.
Many are now hoping the injuries are not too serious.
"We just wanted to come out and say a prayer," one person said.
Wildwood Fire Department and the City's Construction Office are working with the building owners, engineers and contractors to properly secure a hip-roof that is currently unsupported due to the collapse.