The couple was able to escape without serious injuries, KMGH reported.
The mother and the first cub entered Jon Johnson and George Field's home around 8:30 p.m. Monday and started eating a loaf of bread, the Jefferson County Sheriff's office said.
The homeowner's surveillance video shows the bears push through the kitchen screen door, entering the home through the back porch.
WATCH: Surveillance video catches mother bear and her cubs entering Colo. home
"She stuck her nose in and used the nose to push the door open," homeowner Jon Johnson said.
The couple was downstairs when they heard a noise in the kitchen and went to investigate.
That's when they found the bears.
"She [the bear] started to get aggressive and that's when she took a swipe at my nose," Johnson said.
Colorado Parks and Wildlife said the fight between the sow and the man was described as "a boxing match.
"At that point I smacked her right in her nose," said Johnson.
The sheriff's office said Johnson was scratched and cut on his face, chest and arms by the bear but that he was not seriously injured and was treated at the scene.
Johnson's wife, George Field, was not injured during the fight.
However before Field called the police, she also took a swing at the bear with her Louisville Slugger bat.
"I felt that I had like a lightning bolt in my body that was driving that ball bat. I was so scared" Field said. "My adrenaline was, I mean, I can still feel it."
The couple said it took them a few minutes to drive the bears out of the home, leaving blood stains on the carpet and holes in the walls.
Colorado Parks and Wildlife was notified and used dogs to search for the bears, the sheriff's office said.
CPW said the team searched until about midnight, then again starting at 5:30 a.m. Tuesday.
CPW wildlife officers and USDA Wildlife Services officials tracked the bear and found it about 900 yards away from the home around 7 a.m. Tuesday.
Officials euthanized the mother bear but are still looking for the cubs.
CPW policies state that if a bear attacks and injures a human, it must be euthanized.