The agency in charge of providing protection for the president confirmed on Monday that it was investigating the allegations that its members brought prostitutes to their Cartagena hotel on Wednesday, near where the president was to stay.
Colombian police said five US military service members were also involved in the incident.
But Army Colonel Scott Malcom, a spokesman for US Southern Command, said an interim investigating officer who began working to collect evidence in the case found information indicating more than five service members may have been involved.
The top US military officer, General Martin Dempsey, chairman of the joint chiefs of staff, said that the service members had let Obama down by distracting attention from his meeting with Latin American leaders in Cartagena.
The alleged incident occurred before Obama arrived in Cartagena, but the news broke while he was there.
"We let the boss down because nobody's talking about what went on in Colombia other than this incident," Dempsey said at a news conference with Leon Panetta, the US defence secretary.
"I can speak for myself and my fellow chiefs, we're embarrassed by what occurred in Colombia."
Panetta said General Douglas Fraser, the head of Southern Command, had begun an investigation to determine the facts of the incident.
'Rigorous investigation'
Obama said on Sunday that he expected a "rigorous" investigation into the alleged misconduct.
"If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry," Obama said.
"We are representing the people of the United States, and when we travel to another country, I expect us to observe the highest standards."
In the US capital, legislators said they were shocked and upset by the allegations, pointing to the security risks involved.
"I am having a call this evening with the director of the Secret Service, because I find this to be so appalling," said Senator Susan Collins, the top Republican on the senate homeland security and governmental affairs committee, which shares jurisdiction over the Secret Service with the judiciary committee.
"I can't help but think what if the women involved had been spies, what if they had been members of a drug cartel, what if
they had planted equipment or eavesdropping devices?"
General Fraser said he was "disappointed by the entire incident and that this behaviour is not in keeping with the professional standards expected of members of the United States military".
