"He started attacking [Pelosi] and the president, saying that they were responsible for the kids coming over [across the border]," Rangel recalled. "She left to go over there, and then he said, 'I must have struck a nerve,' and that did it [for Pelosi].
"He called her responsible for it, and she went over there and said, 'You're insignificant,'" Rangel said.
Larson said the exchange "wasn't physical."
"It could have been, but I think people got there [first]," he said.
Rep. Mike Thompson (D-Calif.) was the first Democrat to reach Pelosi, Larson said, "and was able to just calmly walk [her] back to our side."
Read more: Pelosi tangles with GOP lawmaker on House floor | TheHill
Follow us: @thehill on Twitter | TheHill on Facebook