"I think there will be quite a few people who will now be pretty firmly of the view that she should resign", Vaizey told BBC Oxford in an interview on Thursday, when asked whether May should step aside.
In a keynote conference speech where the protester and a coughing fit brought her words nearly to a halt, Mrs May won over many members in the hall by promising to reinvigorate the party and offering pledges to younger people and families alike.
But announcements including a £2 billion to boost council home building and a new law to cap energy prices to help voters "left behind" by an unbalanced economy were overshadowed by the plague of mishaps and interruptions.
The intruder was identified by media as comedian Simon Brodkin, who performs under the name of Lee Nelson.
While May's Cabinet publicly rushed to defend her, speaking of her bravery and resilience in finishing the speech, privately they said she was "limping like a broken horse into oblivion", and suggested that talks about her departure would have to be "accelerated".
Mr Shapps was a minister in David Cameron's government but an ally of the former premier said he has misjudged the situation.
Grant Shapps made the headlines after it was revealed he had gathered a list of 30 Tory MPs who are discontented with the Prime Minister.
There is also anger at Foreign Secretary Boris Johnson, whose recent outbursts on Brexit policy dismayed colleagues for overshadowing the Tory conference in Manchester, according to one senior Conservative official.
May knows she has a job to do and seems determined to give the leadership her best shot.
If that was not enough, a never ending cough fit had interrupted the speech, a speech that she desperately needed to work in her favour. May spoke of plans to "reignite home ownership" in United Kingdom and said the government plans to invest an additional £2bn in affordable housing, taking the total budget up to nearly £9bn.
She tweeted: "The plot is by remain MPs to topple the PM, destroy Boris and put a remain leader in place to delay and possibly destroy #Brexit". She sipped water. Her finance minister, Philip Hammond, passed her a lozenge. The understanding audience came to her aid with a lengthy ovation, allowing May to take a drink of water and regain composure. More letters would follow.
And just as she did after the botched general election - in which she was widely perceived to have squandered an extremely strong position - May soldiered on.
She added that MPs would get an update on her Florence speech next week. She didn't provide details about how much such public housing would cost or how many of them would be built.