Mr Johnson, 21, killed himself last year after alleging that he had been bullied by Mr Clarke, at the time a key player in the Conservative election campaign.
A party spokesman called for anyone who had previously felt unable to come forward with bullying allegations to do so now, following the announcement.
In a statement Mr Semple, the chairman of the voluntary party, said: "Having worked hard to establish both an independent investigation and the appointment of Lord Pannick to oversee it; I am absolutely determined that it should be, and seen to be, a thorough and complete process.
“As a father myself, the wishes of Mr and Mrs Johnson are paramount to me and after seeing their interview on BBC TV, I have decided to recuse myself from the Board meeting that will discuss the Clifford Chance report.
“As a Conservative volunteer for 20 years I want to establish the truth of what happened to Elliott. I am convinced the investigation will be fair and thorough and my decision must not be regarded as my accepting any suggestion that I would be less than impartial.
"As Chairman of the voluntary party I will be pressing for the implementation of all the recommendations of the investigation.”
A spokesman for the Tories said: “If anyone felt they were unable to come forward whilst Rob Semple would be in the Board meeting where the Clifford Chance report was discussed, we would urge them now to do so.”
It follows a TV interview in which Mr Johnson's mother Alison complained the family have been treated like suspects in the inquiry.
She said: "We live out in the backwaters so they probably think if we silence them, they will go away quietly but we won’t.
"I want justice for my son. Every day just drifts into another, we don’t know anything."
She added: “We also heard that they wanted to interview my husband and myself individually to find out what our stories were. Well its not a story, it’s the facts. Elliott is now gone, we can only deal with the facts."
The Conservative Party said: "We would like to again express our very deepest sympathies to Elliott's family and friends."
The Conservative party has expelled Mr Clarke from the party and started an investigation, which has been handed entirely to law firm Clifford Chance.
Mr Clarke denies any wrongdoing.
Source: http://telegraph.feedsportal.com/c/32726/f/579309/s/4d31d581/sc/13/l/0L0Stelegraph0O0Cnews0Cpolitics0Cconservative0C121280A530CSenior0ETory0Ewill0Enot0Ebe0Eat0Emeeting0Eto0Ediscuss0ETatler0ETory0Ereport0Eamid0Eimpartiality0Econcerns0Bhtml/story01.htm