The question of whether the law of evidence should be abolished is reminscent of the question of whether taxation should be abolished.
The law of evidence does not consist solely of the hearsay rule.
The law of evidence does not consist solely of rules whose main aim is to enhance the accuracy of fact finding.
The law of evidence does not consist solely of rules that govern the admissibility of evidence.
Broadly conceived, the law of evidence -- Continentals call it the "law of proof" -- includes all legal rules that regulate evidence and fact finding in adjudication.
It is possible that society can realize its purposes -- epistemic and social -- if it leaves evidence-gathering, evidence-assessment, and fact finding completely unregulated, if we say to judges, lawyers, witnesses, and clients, "Go at it -- collecting evidence etc. -- as best you can, and we wish you luck in resolving your disputes about evidence." But it is not very probable that society can best achieve its purposes by abolishing all legal rules that regulate the gathering of evidence, the assessment of evidence, and the drawing of conclusions from evidence. The real battleground is about how evidence-gathering etc. should be regulated, and not whether the law of evidence should be "abolished."