Workshop on Predictive Coding

DESI V Workshop at ICAIL 2013 

Standards for Using Predictive Coding

and Other Machine Learning Algorithms in E-discovery

June 14, 2013

Consiglio Nazionale delle Ricerche, Rome, Italy

DESI V site: http://www.umiacs.umd.edu/~oard/desi5/

DESI V call for submissions: http://www.umiacs.umd.edu/~oard/desi5/Desi5Cfs.pdf

ICAIL 2013 site: http://icail2013.ittig.cnr.it/

 

Purpose

The DESI (Discovery of Electronically Stored Information) workshop series addresses the problem of searching effectively and efficiently for relevant documents, in the context of litigation or investigation, across increasingly complex, enterprise-wide collections within corporate and institutional settings.   The workshops discuss the use of AI and other advanced forms of search techniques in legal settings, as cost-efficient alternatives to traditional Boolean and manual searching.  At DESI V, we intend to focus on best practices and standards for using predictive coding and other forms of machine learning in e-discovery.

Submissions

We invite participation from e-discovery stakeholders and practitioners from the law, government, and industry, along with researchers on process quality, information retrieval, human language technology, human-computer interaction, artificial intelligence, and other fields connected with e-discovery. The dialogue at the workshop will be expected to center around how lawyers are currently using such techniques, how better protocols can be developed that will satisfy the interests of the legal community, and what open questions exist that would benefit from further research into optimizing the use of these techniques in a variety of legal and investigatory settings.

We encourage the submission of research papers and position papers on both supporting technologies for e-discovery (search, text classification, etc.) and on efforts to develop best practices and standards for use of these technologies. Accepted position papers and accepted research papers will be made available on the Workshop's Web page and distributed to participants on the day of the event, and some speakers may be selected from among those submitting position papers. See the Call for Submissions, http://www.umiacs.umd.edu/~oard/desi5/Desi5Cfs.pdf, for further details.  Any questions may be addressed to Doug Oard (oard@umd.edu).

Important Dates

• Research papers due: May 1, 2013
• Position papers due: May 8, 2013
• Accept/Reject notification for research papers: May 15, 2013
• Preliminary Agenda posted: May 22, 2013
• Camera-ready research papers due: May 22, 2013
• ICAIL Conference: June 10-13, 2013
• DESI V Workshop: June 14, 2013

Organizing Committee

Jason R. Baron, National Archives and Records Administration, USA
Jack G. Conrad, Thomson Reuters, Switzerland
Dave Lewis, David D. Lewis Consulting, USA
Debra Logan, Gartner Research, UK
Douglas W. Oard, University of Maryland, USA
Fabrizio Sebastiani, Istituto di Scienza e Tecnologia dell'Informazione, Italy

---

    &&& The dynamic evidence page

Evidence marshaling software MarshalPlan

An Important Message about Fuzzy Logic

The recent award of a major scientific prize to Lotfi Zadeh provoked many admirers from around the world to send along their congratulations on a discussion list devoted to fuzzy logic and soft computing. In so doing, these well-wishers also made a variety of comments about fuzzy logic. Professor Zadeh eventually responded with a substantive comment. (See below.) His comment is important and enlightening in several different ways. Although I will let his comment speak for itself, I do wish to emphasize the astonishing reach of fuzzy logic that Zadeh highlights. It is also worth mentioning that the use of fuzzy logic is apparently accelerating rather than plateauing. There is thus reason to think and to hope that an increasing number of legal scholars (in addition to luminaries such as Kevin Clermont and Lothar Phillips) will decide to use fuzzy logic to explore reasoning about and in law. It is high time that they do so!   

The message from Professor Zadeh: 

Dear members of the BISC Group:

    The BBVA Award has rekindled discussions and debates regarding what fuzzy logic is and what it has to offer. The discussions and debates brought to the surface many misconceptions and misunderstandings. A major source of misunderstanding is rooted in the fact that fuzzy logic has two different meanings -- fuzzy logic in a narrow sense, and fuzzy logic in a wide sense. Informally, narrow-sense fuzzy logic is a logical system which is a generalization of multivalued logic. An important example of narrow-sense fuzzy logic is fuzzy modal logic. In multivalued logic, truth is a matter of degree. A very important distinguishing feature of fuzzy logic is that in fuzzy logic everything is, or is allowed to be, a matter of degree. Furthermore, the degrees are allowed to be fuzzy. Wide-sense fuzzy logic, call it FL, is much more than a logical system. Informally, FL is a precise system of reasoning and computation in which the objects of reasoning and computation are classes with unsharp (fuzzy) boundaries. The centerpiece of fuzzy logic is the concept of a fuzzy set. More generally, FL may be a system of such systems. Today, the term fuzzy logic, FL, is used preponderantly in its wide sense. This is the sense in which the term fuzzy logic is used in the sequel. It is important to note that when we talk about the impact of fuzzy logic, we are talking about the impact of FL. Intellectually, narrow-sense fuzzy logic is an important part of FL, but volume-wise it is a very small part. In fact, most applications of fuzzy logic involve no logic in its traditional sense. 

    What is not widely recognized within the scientific community and the general public, is that fuzzy logic has become a vast enterprise.There are over 280,000 papers in the literature with fuzzy in title. There are 25 journals with fuzzy in title. There are close to 25,000 fuzzy-logic-related patents issued or applied for in the United States and Japan. There is a long list of applications ranging from digital cameras to fraud detection systems. Particularly worthy of note, on one end, is the fuzzy logic subway system in Sendai, a city of over 1 million in Japan. On the other end, numerically, is Omron's 120 million fuzzy logic blood pressure meters.

    Most, but not all of the constituents of fuzzy logic are what are called FL-generalizations of traditional, bivalent-logic-based systems of reasoning and computation. Examples. Fuzzy arithmetic, fuzzy cluster analysis, fuzzy differential equations, fuzzy control, fuzzy linear programming, etc. FL-generalization of a theory or a formalism, T, involves introduction into T of the concept of a fuzzy set, followed by addition of related concepts and techniques. FL-generalization may be applied to any field, any theory, any system, any formalism and any algorithm. The  fundamental importance of FL-generalization derives from the fact that in the real world almost all classes have unsharp (fuzzy) boundaries. As a consequence, FL-generalization opens the door to construction of better models of reality.

    It is of interest to observe that the impact of FL-generalization is growing in visibility and importance in mathematics -- a field in which precision plays a quintessential role. We see a growing number of papers with fuzzy in title in many branches of mathematics, among them topology, algebra, differential equations, group theory, set theory, and functional analysis. What may come as a surprise to many is that Math.Sci.Net database lists over 22,383 papers with fuzzy in title. I did not anticipate that this will happen when I wrote my first paper on fuzzy sets. My expectation was that the concept of a fuzzy set would find its main applications in the realm of soft, human-centered sciences. 

    When it comes to practical application of fuzzy logic, there is a major source of misunderstanding. Fundamentally, fuzzy logic is aimed at precisiation of what is imprecise. But in many of its applications fuzzy logic is used, paradoxically to imprecisiate what is precise. In such applications, there is a tolerance for imprecision, which is exploited through the use of fuzzy logic. Precisiation carries a cost. Imprecisiation reduces cost and enhances tractability. This is what I call the Fuzzy Logic Gambit. What is important to note is  that precision has two different meanings: precision in value and precision in meaning. In the Fuzzy Logic Gambit what is sacrificed is precision in value, but not precision in meaning. More concretely, in the Fuzzy Logic Gambit imprecisiation in value is followed by precisiation in meaning. An example is Yamakawa's inverted pendulum. In this case, differential equations are replaced by fuzzy if-then rules in which words are used in place of numbers. What is precisiated is the meaning of words.

        Some critics have been saying that fuzzy logic is a passing fad. This assessment of fuzzy logic fails to recognize that the world we live in is, in large measure, a world of fuzzy classes, and that science has much to gain from shifting its foundation from classicalAristotelian logic to fuzzy logic. Comments are welcome.
                         Regards to all,

                         Lotfi

&&& The dynamic evidence page

Evidence marshaling software MarshalPlan

Tacit Inferential Reckoning - The Dung Beetle and the Human Animal

If the dung beetle has tacit inferential knowledge, or capacity, why not human beings?
    &&&   The dynamic evidence page
Evidence marshaling software MarshalPlan

Fuzzy Logic and Fuzzy Thinking in Law

I have been thinking about Lotfi Zadeh and his fuzzy logic. Indeed, I have been thinking, off and on, about Zadeh's fuzzy logic for decades. The law is full of fuzzy thinking. If there is a precise way to think about fuzzy ideas, legal scholars, judges, etc., should use it. On reflection and re-reflection and re-re-reflection etc., I do believe that Zadeh has given us important tools for radically better ways of thinking about fuzzy thinking.

It has often been said that the standard probability calculus can do all that fuzzy logic does. But such claims sometimes have the air of gratuitous dicta: one wants to see the proof in such probabilistic pudding. The fuzziness and vagueness of legal concepts (and many other types of concepts) usually can only awkwardly be characterized in terms of uncertainty. Legal principles, concepts, etc., often employ vague notions. There is a difference between a vague concept and an uncertain proposition. Probability theory sometimes deals well with uncertain propositions - but not so well with vague and fuzzy concepts.   In many of its iterations, Zadeh's fuzzy logic does not seem to reach beneath the surface of our fuzzy thinking (e.g., our conventional legal thinking), but seems just to accept the fuzziness of ordinary thinking. It might be said that fuzzy logic, in some its iterations, just describes the natural behavior of fuzzy terms, operators ("and" "or" etc.), and the like, and does not attempt to identify the causes, bases, or foundations of the fuzzy ideas we have and use.  This creates a puzzle: How can a logic that does not purport to identify the foundations, sources, or causes of fuzzy thinking give us useful new thoughts? (This question assumes that fuzzy thinking is a sort of disease.)  

The answer, I think, lies in the notion of tacit knowledge. There is genuine knowledge buried in some or much of our "ordinary" fuzzy thinking. (If that were not the case, few of us would survive even for one day.) Fuzzy logic's proven successes suggest that fuzzy logic may offer a way to uncover, or display, much "innate," or tacit, human knowledge.

These are admittedly deep and possibly murky waters, and I confess I do not have the ability to swim through them easily. I console myself with the thought that the acquisition of knowledge is a collective human enterprise and that there will be others who may be able to build on some of the paltry number of insights I may have acquired over the years. But if it turns out I have not learned much of enduring value about fuzzy thinking, the efforts I have made may have been worth the candle - because people can learn by studying other people's errors. &&&     The dynamic evidence page
Evidence marshaling software MarshalPlan

A Religious Liberty Clinic at Stanford Law School

Ethan Bronner, At Stanford, Clinical Training for Defense of Religious Liberty NYTimes (Jan. 21, 2013):

Backed by two conservative groups, Stanford Law School has opened the nation’s only clinic devoted to religious liberty, an indication both of where the church-state debate has moved and of the growth in hands-on legal education.

Begun with $1.6 million from the John Templeton Foundation, funneled through the Becket Fund for Religious Liberty, the school’s new Religious Liberty Clinic partly reflects a feeling that clinical education, historically dominated by the left’s concerns about poverty and housing, needs to expand.

“The 47 percent of the people who voted for Mitt Romney deserve a curriculum as well,” said Lawrence C. Marshall, the associate dean for clinical legal education at Stanford Law School. “My mission has been to make clinical education as central to legal education as it is to medical education. Just as we are concerned about diversity in gender, race and ethnicity, we ought to be committed to ideological diversity.” Mr. Marshall became a hero to liberals for his work to exonerate death penalty inmates when he was a professor at Northwestern Law School a decade ago.

[snip, snip]

“In framing our docket, we decided we would represent the believers,” said James A. Sonne, the clinic’s founding director, explaining that the believers, rather than governments, were the ones in need of student lawyers to defend them. “Our job is religious liberty rather than freedom from religion.”

Mr. Sonne, who grew up the son of a psychoanalyst in a nominally Episcopalian home near Cherry Hill, N.J., converted to Roman Catholicism while a student at Duke University. He went on to Harvard Law School and later a professorship at Ave Maria School of Law, a Catholic institution. He acknowledges the political coloration of much of the religious-freedom debate but says he does not want his clinic to be seen as a program for conservatives.

His first four students — a Mormon, a Methodist, a Catholic and someone brought up as a Seventh-day Adventist — agree, saying they were drawn to the clinic by the profound questions it raises and the real lawyering it offers, from meeting a potential client to appellate review.

[snip, snip]

Barry Lynn, the executive director ofAmericans United for Separation of Church and State, said he was “shocked that a major law school would accept a gift from Becket,” which he described as “a group that wants to give religious institutions or individuals a kind of preferential treatment, even if that hurts a third party.”

But Hannah C. Smith of Becket, who took part in a panel discussion here on Monday to observe the clinic’s opening, said what liberals like Mr. Lynn call the strict wall of separation is found nowhere in the Constitution. Her group, she said, is working to show that “there are certain God-given rights that existed before the state. God gave people the yearning to discover him. Religious freedom means we have to protect the right to search for religious truth free from government intrusion.”

&&&

The dynamic evidence page
Evidence marshaling software MarshalPlan

Campbell's Law

During a C-Span broadcast of a discussion of Charles J. Wheelan's Naked Statistics: Stripping the Dread from the Data (2013), a member of the audience - a statistician - asked Wheelan if he knew about Campbell's Law. I turned to Wikipedia, of course, and found the following entry (see http://en.wikipedia.org/wiki/Campbell's_law):

Campbell's law is an adage developed by Donald T. Campbell:^[1]

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

The social science principle of Campbell's law is sometimes used to point out the negative consequences of high-stakes testing in U.S. classrooms.

What Campbell also states in this principle is that "achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. (Similar biases of course surround the use of objective tests in courses or as entrance examinations.)"^[1]

Campbell's law was published in 1976 by Donald T. Campbell, an experimental social science researcher and the author of many works on research methodology. Closely related ideas are known under different names, e.g. Goodhart's law, and the Lucas critique. Another concept related to Campbell's law emerged in 2006 when UK researchers Rebecca Boden and Debbie Epstein published an analysis of evidence-based policy, a practice espoused by Prime Minister Tony Blair. In the paper, Boden and Epstein described how a government that tries to base its policy on evidence can actually end up producing corrupted data because it, "seeks to capture and control the knowledge producing processes to the point where this type of ‘research’ might best be described as ‘policy-based evidence’."^[2] (Boden and Epstein 2006: 226)

&&&

Would Lon L. Fulller's students have given him high marks on teaching evaluation forms? I doubt it. I vividly remember hearing my fellow law students complain that Fuller just created confusion in the classroom.

What would or should Fuller have done to raise his teaching evaluation scores (if he cared about them)? I suppose he would have simplified the parable of the Speluncean Explorers and he would given a multiple-choice exam in the jurisprudence course he taught.

&&&


&&&

The dynamic evidence page

Evidence marshaling software MarshalPlan