TAR Case Law
Eighteen Rulings on TAR and Predictive Coding You Need to Know
The use of technology assisted review, often known as TAR or predictive coding, has been approved by the court for over ten years, and serves as a valuable tool for legal teams during the discovery process. TAR uses AI technology to identify and label potentially discoverable documents in an effort to accelerate document review and take the burden off human reviewers. Although the introduction of more innovative generative AI is fast becoming an attractive option for a number of organizations, TAR is still proving its worth.
While the use of TAR in discovery has become commonplace, there are still disputes on whether TAR should be required in cases, whether parties can switch to using TAR in the middle of a case, and who pays for TAR if the requesting party advocates for a specific approach.
Out of the many case law decisions involving the use of TAR over the past ten years, Everlaw has selected eighteen case law decisions that you should know to understand what the courts have been saying about the use of TAR over that time. These cases cover a wide range of uses and applications of predictive coding and TAR, and provide insight into what the future of legal technology looks like. The categories of TAR cases that you need to know include:
Court Approval of TAR: Cases that first illustrated the use and acceptance of TAR in the courts, why it was accepted, and what objections were addressed. The case included in this category is:
Da Silva Moore v. Publicis Groupe & MSL Group (S.D.N.Y. Feb. 24, 2012)
Rulings on Specific TAR Approaches : Cases that illustrate how courts have handled disputes over the approach to TAR. The three cases included in this category are:
Dynamo Holdings v. Commissioner of Internal Revenue (U.S. Tax Ct. July 13, 2016)
FCA US, LLC v. Cummins, Inc. (E.D. Mich., Mar. 28, 2017)
City of Rockford v. Mallinckrodt ARD Inc. (N.D. Ill. Aug. 7, 2018)
Instructions/Protocols for TAR Use: Instructions and protocols issued through the courts on how TAR is to be conducted in those cases. The three cases included in this category are:
In re Actos (Pioglitazone) Products Liability Litigation (W.D. La. July 27, 2012)
Rio Tinto Plc v. Vale S.A. (S.D.N.Y. Mar. 2, 2015)
In re Broiler Chicken Antitrust Litigation (N.D. Ill. Jan. 3, 2018)
Requiring the Use of TAR: What the courts have said to parties that wish to require the responding parties to use TAR. The three cases included in this category are:
Hyles v. New York City (S.D.N.Y. Aug. 1, 2016)
In re Viagra Products Liability Litigation (N.D. Cal., Oct. 14, 2016)
In re Mercedes-Benz Emissions Litigation (D.N.J. Jan. 8, 2020)
Search Term Culling Before TAR: How courts have addressed disputes regarding the use of keyword searches to cull the dataset before TAR. The two cases included in this category are:
In re Biomet M2A Magnum Hip Implant Prods. Liability Litigation (N.D. Ind. Apr. 18, 2013)
Livingston v. City of Chicago (N.D. Ill. Sept. 3, 2020)
Switching Approaches to TAR: What courts have said when parties request to switch to a TAR approach in the middle of discovery. The three cases included in this category are:
Progressive Casualty Insurance Co. v. Delaney (D. Nev. May 19, 2014)
Bridgestone Americas Inc. v. Int’l Bus. Mach. Corp. (M.D. Tenn. July 22, 2014)
In re Valsartan, Losartan, and Irbesartan Prod. Liability Litigation (D.N.J. Dec. 2, 2020)
TAR Cost Disputes and Cost Shifting: How a court handled a dispute over a TAR approach advocated by the requesting party and who would bear the costs associated with that approach. The one case included in this category is:
Lawson v. Spirit Aerosystems, Inc. (D. Kan. June 18, 2020)
Addressing Issues With TAR: How courts addressed issues associated with TAR productions that were identified by the receiving parties in discovery. The two cases included in this category are:
Winfield v. City of New York (S.D.N.Y. Nov. 27, 2017)
In Re Domestic Airline Travel Antitrust Litigation, MDL Docket No. 2656 (D.D.C. Sept. 13, 2018)
Court Approval of Technology Assisted Review
Da Silva Moore
Summary of the Case Ruling
In Da Silva Moore v. Publicis Groupe & MSL Group (S.D.N.Y. Feb. 24, 2012), Magistrate Judge Andrew J. Peck of the U.S. District Court for the Southern District of New York, after instructing the parties to submit proposals to adopt a protocol for ediscovery that includes the use of predictive coding, issued an opinion that approved of the use of “computer-assisted review” of ESI for this case, making it likely the first case to recognize that “computer-assisted review is an acceptable way to search for relevant ESI in appropriate cases.”
Highlights of the Court’s Decision
After providing a background of the Title VII gender discrimination case, Judge Peck went on to reference his article (Search, Forward: Will manual document review and keyword searches be replaced by computer-assisted coding?) to explain computer-assisted review.
Judge Peck also detailed the parties’ negotiation of an agreed protocol for the computer-assisted review for this case and accepted the defendants’ proposal, which included seven iterative “seeding” reviews (with a caveat of what to do if the parties didn’t agree that seven rounds of review were sufficient). Judge Peck also recognized that “computer-assisted review is not a magic, Staples-Easy-Button, solution appropriate for all cases” and noted that “[t]he goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the ‘value’ of the case” (as noted in the 2011 article Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, written by Maura R. Grossman & Gordon V. Cormack).
In the conclusion to his ruling, Judge Peck stated:
This Opinion appears to be the first in which a Court has approved of the use of computer-assisted review. That does not mean computer-assisted review must be used in all cases, or that the exact ESI protocol approved here will be appropriate in all future cases that utilize computer-assisted review. Nor does this Opinion endorse any vendor … nor any particular computer-assisted review tool. What the Bar should take away from this Opinion is that computer-assisted review is an available tool and should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review… Computer-assisted review now can be considered judicially-approved for use in appropriate cases.
Related Cases
Three other cases were the first in their respective jurisdictions to approve TAR: Irish Bank Resolution Corporation Ltd v. Quinn in March 2015 in Ireland, Pyrrho Investments Ltd v MWB Property Ltd in February 2016 in England, and McConnell Dowell Constructors v Santam in December 2016 in Australia.
Why This Case Is Significant
It’s the first case to approve the use of TAR, giving parties seeking to apply TAR technologies and approaches a precedent for its use, settling the question of whether a party can use TAR for good!
Download Da Silva Moore v. Publicis Groupe & MSL Group (S.D.N.Y. Feb. 24, 2012).
Rulings on Specific TAR Approaches
Dynamo Holdings
Summary of the Case Ruling
In the case of Dynamo Holdings v. Commissioner of Internal Revenue (U.S. Tax Ct. July 13, 2016), Texas Tax Court Judge Ronald Buch denied the respondent’s Motion to Compel Production of Documents Containing Certain Terms, finding that “[t]here is no question that petitioners satisfied our Rules when they responded using predictive coding”.
Highlights of the Court’s Decision
Judge Buch previously ruled in September 2014 that “[p]etitioners may use predictive coding in responding to respondent’s discovery request. If, after reviewing the results, respondent believes that the response to the discovery request is incomplete, he may file a motion to compel at that time.” At the outset of this ruling, Judge Buch noted that “[t]he parties are to be commended for working together to develop a predictive coding protocol from which they worked”.
After the respondent filed a motion to compel production of the documents identified in a Boolean search that were not produced in the production set (1,353 of 1,645 documents containing those terms they claimed were not produced), asserting that those documents were “highly likely to be relevant”, Judge Buch noted that “[r]espondent’s motion is predicated on two myths”: 1) the myth that “manual review by humans of large amounts of information is as accurate and complete as possible – perhaps even perfect – and constitutes the gold standard by which all searches should be measured”, and 2) the myth of a perfect response to the respondent’s discovery request, which the Tax Court Rules don’t require.
Judge Buch cited Rio Tinto where Judge Andrew Peck stated: “One point must be stressed – it is inappropriate to hold TAR [technology assisted review] to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.”
Stating that “[t]here is no question that petitioners satisfied our Rules when they responded using predictive coding”, Judge Buch denied the respondent’s motion to compel.
Why This Case Is Significant
This case represents one of the earliest disputes regarding the approach to TAR, which helped set the expectations for the results to be expected from a TAR approach. It was subsequently referenced in another Judge Peck decision – Hyles – as a rationale for not dictating to responding parties the approach they use for discovery.
Download Dynamo Holdings v. Commissioner of Internal Revenue (U.S. Tax Ct. July 13, 2016).
FCA US, LLC v. Cummins, Inc.
Summary of the Case Ruling
In FCA US, LLC v. Cummins, Inc. (E.D. Mich., Mar. 28, 2017), Michigan District Judge Avern Cohn “rather reluctantly” decided a dispute between the plaintiff and defendant on whether the universe of electronic material subject to TAR review should first be culled using search terms by agreeing with the plaintiff that “[a]pplying TAR to the universe of electronic material before any keyword search reduces the universe of electronic material is the preferred method.”
Highlights of the Court’s Decision
In a short ruling, Judge Cohn stated:
While the parties agree on many issues relating to discovery and particularly electronic discovery, they unfortunately cannot reach agreement over whether the universe of electronic material subject to TAR review should first be culled by the use of search terms. FCA takes the position that the electronic material subject to TAR review should not first be culled by the use of search terms. Cummins takes the position that a pre-TAR culling is appropriate. They have left the matter for the Court to decide. The Court does so rather reluctantly. Given the magnitude of the dispute and the substantial matters upon which they agree, the parties should have been able to resolve the discovery issue without the Court as decision maker. Be that as it may, having reviewed the letters and proposed orders together with some technical in-house assistance including a read of The Sedona Conference TAR Case Law Primer, issued in 2017, the Court is satisfied that FCA has the better position. Applying TAR to the universe of electronic material before any keyword search reduces the universe of electronic material is the preferred method. The TAR results can then be culled by the use of search terms or other methods. Accordingly, the Court shall enter FCA's proposed order.
Why This Case Is Significant
It’s one of the few cases where the Court has actually chosen between two approaches to TAR submitted by the parties. Typically, courts have defaulted to the approach preferred by the responding party, but in this case, the Court decided to have both parties adopt the same approach to TAR – perhaps because the case was symmetrical in terms of volume being produced by each party.
Download FCA US, LLC v. Cummins, Inc. (E.D. Mich., Mar. 28, 2017).
City of Rockford v. Mallinckrodt
Summary of the Case Ruling
In City of Rockford v. Mallinckrodt ARD Inc. (N.D. Ill. Aug. 7, 2018), Illinois Magistrate Judge Iain D. Johnston adopted the parties’ proposed order establishing the production protocol for ESI with the inclusion of the plaintiffs’ proposal that a random sample of the null set will occur after the production and that any responsive documents found because of that process will be produced.
Highlights of the Court’s Decision
Here, the parties agreed to a keyword search (non-TAR) based approach but couldn’t agree whether to conduct a random sample of the null set (the documents not returned via search) once the searches were complete.
While noting that “the parties have agreed to use keyword searching”, Judge Johnston evaluated the “pros and cons” of keyword searching as compared to TAR, stating: “With the advent of TAR and improvements made to that process, there appears to be a growing chorus that keyword searching is not best practices, and that TAR is the way to go. Many trusted ESI experts – and courts – understandably sing the praises of TAR, especially its ability to scientifically quantify the successfulness of gathering and producing relevant documents.” However, Judge Johnston ultimately decided that he “will not micromanage the litigation and force TAR onto the parties.
Judge Johnston also ruled that sampling the null set is reasonable under Rule 26(g), stating that “Defendants provide no reason establishing that a random sampling of the null set cannot be done when using keyword searching. Indeed, sampling the null set when using keyword searching provides for validation to defend the search and production process, and was commonly used before the movement towards TAR.”
Judge Johnston also ruled that sampling the null set is proportionate under Rule 26(b)(1), stating: “The Court’s experience and understanding is that a random sample of the null set will not be unreasonably expensive or burdensome. Moreover and critically, Defendants have failed to provide any evidence to support their contention…Indeed, the Court’s experience and understanding is that the random sample will not be voluminous in the context of a case of this magnitude.” As a result, Judge Johnston ordered a random sample of the null set, determining that “Plaintiffs’ proposed 95% confidence level with +/-margin of 2% is acceptable.”
Why This Case Is Significant
While TAR wasn’t used in this case, the potential use of it was discussed and the Court drew a comparison of the use of random sampling of the null set as appropriate for both approaches.
Download City of Rockford v. Mallinckrodt ARD Inc. (N.D. Ill. Aug. 7, 2018).
Instructions/Protocols for TAR Use
In re Actos Products Liability Litigation
Summary of the Case Ruling
In the case of In re Actos (Pioglitazone) Products Liability Litigation (W.D. La. July 27, 2012), in an order entitled “Procedures and Protocols Governing the Production of Electronically Stored Information (‘ESI’)”, U.S. District Judge Rebecca Doherty of the Western District of Louisiana ruled how the parties would treat data sources, custodians, costs, and format of production, among others. The order also contained a “Search Methodology Proof of Concept,” which governed the parties’ usage of TAR during the search and review of ESI.
Highlights of the Court’s Decision
The order stated that the parties “agree to meet and confer regarding the use of advanced analytics”, but also agreed that “Defendants retain the right to review documents after predictive coding but prior to production for relevance, confidentiality and privilege”. They also agreed to meet and confer to select four key custodians whose email would be used to create an initial sample set, after which three experts would train the TAR system to score every document based on relevance. To address concerns about the process, the court provided that both parties would collaborate to train the system, and after the TAR process is completed, the documents would be randomly sampled for quality control. It was also determined that the parties would “meet and confer regarding which relevance score will provide a cutoff for documents to be manually reviewed by defense counsel for production.”
The “Search Methodology Proof of Concept” section comprised ten pages of the 26-page order, describing a comprehensive approach to the TAR process.
Why This Case Is Significant
It’s the first case ruling with a detailed set of instructions and protocol for using TAR (as part of an overall ESI protocol) – only a few months after TAR was approved for the first time in Da Silva Moore!
Download In re Actos (Pioglitazone) Products Liability Litigation (W.D. La. July 27, 2012).
Rio Tinto
Summary of the Case Ruling
In Rio Tinto Plc v. Vale S.A. (S.D.N.Y. Mar. 2, 2015), New York Magistrate Judge Andrew J. Peck approved the proposed protocol for TAR presented by the parties but made it clear that “the Court’s approval ‘does not mean. . . that the exact ESI protocol approved here will be appropriate in all [or any] future cases that utilize [TAR].’”
Highlights of the Court’s Decision
After acknowledging that it had been three years since his decision in Da Silva Moore approved the use of TAR for the first time, Judge Peck went on to state: “In the three years since Da Silva Moore, the case law has developed to the point that it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.”
Judge Peck also referenced an earlier ruling in the Dynamo Holdings case from 2014, calling it “instructive” in its approval of TAR, noting that the tax court ruled that “courts leave it to the parties to decide how best to respond to discovery requests.”
According to Judge Peck, the TAR issue still to be addressed overall “is how transparent and cooperative the parties need to be with respect to the seed or training set(s)”, commenting that “where the parties do not agree to transparency, the decisions are split and the debate in the discovery literature is robust.”
Judge Peck also stated: “One point must be stressed — it is inappropriate to hold TAR to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.”
While approving the parties’ TAR protocol, Judge Peck also indicated that he wrote this opinion, “rather than merely signing the parties’ stipulated TAR protocol, because of the interest within the ediscovery community about TAR cases and protocols.”
Download Rio Tinto Plc v. Vale S.A. (S.D.N.Y. Mar. 2, 2015).
Why This Case Is Significant
This case illustrated just how far Court approval had come for TAR in just three years to the point that it was “black letter law” that it would be permitted. However, it also illustrated how the standards for TAR should be reasonable.
In re Broiler Chicken Antitrust Litigation
Summary of the Case Ruling
In In re Broiler Chicken Antitrust Litigation (N.D. Ill. Jan. 3, 2018), Illinois Magistrate Judge Jeffrey Gilbert appointed a special master (noted TAR expert Maura Grossman) to help the parties resolve ediscovery disputes. Judge Gilbert and Special Master Grossman issued a very detailed procedure (Order Regarding Search Methodology for Electronically Stored Information) for how the parties were to conduct TAR, including search, validation and document sourcing approaches, split into three primary sections: 1) how the parties will act, 2) what search technologies will be used, and 3) an outline of a document review validation protocol.
Highlights of the Court’s Decision
The six-page order set forth expectations regarding:
Transparency and the use of culling technologies prior to search, including de-duplication, email threading, email domains, targeted collections, exception reporting and other culling;
Search methods, divided into “TAR/CAL” (Technology Assisted Review/Continuous Active Learning) and Keyword Search Processes;
Document review validation protocol involving specifications for QC sampling, regardless whether TAR or “exhaustive manual review” was used.
The Order also included an appendix, detailing the recall estimation method for both a review process involving TAR and a review process involving manual review.
Why This Case Is Significant
The Broiler Chicken protocol is one of the most comprehensive protocols covering search and TAR approved by the courts and it is often referenced as a model protocol for consideration in cases involving TAR.
Download In re Broiler Chicken Antitrust Litigation (N.D. Ill. Jan. 3, 2018).
Decisions on Requiring the Use of TAR
Hyles v. New York City
Summary of the Case Ruling
In Hyles v. New York City (S.D.N.Y. Aug. 1, 2016), New York Magistrate Judge Andrew J. Peck, indicating that the key issue before the court in the discovery dispute between parties was whether (at the plaintiff’s request) the defendants can be forced to use TAR, refused to force the defendant to do so, stating “The short answer is a decisive ‘NO.’”
Highlights of the Court’s Decision
Plaintiff’s counsel consulted an ediscovery vendor and proposed that the defendants should use TAR. The defendants declined, because of cost and also concerns that the parties would not be able to collaborate to develop the seed set for a TAR process.
Judge Peck noted that “Hyles absolutely is correct that in general, TAR is cheaper, more efficient and superior to keyword searching” and referenced his “seminal” Da Silva Moore decision and also his 2015 Rio Tinto decision. He also noted that “Hyles’ counsel is correct that parties should cooperate in discovery”, but stated that “[c]ooperation principles, however, do not give the requesting party, or the Court, the power to force cooperation or to force the responding party to use TAR.”
Judge Peck stated: “It certainly is fair to say that I am a judicial advocate for the use of TAR in appropriate cases”, but he also said that he is also “a firm believer in the Sedona Principles, particularly Principle 6, which clearly provides that: Responding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.”
Judge Peck also noted:
Under Sedona Principle 6, the City as the responding party is best situated to decide how to search for and produce ESI responsive to Hyles’ document requests. Hyles’ counsel candidly admitted at the conference that they have no authority to support their request to force the City to use TAR. The City can use the search method of its choice. If Hyles later demonstrates deficiencies in the City’s production, the City may have to re-do its search. But that is not a basis for Court intervention at this stage of the case.
He also cited an earlier ruling in the Dynamo Holdings case from 2014, which stated: “[T]he Court is not normally in the business of dictating to parties the process that they should use when responding to discovery.”
Why This Case Is Significant
This case showed that, while the use of TAR provides benefits in discovery, even the judge who originally approved the use of TAR and was “a judicial advocate” for its use refused to order the defendants to use it, citing Sedona Principle 6.
In re Viagra Products Liability Litigation
Summary of the Case Ruling
In In re Viagra Products Liability Litigation (N.D. Cal., Oct. 14, 2016), California Magistrate Judge Sallie Kim, noting that other courts had declined to force a party to use predictive coding, denied the plaintiff’s motion to force the defendant to use TAR instead of its preferred approach using search terms.
Highlights of the Court’s Decision
In this multi-district litigation (MDL) against drug company Pfizer regarding its drug Viagra and alleged correlations to incidences of melanoma, the plaintiffs urged the Court to order the defendant to use TAR with the plaintiffs’ input to identify the locations of relevant information and the responsive ESI from those locations.
The defendant proposed instead to use search terms to identify potentially relevant documents – describing its preferred methodology as an iterative process – and proposed to exchange lists of proposed search terms and the defendant would agree to run any search terms that appeared on both parties’ lists. At the hearing on the matter, the plaintiffs conceded that no court has ordered a party to engage in TAR over the objection of the party.
Judge Kim noted that “[t]he few courts that have considered this issue have all declined to compel predictive coding”, citing Hyles as an example, stating: “As the court reasoned in Hyles, the responding party is the one best situated to decide how to search for and produce ESI responsive to discovery requests.” In denying the plaintiffs’ motion, Judge Kim ruled: “The Court finds Hyles well-reasoned. Even if predictive coding were a more efficient and better method, which Pfizer disputes, it is not clear on what basis the Court could compel Pfizer to use a particular form of ESI, especially in the absence of any evidence that Pfizer’s preferred method would produce, or has produced, insufficient discovery responses.”
Why This Case Is Significant
This case reiterated the trend to not force parties to conduct TAR, but it’s the first known MDL litigation where the issue was addressed.
Download In re Viagra Products Liability Litigation (N.D. Cal., Oct. 14, 2016).
In re Mercedes-Benz Emissions Litigation
Summary of the Case Ruling
In In re Mercedes-Benz Emissions Litigation (D.N.J. Jan. 9, 2020), Special Master Dennis Cavanaugh (U.S.D.J., Ret.) issued an order and opinion stating that he would not compel defendants to use TAR, and instead adopted the search term protocol negotiated by the parties, with three areas of dispute resolved by his ruling.
Highlights of the Court’s Decision
The plaintiffs proposed that the defendants use TAR (or its proposed Search Term Protocol if that request was denied), asserting that TAR yields significantly better results than either traditional human “eyes on” review of the full data set or the use of search terms. The defendants argued that there is no authority for imposing TAR on an objecting party and also argued this case presented several unique issues that would make developing an appropriate and effective seed set challenging, including language and translation issues, unique acronyms and identifiers, redacted documents, and technical documents.
Special Master Cavanaugh cited both Rio Tinto and Hyles in his ruling and stated:
While the Special Master believes TAR would likely be a more cost effective and efficient methodology for identifying responsive documents, Defendants may evaluate and decide for themselves the appropriate technology for producing their ESI. Therefore, the Special Master will not order Defendants to utilize TAR at this time. However, Defendants are cautioned that the Special Master will not look favorably on any future arguments related to burden of discovery requests, specifically cost and proportionality, when Defendants have chosen to utilize the custodian-and-search term approach despite wide acceptance that TAR is cheaper, more efficient and superior to keyword searching. Additionally, the denial of Plaintiffs’ request to compel Defendants to utilize TAR is without prejudice to revisiting this issue if Plaintiffs contend that Defendants’ actual production is deficient.
While encouraging the parties to work together to develop a reasonable procedure for the validation of the plaintiffs’ search terms, he also ruled: “As no articulable alternative process has been proposed by Plaintiffs, the Special Master will adopt Defendants’ protocol to the extent that it will require the parties, at Defendants’ request, to meet and confer concerning the application of validation procedures described in paragraph 12(a) to Plaintiffs, if the parties are unable to agree to a procedure.”
Why This Case Is Significant
While it’s another refusal to require a party to use TAR, this time it was by a Special Master in the case. The other notable consideration is that the Special Master warned the defendants that their decision not to use TAR would make it more difficult for any arguments regarding burden to be considered favorably.
Search Term Culling Before TAR
In re Biomet
Summary of the Case Ruling
In In re Biomet M2A Magnum Hip Implant Prods. Liability Litigation (N.D. Ind. Apr. 18, 2013), Indiana District Judge Robert L. Miller, Jr. denied the request by the Plaintiffs’ Steering Committee to force the defendant go back to its original set of 19.5 million documents and repeat the predictive coding without performing keyword searching, after the defendant had already performed keyword searching prior to performing predictive coding.
Highlights of the Court’s Decision
The defendant used keyword culling to reduce the universe of documents and attachments from 19.5 million documents to 2.5 million unique documents and attachments, then performed predictive coding to identify responsive documents to be produced from the set of 2.5 million documents. The defendant had already spent “about $1.07 million” and expected to spend a total of “between $2 million and $3.25 million” on ediscovery. Despite that, the plaintiffs’ Steering Committee objected, claiming that the defendant’s use of keyword searching “has tainted the process” and referenced an article which estimated a response rate of 60% for linear review and only 20% for keyword searching.
Judge Miller stated: “What Biomet has done complies fully with the requirements of Federal Rules of Civil Procedure 26(b) and 34(b)(2).” He also noted that “the Steering Committee’s request that Biomet go back to Square One…and institute predictive coding at that earlier stage sits uneasily with the proportionality standard in Rule 26(b)(2)(C).”
Judge Miller also stated: “Even in light of the needs of the hundreds of plaintiffs in this case, the very large amount in controversy, the parties’ resources, the importance of the issues at stake, and the importance of this discovery in resolving the issues, I can’t find that the likely benefits of the discovery proposed by the Steering Committee equals or outweighs its additional burden on, and additional expense to, Biomet.” And he rejected the Steering Committee’s position that the defendant can’t rely on proportionality arguments because they proceeded with document production while the centralization decision was pending.
Why This Case Is Significant
It’s the first known case where a court addressed the question of whether keyword searching can be performed before TAR and, even in a huge case, the Court didn’t find the “do over” request to be proportional.
Download In re Biomet M2A Magnum Hip Implant Prods. Liability Litigation (N.D. Ind. Apr. 18, 2013).
Livingston v. City of Chicago
Summary of the Case Ruling
In the case of Livingston v. City of Chicago (N.D. Ill. Sept. 3, 2020), Illinois Magistrate Judge Young B. Kim denied the plaintiffs’ motion to force the defendant to either use agreed-upon search terms to identify responsive documents and then perform a manual review for privilege or use TAR on the entire ESI collection with an agreed-upon coding system for responsiveness instead of the defendant’s proposed TAR protocol to use TAR to identify responsive documents from the documents retrieved by the search terms.
Highlights of the Court’s Decision
The Court ordered the defendant to apply an initial keyword search using the plaintiffs’ search terms but rejected the plaintiffs’ request that once the initial universe of emails had been identified through keyword searches, the defendant should produce the same without any further review. The defendant subsequently reported that the emails had been collected and searched, resulting in 192,000 unique emails and informed the court that it intended to use TAR to identify relevant responsive documents to be produced. The plaintiffs expressed concern that TAR would exclude responsive documents from the review process, that using TAR for responsiveness review was inconsistent with the Court’s order and requested compliance with the order or for their TAR protocol to be used instead.
Judge Kim stated:
The court agrees with the City that the November 2019 order did not set forth the review methodology that the City must use to identify responsive ESI…While the court anticipated that the parties would need to perform multiple keyword searches in order to narrow the universe of emails, it never directed them to do so. Nor did it suggest that after the searches are performed the City would have to produce the entire batch of documents subject only to a privilege review. In fact, the court specifically rejected Plaintiffs’ proposal that the City produce all of the documents that hit upon their initial search terms without further review. While the City may dump all 1.3 million pages of documents on Plaintiffs with an entry of a Rule 502(d) order, it also has the right to perform a review to produce only those documents that are responsive and relevant. In sum, the City’s responsiveness review is outside the scope of the November 2019 order.
Judge Kim also stated:
Plaintiffs’ insistence that the City must collaborate with them to establish a review protocol and validation process has no foothold in the federal rules governing discovery. Moreover, using TAR on the entire ESI collection—when, as Plaintiffs aptly point out, the parties spent nearly a year litigating the protocol for collecting and searching the City’s ESI—would be wasteful and unduly burdensome, and would further delay the resolution of this almost four-year-old case. For these reasons, the court declines to adopt Plaintiffs’ alternate TAR protocol.
Why This Case Is Significant
This case illustrates how, after more than eight years of court approval of TAR, the disputes are no longer about whether TAR can be used, but how it should be conducted. Parties still have the right to conduct a responsiveness review on keyword search results, using TAR as the approach for review.
Download Livingston v. City of Chicago (N.D. Ill. Sept. 3, 2020).
Switching Approaches to TAR
Progressive Casualty Insurance Co. v. Delaney
Summary of the Case Ruling
In Progressive Casualty Insurance Co. v. Delaney (D. Nev. May 20, 2014), Nevada Magistrate Judge Peggy A. Leen determined that the plaintiff’s unannounced shift from the agreed upon discovery methodology, to a predictive coding methodology for privilege review was not cooperative. Therefore, the plaintiff was ordered to produce documents that met agreed-upon search terms without conducting a privilege review first.
Highlights of the Court’s Decision
The parties agreed to search terms that would reduce the number of potentially responsive documents from approximately 1.8 million documents to approximately 565,000 documents, which the plaintiff would manually review for privileged documents before producing discovery to the defendant. However, the plaintiff elected to apply predictive coding to those documents without consulting with either the court or the requesting party regarding their intentions to change review methodology when they determined that manual review would be too expensive and time-consuming.
While recognizing potential support for the general methodology of predictive coding in ediscovery and stating that a “transparent mutually agreed upon” protocol for it would likely have been approved, Judge Leen took issue that the plaintiff had refused to “engage in the type of cooperation and transparency that its own ediscovery consultant has so comprehensively and persuasively explained is needed for a predictive coding protocol to be accepted by the court or opposing counsel” and instead had “elected and then abandoned the second option—to manually review and produce responsive ESI documents. It abandoned the option it selected unilaterally, without the [defendant’s] acquiescence or the court’s approval and modification of the parties’ stipulated ESI protocol.”
As a result, Judge Leen enforced the second option from the agreed-upon ESI protocol, requiring the plaintiff to produce all 565,000 documents that matched the stipulated search terms without review, with a clawback option in place for privileged documents as well as permission to apply privilege filters to the documents at issue, and withhold those documents that returned as “most likely privileged.”
Why This Case Is Significant
An agreed-upon protocol makes all the difference. Even though courts today don’t typically require notification before a party decides to use TAR to conduct review, all bets are off when there is an agreed-upon protocol. Had the plaintiff cooperated during their process and informed the court and the requesting party regarding their intentions to switch beforehand, the ruling might have been different.
Bridgestone Americas Inc. v. Int’l Bus. Mach. Corp.
Summary of the Case Ruling
Ruling in Bridgestone Americas Inc. v. Int’l Bus. Mach. Corp. (M.D. Tenn. July 22, 2014), while recognizing that he was “allowing Plaintiff to switch horses in midstream”, Tennessee Magistrate Judge Joe B. Brown ruled that the plaintiff could use predictive coding to search documents for discovery, even though keyword search had already been performed.
Highlights of the Court’s Decision
The plaintiff requested that the court allow the use of predictive coding in reviewing over two million documents. The defendant objected on the grounds that the request was an unwarranted change to the original case management order that did not include predictive coding, and that it would be unfair to use predictive coding after an initial screening had been done with keyword search terms.
After conducting a lengthy telephone conference with the parties, Judge Brown stated: “Predictive coding is a rapidly developing field in which the Sedona Conference has devoted a good deal of time and effort to, and has provided various best practices suggestions”. Judge Brown also stated: “the use of predictive coding is a judgment call, hopefully keeping in mind the exhortation of Rule 26 that discovery be tailored by the court to be as efficient and cost-effective as possible.”
Stating that “we are talking about millions of documents to be reviewed with costs likewise in the millions”, Judge Brown permitted the plaintiff “to use predictive coding on the documents that they have presently identified, based on the search terms Defendant provided.” Judge Brown acknowledged that he was “allowing Plaintiff to switch horses in midstream” but he made it clear that “openness and transparency in what Plaintiff is doing will be of critical importance.”
Why This Case Is Significant
A mere two months after the Court’s ruling in the Progressive Casualty Insurance case, this Court chose to allow the plaintiff to “switch horses in midstream” and move to predictive coding. This time, the parties didn’t have an agreed upon protocol here, which fits with the trend that (absent such a protocol), courts were beginning to recognize the ability for responding parties to evaluate and choose their own methodologies for responding to discovery requests.
Download Bridgestone Americas Inc. v. Int’l Bus. Mach. Corp. (M.D. Tenn. July 22, 2014).
In re Valsartan
Summary of the Case Ruling
In In re Valsartan, Losartan, and Irbesartan Prod. Liability Litigation (D.N.J. Dec. 2, 2020), New Jersey Magistrate Judge Joel Schneider rejecting the defendant’s “unilaterally developed and administered TAR” approach because it violated the Court Ordered Protocol and ordered the defendant “to conduct its review of non-responsive documents using the TAR protocol the parties almost finalized but for two provisions” and permitted the plaintiffs “to review 5,000 alleged non-responsive documents of their choosing”.
Highlights of the Court’s Decision
A year after the Protocol was entered, and only two weeks before the first rolling production was due, the defendants informed the plaintiffs for the first time that they would be utilizing a continuous multimodal learning (CMML) platform to assist with review and production. The plaintiffs objected, stating that if they had known the defendants were using TAR, they would not have agreed to limit the review of the custodians’ documents to only those that contained the designated search terms, or “wasted countless hours” negotiating over search terms or agree to the defendants’ plan to layer TAR review with search terms.
Eventually, all TAR related disputes were resolved except for two issues: the defendants wouldn’t agree to the entry of a Court Order to memorialize the parties’ agreement, and they wouldn’t agree to permit the plaintiffs to review 5,000 alleged non-responsive documents to evaluate and validate the defendant’s CMML platform.
Judge Schneider stated that “Teva did not comply with the requirements in the Protocol since it is seeking the Court’s blessing to use a CMML platform it unilaterally adopted…the Protocol requires the parties to meet and confer in good faith to attempt to reach agreement. This does not occur if one side or the other unilaterally adopts a TAR protocol ‘late in the game’ and argues it should be approved by the Court. This also does not occur when a party’s TAR protocol is presented to its adversary as a fait accompli. The time to meet and confer in good faith is before a TAR protocol or CMML platform is adopted and used, not after.”
Therefore, Judge Schneider ruled: “The Court will direct that the TAR protocol to be implemented include the two provisions originally objected to by Teva, neither of which the Court finds controversial or bothersome.”
Why This Case Is Significant
More than six years after the Court’s ruling in the Progressive Casualty Insurance case, bypassing an agreed-upon (and Court ordered) protocol is still ill-advised, even with so much case law approving the use of TAR!
Download In re Valsartan, Losartan, and Irbesartan Prod. Liability Litigation (D.N.J. Dec. 2, 2020).
TAR Cost Disputes and Cost Shifting
Lawson v. Spirit Aerosystems
Summary of the Case Ruling
In Lawson v. Spirit Aerosystems, Inc. (D. Kan. June 18, 2020), Kansas Magistrate Judge Angel D. Mitchell granted the defendant’s Motion to Shift Costs of Technology Assisted Review of ESI to the plaintiff, ruling “the ESI/TAR process became disproportionate to the needs of the case” after having previously warned the plaintiff that inability to focus ESI requests would result in the court shifting costs.
Highlights of the Court’s Decision
After several iterations, the defendant conducted searches of the custodians’ ESI using revised terms provided by the plaintiff, which returned approximately 322,000 documents. Even though the defendant’s sampling exercise suggested that only 5% of the 322,000 documents would be responsive to discovery requests (with those still “largely irrelevant to the dispute”), the plaintiff still pursued conducting the TAR protocol. Only 3.3% of the documents in the TAR set of 322,000 documents were responsive and the defendant estimated its TAR expenses to be approximately $400,000 in vendor costs and $200,000 in law firm fees.
Judge Mitchell stated:
At Lawson’s request, the parties spent months engaged in an ESI discovery process regarding the issue of business overlap between Spirit and non-party Arconic, Inc…using traditional ESI methods involving custodians and search terms. When that process repeatedly yielded low responsiveness rates, the court allowed the parties to proceed— again, at Lawson’s request—with a technology-assisted review (‘TAR’) of approximately 322,000 documents, with the caveat that the court would decide whether to allocate the TAR expenses to Lawson.
Judge Mitchell also stated: “Lawson was unwilling to abandon the largely non-responsive ESI dataset and instead sought continued review via TAR that unnecessarily perpetuated and exacerbated ESI/TAR expenses. The TAR process ultimately yielded a responsiveness rate of only 3.3%. Even the documents that were technically responsive were of marginal (if any) relevance above and beyond what Spirit produced outside of the ESI/TAR process. Thus, the ESI/TAR process became disproportionate to the needs of the case.” Therefore, Judge Mitchell granted the defendant’s motion and ordered the parties to “meet and confer to try to reach agreement on the amount of the TAR expenses”, with further briefing ordered if they could not agree.
Why This Case Is Significant
Sampling is a great way to determine whether to proceed with review, regardless of whether that is TAR or manual review. In this case, the plaintiff’s insistence in proceeding with TAR for a data set with low prevalence (i.e., low percentage of documents likely to be responsive), even after the Court’s warnings that costs could be shifted for continuing to pursue a data set that contained largely non-responsive documents. Even though TAR provides efficiencies during review, a party can’t simply pursue review of documents unlikely to contain many responsive documents without expecting to bear at least some of the costs.
Download Lawson v. Spirit Aerosystems, Inc. (D. Kan. June 18, 2020).
Addressing Issues With TAR
Winfield v. City of New York
Summary of the Case Ruling
In Winfield v. City of New York (S.D.N.Y. Nov. 27, 2017), New York Magistrate Judge Katharine H. Parker, after conducting an in-camera review of the defendant’s TAR process and a sample set of documents, granted in part and denied in part the plaintiffs’ motion, ordering the defendant to provide copies of specific documents where the parties disagreed on their responsiveness and a random sample of 300 additional documents deemed non-responsive by the defendant. But Judge Parker denied the plaintiff’s request for information about the defendant’s TAR process, finding no evidence of gross negligence or unreasonableness in their process.
Highlights of the Court’s Decision
After the plaintiffs complained about the pace of discovery, the Court directed the defendant to complete a linear review as to certain custodians and begin using TAR software for the rest of the collection. The plaintiffs contended that the defendant had over-designated documents as privileged and non-responsive, using an “impermissibly narrow view of responsiveness” during its review process.
After the Court required the defendant to provide a privilege log for a sample set of 80 documents that it designated as privileged in its initial review and the defendant maintained its original privilege assertions over only 20 of those documents, the plaintiffs filed a motion requesting random samples of several categories of documents and also sought information about the TAR ranking system used by the defendant and all materials submitted by the defendant for the Court’s in camera review relating to predictive coding.
Judge Parker ordered the defendant to “expand its search for documents responsive to Plaintiffs’ document requests as it construed this Court’s prior ruling too narrowly”. But Judge Parker also determined “that the City appropriately trained and utilized its TAR system”, noting that the defendant’s seed set “included over 7,200 documents that were reviewed by the City’s document review team and marked as responsive or non-responsive in order to train the system” and that “the City provided detailed training to its document review team as to the issues in the case.” In addition to ordering the defendant to provide copies of specific documents where the parties disagreed on their responsiveness and a random sample of 300 additional documents deemed non-responsive by the defendant, Judge Parker ordered the parties to meet and confer on any disputes “with the understanding that reasonableness and proportionality, not perfection and scorched-earth, must be their guiding principles.”
Why This Case Is Significant
This case represents a detailed look at how a party is conducting TAR and represents how disputes about TAR have shifted from whether it is court-approved to how the process is being conducted. It’s important to ensure a defensible process, as requesting parties will be quick to identify potential issues – just as they would be with any approach to discovery.
Download Winfield v. City of New York (S.D.N.Y. Nov. 27, 2017).
In Re Domestic Airline Travel Antitrust Litigation
Summary of the Case Ruling
In In Re Domestic Airline Travel Antitrust Litigation, MDL Docket No. 2656 (D.D.C. Sept. 13, 2018), District of Columbia District Judge Colleen Kollar-Kotelly granted the Plaintiffs’ Motion for an Extension of Fact Discovery Deadlines (over the defendants’ objections) for six months, finding that defendant “United’s production of core documents that varied greatly from the control set in terms of the applicable standards for recall and precision and included a much larger number of non-responsive documents that was anticipated”.
Highlights of the Court’s Decision
Defendant United produced more than 3.5 million documents to the Plaintiffs, but “due to United’s technology assisted review process (‘TAR’), only approximately 17%, or 600,000, of the documents produced are responsive to Plaintiffs’ requests,” and the plaintiffs (despite having staffed their discovery review with 70 attorneys) required additional time to sort through them, so they requested an extension of six months.
Judge Kollar-Kotelly stated: “United’s core production of 3.5 million documents — containing numerous nonresponsive documents — was unanticipated by Plaintiffs, considering the circumstances leading up to that production” and also that “Plaintiffs devoted considerable resources to the review of the United documents prior to filing this motion seeking an extension”. Finding also that “Plaintiffs’ claim of prejudice in not having the deadlines extended far outweighs any inconvenience that Defendants will experience if the deadlines are extended”, Judge Kollar-Kotelly found “that Plaintiffs have demonstrated good cause to warrant an extension of deadlines in this case based upon Plaintiffs’ demonstration of diligence and a showing of nominal prejudice to the Defendants, if an extension is granted, while Plaintiffs will be greatly prejudiced if the extension is not granted.” Therefore, she granted the motion to request the extension.
Why This Case Is Significant
This case illustrates that the simple use of TAR doesn’t guarantee a successful outcome. Mistakes can happen in any discovery process – with or without TAR – if you don’t test and verify the process and the results. In this case, sampling of the resulting document set could have identified the low prevalence of the document set to be produced, enabling the responding party to make adjustments to identify and eliminate millions of non-responsive documents.
Download In Re Domestic Airline Travel Antitrust Litigation, MDL Docket No. 2656 (D.D.C. Sept. 13, 2018).
Dive Deeper Into TAR and Predictive Coding
With over ten years since TAR was approved in the courts, it’s not surprising the way the case law surrounding it has evolved – from mere approval for the use of TAR to disputes regarding the TAR process itself to issues with the output from TAR. Now, with the introduction of generative AI in the document review process, legal technology and case law will only continue to evolve.
The Sedona Conference TAR Case Law Primer, issued in 2017, was created to analyze decisions from courts that were required to opine on the efficacy of TAR in a variety of circumstances, and explored the evolution in the courts’ thinking from 2012 through the end of 2016. Several of the earliest decisions in our list were discussed in the Primer, but half of the cases in our list – nine cases out of eighteen total cases – have occurred since the Primer was published.
As case law keeps progressing regarding the use and best practices associated with TAR, we understand that there will be newer cases that provide significant guidance as to how TAR should be conducted and what courts will expect.
To learn more about the technological underpinnings of TAR and predictive coding, be sure to check out Everlaw's Beginner's Guide to Predictive Coding. To see predictive coding in action, sign up for an Everlaw demo today.
