We sought to characterize how well computerized "case flags" performed in evaluating critical diagnoses in anatomic pathology. All cases identified by a pathologist at sign-out and flagged as a critical diagnosis in anatomic pathology in 2 hospital laboratories during a 3-year period were reviewed. A subset of all critical diagnoses consisting of only treatable, immediately life-threatening (TILT) diagnoses was selected, and a text search for key words was used to evaluate performance during a 6-month period. During a 3-year period, there were 635 cases (0.5% of all cases) that were flagged as critical diagnoses. A key word search identified 269 TILT cases, which represented 1.8% of all cases during this time; 30 (11.2%) were critical diagnoses, of which 24 (80%) had documentation of a call to the clinician and only 2 (7%) were flagged as a critical diagnosis. Critical diagnoses in anatomic pathology remain poorly defined. Computerized case flags underestimate the number of critical diagnosis cases in a laboratory and cannot identify missed critical diagnoses. A more limited and clearly defined approach to quality assurance of critical diagnoses emphasizing TILT diagnoses and selected key word searches can be performed and requires reviewing only 1.8% of all cases.