| 1 |
Mining for the meanings of a murder : the impact of OCR quality on the use of digitized historical newspapers
|
|
|
|
Abstract:
Digital humanities research that requires the digitization of medium-scale, project-specific texts confronts a significant methodological and practical question: is labour-intensive cleaning of the Optical Character Recognition (OCR) output necessary to produce robust results through text mining analysis? This paper traces the steps taken in a collaborative research project that aimed to analyze newspaper coverage of a high-profile murder trial, which occurred in New York City in 1873. A corpus of approximately one-half million words was produced by converting original print sources and image files into digital texts, which produced a substantial rate of OCR-generated errors. We then corrected the scans and added document-level genre metadata. This allowed us to evaluate the impact of our quality upgrade procedures when we tested for possible differences in word usage across two key phases in the trial's coverage using log likelihood ratio [Dunning 1993]. The same tests were run on each dataset – the original OCR scans, a subset of OCR scans selected through the addition of genre metadata, and the metadata-enhanced scans corrected to 98% accuracy. Our results revealed that error correction is desirable but not essential. However, metadata to distinguish between different genres of trial coverage, obtained during the correction process, had a substantial impact. This was true both when investigating all words and when testing for a subset of “judgment words” we created to explore the murder’s emotive elements and its moral implications. Deeper analysis of this case, and others like it, will require more sophisticated text mining techniques to disambiguate word sense and context, which may be more sensitive to OCR-induced errors.
|
|
Keyword:
digital humanities; newspapers; optical character recognition; scanning systems; XXXXXX - Unknown
|
|
URL: https://hdl.handle.net/1959.7/uws:54807
http://www.digitalhumanities.org/dhq/vol/8/1/000168/000168.html
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Using Case-Based Reasoning in Natural Language Processing
|
|
|
|
In: DTIC AND NTIS (1993)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
Video Scan Rate Conversion Method and Apparatus for Achieving Same.
|
|
|
|
In: DTIC AND NTIS (1992)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
A Whole Word and Number Reading Machine Based on Two Dimensional Low Frequency Fourier Transforms
|
|
|
|
In: DTIC AND NTIS (1990)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
Chinese/Japanese Input Feasibility Study.
|
|
|
|
In: DTIC AND NTIS (1973)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Automatic Typeset Input Technique Evaluation.
|
|
|
|
In: DTIC AND NTIS (1971)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Character Recognition Techniques. Volume 2.
|
|
|
|
In: DTIC AND NTIS (1969)
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Reconstructing detailed dynamic face geometry from monocular video
|
|
|
|
In: http://gvv.mpi-inf.mpg.de/files/SIGGRAPH_ASIA_2013/monfacecap_high.pdf
|
|
BASE
|
|
Show details
|
|
|
|
|
