Department of Medicine, Division of Cardiovascular Medicine
Artificial Intelligence and Robotics | Health Information Technology | Health Services Administration | Health Services Research | Information Literacy
BACKGROUND: The use of electronic health record (EHR) systems with patient engagement capabilities, including viewing, downloading, and transmitting health information, has recently grown tremendously. However, using these resources to engage patients in managing their own health remains challenging due to the complex and technical nature of the EHR narratives.
OBJECTIVE: Our objective was to develop a machine learning-based system to assess readability levels of complex documents such as EHR notes.
METHODS: We collected difficulty ratings of EHR notes and Wikipedia articles using crowdsourcing from 90 readers. We built a supervised model to assess readability based on relative orders of text difficulty using both surface text features and word embeddings. We evaluated system performance using the Kendall coefficient of concordance against human ratings.
RESULTS: Our system achieved significantly higher concordance (.734) with human annotators than did a baseline using the Flesch-Kincaid Grade Level, a widely adopted readability formula (.531). The improvement was also consistent across different disease topics. This method's concordance with an individual human user's ratings was also higher than the concordance between different human annotators (.658).
CONCLUSIONS: We explored methods to automatically assess the readability levels of clinical narratives. Our ranking-based system using simple textual features and easy-to-learn word embeddings outperformed a widely used readability formula. Our ranking-based method can predict relative difficulties of medical documents. It is not constrained to a predefined set of readability levels, a common design in many machine learning-based systems. Furthermore, the feature set does not rely on complex processing of the documents. One potential application of our readability ranking is personalization, allowing patients to better accommodate their own background knowledge.
comprehension, electronic health records, machine learning, readability
Rights and Permissions
Copyright: © Jiaping Zheng, Hong Yu. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 23.03.2018. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.
DOI of Published Version
JMIR Med Inform. 2018 Mar 23;6(1):e17. doi: 10.2196/medinform.8611. Link to article on publisher's site
JMIR medical informatics
Zheng, Jiaping and Yu, Hong, "Assessing the Readability of Medical Documents: A Ranking Approach" (2018). Open Access Articles. 3420.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.