CC BY 4.0Nowak, S.S.NowakBiesner, DavidDavidBiesnerLayer, Y.C.Y.C.LayerTheis, M.M.TheisSchneider, HelenHelenSchneiderBlock, W.W.BlockWulff, BenjaminBenjaminWulffAttenberger, U.I.U.I.AttenbergerSifa, RafetRafetSifaSprinkart, A.M.A.M.Sprinkart2023-06-062023-06-062023-03-11https://publica.fraunhofer.de/handle/publica/442580https://doi.org/10.24406/publica-143410.1007/s00330-023-09526-y10.24406/publica-143436905469To provide insights for on-site development of transformer-based structuring of free-text report databases by investigating different labeling and pre-training strategies. A total of 93,368 German chest X-ray reports from 20,912 intensive care unit (ICU) patients were included. Two labeling strategies were investigated to tag six findings of the attending radiologist. First, a system based on human-defined rules was applied for annotation of all reports (termed "silver labels"). Second, 18,000 reports were manually annotated in 197 h (termed "gold labels") of which 10% were used for testing. An on-site pre-trained model (Tmlm) using masked-language modeling (MLM) was compared to a public, medically pre-trained model (Tmed). Both models were fine-tuned on silver labels only, gold labels only, and first with silver and then gold labels (hybrid training) for text classification, using varying numbers (N: 500, 1000, 2000, 3500, 7000, 14,580) of gold labels. Macro-averaged F1-scores (MAF1) in percent were calculated with 95% confidence intervals (CI). Custom pre-training of transformers and fine-tuning on manual annotations promises to be an efficient strategy to unlock report databases for data-driven medicine.enDeep learningIntensive care unitsNatural language processingRadiologyThoraxjournal article