Table 4 Comparison models (average ± standard deviation); medical-domain models are italized;general models are underlined; best results in bold (P: ‘precision’; R: ‘recall’)
From: Hybrid natural language processing tool for semantic annotation of medical texts in Spanish
| Â | UMLS entities | Negation / speculation | ||||
|---|---|---|---|---|---|---|
| Â | P | R | F1 | P | R | F1 |
Bi-LSTM-CRF | 0.815 | 0.813 | 0.814 | 0.845 | 0.776 | 0.809 |
|  | (± 0.003) | (± 0.002) | (± 0.002) | (± 0.002) | (± 0.006) | (± 0.004) |
mBERT | 0.875 | 0.887 | 0.881 | 0.855 | 0.852 | 0.853 |
|  | (± 0.004) | (± 0.005) | (± 0.001) | (± 0.006) | (± 0.010) | (± 0.004) |
RoBERTa EHR | 0.878 | 0.894 | 0.886 | 0.855 | 0.864 | 0.859 |
|  | (± 0.003) | (± 0.003) | (± 0.002) | (± 0.005) | (± 0.009) | (± 0.006) |
EriBERTa | 0.881 | 0.896 | 0.889 | 0.861 | 0.871 | 0.866 |
|  | (± 0.005) | (± 0.002) | (± 0.003) | (± 0.008) | (± 0.005) | (± 0.006) |
CLIN-X-ES | 0.906 | 0.911 | 0.909 | 0.871 | 0.874 | 0.873 |
|  | (± 0.005) | (± 0.008) | (± 0.003) | (± 0.008) | (± 0.007) | (± 0.005) |
mDeBERTa | 0.904 | 0.916 | 0.910 | 0.861 | 0.871 | 0.866 |
|  | (± 0.001) | (± 0.003) | (± 0.002) | (± 0.009) | (± 0.009) | (± 0.006) |
| Â | Temporal entities | Medication information | ||||
| Â | P | R | F1 | P | R | F1 |
Rules | 0.824 | 0.841 | 0.833 | 0.628 | 0.786 | 0.698 |
Bi-LSTM-CRF | 0.899 | 0.859 | 0.879 | 0.895 | 0.740 | 0.810 |
|  | (± 0.007) | (± 0.005) | (± 0.006) | (± 0.002) | (± 0.015) | (± 0.009) |
mBERT | 0.874 | 0.862 | 0.868 | 0.856 | 0.823 | 0.839 |
|  | (± 0.007) | (± .006) | (± 0.004) | (± 0.007) | (± 0.011) | (± 0.004) |
RoBERTa EHR | 0.900 | 0.900 | 0.900 | 0.873 | 0.871 | 0.872 |
|  | (± 0.011) | (± 0.007) | (± 0.001) | (± 0.022) | (± 0.015) | (± 0.004) |
EriBERTa | 0.922 | 0.908 | 0.915 | 0.874 | 0.828 | 0.850 |
|  | (± 0.004) | (± 0.011) | (± 0.007) | (± 0.024) | (± 0.007) | (± 0.010) |
CLIN-X-ES | 0.899 | 0.895 | 0.897 | 0.883 | 0.863 | 0.872 |
|  | (± 0.017) | (± 0.005) | (± 0.010) | (± 0.021) | (± 0.015) | (± 0.008) |
mDeBERTa | 0.909 | 0.918 | 0.913 | 0.897 | 0.885 | 0.891 |
|  | (± 0.009) | (± 0.006) | (± 0.005) | (± 0.012) | (± 0.014) | (± 0.008) |
| Â | Miscellaneous entities | Experiencer/Temporality attributes | ||||
| Â | P | R | F1 | P | R | F1 |
Bi-LSTM-CRF | 0.721 | 0.536 | 0.613 | 0.891 | 0.816 | 0.852 |
|  | (± 0.006) | (± 0.005) | (± 0.004) | (± 0.007) | (± 0.004) | (± 0.001) |
mBERT | 0.674 | 0.623 | 0.647 | 0.868 | 0.831 | 0.849 |
|  | (± 0.027) | (± 0.015) | (± 0.006) | (± 0.023) | (± 0.014) | (± 0.005) |
RoBERTa EHR | 0.685 | 0.669 | 0.677 | 0.877 | 0.835 | 0.856 |
|  | (± 0.008) | (± 0.004) | (± 0.003) | (± 0.009) | (± 0.008) | (± 0.006) |
EriBERTa | 0.703 | 0.666 | 0.684 | 0.890 | 0.848 | 0.868 |
|  | (± 0.017) | (± 0.009) | (± 0.006) | (± 0.007) | (± 0.008) | (± 0.002) |
CLIN-X-ES | 0.715 | 0.672 | 0.692 | 0.894 | 0.863 | 0.878 |
|  | (± 0.014) | (± 0.016) | (± 0.007) | (±0.013) | (± 0.010) | (± 0.005) |
mDeBERTa | 0.702 | 0.670 | 0.686 | 0.898 | 0.882 | 0.890 |
|  | (± 0.011) | (± 0.007) | (± 0.004) | (± 0.009) | (± 0.008) | (± 0.005) |