Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data

Journal Publication ResearchOnline@JCU
Sidorczuk, Katarzyna;Gagat, Przemyslaw;Pietluch, Filip;Kala, Jakub;Rafacz, Dominik;Bakala, Laura;Slowik, Jadwiga;Kolenda, Rafal;Rodiger, Stefan;Fingerhut, Legana C.H.W.;Cooke, Ira R.;Mackiewicz, Pawel;Burdukiewicz, Michal
Abstract

Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMP Benchmark is available at http://BioGenies.info/AMPBenchmark.

Journal

Briefings in Bioinformatics

Publication Name

N/A

Volume

23

ISBN/ISSN

1477-4054

Edition

N/A

Issue

5

Pages Count

12

Location

N/A

Publisher

Oxford University Press

Publisher Url

N/A

Publisher Location

N/A

Publish Date

N/A

Url

N/A

Date

N/A

EISSN

N/A

DOI

10.1093/bib/bbac343