Title: "Better than classical? The subtle art of benchmarking quantum machine learning models"

Abstract: Benchmarks are one of the main tools to judge ideas in quantum machine learning before noise-free hardware is available. However, the huge impact of the experimental design on the results, the small scale of simulations within reach today, as well as narratives influenced by the commercialisation of quantum technologies make it difficult to gain robust insights. To facilitate better decision-making we have developed an open-source package based on the PennyLane software framework and have used it to conduct a large-scale study that systematically tests 12 popular quantum machine learning models on 6 binary classification tasks used to create 160 individual datasets. We find that overall, out-of-the-box classical machine learning models beat the quantum classifiers. Moreover, removing entanglement from a quantum model often results in as good or better performance, suggesting that "quantumness'' may not be their crucial ingredient for the small learning tasks considered here. Our benchmarks also unlock investigations beyond simplistic leaderboard comparisons, and we identify five important questions for quantum model design that follow from our results.

Bio: Joseph Bowles is a quantum machine learning researcher at Xanadu. His work focuses on understanding the types of biases that can be encoded into quantum machine learning models, and developing models that can be efficiently trained at scale.