Psychometric scaling of TID2013 dataset

Research areas:

• Methodology - Machine learning

Year:

2018

Type of Publication:

In Proceedings

Authors:

• Mikhailiuk, Aliaksei
• Pérez-Ortiz, María
• Mantiuk, Rafal K.

Book title:

International Conference on Quality of Multimedia Experience

ISSN:

2472-7814

BibTex:

@conference{Aliaksei Mikhailiuk2018, author = "Aliaksei Mikhailiuk and Mar{\'i}a P{\'e}rez-Ortiz and Rafal K. Mantiuk", abstract = "TID2013 is a subjective image quality assessment dataset with a wide range of distortion types and over 3000 images. The dataset has proven to be a challenging test for objective quality metrics. The dataset mean opinion scores were obtained by collecting pairwise comparison judgments using the Swiss tournament system, and averaging votes of observers. However, this approach differs from the usual analysis of multiple pairwise comparisons, which involves psychometric scaling of the comparison data using either Thurstone or Bradley-Terry models. In this paper we investigate how quality scores change when they are computed using such psychometric scaling instead of averaging vote counts. In order to properly scale TID2013 quality scores, we conduct four additional experiments of two different types, which we found necessary to produce a common quality scale: comparisons with reference images, and cross-content comparisons. We demonstrate on a fifth validation experiment that the two additional types of comparisons are necessary and in conjunction with psychometric scaling improve the consistency of quality scores, especially across images depicting different contents.", booktitle = "International Conference on Quality of Multimedia Experience", doi = "10.1109/QoMEX.2018.8463376", issn = "2472-7814", title = "{P}sychometric scaling of {TID}2013 dataset", url = "doi.org/10.1109/QoMEX.2018.8463376", year = "2018", }

Abstract:

TID2013 is a subjective image quality assessment dataset with a wide range of distortion types and over 3000 images. The dataset has proven to be a challenging test for objective quality metrics. The dataset mean opinion scores were obtained by collecting pairwise comparison judgments using the Swiss tournament system, and averaging votes of observers. However, this approach differs from the usual analysis of multiple pairwise comparisons, which involves psychometric scaling of the comparison data using either Thurstone or Bradley-Terry models. In this paper we investigate how quality scores change when they are computed using such psychometric scaling instead of averaging vote counts. In order to properly scale TID2013 quality scores, we conduct four additional experiments of two different types, which we found necessary to produce a common quality scale: comparisons with reference images, and cross-content comparisons. We demonstrate on a fifth validation experiment that the two additional types of comparisons are necessary and in conjunction with psychometric scaling improve the consistency of quality scores, especially across images depicting different contents.