147 lines
27 KiB
Markdown
147 lines
27 KiB
Markdown
type:: [[REVIEWS]]
|
||
tags::
|
||
year:: 2024
|
||
venue:: [[TOSEM]]
|
||
full-title:: A Study of Fairness Concerns in AI-based Mobile App Reviews
|
||
date-start:: [[05-03-2024]] - 21:02
|
||
date-submitted::
|
||
deadline:: [[28-03-2024]]
|
||
external-links:: https://mc.manuscriptcentral.com/tosem?URL_MASK=d714e5cc6e364056bba6b4ff7171802b
|
||
status:: [[DONE]]
|
||
deadline-submission::
|
||
file:: zotero://select/library/items/BGIJALY7
|
||
parent::
|
||
todoist:: https://app.todoist.com/showTask?id=7737551998
|
||
|
||
- ### [[Highlights]]
|
||
- “unfair AI-based mobile apps” ([Nasab et al., p. 2](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=2&annotation=SSSLKD96)) #a28ae5
|
||
- “global AI market size, particularly in the form of Machine Learning (ML) and Deep Learning (DL), is predicted to exceed USD 1.6 trillion by 2030” ([Nasab et al., p. 2](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=2&annotation=THWMERI8)) #5fb236
|
||
- “fairness concerns in AI-based app reviews” ([Nasab et al., p. 2](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=2&annotation=HXUURJ47)) #a28ae5
|
||
- “fairness” ([Nasab et al., p. 2](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=2&annotation=BY6YCQWA)) #a28ae5
|
||
- “non-fairness reviews.” ([Nasab et al., p. 2](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=2&annotation=E8A4DZSG)) #a28ae5
|
||
- “fairness concerns in AI systems cannot only stem from technical components and data but can also be attributed to the process of building AI systems, humans, and governance (e.g., decisions made by developers or policies adopted by providers).” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=MXRYLAVK)) #a28ae5
|
||
- “With 6.64 billion smartphone users worldwide (83.32% of the current world population) [86], mobile software apps are the most prevailing type of software system used by a wide range of individuals and groups with different characteristics (e.g., age, education level, cultural background, race, and gender)” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=VCVI36V8)) #a28ae5
|
||
- “This increased risk is attributed to the complex nature of AI technologies, which can inadvertently encode and perpetuate biases present in their training data or opaque decision-making algorithms” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=48PBV9XG)) #a28ae5
|
||
- “no work has focused on providing an accurate and fine-grained view of different types of fairness concerns and their possible root causes in AI-based apps” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=YXC76SNX)) #e56eee
|
||
- “a growing concern exists about ensuring the responsible behavior of AI solutions” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=BBINKECH)) #a28ae5
|
||
- “possibility that they could produce biased results, have errors, or lack sufficient transparency” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=JX8YJKBI)) #a28ae5
|
||
- “responsible use and adoption of AI” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=RVT8CVIY)) #a28ae5
|
||
- “Fairness is a key element of responsible AI” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=HB4R7GHQ)) #a28ae5
|
||
- “no single definition of fairness in AI systems, and varies in different domain” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=CAZZS5BN)) #5fb236
|
||
- “fairness is defined as “the absence of any prejudice or favoritism towards an individual or group based on their inherent or acquired characteristics” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=XMXCBCYY)) #a28ae5
|
||
|
||
*#card*
|
||
- “socio-technical challenge” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=M8CAU6XN)) #a28ae5
|
||
- “TensorFlow Lite that enable executing AI tasks solely on smartph” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=KFYJILTG)) #5fb236
|
||
- “investigate fairness concerns in AI-based app reviews.” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=CGEYA5TT)) #e56eee
|
||
- “fairness review is a review that discusses a fairness concern (topic)” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=JXNWS6HP)) #5fb236
|
||
- “our best-performing classifier can detect fairness reviews with a precision of 94%” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=CBLXDRJ4)) #a28ae5
|
||
- “a finegrained view of different fairness concerns.” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=VSYWLUNX)) #ffd400
|
||
|
||
*Should it be defined as a multi-classification problem?*
|
||
- “the top six fairness concerns in AI-based app reviews” ([Nasab et al., p. 3](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=3&annotation=XYM8GIAR)) #ffd400
|
||
|
||
*Why *\***the**\*** top six? Where have they been introduced/defined?**
|
||
- “K-means clustering technique to summarize fairness concerns for answering RQ1 in Section 5” ([Nasab et al., p. 4](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=4&annotation=U7L5HKY3)) #ffd400
|
||
|
||
*I would not mention RQs here since they have not been introduced. For instance, I would write "... We report about the root causes of fairness concerns in Section 6."...*
|
||
- “n,” ([Nasab et al., p. 4](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=4&annotation=2TI5Y3ND)) #ff6666
|
||
- “six root causes of the fairness concerns: ‘copyright issues’, ‘development complexity’, ‘buggy code’, ‘external factors’, ‘development cost’, and ‘user usage and awareness’.” ([Nasab et al., p. 4](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=4&annotation=Z8ZZ727U)) #ffd400
|
||
|
||
*The definition of these 6 concerns needs to be introduced before the usage done before without any explanation.*
|
||
- “However, it is evident that ‘fairness’ is a complicated notion with different definitions” ([Nasab et al., p. 4](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=4&annotation=GDM5K2CN)) #5fb236
|
||
- “more work is required to clarify different fairness definitions and how they are employed in different scenarios” ([Nasab et al., p. 4](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=4&annotation=ITQYREEQ)) #a28ae5
|
||
- “improving the mitigation techniques on fairness-aware ML models requires a thorough understanding of the root causes of the unintended behaviour of these models” ([Nasab et al., p. 5](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=5&annotation=UQARPNYI)) #a28ae5
|
||
- “biases in the representation, measurement, and learning phase of a model to biases in aggregation, evaluation of results and deployment of model.” ([Nasab et al., p. 5](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=5&annotation=UBN2GGAD)) #e56eee
|
||
- “evaluating the fairness of AI systems comes with three challenges for practitioners: determining the appropriate performance metrics, identifying relevant stakeholders and demographic groups, and collecting datasets.” ([Nasab et al., p. 5](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=5&annotation=DI85SAC2)) #e56eee
|
||
- “Bird et al. [16] proposed the Fairlearn toolkit to help AI practitioners make a trade-off between fairness and model performance.” ([Nasab et al., p. 5](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=5&annotation=WID3626Q)) #a28ae5
|
||
- “Fairify” ([Nasab et al., p. 5](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=5&annotation=JSBGX446)) #a28ae5
|
||
- “This paper aims to deeply analyze the fairness of AI-based apps. This will be achieved by answering the following overarching” ([Nasab et al., p. 6](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=6&annotation=YRTTDCMR)) #ffd400
|
||
|
||
*"Fairness of AI-based apps" OR "Fairness of AI-based apps review?"*
|
||
- “We argue that user reviews of AI-based mobile apps can be a rich resource to uncover fairness concerns from the users’ perspective.” ([Nasab et al., p. 6](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=6&annotation=KZ54V6KW)) #5fb236
|
||
- “socio-technical issues” ([Nasab et al., p. 6](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=6&annotation=EVJ6WMG3)) #5fb236
|
||
- “none of these automated techniques has focused on fairness concerns” ([Nasab et al., p. 6](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=6&annotation=B73RQXB7)) #a28ae5
|
||
- “we plan to experiment with several ML and DL binary classifiers and a clustering technique that detect and summarize fairness concerns from the user reviews of AI-based apps.” ([Nasab et al., p. 6](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=6&annotation=UT5IVSGH)) #5fb236
|
||
- “we decided to use the most relevant reviews sorted by Google Play Store.” ([Nasab et al., p. 7](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=7&annotation=24BFRZ39)) #5fb236
|
||
- “fairness and non-fairness app reviews” ([Nasab et al., p. 7](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=7&annotation=WGIP95WQ)) #5fb236
|
||
- “we collected the top 500K reviews from each app with more than 500K reviews using a Google Play Crawler” ([Nasab et al., p. 7](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=7&annotation=Q8IGFZUZ)) #5fb236
|
||
- “17,968,298 reviews.” ([Nasab et al., p. 7](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=7&annotation=TMX8V5FI)) #5fb236
|
||
- “removed reviews with less than 4 words and non-English reviews” ([Nasab et al., p. 7](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=7&annotation=4WZ9GR8M)) #5fb236
|
||
- “9,475,506 reviews that potentially covered several topics such as feature requests, bug reports, non-technical concerns, etc.” ([Nasab et al., p. 7](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=7&annotation=GS6YBQ6T)) #a28ae5
|
||
- “‘fair’, ‘discrimination’, and ‘bias’ are three commonly used keywords to refer to fairness.” ([Nasab et al., p. 8](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=8&annotation=EPSFEMZ2)) #5fb236
|
||
- “30,019 ‘potential fairness reviews’.” ([Nasab et al., p. 8](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=8&annotation=8WQWRM84)) #ffd400
|
||
|
||
*I would also checked if some filtered out reviewes contain relevant text that could have been included.*
|
||
- “the 30,019 potential fairness reviews” ([Nasab et al., p. 8](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=8&annotation=MGH3JTS3)) #ffd400
|
||
|
||
*By following my previous comment, why not applying KeyBERT to the initial dataset?*
|
||
- “46,700 potential fairness reviews.” ([Nasab et al., p. 8](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=8&annotation=KKW75Y2D)) #ffd400
|
||
|
||
*Why not analyse the initial 9M reviews with the found keywords?*
|
||
- “TF-IDF calculates the significance of a word in a text” ([Nasab et al., p. 9](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=9&annotation=5NUJD3LE)) #5fb236
|
||
- “1,132 fairness and 1,473 non-fairness reviews.” ([Nasab et al., p. 9](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=9&annotation=84XCRKAJ)) #a28ae5
|
||
- “4.1.1 Dataset.” ([Nasab et al., p. 9](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=9&annotation=N76NCMIH)) #5fb236
|
||
- “Word2Vec is a word embedding technique that captures the semantic and syntactic relationships between words in a large corpus of text” ([Nasab et al., p. 9](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=9&annotation=4KVL96FE)) #5fb236
|
||
- “USE works at the sentence level and utilizes pre-trained sentence embedding models to produce sentence vectors” ([Nasab et al., p. 9](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=9&annotation=K7UZWG4P)) #5fb236
|
||
- “average” ([Nasab et al., p. 11](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=11&annotation=RBZDQVMX)) #5fb236
|
||
- “Performance of the cla” ([Nasab et al., p. 11](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=11&annotation=F4KC4EQ7)) #5fb236
|
||
- “has low precision of 0.63,” ([Nasab et al., p. 11](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=11&annotation=EX72DBFN)) #5fb236
|
||
- “high false positives” ([Nasab et al., p. 11](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=11&annotation=Z8KLL4MQ)) #5fb236
|
||
- “Logistic Regression with TF-IDF, USE, and Word2Vec” ([Nasab et al., p. 11](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=11&annotation=NVSGWGLC)) #5fb236
|
||
- “best-performing model,” ([Nasab et al., p. 12](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=12&annotation=2PE3XKI3)) #5fb236
|
||
- “Hence, we aimed to cluster and summarize fairness reviews” ([Nasab et al., p. 13](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=13&annotation=BX2Z8UZV)) #a28ae5
|
||
- “such unfair behaviours may lead to the exclusion of or discrimination towards some users.” ([Nasab et al., p. 13](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=13&annotation=JRBVBTLP)) #5fb236
|
||
- “binary classifier to detect fairness reviews” ([Nasab et al., p. 13](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=13&annotation=RXXVARLI)) #5fb236
|
||
- “Table 2.” ([Nasab et al., p. 14](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=14&annotation=CRZ5YRCT)) #ffd400
|
||
|
||
*The distribution of the identified six categories should be also discussed with respect to the categories of Tab 2. Currently, the reader has to wait the discussion section, to get some hints about this aspect.*
|
||
- “maximizing the similarity between the individual reviews in each cluster and the center of that cluster” ([Nasab et al., p. 14](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=14&annotation=X3BTHY9G)) #5fb236
|
||
- “summarization metric suggested by Nema et al. [72] to identify the suitable value of k in our work.” ([Nasab et al., p. 14](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=14&annotation=ZQT9SIG3)) #5fb236
|
||
- “minimizing the influence of the following two issues in data, i.e., (i) a data point can appear in multiple clusters produced by K-means” ([Nasab et al., p. 14](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=14&annotation=XM63Y3LE)) #5fb236
|
||
- “(ii) a data point discusses several topics or is an outlier.” ([Nasab et al., p. 14](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=14&annotation=ZLB2KRM5)) #5fb236
|
||
- “The silhouette score is a measure of how similar a data point is to the other data points in its cluster compared to how similar it is to the data points in other clusters” ([Nasab et al., p. 15](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=15&annotation=ELPT4HIH)) #a28ae5
|
||
- “Our experiments showed that the highest value for the summarization metric is achieved when k=10” ([Nasab et al., p. 15](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=15&annotation=A2ZSCFQV)) #a28ae5
|
||
- “2 to 10” ([Nasab et al., p. 15](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=15&annotation=W78J4CBI)) #5fb236
|
||
- “To understand the main fairness concern discussed in each of the eight compact clusters, we descendingly sorted the reviews in each compact cluster based on their silhouette scores. The first author manually analyzed the 30 reviews with the highest silhouette scores in each cluster (8 ∗ 30 = 240 reviews in total) and suggested a topic for each cluster. Then, three other authors rechecked all these 240 reviews and the proposed topics. The first author and the rest of authors held several meetings to discuss the topics, refined them and solved any disagreements. Although we had eight compact clusters, as shown in Fig. 3, the qualitative analysis process led to the identification of six distinct fairness concerns that are discussed next in Section 5.2.” ([Nasab et al., p. 15](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=15&annotation=5FZ563SN)) #ffd400
|
||
|
||
*The cluster analysis should be better presented, including the including the details that lead to focus on six concerns starting from initial 10 clusters.*
|
||
- “Fairness Concern 2: Feeling linguistic d” ([Nasab et al., p. 16](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=16&annotation=ALQC8JN2)) #a28ae5
|
||
- “Fairness Concern 1: Receiving different quality of features and services in different platforms and devices” ([Nasab et al., p. 16](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=16&annotation=88AYTPMU)) #a28ae5
|
||
- “Our analysis found that many users complain about the difference between the quality of features and services that an app offers on different platforms and devices.” ([Nasab et al., p. 16](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=16&annotation=PNLM5V25)) #5fb236
|
||
- “expect to be treated equally by app providers.” ([Nasab et al., p. 16](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=16&annotation=PIBSFG6Z)) #5fb236
|
||
- “feeling of inequality” ([Nasab et al., p. 16](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=16&annotation=F2UFJFEW)) #5fb236
|
||
- “Fairness Concern 3: Lack of transparency and fairness in dealing with user-generated content.” ([Nasab et al., p. 16](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=16&annotation=SR3QLDNI)) #a28ae5
|
||
- “Feeling” ([Nasab et al., p. 17](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=17&annotation=AEFQ26QM)) #a28ae5
|
||
- “Fairness Concern 6: Unfair and non-transparent advertisement and subscription poli- cies” ([Nasab et al., p. 17](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=17&annotation=ASJDUTYT)) #a28ae5
|
||
- “Fairness Concern 4: Feeling gender and racial discrimination” ([Nasab et al., p. 17](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=17&annotation=ILQKJUHR)) #a28ae5
|
||
- “Fairness Concern 5:” ([Nasab et al., p. 17](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=17&annotation=MPAK27SP)) #a28ae5
|
||
- “hip and promotion” ([Nasab et al., p. 17](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=17&annotation=QKX3LQU4)) #a28ae5
|
||
- “copyright issues imposed on the apps, platforms, or” ([Nasab et al., p. 18](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=18&annotation=V895R6IC)) #a28ae5
|
||
- “discuss six types of fairness concerns in AI-based app reviews, among which ‘receiving different quality of features and services in different platforms and devices’ and ‘lack of transparency and fairness in dealing with user-generated content’ are the most widely raised fairness concerns.” ([Nasab et al., p. 18](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=18&annotation=WGS565R6)) #a28ae5
|
||
- “the used AI models may make mistakes in identifying the context of a conversation, leading to saying inappropriate messages.” ([Nasab et al., p. 19](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=19&annotation=Z7Q4XBB6)) #a28ae5
|
||
- “App owners report six root causes, namely ‘copyright issues’, ‘development complexity’, ‘buggy code’, ‘external factors’, ‘development cost’, and ‘user usage and awareness’, to justify the fairness concerns raised by users.” ([Nasab et al., p. 20](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=20&annotation=FIIUDE7H)) #a28ae5
|
||
- “We suggest future research efforts to investigate this relationship and delve deeper into the fairness concerns specific to each category.” ([Nasab et al., p. 20](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=20&annotation=GHU2F7MM)) #a28ae5
|
||
- “Other researchers may apply this iterative and multi-faceted approach to reduce false positives, particularly when working with infrequently discussed topics in app reviews.” ([Nasab et al., p. 21](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=21&annotation=6NM9J43D)) #a28ae5
|
||
- “We argue that the popularity of DL models should not result in less attention being paid to traditional ML classifiers, which may outperform DL models in some datasets.” ([Nasab et al., p. 22](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=22&annotation=YF99TMDX)) #a28ae5
|
||
- “We started with a literature-driven set of keywords and expanded this using KeyBERT.” ([Nasab et al., p. 22](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=22&annotation=9HKJLGBR)) #ffd400
|
||
|
||
*I would have used KeyBERT on the initial dataset consisting of 9M app reviews.*
|
||
- “≈9.5 million reviews” ([Nasab et al., p. 22](zotero://select/library/items/BGIJALY7)) ([pdf](zotero://open-pdf/library/items/KY9752ET?page=22&annotation=MEKC65AN)) #5fb236
|
||
- ### [[Comments]]
|
||
- Summary: This research paper examines fairness in AI-powered mobile applications by analyzing user reviews to identify unfair practices. The study began by creating a ground-truth dataset comprising fairness and non-fairness reviews, which was used to develop and evaluate different machine learning and deep learning classifiers. The best-performing classifier achieved a 94% precision rate in identifying fairness reviews. This classifier was then applied to analyze approximately 9.5 million reviews from 108 AI-based apps, identifying around 92,000 reviews that raised fairness concerns. The concerns were categorized into six clusters using K-means clustering and manual analysis. The paper also delves into the app owners' responses, identifying six primary root causes that they reported to justify the fairness concerns raised in the reviews.
|
||
- Comments: This is an interesting and relevant paper. I found the work properly organized and structured. I have some minor concerns that need to be addressed to bring the paper to the proper shape:
|
||
- Pag. 3 - Page Organization: I would not mention RQs here since they have yet to be introduced properly. For instance, I would write, "... We report about the root causes of fairness concerns in Section 6."...
|
||
- Pag. 5, first sentence of Sec.3: "Fairness of AI-based apps" OR "Fairness of AI-based apps review"? The second one, right?
|
||
- Page 7 - Using KeyBERT: The decision to employ KeyBERT on a pre-filtered set of reviews rather than the initial full dataset consisting of 9M reviews may limit the discovery of sentences related to fairness concerns. Utilizing KeyBERT across the broader dataset might uncover additional reviews that can be of interest to be analyzed.
|
||
- Table 2 and its accompanying descriptive text require further elaboration to clarify the relationship between the distribution of the six identified fairness concerns and the application categories listed in Table 2. The reader must wait until the discussion section to gain insights into how fairness concerns and application categories are related.
|
||
- Page 14 - end of Sec. 5.1: The presented cluster analysis has to be improved by more thoroughly describing how the initial ten clusters were refined to focus on six specific fairness concerns.
|
||
- General comment: the paper introduces an interesting set of fairness concerns but needs to clearly delineate whether these concerns could be more effectively addressed through a multi-classification problem framework.
|
||
- To summarize, the paper presents an interesting contribution to the ongoing discussion of fairness in AI-powered mobile applications. Nevertheless, the paper could benefit from addressing a few critical issues to enhance its clarity and presentation.
|
||
-
|
||
- ### [[REVIEWS/Notes]]
|
||
- ### ❓️YELLOW CONCERNS
|
||
- {{query (and [[ffd400]] [[TOSEM-2024-0086]] )}}
|
||
query-table:: true
|
||
query-properties:: [:block]
|
||
- |