Master - Lab - What size of effects can Developer-Centered Security Studies detect?

Supervisor: Anna-Marie (ortloff@cs.uni-bonn.de)

This lab is suitable for a group.

Together with students and colleagues I developed a database with information from studies in developer-centered security, i.e. studies where the participants are e.g. software developers or administrators. We focused especially on information about the data collection and data analysis practices, as well as statistical tests and how this information can be used to conduct power analysis. We collected information on whether the statistics reported in the papers are sufficiently detailed to be able to conduct power analysis, for seven simple and common hypothesis tests, e.g. Fisher’s Exact Test, Wilcoxon-Ranksum-test.

Your task

• For the seven tests, research which data is necessary to be able to conduct a sensitivity analysis (a variant of power analysis, where the detectable effect size is calculated from power, sample size and alpha level) with G*Power and/or another
• Conduct the sensitivity analysis for those tests in the database where sufficient data for sensitivity analysis is reported.
• Discuss what size of effects can be detected by studies in developer-centered security on average (Examine the distribution of effect sizes resulting from the sensitivity analyses)

If more than one student is interested in this topic, you will each focus on different test types for researching sensitivity analysis and get assigned to different specific tests to conduct the analysis for.

Literature to start with:

• Explanation of the database structure / Data-input with the admin interface of the database: https://de.overleaf.com/read/ygxcbgzyzrtn
• Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Download here: https://link.springer.com/content/pdf/10.3758/BF03193146.pdf Especially the paragraph on sensitivity analysis.
• Coe, R. (2002). It’s the effect size, stupid. Download here: https://cebma.org/wp-content/uploads/Coe-2002.pdf
• Lakens, D. (2022). Sample size justification. Access here: https://online.ucpress.edu/collabra/article/8/1/33267/120491/Sample-Size-Justification Especially the part on sensitivity