Ten Years of Teaching Empirical Software Engineering in the context of Energy-efficient Software

Abstract

#

In this chapter we share our experience in running ten editions of the Green Lab course at the Vrije Universiteit Amsterdam, the Netherlands. The course is given in the Software Engineering and Green IT track of the Computer Science Master program of the VU. The course takes place every year over a 2-month period and teaches Computer Science students the fundamentals of Empirical Software Engineering in the context of energy-efficient software.

The peculiarity of the course is its research orientation: at the beginning of the course the instructor presents a catalog of scientifically relevant goals, and each team of students signs up for one of them and works together for 2 months on their own experiment for achieving the goal. Each team goes over the classic steps of an empirical study, starting from a precise formulation of the goal and research questions to context definition, selection of experimental subjects and objects, definition of experimental variables, experiment execution, data analysis, and reporting.

Over the years, the course became well-known within the Software Engineering community since it led to several scientific studies that have been published at various scientific conferences and journals. Also, students execute their experiments using open-source tools, which are developed and maintained by researchers and other students within the program, thus creating a virtuous community of learners where students exchange ideas, help each other, and learn how to collaboratively contribute to open-source projects in a safe environment.

Materials

#

Lecture Slides

#

Introduction

#

Green IT

#

Design and Development of Green Software

#

The experimental process

#

Experiment scoping

#

Experiment validity - Construct validity

#

Experiment planning

#

Measurement theory basics

#

Experiment validity - External validity

#

Experiment design - basics

#

Experiment design - advanced

#

Experiment validity - Internal validity

#

Data analysis

#

Hypothesis testing

#

Experiment validity - Conclusion validity

#

Statistical tests and effect size

#

Data visualization

#

Experiment reporting

#

Software Carbon Intensity

#

Q&A

#

Project Tracks

#

Instructions for Students

#

Expected Contributions

#

Recommended Readings

#

Material For Projects

#

The following is a list of recommended readings specific to each proposed project. In particular, the following list pertains to the sample projects described in materials/lectures/Project tracks.pdf.

• Orosz, Gergely. “Building Mobile Apps at Scale.” 1st ed. Pragmatic Engineer. 2021
• Dawson, Alexander. “The Carbon Impact of Web Standards.” Website Sustainability. 2023 (https://websitesustainability.com/cache/files/research23.pdf)
• Oliveira, Wellington, Bernardo Moraes, Fernando Castor, and João Paulo Fernandes. “Analyzing the Resource Usage Overhead of Mobile App Development Frameworks.” In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, 2023
• Vogel, Lucas, and Thomas Springer. “An in-depth analysis of web page structure and efficiency with focus on optimization potential for initial page load.” In International Conference on Web Engineering, Cham: Springer International Publishing, 2022.
• Vogel, Lucas, and Thomas Springer. “Speed Up the Web with Universal CSS Rendering.” In International Conference on Web Engineering, Cham: Springer Nature Switzerland, 2023
• Dornauer, Benedikt, and Michael Felderer. “Energy-saving strategies for mobile web apps and their measurement: Results from a decade of research.” In 2023 IEEE/ACM 10th International Conference on Mobile Software Engineering and Systems (MOBILESoft), pp. 75-86. IEEE, 2023

Material for Students

#

The following is a list of recommended readings for students providing additional information about course content and material to improve their report. In addition to the list of papers below, the student materials also include assignments taken from previous years.

Data Visualization

#

• Tufte, Edward, and P. Graves-Morris. “The visual display of quantitative information.; 1983.” Diagrammatik-Reader. Grundlegende Texte aus Theorie und Geschichte. Berlin: De Gruyter (2014): 219-230
• Munzner, Tamara. Visualization analysis and design. CRC press, 2014.

Empirical Methods

#

• Dybå, Tore, Vigdis By Kampenes, and Dag IK Sjøberg. “A systematic review of statistical power in software engineering experiments.” Information and Software Technology 48, no. 8 (2006): 745-755.
• Feldt, R. and Chalmers, A., 2010. Guide to research questions. Retrieved December, 5, p.2017.
• de Oliveira Neto, Francisco Gomes, Richard Torkar, Robert Feldt, Lucas Gren, Carlo A. Furia, and Ziwei Huang. “Evolution of statistical analysis in empirical software engineering research: Current state and steps forward.” Journal of Systems and Software 156 (2019)
• Juristo, Natalia, and Ana M. Moreno. Basics of software engineering experimentation. Springer Science & Business Media, 2013.
• Pfleeger, Shari Lawrence. “Experimental design and analysis in software engineering.” Annals of Software Engineering 1, no. 1 (1995)
• Gopen, George D., and Judith A. Swan. “The science of scientific writing.” American scientist 78, no. 6 (1990)

Energy Efficiency

#

• Biørn-Hansen, Andreas, Tor-Morten Grønli, and Gheorghita Ghinea. “A survey and taxonomy of core concepts and research challenges in cross-platform mobile development.” ACM Computing Surveys (CSUR) 51, no. 5 (2018): 1-34.
• Pinto, Gustavo, and Fernando Castor. “Energy efficiency: a new concern for application software developers.” Communications of the ACM 60, no. 12 (2017): 68-75.
• Grønli, Tor-Morten, Andreas Biørn-Hansen, and Tim A. Majchrzak. “Software development for mobile computing, the internet of things and wearable devices: Inspecting the past to understand the future.” (2019).
• Verdecchia, Roberto, Patricia Lago, Christof Ebert, and Carol De Vries. “Green IT and green software.” IEEE Software 38, no. 6 (2021): 7-15.
• Ardito, Luca, Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. “Methodological guidelines for measuring energy consumption of software applications.” Scientific Programming 2019 (2019): 1-16.
• Oliveira, W., Matalonga, H., Pinto, G., Castor, F. and Fernandes, J.P., 2021. Small changes, big impacts: Leveraging diversity to improve energy efficiency. Software Sustainability, pp.123-152.
• Pramanik, P.K.D., Sinhababu, N., Mukherjee, B., Padmanaban, S., Maity, A., Upadhyaya, B.K., Holm-Nielsen, J.B. and Choudhury, P., 2019. Power consumption analysis, measurement, management, and issues: A state-of-the-art review of smartphone battery and energy usage. IEEE Access
• Guldner, Achim, Rabea Bender, Coral Calero, Giovanni S. Fernando, Markus Funke, Jens Gröger, Lorenz M. Hilty et al. “Development and evaluation of a reference measurement model for assessing the resource and energy efficiency of software products and components—Green Software Measurement Model (GSMM).” Future Generation Computer Systems (2024)

Examples of Experiments

#

• Hampau, Raluca Maria, Maurits Kaptein, Robin Van Emden, Thomas Rost, and Ivano Malavolta. “An empirical study on the performance and energy consumption of ai containerization strategies for computer-vision tasks on the edge.” In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering, pp. 50-59. 2022.
• Linares-Vásquez, Mario, Gabriele Bavota, Carlos Bernal-Cárdenas, Massimiliano Di Penta, Rocco Oliveto, and Denys Poshyvanyk. “Api change and fault proneness: A threat to the success of android apps.” In Proceedings of the 2013 9th joint meeting on foundations of software engineering, pp. 477-487. 2013.
• van Riet, Jasper, Flavia Paganelli, and Ivano Malavolta. “From 6.2 to 0.15 seconds–an industrial case study on mobile web performance.” In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 746-755. IEEE, 2020.
• Procaccianti, Giuseppe, Héctor Fernández, and Patricia Lago. “Empirical evaluation of two best practices for energy-efficient software development.” Journal of Systems and Software 117 (2016): 185-198.
• Santos, Eddie Antonio, Carson McLean, Christopher Solinas, and Abram Hindle. “How does Docker affect energy consumption? Evaluating workloads in and out of Docker containers.” Journal of Systems and Software 146 (2018): 14-25.
• Chowdhury, Shaiful, Silvia Di Nardo, Abram Hindle, and Zhen Ming Jiang. “An exploratory study on assessing the energy impact of logging on android applications.” Empirical Software Engineering 23 (2018): 1422-1456.
• Ma, Yun, Xuanzhe Liu, Yi Liu, Yunxin Liu, and Gang Huang. “A tale of two fashions: An empirical study on the performance of native apps and web apps on android.” IEEE Transactions on Mobile Computing 17, no. 5 (2017): 990-1003.
• Janssen, Kalle, Tim Pelle, Lucas De Geus, Reinier Van Der Gronden, Tanjina Islam, and Ivano Malavolta. “On the impact of the critical css technique on the performance and energy consumption of mobile browsers.” In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering, pp. 130-139. 2022.
• Anwar, Hina, Berker Demirer, Dietmar Pfahl, and Satish Srirama. “Should energy consumption influence the choice of android third-party http libraries?.” In Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems, pp. 87-97. 2020.
• Linares-Vásquez, Mario, Gabriele Bavota, Carlos Bernal-Cárdenas, Rocco Oliveto, Massimiliano Di Penta, and Denys Poshyvanyk. “Mining energy-greedy api usage patterns in android apps: an empirical study.” In Proceedings of the 11th working conference on mining software repositories, pp. 2-11. 2014.

Programming Books

#

• Lutz, Mark. Learning python: Powerful object-oriented programming. “O’Reilly Media, Inc.”, 2013.
• Albing, Carl, J. P. Vossen, and Cameron Newham. bash Cookbook: Solutions and Examples for bash Users. " O’Reilly Media, Inc.", 2007.

Statistics

#

• McElreath, Richard. Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman and Hall/CRC, 2018.
• Wasserman, Larry. All of statistics: a concise course in statistical inference. Springer Science & Business Media, 2013.
• Kruschke, John K. “Bayesian data analysis.” Wiley Interdisciplinary Reviews: Cognitive Science 1, no. 5 (2010): 658-676.
• Lakens, Daniël. “Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs.” Frontiers in psychology 4 (2013): 62627.
• Cohen, Jacob. Statistical power analysis for the behavioral sciences. Routledge, 2013.
• Salkind, Neil J. Encyclopedia of measurement and statistics. SAGE publications, 2006.
• Cowan, Glen. Statistical data analysis. Oxford university press, 1998.
• Wickham, Hadley, Mine Çetinkaya-Rundel, and Garrett Grolemund. R for data science. " O’Reilly Media, Inc.", 2023.
• de Oliveira Neto, Francisco Gomes, Richard Torkar, Robert Feldt, Lucas Gren, Carlo A. Furia, and Ziwei Huang. “Evolution of statistical analysis in empirical software engineering research: Current state and steps forward.” Journal of Systems and Software 156 (2019): 246-267.
• Ross, Sheldon. “Probability and statistics for engineers and scientists.” Elsevier, New Delhi 16 (2009): 32-33.

Licensing

#