software testing Recently Published Documents

Total documents.

  • Latest Documents
  • Most Cited Documents
  • Contributed Authors
  • Related Sources
  • Related Keywords

Combining Learning and Engagement Strategies in a Software Testing Learning Environment

There continues to be an increase in enrollments in various computing programs at academic institutions due to many job opportunities available in the information, communication, and technology sectors. This enrollment surge has presented several challenges in many Computer Science (CS), Information Technology (IT), and Software Engineering (SE) programs at universities and colleges. One such challenge is that many instructors in CS/IT/SE programs continue to use learning approaches that are not learner centered and therefore are not adequately preparing students to be proficient in the ever-changing computing industry. To mitigate this challenge, instructors need to use evidence-based pedagogical approaches, e.g., active learning, to improve student learning and engagement in the classroom and equip students with the skills necessary to be lifelong learners. This article presents an approach that combines learning and engagement strategies (LESs) in learning environments using different teaching modalities to improve student learning and engagement. We describe how LESs are integrated into face-to-face (F2F) and online class activities. The LESs currently used are collaborative learning , gamification , problem-based learning , and social interaction . We describe an approach used to quantify each LES used during class activities based on a set of characteristics for LESs and the traditional lecture-style pedagogical approaches. To demonstrate the impact of using LESs in F2F class activities, we report on a study conducted over seven semesters in a software testing class at a large urban minority serving institution. The study uses a posttest-only study design, the scores of two midterm exams, and approximate class times dedicated to each LES and traditional lecture style to quantify their usage in a face-to-face software testing class. The study results showed that increasing the time dedicated to collaborative learning, gamification, and social interaction and decreasing the traditional lecture-style approach resulted in a statistically significant improvement in student learning, as reflected in the exam scores.

Enhancing Search-based Testing with Testability Transformations for Existing APIs

Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag problem , where the fitness landscape presents a plateau that provides no guidance to the search. In this article, we provide a series of novel testability transformations aimed at providing guidance in the context of commonly used API calls (e.g., strings that need to be converted into valid date/time objects). We also provide specific transformations aimed at helping the testing of REST Web Services. We implemented our novel techniques as an extension to EvoMaster , an SBST tool that generates system-level test cases. Experiments on nine open-source REST web services, as well as an industrial web service, show that our novel techniques improve performance significantly.

A Survey of Flaky Tests

Tests that fail inconsistently, without changes to the code under test, are described as flaky . Flaky tests do not give a clear indication of the presence of software bugs and thus limit the reliability of the test suites that contain them. A recent survey of software developers found that 59% claimed to deal with flaky tests on a monthly, weekly, or daily basis. As well as being detrimental to developers, flaky tests have also been shown to limit the applicability of useful techniques in software testing research. In general, one can think of flaky tests as being a threat to the validity of any methodology that assumes the outcome of a test only depends on the source code it covers. In this article, we systematically survey the body of literature relevant to flaky test research, amounting to 76 papers. We split our analysis into four parts: addressing the causes of flaky tests, their costs and consequences, detection strategies, and approaches for their mitigation and repair. Our findings and their implications have consequences for how the software-testing community deals with test flakiness, pertinent to practitioners and of interest to those wanting to familiarize themselves with the research area.

Test Suite Optimization Using Firefly and Genetic Algorithm

Software testing is essential for providing error-free software. It is a well-known fact that software testing is responsible for at least 50% of the total development cost. Therefore, it is necessary to automate and optimize the testing processes. Search-based software engineering is a discipline mainly focussed on automation and optimization of various software engineering processes including software testing. In this article, a novel approach of hybrid firefly and a genetic algorithm is applied for test data generation and selection in regression testing environment. A case study is used along with an empirical evaluation for the proposed approach. Results show that the hybrid approach performs well on various parameters that have been selected in the experiments.

Machine Learning Model to Predict Automated Testing Adoption

Software testing is an activity conducted to test the software under test. It has two approaches: manual testing and automation testing. Automation testing is an approach of software testing in which programming scripts are written to automate the process of testing. There are some software development projects under development phase for which automated testing is suitable to use and other requires manual testing. It depends on factors like project requirements nature, team which is working on the project, technology on which software is developing and intended audience that may influence the suitability of automated testing for certain software development project. In this paper we have developed machine learning model for prediction of automated testing adoption. We have used chi-square test for finding factors’ correlation and PART classifier for model development. Accuracy of our proposed model is 93.1624%.

Metaheuristic Techniques for Test Case Generation

The primary objective of software testing is to locate bugs as many as possible in software by using an optimum set of test cases. Optimum set of test cases are obtained by selection procedure which can be viewed as an optimization problem. So metaheuristic optimizing (searching) techniques have been immensely used to automate software testing task. The application of metaheuristic searching techniques in software testing is termed as Search Based Testing. Non-redundant, reliable and optimized test cases can be generated by the search based testing with less effort and time. This article presents a systematic review on several meta heuristic techniques like Genetic Algorithms, Particle Swarm optimization, Ant Colony Optimization, Bee Colony optimization, Cuckoo Searches, Tabu Searches and some modified version of these algorithms used for test case generation. The authors also provide one framework, showing the advantages, limitations and future scope or gap of these research works which will help in further research on these works.

Software Testing Under Agile, Scrum, and DevOps

The adoption of agility at a large scale often requires the integration of agile and non-agile development practices into hybrid software development and delivery environment. This chapter addresses software testing related issues for Agile software application development. Currently, the umbrella of Agile methodologies (e.g. Scrum, Extreme Programming, Development and Operations – i.e., DevOps) have become the preferred tools for modern software development. These methodologies emphasize iterative and incremental development, where both the requirements and solutions evolve through the collaboration between cross-functional teams. The success of such practices relies on the quality result of each stage of development, obtained through rigorous testing. This chapter introduces the principles of software testing within the context of Scrum/DevOps based software development lifecycle.

Quality Assurance Issues for Big Data Applications in Supply Chain Management

Heterogeneous data types, widely distributed data sources, huge data volumes, and large-scale business-alliance partners describe typical global supply chain operational environments. Mobile and wireless technologies are putting an extra layer of data source in this technology-enriched supply chain operation. This environment also needs to provide access to data anywhere, anytime to its end-users. This new type of data set originating from the global retail supply chain is commonly known as big data because of its huge volume, resulting from the velocity with which it arrives in the global retail business environment. Such environments empower and necessitate decision makers to act or react quicker to all decision tasks. Academics and practitioners are researching and building the next generation of big-data-based application software systems. This new generation of software applications is based on complex data analysis algorithms (i.e., on data that does not adhere to standard relational data models). The traditional software testing methods are insufficient for big-data-based applications. Testing big-data-based applications is one of the biggest challenges faced by modern software design and development communities because of lack of knowledge on what to test and how much data to test. Big-data-based applications developers have been facing a daunting task in defining the best strategies for structured and unstructured data validation, setting up an optimal test environment, and working with non-relational databases testing approaches. This chapter focuses on big-data-based software testing and quality-assurance-related issues in the context of Hadoop, an open source framework. It includes discussion about several challenges with respect to massively parallel data generation from multiple sources, testing methods for validation of pre-Hadoop processing, software application quality factors, and some of the software testing mechanisms for this new breed of applications

Use of Qualitative Research to Generate a Function for Finding the Unit Cost of Software Test Cases

In this article, we demonstrate a novel use of case research to generate an empirical function through qualitative generalization. This innovative technique applies interpretive case analysis to the problem of defining and generalizing an empirical cost function for test cases through qualitative interaction with an industry cohort of subject matter experts involved in software testing at leading technology companies. While the technique is fully generalizable, this article demonstrates this technique with an example taken from the important field of software testing. The huge amount of software development conducted in today's world makes taking its cost into account imperative. While software testing is a critical aspect of the software development process, little attention has been paid to the cost of testing code, and specifically to the cost of test cases, in comparison to the cost of developing code. Our research fills the gap by providing a function for estimating the cost of test cases.

Framework for Reusable Test Case Generation in Software Systems Testing

Agile methodologies have become the preferred choice for modern software development. These methods focus on iterative and incremental development, where both requirements and solutions develop through collaboration among cross-functional software development teams. The success of a software system is based on the quality result of each stage of development with proper test practice. A software test ontology should represent the required software test knowledge in the context of the software tester. Reusing test cases is an effective way to improve the testing of software. The workload of a software tester for test-case generation can be improved, previous software testing experience can be shared, and test efficiency can be increased by automating software testing. In this chapter, the authors introduce a software testing framework (STF) that uses rule-based reasoning (RBR), case-based reasoning (CBR), and ontology-based semantic similarity assessment to retrieve the test cases from the case library. Finally, experimental results are used to illustrate some of the features of the framework.

Export Citation Format

Share document.

Help | Advanced Search

Computer Science > Software Engineering

Title: automated unit test improvement using large language models at meta.

Abstract: This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. We describe the deployment of TestGen-LLM at Meta test-a-thons for the Instagram and Facebook platforms. In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement.

Submission history

Access paper:.

  • Download PDF
  • Other Formats

license icon

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Gemma: Introducing new state-of-the-art open models

Feb 21, 2024

Gemma is built for responsible AI development from the same research and technology used to create Gemini models.

The word “Gemma” and a spark icon with blueprint styling appears in a blue gradient against a black background.

At Google, we believe in making AI helpful for everyone . We have a long history of contributing innovations to the open community, such as with Transformers , TensorFlow , BERT , T5 , JAX , AlphaFold , and AlphaCode . Today, we’re excited to introduce a new generation of open models from Google to assist developers and researchers in building AI responsibly.

Gemma open models

Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. Developed by Google DeepMind and other teams across Google, Gemma is inspired by Gemini, and the name reflects the Latin gemma , meaning “precious stone.” Accompanying our model weights, we’re also releasing tools to support developer innovation, foster collaboration, and guide responsible use of Gemma models.

Gemma is available worldwide, starting today. Here are the key details to know:

  • We’re releasing model weights in two sizes: Gemma 2B and Gemma 7B . Each size is released with pre-trained and instruction-tuned variants.
  • A new Responsible Generative AI Toolkit provides guidance and essential tools for creating safer AI applications with Gemma.
  • We’re providing toolchains for inference and supervised fine-tuning (SFT) across all major frameworks: JAX, PyTorch, and TensorFlow through native Keras 3.0 .
  • Ready-to-use Colab and Kaggle notebooks , alongside integration with popular tools such as Hugging Face , MaxText , NVIDIA NeMo and TensorRT-LLM , make it easy to get started with Gemma.
  • Pre-trained and instruction-tuned Gemma models can run on your laptop, workstation, or Google Cloud with easy deployment on Vertex AI and Google Kubernetes Engine (GKE).
  • Optimization across multiple AI hardware platforms ensures industry-leading performance, including NVIDIA GPUs and Google Cloud TPUs .
  • Terms of use permit responsible commercial usage and distribution for all organizations, regardless of size.

State-of-the-art performance at size

Gemma models share technical and infrastructure components with Gemini , our largest and most capable AI model widely available today. This enables Gemma 2B and 7B to achieve best-in-class performance for their sizes compared to other open models. And Gemma models are capable of running directly on a developer laptop or desktop computer. Notably, Gemma surpasses significantly larger models on key benchmarks while adhering to our rigorous standards for safe and responsible outputs. See the technical report for details on performance, dataset composition, and modeling methodologies.

A chart showing Gemma performance on common benchmarks, compared to Llama-2 7B and 13B

Responsible by design

Gemma is designed with our AI Principles at the forefront. As part of making Gemma pre-trained models safe and reliable, we used automated techniques to filter out certain personal information and other sensitive data from training sets. Additionally, we used extensive fine-tuning and reinforcement learning from human feedback (RLHF) to align our instruction-tuned models with responsible behaviors. To understand and reduce the risk profile for Gemma models, we conducted robust evaluations including manual red-teaming, automated adversarial testing, and assessments of model capabilities for dangerous activities. These evaluations are outlined in our Model Card . 1

We’re also releasing a new Responsible Generative AI Toolkit together with Gemma to help developers and researchers prioritize building safe and responsible AI applications. The toolkit includes:

  • Safety classification: We provide a novel methodology for building robust safety classifiers with minimal examples.
  • Debugging: A model debugging tool helps you investigate Gemma's behavior and address potential issues.
  • Guidance: You can access best practices for model builders based on Google’s experience in developing and deploying large language models.

Optimized across frameworks, tools and hardware

You can fine-tune Gemma models on your own data to adapt to specific application needs, such as summarization or retrieval-augmented generation (RAG). Gemma supports a wide variety of tools and systems:

  • Multi-framework tools: Bring your favorite framework, with reference implementations for inference and fine-tuning across multi-framework Keras 3.0, native PyTorch, JAX, and Hugging Face Transformers.
  • Cross-device compatibility: Gemma models run across popular device types, including laptop, desktop, IoT, mobile and cloud, enabling broadly accessible AI capabilities.
  • Cutting-edge hardware platforms: We’ve partnered with NVIDIA to optimize Gemma for NVIDIA GPUs , from data center to the cloud to local RTX AI PCs, ensuring industry-leading performance and integration with cutting-edge technology.
  • Optimized for Google Cloud: Vertex AI provides a broad MLOps toolset with a range of tuning options and one-click deployment using built-in inference optimizations. Advanced customization is available with fully-managed Vertex AI tools or with self-managed GKE, including deployment to cost-efficient infrastructure across GPU, TPU, and CPU from either platform.

Free credits for research and development

Gemma is built for the open community of developers and researchers powering AI innovation. You can start working with Gemma today using free access in Kaggle, a free tier for Colab notebooks, and $300 in credits for first-time Google Cloud users. Researchers can also apply for Google Cloud credits of up to $500,000 to accelerate their projects.

Getting started

You can explore more about Gemma and access quickstart guides on ai.google.dev/gemma .

As we continue to expand the Gemma model family, we look forward to introducing new variants for diverse applications. Stay tuned for events and opportunities in the coming weeks to connect, learn and build with Gemma.

We’re excited to see what you create!

More Information

Google adheres to rigorous data filtering practices to ensure fair evaluation. Our models exclude benchmark data from training sets, ensuring the integrity of benchmark comparisons.

Related stories

Gemini models are coming to performance max, what is a long context window.

MSC_Keyword_Cover (3)

How AI can strengthen digital security

Shield

Working together to address AI risks and opportunities at MSC

Our next-generation model: gemini 1.5.

AI Evergreen 1 (1)

How we’re partnering with the industry, governments and civil society to advance AI

Let’s stay in touch. Get the latest news from Google in your inbox.

A walk through of software testing techniques

Ieee account.

  • Change Username/Password
  • Update Address

Purchase Details

  • Payment Options
  • Order History
  • View Purchased Documents

Profile Information

  • Communications Preferences
  • Profession and Education
  • Technical Interests
  • US & Canada: +1 800 678 4333
  • Worldwide: +1 732 981 0060
  • Contact & Support
  • About IEEE Xplore
  • Accessibility
  • Terms of Use
  • Nondiscrimination Policy
  • Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Testing research software: a survey

  • Published: 26 July 2022
  • Volume 27 , article number  138 , ( 2022 )

Cite this article

  • Nasir U. Eisty 1 &
  • Jeffrey C. Carver   ORCID: orcid.org/0000-0002-7824-9151 2  

707 Accesses

Explore all metrics

Research software plays an important role in solving real-life problems, empowering scientific innovations, and handling emergency situations. Therefore, the correctness and trustworthiness of research software are of absolute importance. Software testing is an important activity for identifying problematic code and helping to produce high-quality software. However, testing of research software is difficult due to the complexity of the underlying science, relatively unknown results from scientific algorithms, and the culture of the research software community.

The goal of this paper is to better understand current testing practices, identify challenges, and provide recommendations on how to improve the testing process for research software development.

We surveyed members of the research software developer community to collect information regarding their knowledge about and use of software testing in their projects.

We analysed 120 responses and identified that even though research software developers report they have an average level of knowledge about software testing, they still find it difficult due to the numerous challenges involved. However, there are a number of ways, such as proper training, that can improve the testing process for research software.

Conclusions

Testing can be challenging for any type of software. This difficulty is especially present in the development of research software, where software engineering activities are typically given less attention. To produce trustworthy results from research software, there is a need for a culture change so that testing is valued and teams devote appropriate effort to writing and executing tests.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

types of software testing research paper

https://se4science.org/workshops/

The survey data is available in a public repository but set to private until publication of this paper (Carver and Eisty 2021 )

https://www.istqb.org/

Ackroyd K S, Kinder S H, Mant G R, Miller M C, Ramsdale C A, Stephenson P C (2008) Scientific software development at a research facility. IEEE Softw 25(4):44–51. https://doi.org/10.1109/MS.2008.93

Article   Google Scholar  

Ammann P, Offutt J (2016) Introduction to software testing, 2nd edn. Cambridge University Press, Cambridge

Book   Google Scholar  

Bourque P, Fairley RE (eds) (2014) SWEBOK: guide to the software engineering body of knowledge, version 3.0 edn. IEEE Computer Society, Los Alamitos. http://www.swebok.org/

Carver JC, Eisty N (2021) Testing research software: survey data. https://doi.org/10.6084/m9.figshare.16663561 . https://figshare.com/articles/dataset/_/16663561/0

Clune T, Rood R (2011) Software testing and verification in climate model development. IEEE Softw 28(6):49–55. https://doi.org/10.1109/MS.2011.117

Drake J B, Jones P W, George R Carr J (2005) Overview of the software design of the community climate system model. Int J High Perform Comput Appl 19 (3):177–186. https://doi.org/10.1177/1094342005056094 https://doi.org/10.1177/1094342005056094

Easterbrook S M (2010) Climate change: a grand software challenge. In: Proceedings of the FSE/SDP workshop on future of software engineering research, FoSER ’10. https://doi.org/10.1145/1882362.1882383 . http://doi.acm.org/10.1145/1882362.1882383 . ACM, pp 99–104

Easterbrook S M, Johns T C (2009) Engineering the software for understanding climate change. Comput Sci Eng 11(6):65–74. https://doi.org/10.1109/MCSE.2009.193

Eddins S L (2009) Automated software testing for matlab. Comput Sci Eng 11(6):48–55. https://doi.org/10.1109/MCSE.2009.186 https://doi.org/10.1109/MCSE.2009.186

Eisty N U, Thiruvathukal G K, Carver J C (2018) A survey of software metric use in research software development. In: 2018 IEEE 14th international conference on e-science (e-science). https://doi.org/10.1109/eScience.2018.00036 , pp 212–222

Farrell P E, Piggott M D, Gorman G J, Ham D A, Wilson C R, Bond T M (2011) Automated continuous verification for numerical simulation. Geosci Model Dev 4(2):435–449. https://doi.org/10.5194/gmd-4-435-2011 https://doi.org/10.5194/gmd-4-435-2011 . https://www.geosci-model-dev.net/4/435/2011/

Hannay J E, MacLeod C, Singer J, Langtangen H P, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software?. In: 2009 ICSE workshop on software engineering for computational science and engineering. https://doi.org/10.1109/SECSE.2009.5069155 , pp 1–8

Heaton D, Carver J C (2015) Claims about the use of software engineering practices in science: a systematic literature review. Inf Softw Tech 67:207–219. https://doi.org/10.1016/j.infsof.2015.07.011 . http://www.sciencedirect.com/science/article/pii/S0950584915001342

Hill C (2016) Socio-economic status and computer use: designing software that supports low-income users. In: 2016 IEEE Symposium on visual languages and human-centric computing. https://doi.org/10.1109/VLHCC.2016.7739651 , pp 1–1

Hochstein L, Basili V R (2008) The asc-alliance projects: a case study of large-scale parallel scientific code development. Computer 41(3):50–58. https://doi.org/10.1109/MC.2008.101

Hook D, Kelly D (2009) Testing for trustworthiness in scientific software. In: 2009 ICSE Workshop on software engineering for computational science and engineering. https://doi.org/10.1109/SECSE.2009.5069163 https://doi.org/10.1109/SECSE.2009.5069163 , pp 59–64

Kanewala U, Bieman J M (2014) Testing scientific software: a systematic literature review. Inf Softw Technol 56(10):1219–1232. https://doi.org/10.1016/j.infsof.2014.05.006

Katz D S, McInnes L C, Bernholdt D E, Mayes A C, Hong N P C, Duckles J, Gesing S, Heroux M A, Hettrick S, Jimenez R C, Pierce M, Weaver B, Wilkins-Diehr N (2019) Community organizations: changing the culture in which research software is developed and sustained. Comput Sci Eng 21(2):8–24. https://doi.org/10.1109/MCSE.2018.2883051

Kelly D, Sanders R, Saint R, Floor P, Sanders R, Kelly D (2008) The challenge of testing scientific software. In: Proceedings of the conference for the association for software testing, pp 30–36

Kelly D, Hook D, Sanders R (2009) Five recommended practices for computational scientists who write software. Comput Sci Eng 11(5):48–53. https://doi.org/10.1109/MCSE.2009.139

Kelly D, Thorsteinson S, Hook D (2011) Scientific software testing: analysis with four dimensions. IEEE Softw 28(3):84–90. https://doi.org/10.1109/MS.2010.88

Lemos G S, Martins E (2012) Specification-guided golden run for analysis of robustness testing results. In: 2012 IEEE Sixth international conference on software security and reliability. https://doi.org/10.1109/SERE.2012.28 , pp 157–166

Miller G (2006) A scientist’s nightmare: software problem leads to five retractions. Science 314(5807):1856–1857. https://doi.org/10.1126/science.314.5807.1856 . https://science.sciencemag.org/content/314/5807/1856 , https://science.sciencemag.org/content/314/5807/1856.full.pdf

Murphy C, Raunak M, King A, Chen S, Imbraino C, Kaiser G, Lee I, Sokolsky O, Clarke L, Osterweil L (2011) On effective testing of health care simulation software. Technical Reports (CIS). https://doi.org/10.1145/1987993.1988003

Nguyen-Hoan L, Flint S, Sankaranarayana R (2010) A survey of scientific software development. In: Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’10. https://doi.org/10.1145/1852786.1852802 . http://doi.acm.org/10.1145/1852786.1852802 . ACM, pp 12:1–12:10

Post D E, Kendall R P (2004) Software project management and quality engineering practices for complex, coupled multiphysics, massively parallel computational simulations: Lessons learned from asci. Int J High Perform Comput Appl 18(4):399–416. https://doi.org/10.1177/1094342004048534

Sanders R, Kelly D (2008) Dealing with risk in scientific software development. IEEE Softw 25(4):21–28. https://doi.org/10.1109/MS.2008.84

Segal J (2005) When software engineers met research scientists: a case study. Empir Softw Eng 10. https://doi.org/10.1007/s10664-005-3865-y

Segal J (2009) Software development cultures and cooperation problems: a field study of the early stages of development of software for a scientific community. Comput Supported Coop Work 18 (5):581. https://doi.org/10.1007/s10606-009-9096-9

Vilkomir S A, Swain W T, Poore J H, Clarno K T (2008) Modeling input space for testing scientific computational software: a case study. In: Bubak M, van Albada GD, Dongarra J, Sloot PMA (eds) Computational science—ICCS 2008. Springer, Berlin, pp 291–300

Download references

Acknowledgements

We thank the study participants and NSF-1445344.

Author information

Authors and affiliations.

Department of Computer Science, Boise State University, Boise, ID, USA

Nasir U. Eisty

Department of Computer Science, University of Alabama, Tuscaloosa, AL, USA

Jeffrey C. Carver

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Nasir U. Eisty .

Ethics declarations

Conflict of interest, additional information.

Communicated by: Dietmar Pfahl

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Definitions Provided

We refer to Figs.  1 and  2 for the survey question. In this section we listed the definitions we provided in the actual survey.

Acceptance testing —Assess software with respect to requirements or users’ needs.

Architect —An individual who is a software development expert who makes high-level design choices and dictates technical standards, including software coding standards, tools, and platforms.

Assertion checking —Testing some necessary property of the program under test using a boolean expression or a constraint to verify.

Backward compatibility testing —Testing whether the newly updated software works well with an older version of the environment or not.

Branch coverage —Testing code coverage by making sure all branches in the program source code are tested at least once.

Boundary value analysis —Testing the output by checking if defects exist at boundary values.

Condition coverage —Testing code coverage by making sure all conditions in the program source code are tested at least once.

Decision table based testing —Testing the output by dealing with different combinations of inputs which produce different results.

Developer —An individual who writes, debugs, and executes the source code of a software application.

Dual coding —Testing the models created using two different algorithms while using the same or most common set of features.

Equivalence partitioning —Testing a set of the group by picking a few values or numbers to understood that all values from that group generate the same output.

Error Guessing —Testing the output where the test analyst uses his / her experience to guess the problematic areas of the application.

Executive —An individual who establishes and directs the strategic long term goals, policies, and procedures for an organization’s software development program.

Fuzzing test —Testing the software for failures or error messages that are presented due to unexpected or random inputs.

Graph coverage —Testing code coverage by mapping executable statements and branches to a control flow graph and cover the graph in some way.

Input space partitioning —Testing the output by dividing the input space according to logical partitioning and choosing elements from the input space of the software being tested.

Integration testing —Asses software with respect to subsystem design.

Logic coverage —Testing both semantic and syntactic meaning of how a logical expression is formulated.

Maintainer —An individual who builds source code into a binary package for distribution, commit patches or organize code in a source repository.

Manager —An individual who is responsible for overseeing and coordinating the people, resources, and processes required to deliver new software or upgrade existing products.

Metamorphic testing —Testing how a particular change in input of the program would change the output.

Module testing —Asses software with respect to detailed design.

Monte carlo test —Testing numerical results using repeated random sampling.

Performance testing —Testing some of the non-functional quality attributes of software like Stability, reliability, availability.

Quality Assurance Engineer —An individual who tracks the development process, oversee production, testing each part to ensure it meets standards before moving to the next phase.

State transition —Testing the outputs by changes to the input conditions or changes to ’state’ of the system.

Statement coverage —Testing code coverage by making sure all statements in the program source code are tested at least once.

Syntax-based testing —Testing the output using syntax to generate artifacts that are valid or invalid.

System testing —Asses software with respect to architectural design and overall behavior.

Test driven development —Testing the output by writing an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test.

Unit testing —Asses software with respect to implementation.

Using machine learning —Testing the output values using different machine learning techniques.

Using statistical tests —Testing the output values using different statistical tests.

Appendix B: List of Testing Techniques

This appendix provides the list of testing techniques respondents mentioned they were familiar with in response to the survey question Q9. The numbers in the parenthesis represent how many respondents indicated that testing technique.

2.1 B.1 Testing Methods

Acceptance testing (9), Integration testing (43), System testing (14), Unit testing (87)

2.2 B.2 Testing Techniques

A/B testing (1), Accuracy testing (1), Alpha testing (1), Approval testing (2), Answer testing (1), Assertions testing (3), Behavioral testing (1), Beta testing (1), Bit-for-bit (1), Black-box testing (4), Built environment testing (1), Builtd testing (1), Checklist testing (1), Checksum (1), Compatibility Testing (1), Concolic testing (1), Correctness tests (1), Dependencies testing (1), Deployment testing (1), Dynamic testing (3), End-to-end testing (2), Equivalence class (1), Engineering tests (1), Exploratory tests (1), Functional testing (6), Fuzz testing (12), Golden master testing (1), Install testing (1), Jenkins automated testing (1), Load testing (1), Manual testing (2), Memory testing (6), Mock testing (6), Mutation testing (5), Penetration testing (1), Performance testing (6), Periodic testing (1), Physics testing (1), Property-based testing (2), Random input testing (2), Reference runs on test datasets (1), Regression testing (39), Reliability testing (1), Resolution testing (1), Scientific testing (2), Security testing (1), Smoke test (2), Statistical testing (1), Stress test (1), Usability testing (1), Use case test (1), User testing (2), Validation testing (9), White-box testing (2)

2.3 B.3 Testing Tools

CTest (3), gtest (1), jUnit (1)

2.4 B.4 Other types of QA

Code coverage (16), Code reviews (2), Documentation checking (3), Static analysis (6)

2.5 B.5 Others

Agile (1), Asan (1), Automatic test-case generation (1), Behavior-Driven Development (1), Bamboo (1), Benchmarking (1), Caliper (1), Code style checking (1), Coding standards (1), Comparison with analytical solutions (1), Continuous integration (33), Contracts (1), DBC (1), Design by contract (1), Doctests (1), Formal Methods (2), GitLab (1), License compliance (1), Linting (1), Method of exact solutions (1), Method of manufacture solution (2), Monitoring production apps (1), Msan (1), N-version (1), Nightly (1), Pre-commit (1), Profiling (1), Release (1), Run-time instrumentation and logging (1), Squish (1), Test-driven development (18), Test suites (1), Tsan (1), Visual Studio (2)

Rights and permissions

Reprints and permissions

About this article

Eisty, N.U., Carver, J.C. Testing research software: a survey. Empir Software Eng 27 , 138 (2022). https://doi.org/10.1007/s10664-022-10184-9

Download citation

Accepted : 26 May 2022

Published : 26 July 2022

DOI : https://doi.org/10.1007/s10664-022-10184-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Software testing
  • Research software
  • Software engineering
  • Find a journal
  • Publish with us
  • Track your research
  • Venue: International Institute of Information Technology, Bangalore
  • Registration
  • Travel Grants
  • Student Accommodation
  • Accommodation
  • Complete Program
  • Your Program
  • 17th Innovations in Software Engineering Conference (ISEC 2024)
  • Invited Talk
  • Research Papers
  • Tutorials and Workshops
  • Doctoral Symposium
  • Software Engineering in Practice
  • Student Posters Session
  • 17th Innovations in Software Engineering Conference (ISEC 2024) Committees
  • Organizing Committee
  • Steering Committee
  • Track Committees
  • Contributors
  • People Index
  • Sponsorship

Test of Time Paper: Automatic generation of sequence diagram from use case specification

This program is tentative and subject to change., program display configuration.

small-avatar

Jitendra Singh Thakur

Indian institute of information technology, design and manufacturing (iiitdm), fri 23 feb displayed time zone: chennai, kolkata, mumbai, new delhi change.

IMAGES

  1. Types of Software Testing Explained

    types of software testing research paper

  2. 7 Common Types of Software Testing [Complete Guide + FAQ]

    types of software testing research paper

  3. 15 Different Types of Software Testing Methodologies

    types of software testing research paper

  4. (PDF) Comprehensive Study of Software Testing Techniques and Strategies

    types of software testing research paper

  5. Testing Types |Types of Software Testing

    types of software testing research paper

  6. Types of Testing

    types of software testing research paper

VIDEO

  1. SOFTWARE TESTING LIFE CYCLE||SOFTWARE TESTING SYLLABUS||TEST PLANNING||TEST CASE DESIGN Part6#shorts

  2. Objectives of Software Testing

  3. Introduction Software Testing

  4. Introduction to Software Testing Week 1

  5. Software Testing Material Introduction Video

  6. testing templates

COMMENTS

  1. Software Testing Techniques: A Literature Review

    Software Testing Software Testing Techniques: A Literature Review Conference: 2016 6th International Conference on Information and Communication Technology for The Muslim World (ICT4M)...

  2. PDF Software Testing Fundamentals—Concepts, Roles, and Terminology

    Paper 141-30 Software Testing Fundamentals—Concepts, Roles, and Terminology John E. Bentley, Wachovia Bank, Charlotte NC ABSTRACT SAS® software provides a complete set of application development tools for building stand-alone, client-server, and Internet-enabled applications, and SAS Institute provides excellent training in using their software.

  3. Research on software testing techniques and software automation testing

    Abstract: Software Testing is a process, which involves, executing of a software program/application and finding all errors or bugs in that program/application so that the result will be a defect-free software. Quality of any software can only be known through means of testing (software testing).

  4. Software Testing Techniques: A Literature Review

    Software Testing Techniques: A Literature Review Abstract: With the growing complexity of today's software applications injunction with the increasing competitive pressure has pushed the quality assurance of developed software towards new heights.

  5. PDF Artificial Intelligence in Software Testing : Impact, Problems

    Preprint submitted to Elsevier Over the past few decades, there has been substantial sig-nificant growth in the software industry driven by the re-cent advances in AI. Artificial Intelligence is gradually changing the landscape of software engineering in general [5] and software testing in particular [6] both in research and industry as well.

  6. Mapping the structure and evolution of software testing research over

    Highlights • Popular keywords relate to automation, creation/assessment, quality, and embedded. • Testing research can be divided into 16 topics and 18 subtopics. • Test creation, generation, evolution, and oracles are well-connected topics. • Emerging keywords: web, mobile, machine learning, energy, program repair, fuzzing. •

  7. A Literature Review on Software Testing Techniques

    Testing of software is an inescapable portion of software testing life cycle, and also thinking of its criticality during the pre-post process of development makes it a problem that must be dealt with improved and well-organized methodologies/techniques. A lot of studies have been conducted in order to enhance the performance of software.

  8. software testing Latest Research Papers

    pp. 1-25 Author (s): Peter J. Clarke Debra L. Davis Ingrid A. Buckley Geoff Potvin Mandayam Thirunarayanan ... Keyword (s): Social Interaction Collaborative Learning Student Learning Software Testing Traditional Lecture Face To Face Engagement Strategies Testing Class Improve Student Learning Pedagogical Approaches

  9. A Literature Review of Software Test Cases and Future Research

    A Literature Review of Software Test Cases and Future Research Abstract: Nowadays, software testing is becoming more critical to ensure that software will function properly in a dynamic environment. Consequently, different types of resources such as effort, time, and fund invested on software testing activities increase significantly.

  10. [PDF] Software Testing

    This chapter gives an overview of the entire range of software testing with suggestions for their implementation and one focus is on testing in an agile development process why the different types of software tests are important and their cycle and methodology are described. Testing software is a process of program execution with the intent to find errors. For this purpose, various testing ...

  11. A Decade of Intelligent Software Testing Research: A Bibliometric Analysis

    The paper showcased, through the keywords network graph, some types of software testing, such as regression testing and metamorphic testing. However, there are many other types of testing, such as security testing or usability testing, that can also benefit from AI techniques.

  12. PDF Software Testing: A Research Travelogue (2000-2014)

    for software testing. Where the Future of Software Engineering (FOSE) track is concerned, two such papers have appeared: Mary Jean Harrold's 2000 paper, "Testing: A Roadmap" [88] (already mentioned), and Antonia Bertolino's 2007 paper, "Software Test-ing Research: Achievements, Challenges, Dreams" [19]. We en-

  13. PDF Software Testing Techniques

    This retrospective on a fifty-year of software testing technique research examines the maturation of the software testing technique research by tracing the major research results that have contributed to the growth of this area.

  14. Software Testing Research and Practice

    The paper attempts to provide a comprehensive view of the field of software testing. The objective is to put all the relevant issues into a unified context, although admittedly the overview is biased towards my own research and expertise. In view of the vastness of the field, for each topic problems and approaches are only briefly tackled, with ...

  15. Software-testing education: A systematic literature mapping

    A systematic literature mapping (SLM) of 204 papers in software-testing education. • We review a summary of pedagogical approaches (how to best teach testing). • We review the challenges in testing education and insights on how to overcome those. Context Objective Method Results Conclusion Keywords Software testing Software-testing education

  16. Testing Research Software: A Case Study

    Background: The increasing importance of software for the conduct of various types of research raises the necessity of proper testing to ensure correctness.The unique characteristics of the research software produce challenges in the testing process that require attention. Aims: Therefore, the goal of this paper is to share the experience of implementing a testing framework using a statistical ...

  17. [2402.09171] Automated Unit Test Improvement using Large Language

    This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. We describe the deployment of TestGen-LLM at Meta test-a-thons for the ...

  18. Gemma: Google introduces new state-of-the-art open models

    At Google, we believe in making AI helpful for everyone.We have a long history of contributing innovations to the open community, such as with Transformers, TensorFlow, BERT, T5, JAX, AlphaFold, and AlphaCode.Today, we're excited to introduce a new generation of open models from Google to assist developers and researchers in building AI responsibly.

  19. A walk through of software testing techniques

    Software Testing is a process of program's execution with having the intentions of finding errors in order to attain software with null defect. For this purpose various testing technique have been used over the time. Software testing is an intense arena of research and a lot of development work has been done in the same. This field is gaining more and more of importance in the future. Many ...

  20. Sustainability

    In this paper, a self-developed multi-functional high-temperature rock triaxial servo control testing machine was used to carry out uniaxial compression tests on the granite after the cooling and heating cycles under real-time temperature. The physical and mechanical properties of two types of granite damaged by hot and cold cycling under real-time temperature were discussed, and the changes ...

  21. Testing research software: a survey

    Background Research software plays an important role in solving real-life problems, empowering scientific innovations, and handling emergency situations. Therefore, the correctness and trustworthiness of research software are of absolute importance. Software testing is an important activity for identifying problematic code and helping to produce high-quality software. However, testing of ...

  22. Test of Time Paper: Automatic generation of sequence diagram from use

    Innovations in Software Engineering Conference, ISEC (Formerly known as India Software Engineering Conference) is the annual conference of iSOFT, the India chapter of ACM SIGSOFT (isoft.acm.org) under the umbrella of ACM India. The 17th edition of the conference will be held in International Institute of Information Technology Bangalore, India. ISEC will bring together researchers and ...

  23. OpenAI will now let you create videos from verbal cues

    Artificial intelligence leader OpenAI introduced a new AI model called Sora which it claims can create "realistic" and "imaginative" 60-second videos from quick text prompts.