testing

Artifact Flow Teamwork Sequence DiagramPop-up in a separate window or download this Visio Use Case diagram.This shows the flow of deliverables among major participants (the stick figures in the use case diagram above).The darker vertical lines illustrate the principal exchanges of artifacts of information — the teamwork necessary among participants.Each artifact's color identify one of the 4 metalayers (packages) of abstraction defined by the UML 2.0 Test Profile (U2TP) standard:1. Test Architecture defining concepts related to test structure and test configuration (containing the relationship of elements involved in a test project)2. Test Behaviors defining concepts related to the dynamic aspects of test procedures — structural (static) Model -> Test Execution Directives (The interface for testing)3. Test Data (the structures and meaning of values to be processed in a test)4. Test Time defining concepts for a time quantified definition of test procedures (constraints and time observation for test execution)The word "Profile" in U2TP means that it conforms to UML standards.The seven basic types of testing as defined by SourceLab's "CERT7" are:Type of Testing1. Unit Testing2. Functional Testing3. Security Testing4. Stress Testing5. ScalabilityTesting6. Reliability Testing7. Integration TestingTesting TerminologiesMercury Interactive's Quality Center (formerly TestDirector) product organizes each requirement for testing as a test subject for each AUT under a Test Plan Tree hierarchy. Both manual and automated scripts can be specified in TestDirector. Each test script selected from the tree becomes a Test Step in a Test Set actually executed by TestDirector.In Rational's TestManager, a test plan contains test cases organized within test case folders. These are read by Rational's ClearQuest defect tracking system.Why Bother with UML Test Profiles?You can describe a system precisely using UML. That's the reason why UML was invented. The visual representation of test artifacts aims for common and thus hopefully unambiguous intrepretation of test designs.UML is the language (lingua franca) "spoken" by "professional" system architects and developers. Testers need to understand the Model Drive Architectures (MDA) that they design and build.The UML standard Test Profiles includes specification of plain-text XML which (at this point, theoretically) enables tool independent interchange of test profile information.Soon, test tools will require testers to augment UML created by architects to specify testing at a higher level of abstraction instead of crafting scripts as automation testers do now. Testers will specify executable UML action semantics which are automatically compiled into platform-specific test components used to conduct testing.The problem with UML's MOF-based Metamodel for Testing is that its object-oriented approach is not presented in a sequential way. That's what this page aims to do.Test OutputsThe purpose of testing is to obtain information needed to make decisions about a System Under test (SUT).Test LogsWith UML: A Test Log is an interaction resulting from the execution of a test case. It represents (remembers) the different messages exchanged between test components and the SUT and/or the states of the test components involved.A log is associated with verdicts representing the adherence of the SUT to the test objective of the associated test case.The names of test log files usually differ by vendor:Logs output by the testing tool:WinRunner, test LogsLoadRunner output.txt and run logs.Logs output by scripts within test tools.Java JVM verbose logsWindows OS Application logs and Security logs.An application's stdout and stderr files.Test ResultsTest Results are Key Measures in TestNumber ofTime needed to run scriptWork Load Analysis ModelSee Performance Testing using LoadRunnerTest Evaluation SummaryPerformance Reports and Enhancement Requests, Defect Reports, or something.Test Plan SectionsTest PlanTest Interface SpecsWith U2TP, a test suite has a constraint: it must contain exactly one property realizing the Arbiter interfaceA structured classifier acting as a grouping mechanism for a set of test cases. The composite structure of a test suite is referred to as test configuration. The classifier behavior of a test suite is used for test control.Test Environment ConfigurationTest Automatation ArchitectureTest ClassesTest ScriptsStarting from the upper right corner of this diagram from the UML 2.0 Test standard document: The System Under Test (SUT)The system under test (SUT) is a part and is the system, subsystem, or component being tested. An SUT can consist of several objects.In UML testing profiles, the system under test (SUT) is not specified as part of the test model, but as a <> which the test architecture package imports the complete design (UML) model of the SUT in order to get the right to access the elements to be tested.The SUT is exercised via its public interface operations and signals by the test components.It is assumed that no further information can be obtained from the SUT as it is a black-box.Properties of the SUT<>Test ArchitectureTest ElementsThe architecture section defines two test elements: test suites and test components.A test suite contains :a collection of test casesan instance of the arbiter interface, normally an instance of the SUT, and(optionally) high level behavior used to control the execution of the test cases.Test components are the various elements which interact with the SUT to realize the test cases defined in the test suite.SchedulerThe Scheduler is a predefined interface defining operations used for controlling the tests and the test components.startTestCase() finishTestCase(t : Test Component) createTestComponent(t: TestComponent)To AGEDIS[3], the Scheduler is a property of a test context used to control the execution of the different test components. The scheduler keeps information about which test components exist at any point in time, and collaborate with the arbiter to inform it when it is time to issue the final verdict. It keeps control over the creation and destruction of test components and it knows which test components take part in each test case.Test Validation ActionValidation actions are performed by a test component. It sets the local verdict of that test component. Evaluation actions evaluate the status of the execution of a test case. The action assesses the SUT observations and/or additional characteristics/parameters of the SUT.Every validation action causes the setVerdict operation on the arbiter implementation to be invoked.Test VerdictA Verdict is an assessment of the correctness of the SUT.AGEDIS notes that a verdict is a property of a test case or a test context to evaluate test results and to assign the overall verdict of a test case or test context respectively. Test cases yield verdicts.Verdicts can be used to report failures in the test system. Predefined verdict values are:Pass - indicates that the test behavior gives evidence for correctness of the SUT for that specific test case.Fail - describes that the purpose of the test case has been violated.Inconclusive - used where neither a Pass nor a Fail can be given.Error - used to indicate errors (exceptions) within the test system itself.Verdicts can be user-defined. The verdict of a test case is calculated by the arbiter.ArbiterWith U2TP, an Arbiter is a predefined <> defining operations used for arbitration of tests.Test cases, test contexts, and the runtime system can use realizations of this interface to assign verdicts of tests and to retrieve the current verdict of a test. The arbitration algorithm can be user-defined. Some sample operations:getVerdict() : Verdict Returns the current verdict.setVerdict(v : Verdict) Sets a new verdict value.There is a default arbitration algorithm based on functional conformance testing which generates Pass, Fail, Inconc, and Error as verdict, where these verdicts are ordered as Pass < name="TestArchFullz">Starting from the middle of this diagram from ___ presenting at the U2TP Consortium:The Test ArchitectureTest ClassTest PackageCollaborationTest CasesA test case is a specification of one case to test the system including what to test with, which input, result, and under which conditions. It is a complete technical specification of how the SUT should be tested for a given test objective.A test case is defined in terms of sequences, alternatives, loops, and defaults of stimuli to and observations from the SUT. It implements a test objective. A test case may invoke other test cases. A test case uses an arbiter to evaluate the outcome of its test behavior.A test case is a property of a test context. It is an operation specifying how a set of cooperating test components interacting with a system under test realize a test objective. Both the system under test and the different test components are parts of the test context to which the test case belongs.Test SuiteA collection of test cases is called a Test Suite.Test OperationsTest ContextWith U2TP, each <> is a structured classifier acting as a grouping mechanism for a set of test cases.The composite structure of a test context is referred to as a test configuration.The classifier behavior of a test context is used for test control.Each test context must contain exactly one property realizing the Arbiter interface and the Scheduler interface.Test ConfigurationA Test Configuration is the collection of test component objects and of connections between the test component objects and to the SUT. The test configuration defines both (1) test component objects and connections when a test case is started (the initial test configuration) and (2) the maximal number of test component objects and connections during the test execution.Test PartA part of the test system representing miscellaneous components that help test components to realize their test behavior. Examples of utility parts are miscellaneous features of the test system.Test UtilityStructured ClassifierTest ComponentsWith U2TP, a test component is a structured classifier participating in test behaviors. A test component is commonly an active class with a set of ports and interfaces. Test components are used to specify test cases as interactions between a number of test components. The classifier behavior of a test component can be used to specify low level test behavior, such as test scripts, or it can be automatically generated by deriving the behavior from all test cases in which the component takes part.Test component objects realize the behavior of a test case.A test component has a set of interfaces via which it may communicate via connections with other test components or with the SUT.A test component object executes a sequence of behaviors against the SUT in the form of test stimuli and test observations. It can also perform validation actions, and can log information into the test trace. Whenever a test component performs a validation action it updates its local verdict.Test Behaviors PackageTest DefaultsTest Defaults are the set of concepts (in addition to the UML 2.0 behavioral concepts) to specify test behaviors, their objectives, and the evaluation of systems under test.Test ControlsA test control is a specification for the invocation of test cases within a test context. It is a technical specification of how the SUT should be tested with the given test context.Test EventsTest Case InvocationA test case can be invoked with specific parameters and within a specific context. The test invocation leads to the execution of the test case. The test invocation is denoted in the test log.Test ObjectivesA test objective is a named element describing what should be tested. It is associated to a test case.Test StimulusA stimulus is an item of test data sent to the SUT in order to control it and to make assessments about the SUT when receiving the SUT reactions to these stimuli.Test ObservationAn Observation is Test data reflecting the reactions from the SUT and used to assess the SUT reactions which are typically the result of a stimulus sent to the SUT.CoordinationConcurrent (and potentially distributed) test components have to be coordinated both functionally and in time in order to assure deterministic and repeatable test executions resulting in well-defined test verdicts. Coordination is done explicitly with normal message exchange between components or implicitly with general ordering mechanisms. Default Default is a behavior triggered by a test observation that is not handled by the behavior of the test case per se. Defaults are executed by test components.Test Log ActionAn action to log information in the test log.Test CaseTest Data PackageIn general, test data refers to the specification of types and values that are received from or sent to the SUT. Data can be specified as being static or dynamic, where:static data refers to the definition of types, and values given as arguments or read-only attributes.dynamic data refers to the manipulation of values during the execution of a test behavior.UML 2.0 does not have a concrete syntax for values and expressions, but does allow the use of OCL (Object Constraint Language).Test timingsParticipantsTest TeamFocusRoleLast NameFirst NameLocationPhoneCell, etc.ResultsManagementTest ManagerFunctionality (Subject Matter) SpecialistsTechnicalManagementTest ArchitectTechnicalProcessesTest LeadTest Automation SpecialistTechnologistsGUI TesterWebsite TesterWireless TesterDatabase TesterHW Driver TesterSecurity SpecialistDevelopment TeamFocusRoleResultsManagementDevelopment Manager (Executive)Project ManagerTechnicalManagementSystem/Product ArchitectTechnicalProcessesDev. LeadConfiguration/Build ManagerSecurity SpecialistTechnologistsGUI DeveloperComponent DeveloperDatabase DeveloperWireless DeveloperHW Driver DeveloperOperations TeamFocusRoleResultsManagementFacility Manager (Executive)Operations ManagerTechnicalManagementNOC ArchitectTechnicalProcessesShift LeaderConfiguration/Build ManagerSecurity SpecialistTechnologistsImplementerSchedulerBackup/Recovery SpecialistDBACapacity/Performance SpecialistTesting Participants' RolesSubjects of TestingThis list of quality metrics should be meaningful to testers, developers, and managers -- as this list is often used by testers to report to management an estimate of the completeness of development on application features.Product IntegrationProduct installability (From CD or other drive, CD copied to a local folder, CD copied to a network drive, etc.)Application TestabilityApplication Functionality (detailed below)Application Performance (Turnaround Cycle Time, Capacity of transactions handled per minute)Product Security:AuthenticationSite Administrators (used for doing configurations)Users (Customers) - Normal and SuperUsersBackup operatorsCorporate AdministratorsManagersGuests (temporary employees, consultants, auditors, etc.)Automated agents (doing backup, restore, etc.)AuthorizationLogin/LogonMain menu and sub menu presentationLogout/LogoffIndividual menu item invocationSpecific functions (Printing, Approval, etc.)AuditSecure TransmissionBackward and forward compatibility. Can files created in a prior version of the product be opened in a current version?Backward and forward interoperability. For two products that use data created with the other product, will files created in an older version of one product open in a newer version of its companion product?These may be separated into client and server tiers.Test Guidelines & StrategiesFirst, test for installability. From CD, from CD copied to a local folder, from CD copied to a network drive.Test for main-line functionality (such as logon, menu selection, logoff)It is usually not possible to sequentially test all menu items (such as "File, Edit, View, Format, Tools, Help," etc.) because scenarios of actions are necessary to establish pre-conditions for testing. The most efficient approach is to reuse the script to test a predecessor function to establish the pre-conditions to automate testing for dependent functions requiring pre-conditions.Test PrioritiesWhen a requirement is added using Mercury TestDirector, the product requires specification of a Priority:1 - Low 2 - Medium 3 - High 4 - Very High 5 - UrgentNote: WinRunner does not recognize the F4 key normally used to drop an activated list.Define examples of what each priority level means in your organization and when each value is appropriate and not appropriate.Test IdeasSingle point of action (a single screen)Action: Mnemonic “CRUD”Create (Add) -- New data percolated to subsidiary tables?Initial AddAdd same key again (catch duplicate?)Add another key (to test for conflicts with previous add action)ReadSearch for keywordsBrowse sequentiallyRead Previous when already at the beginning (negative test)Read NextRead Next when already at end (negative test)Read FirstRead LastUpdatePercolated to subsidiary tablesDeleteDelete again (attempt)Recreate (to see if subsidiary tables have removed effects of previous delete)These actions are performed under these different circumstances:InstantiationEmpty conditionSingle instanceMultiple instances (for tables allowing non-unique keys)All the above are repeated for each hierarchical configuration:HierarchyNo parentno possible parents populatedpossible parents populatedSingle parentMultiple parentsSkip generation (grandfather with no parent)Dual generation (grandfather and father)Cyclic (child is also parent of another parent)Each additional origin of action (performed from another screen)Same actions as aboveAll the above are repeated for each data of these states:Validity:All validOnly Child validOnly Parent validOnly GrandParent validPortions ©Copyright 1996-2007 Wilson Mar. All rights reserved. Privacy Policy Last updatedRelated Topics:· Software Testing· Test Analysis· Defect Reporting· Data Driven Testing· Transition Testing· Free Training!· Technical Support