Skip to main content

GUI Testing... requirements

Alsalam alikom wa ra7mat Allah wa barakatoh

Today, I was investigating -along with one of my friends- in Open Source GUI Testing tools... honestly there are quite a lot of them but non are smart enough.

Let's first define the goal of a GUI Testing Tool..
  1. Automated capture of different user scenarios,
  2. Generate editable script for different scenarios,
  3. Goal setting: define the success/failure state of a test run,
  4. Integratable with other Testing tools (NUnit, JUnit... etc) to provice a complete report about the test run,
  5. Requires 0-code changes,
  6. Can generate valid random inputs for testing (Textbox values mainly),
  7. Capture screenshots if something failed/succeeded/always,
It's a big list, I know, and no free/opensource framework provides all (or even well implemented subset) of them.

My Investigation scope didn't extend to commerical products (which I believe will have some more functionalities/polish interface...etc).

Here is a list of some apps that may cover some of these features for .NET Winforms:
  1. ADLab Quail:
    • Built on Windows Workflow Foundation technology
    • No actions recording capabilities
    • Able to take a snapshot of any control (a snapshot is a copy of the state of all/subset of properties)
    • Goal setting by defining the desired state after specific action (button press ... etc)
    • Requires adding 2 lines of code to the testee application to launch the testing tool
  2. NUnitForms:
    • Integratable with NUnit famous unit testing tool,
    • Supports recording actions and converting them to C# code
    • Goal setting by adding assert statements
    • Supports running script to automate testing (script and c# code are NOT interchangable)
    • Can capture screenshots
  3. White:
    • Easy to use .NET Lib for writing automated UI testing
    • Easy to integrate with any Unit Testing Tool
    • No recording
    • No Screenshots
  4. Test Run:
    • Free MSDN magazine code :)
    • Lower level .NET code for writing automated UI testing (lower than White)
    • No Recording
    • No Screenshots
    • A LOT of code

MS's released a new testing tool called UI Automation, that supports WPF only... I didn't check it :)

As you can see, non of the above tools support our simple-7 rules, there are some available apps that uses Genetic algorithms to generate reasonable valid inputs but they do nothing but this :)

I believe the field is still widely open for those who are willing to contribute and put some innovative ideas over there.

Thanks,

Comments

  1. It looks like an "ideal" list :D
    - momken nsheel el point bta3et "Requires 0-code" ne7'alleha, requires less code...
    in case of putting only assertion code that would be fine, (the other graphical option, is setting the form values, and clicking the "Valid" -button with a green check icon-, to identify a valid state, and vice-versa. I never heard about an easy tool :D )

    - Integration is a perfect point, and even makes unit testing easier
    - Generation of random inputs, looks be a "very" sophisticated option! :D ...not mandatory

    Nize Bost

    ReplyDelete
  2. "ideal" list, that's exactly what it meant to be :D... I was trying what a GUI testing tool would look like in the perfect world...

    - 0-code is the optimal, less code is nice :D
    I agree that currently we can deal with a few lines of code... :D

    - ..

    - Actually I read a paper (http://sigchi.org/chi96/proceedings/papers/Kasik/djk_txt.htm) which proposes to use Genetic Algorithms to do this heavy task..
    I agree it's very sophisticated but this's how human needs evolve ;)

    Thanks,

    ReplyDelete
  3. Modammed,

    I'm an author of ADLab Quail.

    Next release will have capturing screenshots build-in although it's easy to add it by yourself right now. Complete randomizing text inputs is more complicated and less useful then it's looks like, because you need to know which of random inputs suppose to fail (succeed), or which actions they need to cause.
    I do, however, like idea of making a certain type of input easier (i.e. put random literal string to numeric only control etc). Not sure if it's needs to be a menu in design time or special activity.

    As far as recording capabilities, I personally do not like it because it's makes too easy to create huge test scripts that clattered with unneeded code and, after changes in software, practically impossible to modify. I do prefer small, targeting scripts and Quail made just like I wanted it :)

    Nevertheless, I really appreciate your input. if you think of any other features you would like to have please let me know.

    Ilya

    ReplyDelete
  4. Ilya,

    I would like to thank you for taking time to post a comment here.

    - What I meant by capturing screenshot is not just a simple button to do it.. I meant to automatically capture the steps that lead to a Fail state (or success)... this way, it'll be easier to report a bug (along with the steps to regenerate it) later... if this can be added to the next release, it'll be a GR8 addition

    - In some certain type of software, it's really important to automatically generating a LOT of valid inputs, to test performance/fault tolerenace... etc. I agree that generating all possible cases for the input may seem really complicated to achieve... and this is why men should do their job.

    - Recording capabilities, your neat software doesn't generate scripts, that's why it's unique in design, what I imagine the recording will be is -thanks mohammed for the green icon thing-:
    * I'll right click on my project in VS, choose Generate Quial Workflow
    * My applicaiton will show up, with a red circle somewhere in the screen to tell me "I'm recording"
    * I'll start reacting with the application in 2 ways:
    *** Either by clicking a button/menu..etc and this will generate an Action Activity
    *** Or, Selecting a control, contents, dialog then clicking the right green button or cross red button to indicate this is a good behaviour or a bad one..
    * I'll stop the recording and the workflow will be generated for me

    That may seem complicated but I feel it's still controllable and will enable us to easily generate test scripts...

    I appreciate your reply,
    Thanks,

    ReplyDelete

Post a Comment

Popular posts from this blog

Exploiting Double Free Vulnerabilities...

Alsalam alikom wa ra7mat Allah wa barakatoh



Huh!! that's what I said when I first saw that title... but let me explain...


Double Free means that you try to free a pointer two times (which is logically can't work).
Actually windows SP2 and later (even Vista) this can be done (in somehow) and can actually corrupt the heap (Vista will shout at your face if u did) and that can make you able to use and browse the heap as you want..

Facts to know about how Windows frees your pointers:
- There is something called Lookaside buffer (fast access, small size) and another thing is FreeList(slower access, the whole memory).
- Chunk is an object of the DataStructure that holds mainly 2 things: pointer to where the next free Chunk is and pointer to the previous free one (think about it like a node in a linked list)
- The first 4 bytes of the Chunk is the BLink (BackLink) and the second 4 bytes is the FLink
- delete ptr1;
delete ptr2;
Windows takes your Chunk (for ptr1) and puts it in the Lookaside s…

Windows7 adds Math Input Panel

Alsalam alikom wa ra7mat Allah wa barakatoh…I was reading a windows team post about Input Panels improvements in Windows7 [here]. When at the end I saw a very interesting –intuitive if you wish- new thing… which is, as you guessed, the Math Input Panel…Yes, that crappy font is mine… I “drew” that by mouse as I don’t have a tablet pen/pc.You can then paste it directly into word and it’ll recognize it as an editable equation…During my tests, the output panel (the top part) hanged, but I liked that the drawing panel was still responsive and I could still write/erase… till the top one started to respond again…One other thing to know, after you click Insert (that button down there) it copies the equation in MathML [Wikipedia link] format.. which is a standard way of representing equations and hence any application that recognizes the format can insert it not as an image but as a nice editable equation…If you think it recognized something wrong, you can click “Select and Correct” then draw …

Visual Studio 2008 Not saving changes or project properties?

Alsalam alikom wa ra7mat Allah wa barakatoh (Peace upon you)I’ve recently ran into problems with VS 2008. Summarized here:When you try to edit the project properties (specially C++ projects) you are faced with a little nice message saying “Exception from HRESULT: 0xF9F0F308”. Sometimes when you are editing a file (specially large ones), VS doesn’t recognize you’ve made changes (ie doesn’t display that ‘*’ in the files tabs) hence, when you save, nothing actually gets saved. For those 2 problems, a friend explained the problem and a work around (till they officially release a fix)…Open up a Visual Studio 2008 Command Prompt Run cd "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE" Make a backup copy of devenv.exe in case something does not work right.
ie. copy devenv.exe devenv.exe.bak Run editbin /largeaddressaware:no devenv.exe Happy VSing… :)