No systematic (regression) testing done yet

[Christopher Kramer]

Originally reported on Google Code with ID 126

We don't really do any really good systematic testing yet, especially regression testing
is hard because not yet automated.

We should introduce automated test cases. The only solution for web-applications I
know is Selenium http://seleniumhq.org/.

Once we have a good base of test cases, it is a lot easier to check whether a certain
change breaks something (else).

And I think Selenium is really cool I'd really like to use it for this project ;-)

Reported by crazy4chrissi on 2012-10-30 23:19:32

[dreadnaut]

We could also do non-web testing, once we have a bit more structure in the code. We
can add a noop parameter so we can include() the main file in other scripts, and run
a normal suite of tests.

Reported by dreadnaut on 2012-11-22 19:54:38

[Christopher Kramer]

Although normal "non-web-testing" is fine to test a bug part of the code, some things
in the end cannot be tested.

The interplay of JavaScript and PHP for example is almost impossible to test.
And all the other UI-stuff is a lot easier to test with browser-based tests like Selenium.

Of course we could have unit-tests to test our classes and Selenium-tests to test the
UI.
But we need to restructure the code quite a lot before we can do good unit-testing
I'm afraid.

Reported by crazy4chrissi on 2012-11-22 22:47:48

[Christopher Kramer]

Lol. "a bug part" was meant to be "a big part" ;-)

Reported by crazy4chrissi on 2012-11-23 21:11:39

[dreadnaut]

Just noticed that PHPUnit can work with Selenium¹. I was then going to suggest PHPUnit
as a testing framework, but the latest versions requires either php 5.3.3 (3.7) or
5.4.7 (3.8). It's actually becoming difficult to find non-namespaced code (< 5.3).

Until now, I've tried to keep everything 5.2 friendly, but it would be easier to have
different version requirements for main (single file/split file) code and code that
is only used for development or support (build script, tests).

Preferences?

[1] http://www.phpunit.de/manual/3.7/en/selenium.html

Reported by dreadnaut on 2013-04-06 14:12:01

[Christopher Kramer]

I think it is fine to require php 5.3.3 or even 5.4.7 for test scripts. The problem
with php versions is that you cannot update php yourself _on shared hosting webservers_.
But you usually do not _develop_ code on a shared hosting webserver but on a localhost.
Therefore, you can easily update php yourself in your development environment.
So in my opinion, it is definitely okay to require recent php versions for test scripts.
I mean, you can still modify phpliteadmin yourself with older php version, you just
cannot run the tests.

I am not sure if the build script is exactly the same. Although only used by developers,
not being able to run the build script means that you'll have a hard time modifying
parts of PLA like the JS. I'd have a better feeling if the build script did not require
php 5.4, but at most 5.3. Otherwise we might stop some people from modifying PLA.

PHPUnit definitely is the right way to test php units. I did not know so far that it
also supports selenium. This makes it even cooler, so we cannot only test logic units
with it but also test GUI including browser behaviour.

Reported by crazy4chrissi on 2013-04-06 14:25:30

[dreadnaut]

I'll start using PHPUnit for new classes then, we'll see how it goes. In case anyone
in interested in the setup, I'm installing it through Composer and keeping tests files
under 'trunk/tests/'.

composer.json
{
    "require-dev": {
        "phpunit/phpunit": "3.7.*"
    }
}

In the future, I won't commit anything under vendor/ to repository.


[For fun]
We could also experiment with BDD and Behat (http://docs.behat.org/) which also supports
Selenium :)

Reported by dreadnaut on 2013-04-06 15:02:41

[Christopher Kramer]

Okay. Let's see.
I think when I find the time, I will need to create a test suite for the alterTable
function (currently part of the Database class - famous issue #12). When I modify it,
I always fear that changing a regex slightly might crush other cases which I currently
don't have in mind. Having a good test suite would be really helpful for regression
testing there. In fact I consider this method the only one in PLA that really has some
kind of complicated logic ;-)

Test cases for this should be easy to create, a CREATE TABLE statement of the initial
table, an ALTER TABLE statement for the alter operation and a couple of post-checks,
mainly comparing the actual result of SELECT table_info() with the expected one.

Reported by crazy4chrissi on 2013-04-06 16:00:35