The Hostess Project

Goal

The goal is to provide a set of conformance tests for Oberon language consumers (e.g. compilers and language processors). The tests address issues like syntax, type semantic, declarations, operators, statements, predefined functions, and parts of the run-time system required by the language (e.g. memory allocation). Libraries are not covered by the tests.

Test Environment

A Test Environment consists of

test suite
- test cases
- test schedule
test framework
tested application

The Hostess Project provides the test suite only. The test framework is provided by the developers of the tested applications.

Test Files and Test Cases

Definitions:

[CLIENT] the program to be evaluated for conformance
[TEST CASE] the test to be performed which is a tuple
- [TEST FILE] the file to test
- [TEST RESULT] the expected outcome of the test
- [LANGUAGES] if specified, the languages for which this test applies, otherwise all languages are meant.

Submitting a test case to a client has the following result:

[SKIP] the test was ignored
[CRASH] the test causes an exception in the client
[REJECT] the test is rejected by the client (doesn't belong to the language)
[ACCEPT] the test is accepted by the client (belongs to the language)
[ERROR] execution throws an exception
[RUN] execution without exception
[OUTPUT] execution generates the expected output

ERROR, RUN and OUTPUT results imply ACCEPTED.

Ideally, a test case should cover only one particular feature. This is required for REJECTED and ERROR test cases, since one error may cover the other ones. ACCEPTED and RUN test cases should be partitioned in the same way, though this is not mandatory.

The [TEST SCHEDULE] defines the sequence of the tests and their expected result. A test is [PASSED] if the test result is the same as the expected result, otherwise the test is [FAILED].

ERROR, RUN, and OUTPUT are all more specialized cases of ACCEPT. For processors that do not produce executable code, all can be considered ACCEPT. For processors that produce a single output file, the OUTPUT variant can also be useful, of course.

Test outcomes SKIP and CRASH are always considered FAILED tests.

Support for Language Dialects

ACCEPT, ERROR and RUN test cases may be targeted at a language extension or dialect. A client for a different dialect PASSes the test if the result is REJECT, if the result is the same as the requirement the test is [TOLERATED], otherwise it FAILs.

Pseudo-code:

VAR
  cur_language: the language to be certified
  lang: the lang attribute from the test case element
  expected_result: the result attribute from the test case element
  result: the actual result of the test

IF (cur_language IN lang) THEN
  IF expected_result = result THEN
    RETURN PASS
  ELSE
    RETURN FAIL
  END
ELSE
  IF result = REJECT THEN
    RETURN PASS
  ELSIF result = expected_result THEN
    RETURN TOLERATE
  ELSE
    RETURN FAIL
  END
END

Informal Tests

An [INFORMAL TEST CASE] tests a feature that is not derived from a language specification, but can be found in existing processor implementations. The result is not relevant for the certification of the compiler (unless it is CRASH, which is a FAIL result).

Test Suite Outcomes

A test suite:

[COMPLIANT] to a language if all the tests are PASSED
[ACCEPT] a language if all the tests are PASSED or TOLERATED.

Report

A report describes the outcomes of the test suite applied to a client. A report must include:

Time / Date of execution
Test-suite Version
OS Version
Model + CPU + RAM Size
Client Version and Settings
Test Suite Outcome

Optional Information:

List of failed test cases
List of tolerated test cases
Result of the informal test cases
Result of all test cases

Implementation Details

All the test case files must begin with the prefix "HOT". The module must have the same name as the file. The maximal file name length is 32 characters. Test cases are ASCII files.

Modules to be executed must implement a command "Test", which is called by the framework when running this test case.

ASSERT(c, n) and HALT(n): 20 <= n < 256.

License

The test suite is released under the GPL.

Acknowledgments

Patrik Reali
Michael van Acken