Questions about devising tests

Would you tell us how much it costs to develop a new test or assessment?
Currently we are not able to provide details concerning this matter. Multicheck is a classical small business which so far has mainly put its effort into developing aptitude assessments. The management and accounting information systems are still in the very early stages of development.

What is the minimum number of questions per scale necessary to gain meaningful results?
In the personality questionnaires we usually begin with 15 questions per scale during the testing phase and aim to reduce this to 10 questions (in our opinion the number should not be smaller). In the Multichecks the number varies from 10 to 80 per theme (depending on the type of questions).

How long does it take till a finished test product is developed from a test idea?
About nine-months elapsed between the test concept and the successful introduction of the product. The market does not wait!

For whom do you develop tests/who are your customers and how do you proceed in the development of tests?
The Multichecks were developed at the request of associations of apprentice supervisors and large companies (for example CS, UBS, Coop, Migros). Together we chose the fields to be examined and also included questions in the tests which they had used beforehand. As we already have decades of experience with vocational aptitude analyses at the “Feusi Bildungszentrum” we have a great pool of questions available.

How long is the half-life of a test?
We change individual questions concerning the knowledge which was acquired at school every year and after two to three years we replace a theme with a new one. For the questions relating to potential (ability to think logically, concentration etc.) we replace individual themes after three to four years.

In the event of a very bad reliability and validity of questions: How do you handle poor results and what conclusions do you draw from them? Does MC reformulate and retest the questions or are they even removed from the questionnaires?
If the pool of good questions is sufficient we might remove such items completely. However, usually we replace such questions with new ones and review them again the following year.

Questions concerning the test quality

Are quality and usability examinations (reliability, validity, predictive accuracy etc.) also performed on existing analyses, for example the Multicheck vocational aptitude test? If so, what are the results?

Concerning Multicheck commercial sector: In his master thesis, Marcel Widmer of the University of Berne analyzed the Multicheck Commercial sector. The result led to reviewing and redesigning this specific Multicheck. A comparison between the Multicheck and the qualification analysis results shows good results regarding sensitivity and acceptable correlations. However, one has to consider that we only received statistical data from banks, and obviously, only results from people who successfully completed their apprenticeship.

Concerning Multicheck retail sale: We are currently examining this specific Multicheck on the predictive accuracy with data from the two large retailers Migros and Coop. The results are even slightly superior to the Multicheck for the commercial sector.

Concerning the other Multichecks: In the other MC we have concentrated on standardization and fixing the level of difficulty for the questions. The results of this study will be published in a report by the end of February 2009.

According to „Rauchfleisch“ (2005), reliability is the basis of validity. In this context it is necessary that changes should be included with time. Are reliability examinations carried out according to this demand? If so, how often and with what results?
To be precise this would imply accomplishing a reliability examination, a parallel test and a split-half method for every test in order to gain a valid reliability value. Obviously this would not only be very time-consuming but also very expensive.

How do you fulfil the quality requirements for the reliability? Are there any minimum standards which a test has to fulfil?
The above mentioned methods are very time-consuming which is probably one reason why such tests are generally developed in universities. Multicheck cannot expend such an effort. Thus we are in favour that analyses of our products are carried out in bachelor theses for example. Furthermore, in this way the neutrality of the investigation results is guaranteed.

Do you verify the results of your tests regarding the predictive validity over time? Do you have standards, requirements or methods concerning quality assurance?
We have verified the predictive validity for the Multicheck Commercial and Retail sale sectors. For such verifications we depend on the collaboration of the specific companies which can provide us with the necessary data. However, for reasons of data protection this is complicated. Even though strictly speaking, Multichecks are not conventional tests, it is nevertheless still our aim to fulfil the criteria for test quality.

Does the test central establish criteria which have to be fulfilled in order for you to accept a new test? What are these criteria?
We do not collaborate with the test central and therefore they cannot make regulations for us. The “Fachgruppe Diagnostik des SDBB” publishes “Labels” for different tests and also has released a “Label” on Multicheck, however without ever having seen a Multicheck.

In some companies the selection of apprentices regarding the Multicheck result is regarded critically since the result only represents a “snap-shot”. How does MC consider the fact that an evaluation of the professional potential may depend on the moment at which the MC was completed?
This issue arises for every test and exam. For this reason it is possible to repeat the Multicheck, in case “a bad day” was responsible for the outcome of the result. Experience has however shown that this “snap-shot” does in fact present a very reliable picture.

Does MC compare its questionnaires to other existing, scientifically verified questionnaires, for example the “FPI” in case of the achievement motivation analysis?
The “classical Multichecks” were not systematically compared to other performance questionnaires. We were able to fall back on our experience with internal, school-related assessments and additionally used questions out of tests from apprentice supervisors. The results of the validity investigations between Multicheck and other qualification assessments show that regardless of the unscientific approach, Multicheck is a valid method. However we want to examine and compare the personality questionnaires carefully with results from other verified methods. For this reason we are very interested in work done in a study at the FHNW.

Is it not possible to prepare oneself specifically for the Multicheck, maybe by solving classical intelligence tests?
We recommend the following preparation:
- Revision of school work
- Solving the question examples and trial test versions supplied free of charge on the website from Multicheck
- You can find countless intelligence tests on the internet with logical reasoning or concentration questions etc. which allow you to practise.
However, often the results of such tests however do not fulfil scientific requirements. All the same you can gain a certain amount of training by doing classical intelligence tests, because the range of types of questions is quite similar in most intelligence tests.

Questions about the random samples

How strongly are gender-related differences considered whilst developing the tests (for example the mathematical and linguistic differences in aptitude assessments)?
We follow the requirements for the apprenticeship/job. The gender of the job applicant is completely irrelevant.

Concerning the achievement motivation analysis: We assume that mainly young people take the Multicheck test after completing the obligatory time at school. In the scale of the achievement motivation I therefore think that the questions are out of touch with reality. I do not believe that young people will affirm the statement “Sometimes my family suffers when I work long hours”. Firstly they have scheduled working hours and secondly they care a lot about friends and family at this age. Hence I find it difficult to recognize achievement motivation here.
Analyses of the personality are intended for people who are about to finish their initial training or already have completed it and would like to gain further qualifications. Therefore the questions are related to the working environment.
We do not believe personality analyses to be meaningful for people during their phase of choosing an occupation. The personality may change a lot during this time and adolescents very often dislike such personality questionnaires, in which case the answers are not reliable.

What are your criteria for choosing a pre-test or comparison group to gain control data? What dimensions does a random sample need to have, to be useful and to be able to draw serious conclusions from it?
Thanks to our collaboration with the “Feusi Bildungszentrum” in Berne we have the possibility to test people from various age-groups and educational levels (from the fifth class up to universities of applied sciences). We also have the opportunity to carry out public trial tests in other schools and are able to use first year apprentices as control groups. The first pre-examinations are undertaken in quite small groups (about 50 people). The results of the first 100 people are used for a subsequent adjustment, the results being released only afterwards. Thus we achieve a sufficient number.
After one year the adjustments are checked on the basis of the results of a whole year. Our numbers for the last year:
Commercial sector: ca. 12000 people
Retail sale: ca. 10300 people
Technical: ca. 4800 people
Industry: ca. 2600 people
Health services: ca. 2600 people
Beauty: ca. 250 people (new)
However we cannot use students from the “Feusi Bildungszentrum” too often for trial tests as this would interfere with the regular schooling.

Questions about the Multicheck company

What qualifications are required to be able to work in your company?
You need to be trained in one of the following fields or professions:
- Psychology or career/occupational counselling
- Teacher
- Marketing / Business administration
- Information technology
Furthermore it is essential to be keen to work in that specific professional area.

What advice can you give a company which would like to make use of MC? How should the results be handled if for instance out of 20 job applications with MC results the company would like to employ five candidates?
The company should examine all the aspects of an application as recommended by Multicheck:
- Letter of application and CV
- Multicheck
- School reports
- References
- Trial apprenticeship
- Job interview
Besides the overall Multicheck result it is important to also examine the detailed results of the test. Multicheck offers seminars for HR managers to learn how to interpret the certificates correctly.

General questions / Questions to the test experts

As a test expert can you give us hints how to distinguish at a glance a reputable from a dubious test? What values are most important for you if you personally attend a test? What do you pay especial attention to?
I think the following points are very important:
- Is it made clear to me what the aim of the test is?
- Once I receive the result, do I also receive information on how to interpret my results?
- How many questions lead to the test result? If, for example, out of 30 questions and statements a report of 20 pages is developed, I doubt that the test is especially meaningful.
- The more promising a test sounds, the more sceptical I get…