We’ve tested FutureSelves extensively and researched its effectiveness
with clients, counsellors and case managers.
An approach that benefits all stakeholders
Counsellors and case managers like it because it opens up their clients faster, which enables better-targeted assistance.
Clients like it because they feel more motivated and better about where they are going.
Counsellors and case managers like it because it helps clients overcome barriers and address what’s stopping them from making progress.
Measuring the effectiveness of FutureSelves
Our studies have found that counsellors rate FutureSelves as better at dealing with whole of life issues, motivation, and balance and tradeoffs with different parts of life. It is also particularly good for dealing with the barriers that are stopping people.
A sample graph from one of our studies (right) shows how FutureSelves compared against general career counselling with adult career development clients.
Validity of the FutureSelves instrument
We use a range of different methods and approaches to assessing and improving validity. We agree with the following statement:
“The subject of validity is complex, controversial, and peculiarly important in behavioural research” (Kerlinger, 1986, p.416).
We also agree with a point made by the psychometrician Kline that instruments are often wrongly judged valid by their measurement characteristics rather than by their usefulness, which can require a different set of criteria (Kline, 2000). Most of our research is on the effectiveness of the instrument to encourage change and development, as well as the validity of a tool as a measure.
Face Validity
We conduct focus groups, surveys and interviews with users as diverse as long-term unemployed, career practitioners, case managers and university students to ensure people respond to it as an accurate reflection of their self-concept.
Criterion & Concurrent Validity
These two similar concepts concern how well it relates to similar constructs. For example, we measured the concurrent validity of an adult career transitions item bank and found it had high concurrent validity with self-esteem, optimism/pessimism and life satisfaction.
Content Validity
This concerns judgement about whether the items adequately reflect what is being measured. What this means in practice is that content validity is a lot about how items are developed and chosen.
Items for FutureSelves are developed using
a range of different techniques, including a combination of the following:
Surveys
Surveys are in the form of questionnaires, focus groups or interviews with users exploring their hopes and fears. These are then analysed – for those technically minded, we use template analysis.
Expert generation & judgement
The use of expert generating items and vetting those that exist. For example, one group of experts generated a list of items in the relationship section of the life coaching item banks, and these were then checked and screened by a second group.
Item analysis
This involves analysing how people respond to certain items or potential items. Items that are ‘unpopular’ or redundant can then be removed.
Comparison against instruments
Checking to ensure items are consistent with other research that has looked at “what has been measured.” For instance, items in the “Personal beliefs and values” section were checked against the Profile of Mood States.