How Artificial Intelligence is Transforming Assessment

Centre Simulations: Does an AI-driven Simulation

Produce the Same Predictive Validities as the Traditional
Methods.

Dr. Pieter Bronkhorst

[email protected]
 Agenda
• The impact of technology on the talent assessment landscape

• The origins of Artificial Intelligence (AI)

• What is AI (definitions)

• Risks of AI

• AI solutions in talent assessment (Assessment Centres)

• The development of AI driven AC simulations

• Construct validity of two AI driven simulations

• Criterion validity of two AI driven simulations

• The way forward

The Talent Assessment Landscape

Six Seismic Shifts

Enhancing the candidate experience with video clips and skype role
plays.

But is this really a seismic shift to how we do it?

Because it simply uses audio visual (video) and communications

(Skype) enhancements, it is a skin, but to what extent does it enhance
the mechanics. (the black box)

So what is the next major frontier?

 Since the emergence of the virtual assessment
centre, what new innovations became apparent?
What is Artificial Intelligence
 Artificial
neurological
networks
 Artificial Intelligence involves
the building of artificial
neurological networks and is
underpinned by a multitude of
statistical algorithms to
emulate the human mind
 In short, artificial intelligence is
about mimicking the human
cognitive and behavioural processes
to solve more complex problems
faster.
 Turin’s test of Artificial Intelligence

When the user cannot discern whether s/he is dealing

with a human being or a computer.

The Enigma Machine (Alan Turin)

The Imitation Game
Bletchley Park code breakers
Origins of Artificial Intelligence
AI Solutions in Assessments
 System

3 Levels Interpretive

Assessment
 System

3 Levels Interpretive

Assessment
 System

3 Levels Interpretive

Assessment
The Challenge
How do we overcome these challenges?

By creating Artificial Neurological

Networks
 System

3 Levels Interpretive

Assessment
The Development of AI driven AC
Simulations
 Advantages of AI driven Simulations
• Capturing real-time data as it happens.

• Elimination of assessor central tendency: better variance across the scale.

• Equality: the system cannot identify race, gender or age.

• Engagement: the candidate finds it modern, engaging and novel.

• Depth of Information: a tapestry of information collection; not possible by even an experienced

assessor.

• Reliability: the system algorithms score reliably as it is not subject to assessor fatigue, lapses in
concentration, time pressures, invariable bias that may creep in despite the assessor’s best efforts.
 Development of AI driven Simulations

• With the ever present client concerns regarding AC’s cost, the associated issues with inter-rater
reliability, central tendency in ratings, declining validities (0,50 in 1960s to 0,27 more recently) and
the modernisation of work, we embarked on applying the emergent technologies of Artificial
Intelligence and Gamification to the design of simulations.

• As we already had a strong track record in systems development it did not require the development
of an entirely new capability, simply a shift in focus.

• To date, two simulations, an interpretive engine and a talent classification engine, all driven by AI
have been developed.
 Redesigned Problem Analysis Role Play

• Old Scenario: A business case is presented to the candidate with instructions to the effect that s/he has a specified
time to interview a “resource person” with all the information, gather information, then to conclude possible causes
and recommendations.

• We developed an AI driven software application.

• Upon login, candidate is navigated through the process as a human assessor would have done.

• System tracks every action, question and suggestion, creating a spiderweb of data.

• Measures dimensions such as Sourcing and Investigating, Evaluating and Analysing, Solutioning, Concentration,
Level of Thinking.
 Redesigned Staff Role-Play

• Old Scenario: A people management scenario is presented to the candidate with instructions to have a discussion
with a staff member in order to deal with four points on an agenda.

• We developed an AI driven software application.

• Upon login, candidate is navigated through the background information and then starts a discussion, interfacing
with the system. Candidate probes, make statements and asks questions, AI responds and facilitates a dialogue with
the candidate.

• System tracks every action.

• Measures dimensions such as Motivation, Delegation, Recognition, Understanding.

But does it work?
Research Sample characteristics and
assessments done
• Exported the assessment data of the last 1,000 candidates from various clients (2018)

• All assessments done for selection, restructuring, mergers, acquisitions or leadership

development.

• Most completed all of the following assessments:

– Assessment Centre: 5 simulations (3 assessor driven and 2 AI driven)
– Cognitive abilities: BCT
– Business Acumen: OIS
– Personality: 13 traits: Organisation Personality Construct Scale (OPCS)
– Management Styles: 8 Styles: South African Organisation Construct Scale (OPCS)
– Values: 9 values: South African Organisation Construct Scale (OPCS)
– Strategic Interests: 9 Interests: Work Type Orientation Scale (WTOS)
– Operational Interests: 16 Interests: Work Orientation Scale (WOS)
 Age
Gender

From 102 different organisations

Position Type
 Distribution across position level
30

25

20

15

10

0
Supervisory Junior Management Middle Management Senior Management Top Management
Percentage
• Does AI technology work in the assessment space?

• Candidate experience?

• Does it deal with central tendency and variation limitations?

• Does it deal with assessor bias?

• Is it reliable?

• Does it have construct validity?

• Does it have discriminant validity?

• Does it have criterion validity?

Does the Technology Work?
 Does AI technology work?

• The assessment process flows well.

• The system is stable and can handle large volumes.

• The software algorithms certainly produce the correct calculations.

• Candidate experience is positive based on feedback as they feel that

it is more “gamified” and it provides variation from the other more
traditional simulations such as the in-box.

• The AI neurological network produces a result without any human

intervention.
Internal Structure and Reliability
 Traditional AC Reliability
Staff Management 0.90
General Management (Inbox) 0.88
Customer Relationship Management 0.91
Interaction Management (Role-Play) 0.64
Project Management 0.75

AI: Problem Analysis / Decision effectiveness constructs Reliability

Investigation and Exploration 0,80
Cause Identification 0,81
Concentration 0,69
AI: Staff Role Play constructs Reliability
Motivational Behaviour 0,93
Directional Behaviour 0,91
Convergent versus Discriminant?
 Where are the AI simulations similar?

BIG THREE FOUR TRADITIONAL SIMULATIONS AI DRIVEN SIMULATIONS CONSTRUCTS COGNITIVE

Digital Role
GENERAL AVERAGE Customer Dig RP (PA) Business Organisation
CONSTRUCT VALIDITY N=1,000 LEADERSHIP EMOTIONAL
THOUGHT General Project
Relationship
Staff Play
Pred
Digital Role Insight and
acumen & planning and
Productivity Business Cognitive
LEADERSHIP Mngmnt Mngmnt Mngmnt (Problem Play (Staff) Undertanding and Output Knowledge Ability
ABILITY MATURITY Mngmnt Indicator innovation management
Analysis)
AVERAGE COMPETENCE 1,000 0,921 0,763 0,843 0,847 0,919 0,936 0,471 0,429 0,266 0,985 0,905 0,905 0,770 0,422 0,472
AVERAGE EMOTIONAL MATURITY 0,921 1,000 0,660 0,734 0,739 0,896 0,908 0,441 0,412 0,285 0,898 0,814 0,793 0,693 0,386 0,435
THOUGHT LEADERSHIP 0,763 0,660 1,000 0,628 0,589 0,758 0,742 0,498 0,575 0,235 0,761 0,788 0,650 0,571 0,447 0,467
General Management 0,843 0,734 0,628 1,000 0,704 0,683 0,685 0,405 0,382 0,205 0,818 0,768 0,771 0,805 0,329 0,376
Project Management 0,847 0,739 0,589 0,704 1,000 0,723 0,739 0,531 0,492 0,204 0,836 0,700 0,922 0,817 0,388 0,418
Customer Relationship Management 0,919 0,896 0,758 0,683 0,723 1,000 0,823 0,426 0,394 0,263 0,911 0,864 0,801 0,642 0,419 0,465
Staff Management 0,936 0,908 0,742 0,685 0,739 0,823 1,000 0,397 0,344 0,266 0,930 0,837 0,809 0,632 0,390 0,428
Digiatal Role Play (Problem Analysis) 0,471 0,441 0,498 0,405 0,531 0,426 0,397 1,000 0,889 0,198 0,471 0,372 0,515 0,655 0,336 0,548
Dig RP (PA) Pred Indicator 0,429 0,412 0,575 0,382 0,492 0,394 0,344 0,889 1,000 0,248 0,423 0,348 0,479 0,741 0,366 0,524
Digital Role Play (Staff) 0,266 0,285 0,235 0,205 0,204 0,263 0,266 0,198 0,248 1,000 0,281 0,239 0,225 0,210 0,226 0,226
Managerial Understanding and Insight 0,985 0,898 0,761 0,818 0,836 0,911 0,930 0,471 0,423 0,281 1,000 0,877 0,864 0,734 0,423 0,473
Business acumen, creativity & innovation 0,905 0,814 0,788 0,768 0,700 0,864 0,837 0,372 0,348 0,239 0,877 1,000 0,771 0,643 0,406 0,453
Organisation planning and management. 0,905 0,793 0,650 0,771 0,922 0,801 0,809 0,515 0,479 0,225 0,864 0,771 1,000 0,865 0,416 0,440
Productivity and Output 0,770 0,693 0,571 0,805 0,817 0,642 0,632 0,655 0,741 0,210 0,734 0,643 0,865 1,000 0,377 0,403
OIS Average 0,422 0,386 0,447 0,329 0,388 0,419 0,390 0,336 0,366 0,226 0,423 0,406 0,416 0,377 1,000 0,505
BCT Total Score 0,472 0,435 0,467 0,376 0,418 0,465 0,428 0,548 0,524 0,226 0,473 0,453 0,440 0,403 0,505 1,000
 Where are the AI simulations different?

BIG THREE FOUR TRADITIONAL SIMULATIONS AI DRIVEN SIMULATIONS CONSTRUCTS COGNITIVE

Digital Role
GENERAL AVERAGE Customer Dig RP (PA) Business Organisation
CONSTRUCT VALIDITY N=1,000 LEADERSHIP EMOTIONAL
THOUGHT General Project
Relationship
Staff Play
Pred
Digital Role Insight and
acumen & planning and
Productivity Business Cognitive
LEADERSHIP Mngmnt Mngmnt Mngmnt (Problem Play (Staff) Undertanding and Output Knowledge Ability
ABILITY MATURITY Mngmnt Indicator innovation management
Analysis)
AVERAGE COMPETENCE 1,000 0,921 0,763 0,843 0,847 0,919 0,936 0,471 0,429 0,266 0,985 0,905 0,905 0,770 0,422 0,472
AVERAGE EMOTIONAL MATURITY 0,921 1,000 0,660 0,734 0,739 0,896 0,908 0,441 0,412 0,285 0,898 0,814 0,793 0,693 0,386 0,435
THOUGHT LEADERSHIP 0,763 0,660 1,000 0,628 0,589 0,758 0,742 0,498 0,575 0,235 0,761 0,788 0,650 0,571 0,447 0,467
General Management 0,843 0,734 0,628 1,000 0,704 0,683 0,685 0,405 0,382 0,205 0,818 0,768 0,771 0,805 0,329 0,376
Project Management 0,847 0,739 0,589 0,704 1,000 0,723 0,739 0,531 0,492 0,204 0,836 0,700 0,922 0,817 0,388 0,418
Customer Relationship Management 0,919 0,896 0,758 0,683 0,723 1,000 0,823 0,426 0,394 0,263 0,911 0,864 0,801 0,642 0,419 0,465
Staff Management 0,936 0,908 0,742 0,685 0,739 0,823 1,000 0,397 0,344 0,266 0,930 0,837 0,809 0,632 0,390 0,428
Digiatal Role Play (Problem Analysis) 0,471 0,441 0,498 0,405 0,531 0,426 0,397 1,000 0,889 0,198 0,471 0,372 0,515 0,655 0,336 0,548
Dig RP (PA) Pred Indicator 0,429 0,412 0,575 0,382 0,492 0,394 0,344 0,889 1,000 0,248 0,423 0,348 0,479 0,741 0,366 0,524
Digital Role Play (Staff) 0,266 0,285 0,235 0,205 0,204 0,263 0,266 0,198 0,248 1,000 0,281 0,239 0,225 0,210 0,226 0,226
Managerial Understanding and Insight 0,985 0,898 0,761 0,818 0,836 0,911 0,930 0,471 0,423 0,281 1,000 0,877 0,864 0,734 0,423 0,473
Business acumen, creativity & innovation 0,905 0,814 0,788 0,768 0,700 0,864 0,837 0,372 0,348 0,239 0,877 1,000 0,771 0,643 0,406 0,453
Organisation planning and management. 0,905 0,793 0,650 0,771 0,922 0,801 0,809 0,515 0,479 0,225 0,864 0,771 1,000 0,865 0,416 0,440
Productivity and Output 0,770 0,693 0,571 0,805 0,817 0,642 0,632 0,655 0,741 0,210 0,734 0,643 0,865 1,000 0,377 0,403
OIS Average 0,422 0,386 0,447 0,329 0,388 0,419 0,390 0,336 0,366 0,226 0,423 0,406 0,416 0,377 1,000 0,505
BCT Total Score 0,472 0,435 0,467 0,376 0,418 0,465 0,428 0,548 0,524 0,226 0,473 0,453 0,440 0,403 0,505 1,000
Does it have criterion validity?
 Major Hotel and Casino Group
N= 65
Criterion = Level Attained
Method = M-Plus

SST BIG THREE TRADITIONAL SIMULATIONS ART INT CONSTRUCTS COGNITIVE

CONSTRUCT AND PREDICTIVE VALIDITY Managerial Business
MAJOR HOTEL GROUP GENERAL Customer Organisation
Position EMOTIONAL THOUGHT General Project Staff Digital Role Understandi acumen, Productivity Business Cognitive
LEADERSHIP Relationshi planning and
N = 65 Level SST MATURITY LEADERSHIP Mngmnt Mngmnt Mngmnt Play (Staff) ng and creativity & and Output Knowledge Ability
ABILITY p Mngmnt mngmnt.
Insight innovation

Position Level (SST) 1,00 0,51 0,36 0,41 0,39 0,48 0,31 0,44 0,43 0,47 0,36 0,51 0,42 0,32 0,07
GENERAL LEADERSHIP ABILITY 0,51 1,00 0,81 0,78 0,75 0,63 0,79 0,83 0,33 0,97 0,79 0,77 0,68 0,38 0,32
EMOTIONAL MATURITY 0,36 0,81 1,00 0,55 0,43 0,34 0,71 0,84 0,29 0,77 0,57 0,56 0,44 0,40 0,19
THOUGHT LEADERSHIP 0,41 0,78 0,55 1,00 0,67 0,51 0,71 0,56 0,34 0,81 0,71 0,56 0,58 0,37 0,26
General Management 0,39 0,75 0,43 0,67 1,00 0,46 0,44 0,41 0,14 0,70 0,78 0,50 0,74 0,27 0,22
Project Management 0,48 0,63 0,34 0,51 0,46 1,00 0,50 0,32 0,43 0,63 0,47 0,80 0,74 0,41 0,41
Customer Relationship Management 0,31 0,79 0,71 0,71 0,44 0,50 1,00 0,54 0,27 0,82 0,55 0,64 0,51 0,30 0,31
Staff Management 0,44 0,83 0,84 0,56 0,41 0,32 0,54 1,00 0,32 0,79 0,61 0,60 0,34 0,33 0,19
Digital Role Play (Staff) 0,43 0,33 0,29 0,34 0,14 0,43 0,27 0,32 1,00 0,32 0,21 0,44 0,35 0,49 0,31
Managerial Understanding and Insight 0,47 0,97 0,77 0,81 0,70 0,63 0,82 0,79 0,32 1,00 0,72 0,70 0,62 0,40 0,35
Business acumen, creativity & innovation 0,36 0,79 0,57 0,71 0,78 0,47 0,55 0,61 0,21 0,72 1,00 0,56 0,57 0,25 0,16
Organisation planning and management. 0,51 0,77 0,56 0,56 0,50 0,80 0,64 0,60 0,44 0,70 0,56 1,00 0,78 0,42 0,34
Productivity and Output 0,42 0,68 0,44 0,58 0,74 0,74 0,51 0,34 0,35 0,62 0,57 0,78 1,00 0,40 0,34
Business Knowledge 0,32 0,38 0,40 0,37 0,27 0,41 0,30 0,33 0,49 0,40 0,25 0,42 0,40 1,00 0,54
Cognitive Ability 0,07 0,32 0,19 0,26 0,22 0,41 0,31 0,19 0,31 0,35 0,16 0,34 0,34 0,54 1,00
 Retail and Distribution Group
N= 55
Criterion = Supervisor rating of
Performance
Method = Correlation

AI DRIVEN PROBLEM ANALYSIS

Sample: 55 E-Band Leaders in one of top 3 OVERALL

Retail Organisations in South Africa SCORE

PERFORMANCE: Commercial 0,474

PERFORMANCE: Technical 0,317
PERFORMANCE: Overall 0,502
 Mixed Industry
N= 101
Criterion = Level Attained
Method = M-Plus

AI driven Problem Analysis

N = 101 Position Level

Mixed Occupation, Industry and Job Level. Attained

Average Problem Analysis 0,482

Average Fact finding 0,311
Average Reasoning 0,340
Average Judgment 0,476
Average Concentration 0,431
Conclusion

The way forward

 Summary
• The AI simulations seem to have a robust internal structure and consistency measuring a number
of constructs / factors.

• The AI simulations show similar reliabilities to the traditional simulations.

• Both AI simulations show promising criterion validity (Early days; small samples)

• Both AI driven simulations show some relationship with the traditional simulation cluster, yet not
at the same level as the intra-traditional cluster.

• Our conclusion is that the simulations are close enough to the AC methodology yet probably
measuring new or untapped constructs as well.

• Maybe these two simulations more accurately emulate the modern world of work which
essentially is becoming increasingly more digital and therefor we are measuring new or
previously untapped behavioural constructs not elicited by traditional simulations..
 Conclusions and the way forward
• From a development point of view: Whilst the development process was a learning curve, we have
now built a strong understanding of how to do it and how to further enhance the simulations.

• From an efficacy point of view: The initial research is encouraging in that these simulations do
measure human behaviour and competencies, for now at similar levels of reliability and validity as
traditional simulations, but offers the potential to do it even better as we become more sophisticated
in our designs.

• As AI develops the role of the assessor will gradually diminish.

• Quite a significant reduction in the cost of the overall AC process is achieved.

• The good news to the psychologist, rather than being dispensed, will shift focus to the machine
learning in the black box as well as more face time with their client, strategizing the talent landscape
of the organisation with much more powerful information.

• AI driven simulations have a distinct impact on our current paradigm and definition of what an AC is
and constitutes.
The End
Dr Pieter Bronkorst
[email protected]
EvaleX Leadership assessment
combines Psychometrics with
Simulations in a unique way through
artificial intelligence