Stage 4

4.1 System Evaluation

Evaluation is known as a process that would help to examine the system that has been
made. It would collect and analyse the information about the system’s characteristic, activity and
also the outcome of it. Evaluation is done to help improvise the effectiveness of the system and
make correction of the error that has been identified (Zint, N.D). It is also used to decide if the
system should be carried out at the current level or to be increased, decreased or even at certain
case to be discontinued. It is not about seeing if the program that was initiated is a success or a
failure but instead it is about getting more data on creating a much more effective and efficient
system that would be delivered (PolicyNL, N.D).

4.1.1 Type of Evaluation

There are several kinds of evaluation that can be carried out based on the evaluation’s
purpose and what is being evaluate. Generally, evaluation can be categorised into two type which
are formative and summative evaluation.

 Formative Evaluation
 This is mainly used to provide information for the purpose to make improvement
by examining the deliverability, implementation, personnel and procedures
(PolicyNL, N.D).
 Needs Assessment
 Examine who need the program, how much importance is the need of it
and how to achieve those need. This assessment helps to identify what the
services that is needed by the client and isn’t being served by the program.
It also helps to provide information on what new needs should be
implemented to meet the client’s requirements (Zint, N.D).
 Process/Implementation Evaluation
 This type of evaluation would examine the program’s implementation and
see if the program if operating as what is being planned. This evaluation
can be done either one time or continuously. The result obtain is used to
 improvise the current system. This evaluation focus on how satisfied is the
user towards the program (Zint, N.D).
 Summative Evaluation
 This is done mainly after the program is completed to tell how well established
and toil what extend the program is meeting the requirement (Zint, N.D).
 Outcome Evaluation
 This evaluation checks out on what extend is the program achieving the
outcome. These are the short- and medium-term change in user that are
directly related to the program. This evaluation may identify the
improvement in user’s skill, knowledge, attitude and behaviour.
 Impact Evaluation
 Determine any long-term change that would have occurred due to the
program. The impacts are normally on the entire community, organization,
environment or even society which are known as the net effects. Impact
evaluation focusses on the educational, human health or the environment
quality.

Figure 1: Formative and Summative Assesment Flow (Zint, N.D).

4.1.2 Importance of Evaluation

Evaluation has a very important role in a program. There are many benefits that comes
along when an evaluation is being performed. Based on Zint (N.D), below are the
importance/benefit of carrying out evaluation.

 Improve program design and the implementation

 Through evaluation area of improvement can be identified easily and help to
realise what the is goals efficiently. It can help to assess and adapt the activities to
make it as effective as they can be.
 Demonstrate program impact
 Through evaluation a program’s success/progress can be demonstrated. The
information collected would allow the communication of the program’s impact to
people using it.

4.1.3 Barrier for Proper Evaluation

It is very common for barriers to occur when evaluation occurs. Some of the barriers that
occur would disturb the flow of the evaluation making a proper evaluation to not be conducted.
Below are some of the common barriers which would lead to improper evaluation (Metz, 2017).

 Lacking of Management Support

 If the management doesn’t support then it would be difficult to collect data that is
needed and conduct the evaluation. Getting a higher authority with more
knowledge opinion would be difficult as the evaluation will only be conducted by
the lower level.
 Lacking of Skills and Resource
 Lack of skills and resource in getting and analysing the data would be difficult
and thus would produce an inaccurate result. Most of the organization doesn’t
have time or a person to go interview a person to get information of the program.
Thus, this would lead to false assumptions from the missing and inaccurate data.
 Lacking of Relevant Data
 Without the proper data certain evaluation information wouldn’t be answered
leading to inaccurate result of the final evaluation.
  Fear of Consequence
 Certain organization would not conduct evaluation as they are afraid that the
result of the evaluation would make them to eliminate or change the program.
Due to this fear, they would stop themselves from conducting the evaluation.

4.1.4 Evaluation Paradigm

Evaluation Paradigm is known as evaluation that is done through guided set of belief
which are paired with theories. Evaluation paradigms are frequently associated with a certain
field in that they have a major impact on how individuals in that discipline think about
evaluation. Each paradigm is related with certain procedures and strategies. There are four type
of evaluation paradigm which are as below (ZeePedia, N.D).

Evaluation paradigm Input Explanation

 This evaluation can be done at any time of

the stage.
 Emphasis on fast input rather than doing
careful document finding.
Feedback gained from  The evaluation is done in short timing.
Quick and Dirty
user  The user involvement in this evaluation
will be high.
 The data collected is either informal and
descriptive as it is either in written notes,
sketches or even in verbal.
 User’s performance is measured by the
number of error and time taken to
complete a certain task.
Evaluating user’s
Usability Testing  Observational data is used to do the
performance
evaluation
 User satisfaction questionnaire with
interview is also done.
  Done in natural setting to understand more
on what the user would normally do.
 Can help to identify the opportunity of
Studies done in natural
Field Studies new technology.
setting
 Get the requirement need for design
 Qualitative techniques like interview and
observation are done.
 Expert apply their knowledge on the user
and guided by heuristic to get the usability
Using expert’s
Predictive Evaluation problem.
knowledge
 User does not need to be there.
 Quick process and cost friendly.

4.2 DECIDE Framework

DECIDE framework is used to make decision and make improvement based on the
feedback that has been received (Guldbrandsson et al., 2015). This is also used to ensure that all
the prototype of MyTrain has meet the user’s requirement and satisfy the user. All the feedback
that has been received by the user which are the general public of MyTrain application will be
used to make refinement towards the prototype.

4.2.1 Determine the goals the evaluation addresses (D)

This is used to determin what is the overall goal of conducting this evaluation. The goals
that want to be achieved over here would make an influence in the type of evaluation paradigm
that would be chosen for the study.

 Goal 1: To make sure the application meets all the Requirement in Stage 1
 Goal 2: To ensure Learnability has been achieved in the prototype.
 Goal 3: To ensure Memorability has been achieved in the prototype.
 Goal 4: To ensure Efficiency has been achieved in the prototype.

4.2.2 Explore the specific question to be answered (E)

 Evaluation needs goals and question in order to be guided so that time would not be
wasted. It is important to derive question that would be asked to ensure that the goals are being
implemented and achieved.

 Goal 1: To make sure the application meets all the Requirement in Stage 1
o Question 1: Is the user interface friendly?
o Question 2: Is the kebab menu functional?
o Question 3: Can user access news section?
o Question 4: Can user use the filter and search function?
o Question 5: Can user access the train schedule?
o Question 6: Can user check the train route map?
o Question 7: Can user purchase ticket?
o Question 8: Can user view the information of card?
o Question 9: Can the user access customer service
o Question 10: Can the user choose the language preferred?
o Question 11: Can user get the best travelling route based on input given?
o Question 12: Can user make various payment?
o Question 13: Can user login in into the MyTrain mobile application?
 Goal 2: To ensure Learnability has been achieved in the prototype.
o Question 1: Is the user interface visible for the user?
o Question 2: Does the user interface inherent aesthetic and minimalist design?
o Question 3: Does the icon help on understanding better and straightforward?
o Question 4: Does the user interface use uncomplicated language and jargons?
 Goal 3: To ensure Memorability has been achieved in the prototype.
o Question 1: Does the user recognize the icon that are being used?
o Question 2: Is the user interface consistent throughout the prototype?
o Question 3: Is the fast access in the menu that is needed by user?
 Goal 4: To ensure Efficiency has been achieved in the prototype.
o Question 1: Is the drop-down option functional?
o Question 2: Is the button clear enough for user to know on what to press?
 o Question 3: Is the pop out indicating a feedback based on action useful to the
user?

4.2.3 Choose the evaluation paradigm and techniques to answer the questions (C)

After identifying the goals and deriving specific question to answer the goal’s
requirement, the evaluation paradigm and technique need to be identified. The two technique that
will be used to do the evaluation is Usability Testing and Heuristic Evaluation. These two
techniques would help to obtain perspective of user and the expert which will be used to gain a
better insight on the design that has been created.

4.2.4 Identify the practical issue (I)

Before conducting the evaluation, practical issues need to be identified. There is various
resource that is need to conduct the evaluation and it is not limited to user, schedule and
equipment.

Resources that will be used for Usability Testing:

 The responded would be primary user of the application.

 10 primary users would be selected to test the design
 The equipment that would be used to carry out the testing is by using iPhone running
IOS 14.0 and Moqups App where the prototype is being run with the resolution of 2 532
x 1 170
 The test would be conducted individually.
 The test would allocate a time frame of 50 minutes for per session.

Resources that will be used for Heuristic Evaluation:

 Two experts will be used to conduct Heuristic Evaluation
 The equipment that would be used to carry out the testing is by using iPhone running IOS
14.0 and Moqups App where the prototype is being run with the resolution of 2 532 x 1
170
 The test would be conducted individually.
 The test would allocate a time frame of 120 minutes/2 hours for per session.
4.2.5 Decide how to deal with the ethical issue (D)

In order to conduct a proper evaluation, it is very important to gain the respondent’s

consent and approval. For a proper evaluation, ethical issues that may arise needs to be properly
addressed and a proper guideline needs to be followed to avoid any further issue. Below are the
guidelines that needs to be followed to make minimal ethical issue.
 Privacy of the responded would always be protected and would be confidential.
 Consent form will be filled to avoid any breach of ethical misconduct and must be signed
by both parties.
 Responded are aware of purpose of the evaluation being conducted (the aim and goals).
 The responded will be treated with respect and kindness.
 The responded has the rights to back out at any moment during the evaluation if they feel
something is wrong but a reason must be given before they back out.
 Responded has the rights to question anything relating to the program and their questions
must be answered.

4.2.6 Evaluate, interpret and present the date (E)

The data that are being gathered is mainly in qualitative form with quantitative data. A
Summary Report will be created in the end after both evaluations has been conducted where
Usability Testing would be collecting data from primary user and heuristic evaluation would be
getting inputs from experts’ user. These two evaluations being conducted would help to correct
the existing user interface and system so that it can be much better and meet more requirement
than the existing ones.
4.3 Usability Testing
4.3.1 Usability Instruction

Welcome to the MyTrain Usability Study

For this study we are interested in gaining a better understanding of the problems people have
when using MyTrain application. MyTrain application has been built for user to Check Train
Schedule, Check and Mange Card Details, Explore Map, Purchase Ticket, Access Customer
Service, Get Destination Route, Get Display of Nearest Attraction, Get Latest News and many
more.

The following pages have tasks for you to complete that will help us achieve that better
understanding.

While you are completing these tasks, it is important for us know what is going on inside your
mind. Therefore, as you complete each task, please tell us what you are looking at, what you
are thinking about, what is confusing to you, and so forth.

Task 1 #: Sign In
(a) First open MyTrain
(b) Click on SignIn button using either social medium platfrom (Goggle, Facebook,
Twitter)
(c) Explore MyTrain

Task #2: Reset Password

(a) Click on SignIn button
(b) Click on forgot password
(c) Insert email and new password
(d) Click on reset password to reset password
(e) Click on SignIn and sign in using email and new password

Task #3: Guest Sign In

(a) Click on Sign In As Guest button

Task #4: Register

(a) Click on Register button
(b) Click up all the details
(c) Tick on all the checkbox
(d) Click on Register Now button

Task #5: Explore Homepage

(a) Click on HomePage
(b) Explore the Homepage by click on the fast access options (Ktm schedule, MyRapid
schedule, maps)

Task #6: Check Train Schedule

(a) Click on Schedule
(b) Explore all the train option schedule

Task #7: Check Card Details

(a) Click on Profile
(b) Check the card’s details

Task #8: Top Up Card

(a) Click on Profile
(b) Click on Top Up button
(c) Select the amount to be top up
(d) Choose payment method
(e) Make payment

Task #9: Renewal of Card

(a) Click on Profile
(b) Click on Renew Card
(c) Enter details that are required
(d) Click Pay Now

Task #10: Check Transaction History

(a) Click on Profile
(b) Click on Transaction History
(c) Select option of daily, monthly, yearly or custom
(d) Select the train line option
(e) View the transaction history

Task #11: Check Company Information

(a) Click on More
(b) Click on About Us
(c) View the Company’s Information

Task #12: Explore Map

(a) Click on More
(b) Click on Display Map
(c) Zoom In and Out of the Map

Task #13: Purchase Ticket

(a) Click on More
(b) Click on Ticket Purchase
(c) Choose date, depart station, destination station and quantity
(d) Click Confirm to make purchasement
(e) Select payment method
 (f) Make payment

Task #14: Talk with customer service

(a) Click on More
(b) Click on Customer Service
(c) Chat with the customer service

Task #15: Get Destination Route

(a) Click on More
(b) Click on Destination Route
(c) Select start and end of destination
(d) Click Search
(e) View the destination route

Task #16: Display Nearest Attraction

(a) Click on More
(b) Click on Attractions
(c) Explore the attraction

Task #17: Get News

(a) Click on More
(b) Click on News
(c) Explore the News

Task #18: Logout

(a) Click on More
(b) Click on Logout
(c) Click on Log Out button
(d) Click on confirmation
MYTRAIN SYSTEM TEST
[CHANPREET KAUR DHANOA]
 [19TH FEBRUARY 2022]
4.3.2 Introduction
MyTrain mobile application is an application that is created to enhance the traveler’s
experience when using the public transportation. An effective application does not only help the
passengers to save time and money, but it will also attract more people to use the public
transportations, indirectly reduces the carbon footprint. Through this application the users are
able to view destination map, edit card profile, top up, purchase ticket and many more.

Usability Test is conducted to check if the users are able to complete the tasks provided
based on the system that is being created. 10 responded has been selected to conduct the testing
where they need to complete 18 given tasks. Throughout the usability testing the session is being
recorded so it can be analyzed and potential improvement of the system can be identified.

General public has been chosen to conduct the usability studies. The test was run on an
iPhone running IOS 14.0 . The session is being recorded with a camera and audio for future
evaluation. Participant task completion rate, satisfaction rating, feedback and question where also
captured in the session.

4.3.3 Executive Summary

MyTrain team has conducted usability testing on the 19th February 2022. The purpose of
the test being conducted is to check evaluate the usability of the interface, system infrastructure
and the flow of information.

10 volunteers from the General Public have been selected to conduct the usability testing.
The session lasted for 45 minutes whereby the volunteers has to complete 18 given tasks. Before
starting the test, all the volunteers are been given consent form for them to fill it up to avoid
ethical issue. They are also been briefed about their rights and gave them much information
needed about the goals of the system. All the question asked by the volunteers are being
answered and they are being treated which full kindness and respect.
 Most of the participant found that the MyTrain application has a clear and
straightforward interface. 80% of the participate (8/10) has stated that the application is easy to
be used and they would utilize it in the future.

While the test is being conducted there are minor issue that has been identified which are:

 Some buttons are not functional whereby it requires multiple clicks to works
 Lack of language option
 Error during payment
 Lacking of filter option in the station selection for ticket purchasement.

This report contains respondent task completion rate, time taken to complete task,
limitation and recommendation for improvement to be done.

4.3.4 Methodology

 Session
The participant who are the general public which agreed to volunteer were
recruited to do the usability testing. A consent form was handed to the participant and
they were explained about their right throughout the whole evaluation. They were also
been explained about the goals of the system.
The session lasted for 45 minutes for each individual. After signing the consent
form the participant would be given a user background form to gain more about their
background. After that the participant read the task that needs to be done and conduct it
accordingly. They are being observed and recorded. Upon completing the task, a post
evaluation form is given to the participant where it is rated using a 5-point Likert scale
(strongly disagree to strongly agree):
 Is the user interface friendly?
 Is the user interface visible for the user?
 Does the user interface inherent aesthetic and minimalist design?
 Is the icon used help for better understanding?
 Does the interface use uncomplicated jargon and simple words?
  Is the button usage clear?

There are 3 subjective questions are also asked to get much better insight about
the application .

 What is the most liked feature of the application?

 What is the least liked feature of the application?
 Any suggestion to improve the system? Any suggestion to improve the
system?

 Participant
In total of 10 people were chosen from general public to conduct the evaluation.
The test was held on the 19th February 2022 where all the 10 selected participate has to
complete the provided task. There was a total of 5 female and 5 male participants chosen
where each of them has different educational background and field of expertise.

Educational Background

High School Pre-University Diploma Degree Masters PhD

3 2 1 2 2 1

Field of Expertise

Computing & Business Engineerin Designing Others

Technology g

1 2 1 2 4

Evaluation/Task to Accomplish

Below is the task that need to be completed by the participant

 Task 1: Sign In
 Task 2: Reset Password
  Task 3: Guest Sign In
 Task 4: Register
 Task 5: Explore Homepage
 Task 6: Check Train Schedule
 Task 7: Check Card Details
 Task 8: Top Up Card
 Task 9: Renewal of Card
 Task 10: Check Transaction History
 Task 11: Check Company Information
 Task 12: Explore Map
 Task 13: Purchase Ticket
 Task 14: Talk with customer service
 Task 15: Get Destination Route
 Task 16: Display Nearest Attraction
 Task 17: Get News
 Task 18: Logout

4.3.5 Result
Task Completion Rate

The participants ability to complete all the given task is been monitored closely by the
developers.

All the participant (100%) were able to complete task 1 (Sign In ), task 2 (Reset Password), task
3 (Guest Sign In), task 7 (Check Card Details), task 8 (Top Up Card), task 10 (Check
Transaction History), task 12 (Explore Map), task 14 (Talk with customer service) and task 17
( Get News). 9 out 10 (90%) participant was able to complete task 4 (Register) , task 5 (Explore
Homepage), task 6 (Check Train Schedule), task 11 (Check Company Information) , task 13
( Purchase Ticket) , task 15 (Get Destination Route) , task 16 (Display Nearest Attraction ) and
task 18 (Logout). 7 out of 10 (70%) participants were able to complete task 9 (Renewal of Card)
Participan 1 2 3 4 5 6 7 8 9 10 Rate(%)
t

Task 1           100

Task 2          100

Task 3           100

Task 4          90

Task 5          90

Task 6          90

Task 7           100

Task 8           100

Task 9        70

Task 10           100

Task 11          90

Task 12           100

Task 13          90

Task 14           100

Task 15          90

Task 16          90

Task 17           100
 Task 18          90

Time to Complete Task

Each task that is been conducted has the time take to complete the task recorded. Below
is the table which show how much of time is taken by different individual for completing
different task. The time is calculated in milliseconds (ms).

Participant 1 2 3 4 5 6 7 8 9 10 Average
Time

Task 1 300 200 200 200 250 100 300 200 250 100 210

Task 2 300 200 100 200 200 100 150 300 100 120 117

Task 3 300 200 100 200 250 100 350 400 430 100 243

Task 4 100 X 500 100 700 650 200 350 120 150 318.89

Task 5 190 200 180 330 190 550 X 350 330 130 272.22

Task 6 190 250 130 210 190 100 200 200 100 300 187

Task 7 200 90 300 200 200 100 200 250 100 300 194

Task 8 300 100 300 200 300 500 100 700 650 100 325

Task 9 450 X 100 X 450 200 X 200 100 100 228.57

Task 10 150 120 190 250 150 200 200 250 430 650 259

Task 11 120 190 250 100 300 200 300 X 200 250 212.22
Task 12 180 330 200 100 150 190 400 300 200 300 235

Task 13 130 210 250 100 350 330 X 130 210 250 217.78

Task 14 150 120 200 300 200 210 700 650 300 300 313

Task 15 200 200 100 X 150 120 180 330 200 150 181.11

Task 16 250 X 330 100 350 190 130 210 250 350 240

Task 17 350 350 300 100 190 330 190 250 130 210 240

Task 18 400 170 300 100 330 210 X 100 350 190 238.89

Summary on Task Completion, Error and Time Taken.

Task Task Completion Error Time Taken

Task 1 10 0 210

Task 2 10 2 117

Task 3 10 3 243

Task 4 9 1 318.89

Task 5 9 2 272.22

Task 6 9 2 187

Task 7 10 5 194

Task 8 10 8 325

Task 9 7 1 228.57
 Task 10 10 3 259

Task 11 9 4 212.22

Task 12 10 1 235

Task 13 9 0 217.78

Task 14 10 0 313

Task 15 9 1 181.11

Task 16 9 2 240

Task 17 10 4 240

Task 18 9 2 238.89

Task 4 indeed would take a longer time to complete because it is registration process
where it will take some time for user to fill in their details properly and precisely. Most the error
comes from task 8 whereby it is the top up card section. The error mainly occurs during the
payment and the error needs to be refined.

Overall Metric
After completing all the task, the participate is given a few question that consist of 5
point scale range Likert Scale from 1 (Strongly Disagree) to 5 (Strongly Agree). There are also
subjective question being asked such as their likes, dislike and recommendation of the
application. This question is being asked to know more about participant opinion about the
application. Below is the table made that display the rating.

Post Task Question

Strongly Disagree Neutral Agree Strongly Percentage

Disagree Agree Agreed
(2) (3) (4)
 (1) (5) (%)

Is the user
8 2 100
interface friendly?

Is the user
interface visible 2 3 5 80
for the user?

Does the user

interface inherent
aesthetic and 3 7 100
minimalist
design?

Is the icon used

help for better 5 5 100
understanding?

Does the interface

use
uncomplicated 3 7 100
jargon and simple
words?

Is the button
3 7 100
usage clear?
*Percentage Agreed = Agree + Strongly Agreed
Most Liked Feature
Below are what is most liked feature by the 10 participants about the application

Participant Most Liked Feature

1 The ability to check current card balance.

2 Best travel route finder with cheapest fare.

3 The real time train schedule is really helpful.

4 Able to access the attraction stops easily.

5 Card balance checking.

6 Checking of real time train schedules.

7 Simple and easy to understand user interface.

8 Attraction stops and the best route finder.

9 Customer service feature

10 Ability to use the application without restriction even though as guest

Least Liked Feature

Below are what is least liked feature by the 10 participants about the application

Participant Least Liked Feature

 1 Nothing

2 Ticket purchase section does not have the categorization on the type of train service
ticket to be purchased.

3 Ticket purchase section because too many train stations to choose from, don't have
proper sectioning

4 Nothing

5 Confirm payment button is not restricted even though some fields have yet to be filled.

6 Payment section as no restriction is made for the pay now button.

7 Payment section as no restriction is made for the pay now button.

8 None

9 No option to increase text size.

10 None

Recommendation
Below are the recommendations given by the 10 participants about the application

Participant Recommendation

Include a feature for users to find out which train stop has public facilities, mainly
1
the toilet.
Restrict the buttons from being accessible when the necessary information has yet to
2
be filled in completely.
Include ticket voucher options if applicable instead of manual typing.
3
 Create an option for which users can choose the type of train service interested and
4
limit the destination and arrival station scope to the said service instead of including
all the type of train service into one.
Add restrictions to the buttons to avoid accidental pressing
5
None
6
Allow users to adjust the text size according to preference.
7
Section the train stations accordingly.
8
Maybe include other languages for users with limited language knowledge.
9
Add more system functionalities such as stops with public facilities or QR code
10
generation for purchased train tickets.

Impact of Usability Testing

After successfully conducting the Usability Testing there has been both positive and
negative response received regarding the application. Based on the feedback and observation
done there will be a few changes that would be made to improvise the application. Below are the
changes that would be made:

 Make refinement on the payment page

 Add feature where user would be able to know which station has public washroom.
 Restrict button until all information has been filled
 Include ticket voucher option
 Add more filter option in the destination route option
 Give user more freedom in making text size changes
 Add more language option

All these changes would affect the design guideline as more feature would be add and it
would improvise more on the efficiency of the system and make it much easier for user to use it
concreting the Learnability and Memorability design principle
Conclusion
It can be concluded that the MyTrain application is user friendly and can be used by
users. Although there are a few issues that has been identified but it is not a severe issue and with
a little of redesigning, it would be much more efficient for the user to use. The recommendation
given would definitely be considered to be implemented as it would improve the user satisfaction
level while they are using the application.
Questionnaire for Evaluation

User Background
Post Evaluation Question
4.4 Heuristic Evaluation
 Client : MY TRAIN
Discipline : Usability
Project : Evaluated part of the MY TRAIN mobile
application to find usability issues in the
design.

Introduction

A heuristic evaluation for MY TRAIN mobile application was conducted on February

21st, 2022. This report presents various findings, limitation and recommendation that was arise
through the evaluation that was being conducted.

The mobile application has been designed for general public user that involves both the
combination of a novice user and intermediate user. The primary use of this mobile application is
for the general public user is to (1) Sign In and Forget Password, (2), Guest Sign In, (3) Register,
(4) Fast Access Through Homepage, (5) Check Train Schedule, (6) Check and Mange Card
Details, (7) Top Up Card, (8) Renewal of Card, (9) Check Transaction History, (10) Company’s
Information, (11) Explore Map, (12) Ticket Purchase, (13) Customer Service, (14) Destination
Route, (15) Display of Nearest Attraction, (16) News, (17) Logout.

The reviewer spent approximate 2 hours or 120 minutes to evaluate the application that
has been created and evaluated it using the accepted heuristic guidelines.

The evaluation was carried out on using iPhone running IOS 14.0 and Moqups App
where the prototype is being runned with the resolution of 2 532 x 1 170
Detailed Findings

The following table contains detailed findings of all the usability issues that have been detected
by the reviewer. A severity rating has been assigned to each issue, depending on the guidelines.
The findings can be grouped by type of heuristic as follow; “Aesthetic and Minimalist Design”,
“Visibility Of System Status”, “Match Between System and The Real World”, “Recognition and
Recall”, “Consistency and Standards”, and “Flexibility and Efficiency of Use”

HIGH MEDIUM LOW

H M L

Reviewer 1: Muhammad Hafiz Bin Mohd Hud

Issue: Aesthetic and Minimalist Design

Severity Location Issue Recommendation
L Home Page – Train Redundant labelling Remove the bolded
Schedule for the schedule label on top of the
buttons. There is button.
label to indicate the
train line on top of
the button and on the
button itself.
L KTM Schedule, ETS Redundant Only display train
Schedule, Skypark information. times that are still
Link Schedule Information available
regarding train times
that have passed and
unavailable are no
longer useful to the
user
L Welcome Page The icon logging in Remove the labelling
via social profile of the buttons
 buttons are uniquely
identifiable without
the need of the labels
M Showing Route If the interface for Use a collapsible /
showing route is kept expandable list where
similar as shown in if collapsed, it will
the prototype even in only show route
the case of multiple details that would
routes found, the help the user decide
information will be the route to take such
cluttered as travel time, number
of line changes or
number of stops from
departure station to
destination and only
display the full route
if expanded

Issue: Visibility of System Status

Severity Location Issue Recommendation

H Payment Method After selecting the Add a confirm button
payment method, at the bottom of the
especially after page
adding a card, there
is no indication of
how to proceed. Only
possible course of
action is pressing the
back button which
usually indicates to
 cancel the operation.
L Home page No indication that the Add a text to notify
map is clickable to the users that it can be
enlarge it enlarged by clicking
on it
M Loading Page No indication of what Add any form of
the app is loading for. indication such as text
or pictures to indicate
the loading page is
trying to do. For
example, if it’s a
loading page after
adding a card, add a
text saying that it is
connecting and
getting authorisation
from the bank on top
of the loading
animation

Issue: Match Between System and The Real World

Severity Location Issue Recommendation

L Profile Page ‘Concession’ is Use easier understood
unlikely to be terms like ‘Type of
understood by most Card’ where users are
of the users more likely to
understand the
information. In this
case, for a student
type card, the fare
 prices might be
cheaper

Issue: Recognition and Recall

Severity Location Issue Recommendation

L Profile Page The icon for the top Replace the icon with
up button is not an icon that is more
representative of an suitable for a top up
action for top up function. Maybe a
wallet with a plus on
it.
L Transaction History The icons used for Use a more suitable
Page expenditure and top icon that can indicate
up is associated more spendings and top ups
with rising and falling
of stock prices

Issue: Consistency and Standards

Severity Location Issue Recommendation

L Log Out Page Most apps do not use Remove the log out
a dedicated log out page. Move the pop-
page. The most they up message box that
would use is by is implemented to the
popping up a More page which
message box to ask pops up when the user
for confirmation selects log out

Issue: Flexibility and Efficiency of Use

Severity Location Issue Recommendation

 M Destination Route Users that have Add a button that
search for a route that directs users to
they desire to take purchase all the
will have to tickets for the
memorise the station destination route that
and train lines that they have chosen
they would need to
purchase.

Reviewer 2: Tan Sheng Jeh

Issue: Aesthetic and Minimalist Design

Severity Location Issue Recommendation

Medium More Page The More Page have Reduce the number of
too many buttons buttons.
which make the
design a little packed.
Medium Home Page The train map is too Allow user to zoom in
small to view on the train map

Issue: Visibility of System Status

Severity Location Issue Recommendation

High Top up Page After reloading, it Fix the issue by
stuck at the loading linking some page
screen without after that loading or
moving and no guide user on what to
indication for the click.
user on what to click.
 Medium Destination route After viewing the Fix the issue by
page destination, the page giving user some
does give user any indication on what
indication on what to they can click next.
click

Issue: Match Between System and The Real World

Severity Location Issue Recommendation

Low Profile Page The word Change it to a simpler
“Concession” might word like “Role” or
not be understood by “Title” so that more
some users. users can understand
it.
Medium Notification Page Don’t have clear Put a Clear
button to clear Read notification button to
notification message eliminate old
which might filled up notification message
the whole
notification center.

Issue: Recognition and Recall

Severity Location Issue Recommendation

Low Profile Page Renew Card icon is Replace the icon with
not well designed another design.
because it hardly can
be recognized as a
Renew icon.
Low Profile Page Top up icon is not Replace the icon with
well designed another better design
 because it hardly can that looks more like
be recognized as a Top up wallet icon.
Top up icon.

Issue: Consistency and Standards

Severity Location Issue Recommendation

Medium Profile Page Lack of icons on Put some icons beside
information like card the word so that the
holder, balance, and design can be
expiry date. consistent.
High Payment Method Won’t proceed to Link user to payment
Page payment page after page after clicking
selecting Online Online Banking
Banking option option.
unlike other payment
method option.

Issue: Flexibility and Efficiency of Use

Severity Location Issue Recommendation
High Ticket Purchase Page Confirm button not Fix the button
working after several freezing issue.
click.
Medium Top up Page Don’t have choice to Give users more
top up amount other option by giving them
than those amounts more choices on top
that are being shown. up.

Executive Summary
 There was two respondent that was used to conduct the heuristic evaluation to identify
usability issue in MyTrain application. The first reviewer is Muhammad Hafiz Bin Mohd Hud
where he manages to identify 12 usability issue: 8 were Low, 3 were Medium and 1 were High.
The second reviewer is Tan Sheng Jeh where 12 usability issue were identified: 3 were Low, 6
were Medium and 3 were High.

Below are similar issues being identified by the two reviewers:

 The train map is too small to view and can’t be enlarged

 The icon for the top up button is not representative of an action for top up
 After selecting the payment method, the payment doesn’t get proceed.
 Word such as “concession is unlikely to be understood by user
 No indication why the app is being load for

Following is the uniquely identified issue by reviewer:

 Redundant labelling for the schedule buttons in home page
 Redundant information regarding train times that have passed
 Social media icon is uniquely identified without needing labels
 Interface for showing route is kept similar as shown in the prototype even in the case of
multiple routes found.
 Icon used for expenditure and top up, renew card isn’t suitable
 Most mobile app doesn’t need a dedicated log out page
 There isn’t an option to immediately make purchase after a route has been searched
 More pages compact with buttons
 Page doesn’t give indication what to do what viewing destination
 Notification doesn’t have clear option to clean notification
 Lack of icon on profile page for card.
 Confirm button in ticket purchase page isn’t functioning well
 User doesn’t have option to top up on desired amount that they want.

Impact
 The existing design of MyTrain mobile application has achieved the usability guideline
but there is also room for improvement to be made. There are a few recommendations that has
been giving by the expert in order to enhance the existing design guideline which are
Learnability, Memorability and Efficiency in the mobile application. Thus, the feedback given
by them will be used to modify the existing system. Here are few of the modification that would
be incorporated into the design:

 There train map would have a magnifying tool to enlarge

 Icons for top up, expenditure, renew card will be redesigned
 Add more icon in the profile card page
 Payment method will be refined to ensure payment gets proceed
 Simpler words will be used and replace words like “concession”
 Loading app will having indication to show why it is loading
 Bolded label on top of the button of schedule will be removed
 Only display train schedule that are available and remove those that has passed
 Use collapsible where it will only show certain train detail rather than compacting
everything together
 Remove dedicated log out page and use the pop out to log out
 Add a button that directs users to purchase all the tickets for the destination route that
they have chosen
 Simplify the more page with lesser button for better visibility
 Add a clear option to clean notification that has been read
 Refine the ticket purchase button’s functionality.
 Give users more option by giving them more choices on top up.
Reference