top of page

Compare Personal Finance Apps
-Usability Test

GRADUATE SCHOOL PROJECT
HCI
2023

An Usibility Analysis was conducted with several users to test the superiority of the new design wrt the the Baseline App and it was proven more useful for expert users, they completed the tasks faster with more ease and lesser  mental load. 

However, the new app did not show any significant improvement with inexperienced users.

Experiment 1.png

Introduction

This experiment was conducted as a part of the Graduate school project in which I created. It is an application which helps people with setting up and achieving their long term and short term financial goals. Different tasks were conducted by 6 participants on the existing app and also on the new prototype and results were analyzed.

Research Objectives

O1: In order to verify the new prototype's superiority for users, compare existing financial goal-setting apps with the new prototype

O2: Determine whether users' experience in financial goal-setting affects their preference for the new prototype

Expected Results

Through this improved application, users can reduce the workload in the process of finding and learning investment alternatives, as well as the enhanced usability of the app itself.

Increase.png

Effectiveness

Efficiency

Satisfaction

Decrease.png

Workload

Independent Variable

IV1: Difference of interface

• Existing app (Simplifi) - Baseline

• New app (Success Trail)

IV2: Experience in financial goal-setting

 No prior experience in financial goal-setting

• Have prior experience in financial goal-setting

Research Hypothesis

Hypothesis.png

Dependent Variables (Measurement)

Picture1.png

Experiment Design

Within Subject Design

This design approach involves systematically exposing participants to multiple iterations of the experimental manipulations or interventions, allowing for a comprehensive evaluation of the effects within the same individuals.

In a within-subject design, participants serve as their own control group, eliminating potential individual differences and increasing the statistical power of the study. By comparing participants' responses across different treatments or conditions, researchers can effectively isolate and examine the specific impact of each manipulation.

Within subject matrix.png
Latin Square scheme

It is a systematic and balanced method used in experimental designs to counterbalance the order of treatments or conditions. It ensures that each treatment occurs once in each position or order across participants, minimizing the potential confounding effects of order.

Participants

Number of Participants

3 male and 3 female 

Man.png
Woman.png

Level of Experience

Experienced

Participants who have set and achieved financial goals previously    

Inexperienced 

Participants with no prior experience in setting financial goals                       

Add.png
Remove.png

Procedure

1.Read and sign a consent form (5 min)

2.Introduced to the experimental area (5 min)

3.Briefed about the app and why the experiment is being conducted (3 min)

4.Introduced the prototype, explaining the procedure to the user, the purpose of the experiment, explaining the consent form, and time for the participant to read and sign the form (5 min)

5.Perform the Tasks with Apps (new prototype vs. baseline) (20 min)

     - Fill out the NASA-TLX after each task

     - Fill out the SUS after using each App

1.User the open-ended questions (10 min)

2.Debriefing (5 min)

Task 1: Set up a financial goal and establish a goal achievement plan

The research team presumed that users had already established their profiles, including their names, contact number, and background information. In the primary scenario, users aim to define a financial objective and create a plan to achieve that goal. New users visit the platform to set up their first financial goal and input specific information about it. They can then choose a plan from a list of suggested achievement plans. Once they select a particular goal achievement plan, they are provided with the details of the plan. After confirming the plan, they receive feedback on their selection.

1.png
4.png
5.png
7.png
6.png

Results

ANOVA Table – Perceived Workload
Task 1 Anova.png

There were no significant differences for Main Effects and Interaction Effect.

ANOVA Table – Performance (Number of Errors)
Task 1.2 Anova.png
Task 1 Errors.png

New app caused fewer errors (interface misleading & regressive behaviors) than Baseline.

ANOVA Table – Performance (Number of Errors)
Task 1.2 Anova.png
1.2 Errors.png

Experienced users made more errors (interface misleading & regressive behaviors) than inexperienced ones.

ANOVA Table – Performance (Number of Errors)
Task 1.2 Anova.png
1.3 Errors.png

Experienced users made more interface misleading errors when using Baseline than using Our App.

On the other hand, for inexperienced users, there was no significant difference in the number of interface misleading errors by interface type.

Task 2: Edit the existing financial plan

The third task is for users who want to modify their financial goals. They can click on the goal to monitor their progress or edit the goal directly. Additionally, users can use a shortcut action by swiping left and clicking on the “...” icon. From there, they can edit their goal settings, plan settings, or deposit/withdraw money.

2.1.png
2.2.png
2.3.png
2.4.png
2.5.png

Results

ANOVA Table – Perceived Workload
T2 PW.png
T2 Errors.png

Experienced users felt lesser physical workload and time pressure when performing the second task of editing the financial plan than the inexperienced users.

Lower is better

ANOVA Table – Performance (Number of Errors)
T2 2 Errors.png
T2 2 Exp.png

Experienced users tend to made more errors (interface misleading) than inexperienced ones.

Task 3: Deposit/withdraw money

The fourth task is for users who want to withdraw a specific amount from one goal and add it towards another based on their new priority. They can click on the withdraw or deposit button to make the transaction.

3.5.png
3.3.png
3.4.png
3.2.png
3.1.png

Results

ANOVA Table – Perceived Workload
T3 1.png
T3.png

Experienced users tend to felt lesser time pressure when performing depositing/withdrawing money than inexperienced ones

Lower is better

ANOVA Table – Perceived Workload
T3 1.png
T3 25.png

Experienced users reported that
Our App felt less mentally demanding than Baseline.

On the other hand, inexperienced users responded that the Baseline app felt less mental workload than Our App.

Lower is better

ANOVA Table – Perceived Workload
T3 1.png
T3 26.png

There was no difference in the time pressure
between inexperienced and experienced userswhen using the Baseline.

 

However, when using Our App,
inexperienced users tend to feel
more time pressure than experienced users.

Discussions

Our app seems to bring more utility to experienced users.

Experienced users made lesser errors, completed task 1 faster, and performed task 3 with less mental and physical workload.

Discussion 1.png

Our app needs to be improved for new users without previous experiences.

Inexperienced users performed task 3 with Our App with more mental & physical workload.

Discussion 2.png

Conclusions

To summarize, this is a powerful tool for managing personal finances, but is based on several assumptions that users should be aware of. Users can create a budget that is accurate and effective for their specific financial situation by understanding these assumptions and making adjustments as needed. To achieve long-term financial goals, it's important to remember that this app is just one component of an overall financial management strategy, and it should be used in conjunction with other strategies such as saving, investing, and debt management.

References

●Alt, R., Beck, R., & Smits, M. T. (2018). FinTech and the transformation of the financial industry. Electronic markets, 28, 235-243.

●Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.

●Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139-183). North-Holland.

●ISO. (1998). ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability. International Organization for Standardization.

●Kerényi, Á., & Müller, J. (2019). Brave new digital world?–Financial technology and the power of information. Financial and Economic Review, 18(1), 5-32.

●Lee, I., & Shin, Y. J. (2018). Fintech: Ecosystem, business models, investment decisions, and challenges. Business horizons, 61(1), 35-46.

●Mifsud, J. (2015). Usability metrics–a guide to quantify the usability of any system. Usability Geek.

●Whiteside, J., Bennett, J., & Holtzblatt, K. (1988). Usability engineering: Our experience and evolution. In Handbook of human-computer interaction (pp. 791-817). North-Holland.

bottom of page