|
|
Research
|
Usability
|
Usage
|
Health Metrics and Overall Performance
|
| Why
|
This helped identify what needed to be fixed
|
This helped verify that the suggested solutions were correct
|
This helped verify that the solutions performed better than the previous versions
|
This helped estimate the long-term effects and benefits of using the new skin at scale
|
| What
|
Qualitative research across multiple languages and wikis, including English Wikipedia, was performed to determine the main issues with the old Vector skin
|
Qualitative methods such as user testing, prototype testing, and interviews were performed for individual features as well as the overall skin
|
Quantitative methods, including A/B testing and subsequent monitoring was performed to validate the results of qualitative testing on wikis, to show how features and the skin were used in reality
|
In addition to testing individual features, qualitative health metrics were identified and monitored to ensure the usage of the new skin was on-par or better than the old Vector skin
|
| Results
|
- Too much information density and navigational elements on the page made it difficult to focus on a particular reading or editing task.
- Readers were unable to understand the purpose, terminology, and concepts of available tools such as the main menu, user menu, and page tools.
- Readers found it difficult to navigate the page and spent a lot of time scrolling
- Readers found it difficult to search for information that is relevant to them.
- Multilingual readers were not aware of language switching functionality.
- Readers wanted to have an easy way to view the table of contents throughout the page.
- Editors and other contributors wanted easier access to tools they used frequently.
- Editors found the hierarchy of the navigation to be confusing for newcomers.
- Editors expressed interest in having fewer, but more prominent, ways to begin contributing.
- Editors expressed the need to be able to configure the interface via gadgets and user scripts.
- Research into the width of the content indicated that Wikipedia's character count per line was more than three times the recommended value.
|
Header
|
|
Usage
|
| In prototype testing with editors, the large majority of users reported positive to neutral experiences with the new header and logo configuration. 8% of editors tested reported negative experiences with the new logo.
|
- As of May 2023, on all large wikis with Vector legacy as the default, Vector 2022 is the most-used non-default skin.
- German Wikipedia with a uniquely high rate of Monobook users is an exception. Vector 2022 and Monobook usage there are almost equal.
- Typically, the most-used non-default skin is Monobook. The third most-used non-default skin is Timeless.
|
| Collapsible sidebar
|
Pageviews
|
- In prototype testing with editors, the majority of users who provided feedback liked the collapsed sidebar for personal use, and especially for the purposes of reading.
- In a user study with readers, participants reported a strong preference towards the collapsed version of the menu, stating that it created a better reading experience than the current version.
|
- Logged-in and logged-out users have very different behavior while using the sidebar.
- Both logged-out and logged-in users used the collapsible sidebar frequently.
- Logged-out users tended to collapse the sidebar and leave it collapsed 81% of the time.
- However, the rate of uncollapsing indicated that users are aware that, were they to need to navigate to an item in the sidebar, that option was available to them.
- Logged-in users tended to collapse and then uncollapse the sidebar at fairly even rates, indicating that they were collapsing it when necessary and uncollapsing to use the necessary functionality.
|
- Wikis with Vector 2022 as default did not experience any decreases in pageviews attributed to Vector 2022.
- Note: without direct A/B tests across all users, which is not privacy-friendly, increases in pageviews due to the new experience are difficult to prove.
- This is why a threshold of no significant increase or decrease is used instead.
- However, wikis with Vector 2022 as the default are receiving a higher ratio of pageviews when compared to the 2019 benchmark than English Wikipedia or all other Wikipedias with the exclusion of English.
|
| Limited content width
|
Edit rates
|
- In prototype testing with editors, most editors appreciated the shorter line lengths and agreed that the feature created a more comfortable reading experience.
- A significant number of editors disliked the whitespace around the content and felt that it was wasted space.
- In user testing with readers, participants reported a strong preference to the limited content width, stating that it improved the reading experience.
- Previous research indicated that users read more accurately and more quickly at limited line widths.
|
Limiting the content width did not show any significant effects of the health metrics: pageviews, edit count, and account creation across the pilot wikis.
|
Wikis with Vector 2022 as default did not experience any decreases in edit rates attributed to Vector 2022.
|
| Search
|
Account creation
|
| In user testing performed on usertesting.com, all participants were able to find the result they were looking for using the new search.
|
- There was an average 28.9% increase in the search sessions initiated across early adopter wikis in the AB test for the group using the new search widget compared to the previous search widget for logged-in users.
- For logged-out users, there was a 13.4% increase in search sessions initiated following the deployment of the new search widget.
|
- Wikis with Vector 2022 experienced minor increases in account creation attributed to Vector 2022.
- A thorough A/B test will have to be performed to establish whether these changes are significant.
|
| Language switching
|
- In prototype testing with editors, the majority of editors preferred the new location of the language selector.
- However, a significant number of editors expressed concern with having an extra click in order to switch languages.
- In user testing with readers, participants reading in non-English languages, as well as multilingual participants reading in Englsh were able to find the new language selector more easily and to switch to a different language quicker.
|
- In an A/B test for the language switching functionality with logged-in users, there was a 44.57% decrease in total clicks on language links. This led the Web team to iterate on the feature.
- After iterating on the feature to include guidance on where the new functionality is located, there was an increase in language switching for logged-out users using the new version of the language switcher and comparable usage of the new and old versions of the language switcher for logged-in users.
|
| Sticky header
|
- In user testing with readers and editors, two issues were studied: the ease of use of a sticky header, and the highest-priority functionality to include within the sticky header.
- Based on these results, the icons within the current sticky header were chosen.
- In prototype testing with logged-in users, an overwhelming majority of test participants reported positive experiences with the sticky header.
|
- There was an average 15% decrease in scrolls per session by logged-in users on the 15 pilot wikis in the treatment group (with the new sticky header), compared to the control group (without the sticky header).
- This indicates that the hypothesis was correct - adding the sticky header to the page reduced the need to scroll to the top of the page significantly.
|
| User menu
|
|
- In prototype testing with editors showed that the new user menu was easy to use and participants could intuitively navigate to pages within the menu from their first try.
- Some participants indicated they would like more control over which items are shown within and outside the menu.
- User testing with readers and editors indicated that the menu items and icons were intuitive to use and understand.
|
| Table of contents
|
- User testing with readers in multiple countries revealed that readers and editors prefer a table of contents that is persistent throughout the page, without any collapsible elements.
- In prototype testing with editors, the overwhelming majority of the feedback received on the table of contents was positive. Logged-in users also reported potential changes to the proposed ToC that would improve its usability.
|
- Among the sessions with at least 1 click on ToC, the treatment group (the group that is exposed to the new ToC) has more clicks on ToC than the control group (the group exposed to the old ToC).
- The data model predicts 53% more clicks on new ToC with logged-in users and 45.5% more clicks on new ToC with anonymous users when compared to the previous ToC.
- This trend is consistent across all edit count buckets: i.e. logged-in users clicked on the ToC at roughly the same frequency regardless of how many edits they had previously made.
|
| References
|
|
|
|
|
