Potential test question
@Parkywiki noted [i] how at en.wiki's Teahouse, many newcomers mistakenly add the question they are wanting help with to the page via the "Edit" button. In doing so, they end up adding said question to the end of an unrelated topic.

The above leads me to wonder whether it would be worthwhile asking the participants in this test what they understand to be the relationship between the "Add topic" and "Edit" tabs.

i. https://w.wiki/itL

Thanks @Parkywiki and @ppelberg .
We will in fact need to have a minimum of two usability tests: one for junior contributors on usertesting.com and another for senior contributors - distributed via mediawiki.

My instinct is to not outright as the tester the relationship between "Add topic" and "Edit" but instead to watch the test and see which tab they are drawn to. I have a hunch that this will continue to be an issue with the new design because we haven't provided any additional affordance or made a copy change to the button UIs, but it would be good to validate these assumptions.

Junior Contributor Usability test

We should repeat the usability test that we ran on the current experience with the prototype. Here's what that looked like:

Opening scenario
Imagine that you were given the name of another user (Alice) on Wikipedia who wants to help you become a better editor. You want to write to this user to ask them for help, so you go to their user page and attempt to start a discussion with them there.

Note:
*This is a test environment so you will not actually be editing or vandalizing Wikipedia.
*You do not need to log in for the purpose of the test.
*Please talk out loud and describe what you are thinking as you are attempting to complete the tasks.

User tasks + questions

1. Without tapping on anything, what are your first impressions of this page? Does anything stand out to you? Is anything confusing or surprising?
2. How does the page you are looking at compare to what you expected to see after reading the opening scenario?
3. Log in using the button in the upper right corner of the page. Use the username: XXXX and the password XXXX
4. Add a new message to the user on this page that says: “Hello it’s nice to meet you, can you help me write an article for this important person?”
5. Bold the words “ Hello, it’s nice to meet you.”
6. Publish your question
7. Now check to see if you can find the question that you posted on the person's talk page.
8. Describe how that felt. Was it obvious?
9. On a scale of 1-10, with 10 being extremely confident, how confident are you that you posted your question correctly?
10. How do you expect to know when this person responds to you?

Senior Contributor Usability test

I think that it would be good to simulate the experience of a long page (such as the teahouse) as well as possibly another instance where there is preloaded content (such as the ask a question section on Teahouse). Will that be possible to do @Esanders? Once I have an answer to that I can post the proposed protocol for that as well.

Okay according to @Esanders it is possible to set up the different pages on the wikis, so I will write the instructions that should be posted up on our project page and then @ppelberg and @Whatamidoing-WMF can review and tweak.

Senior Contributor Usability Test

(via Media Wiki)

Subject: Test Usability of New Discussion Workflow

Thank you to all of those who provided feedback on the initial New Discussion designs [LINK]. We have incorporated relevant feedback and created a prototype [LINK] for you to test.

Are you able to complete your typical workflow using this prototype on LINK 1: Teahouse and LINK 2: Teahouse (ask a question pre-populated form)?

When you are ready to share what you have to say, we would value you doing the following:

1. "Start a new topic" on this talk page
2. Name this new topic: "New Discussion Usability Test: YOUR USERNAME"
3. Write the answers to you have to the following feedback questions.

• Can you compare the designs being proposed here to to the current experience? Can you accomplish your current workflow for adding new discussions with this proposal?
• What other improvements do you think would be valuable for us to consider making to the new tool?

...of course, if other comments/questions come to mind as you are reviewing the designs, please share them.

Thank you for putting this together, @iamjessklein.

Comments and questions in response to what you shared in T243249#6588386 and T243249#6597381 are in-line below. First some, more meta questions.

[Meta] questions

• Are you able to articulate what you see as the objectives of these usability tests in this task's description? I appreciate this is something we know and have talked about. Tho, I want to make sure we make it explicit.
• What screening questions do you think we should be using to select the Junior Contributor test participants?

Follow up comments/questions

In T243249#6588386, @iamjessklein wrote:

My instinct is to not outright as the tester the relationship between "Add topic" and "Edit" but instead to watch the test and see which tab they are drawn to. I have a hunch that this will continue to be an issue with the new design because we haven't provided any additional affordance or made a copy change to the button UIs, but it would be good to validate these assumptions.

Sounds good.

Junior Contributor Usability test

We should repeat the usability test that we ran on the current experience with the prototype. Here's what that looked like:

Opening scenario
Imagine that you were given the name of another user (Alice) on Wikipedia who wants to help you become a better editor. You want to write to this user to ask them for help, so you go to their user page and attempt to start a discussion with them there.

• Can we replace any language in the test that may be present in the interface (e.g. "add topic" or "start a discussion" with more generic words, as you've done in with the User tasks + questions? Thinking: we ought to use words people are likely to have in mind so as not to subtly prime them to look for those particular bits of language on the page.
• For the Junior Contributor test, can we sure that we run test the following scenarios?
  • User talk (as you currently have written)
  • Article talk with {{Talk header(s)}} present

Note:
*This is a test environment so you will not actually be editing or vandalizing Wikipedia.
*You do not need to log in for the purpose of the test.
*Please talk out loud and describe what you are thinking as you are attempting to complete the tasks.

User tasks + questions

• Can we add a step that asks people Cancel a question they've started drafting?

1. Without tapping on anything, what are your first impressions of this page? Does anything stand out to you? Is anything confusing or surprising?
2. How does the page you are looking at compare to what you expected to see after reading the opening scenario?
3. Log in using the button in the upper right corner of the page. Use the username: XXXX and the password XXXX
4. Add a new message to the user on this page that says: “Hello it’s nice to meet you, can you help me write an article for this important person?”

• How might we write this in a more abstract/less prescriptive way? Thinking: we're wanting to see how people interact with the "form" and what – if anything – they think ought to be written in each field. As it's currently written, I worry we're leading people too strongly to fill in the topic body thereby distracting them from considering whether they ought to write something in the subject field.

In T243249#6597381, @iamjessklein wrote:

Senior Contributor Usability Test

(via Media Wiki)

Subject: Test Usability of New Discussion Workflow

Thank you to all of those who provided feedback on the initial New Discussion designs [LINK]. We have incorporated relevant feedback and created a prototype [LINK] for you to test.

Are you able to complete your typical workflow using this prototype on LINK 1: Teahouse and LINK 2: Teahouse (ask a question pre-populated form)?

• For LINK 1, what do you think about asking Senior Contributors to test on a page like Wikipedia:Village_pump_(technical)? Thinking: this will create an opportunity for Senior Contributors to experience what it might be like to use the New Discussion Tool themselves and posting on a page like Wikipedia:Teahouse will create a complimentary opportunity for them to experience the tool as if they were a Junior Contributor who I assume will most likely be the people asking questions/starting new topic at the Teahouse.

@ppelberg - I updated the protocol. We can test it on article and user talk pages for junior contributors and on village pumps for senior contributors. In order to have a 1:1 comparison from the initial control usability tests (of the current experience) I'd like us to ensure that the protocols have parity. As such, re:

How might we write this in a more abstract/less prescriptive way? Thinking: we're wanting to see how people interact with the "form" and what – if anything – they think ought to be written in each field. As it's currently written, I worry we're leading people too strongly to fill in the topic body thereby distracting them from considering whether they ought to write something in the subject field.

I don't see any tasks here that speaks directly to the topic, but please feel free to propose an edit to the protocol for me to review in regards to this point.

TEST 1: ARTICLE PAGE (JR)

• prep url page
• make sure 5 user accounts are made
• make sure each of the accounts talk pages/user pages are fresh
• draft tests on usertesting.com

TEST 2: USER PAGE (JR)

• prep url page
• make sure 5 user accounts are made
• make sure each of the accounts talk pages/user pages are fresh
• draft tests on usertesting.com

TEST 3: TEAHOUSE (SR)

• prep url page
• @ppelberg to confirm draft test instructions

In T243249#6633925, @iamjessklein wrote:

@ppelberg - I updated the protocol. We can test it on article and user talk pages for junior contributors and on village pumps for senior contributors.

Sounds good. I've reviewed what's contained in the "Testing the prototype" section and it looks good.

A resulting question:

• @iamjessklein: can you please point me to where the draft test instructions for the Teahouse are? I did not see them in the Usability test of New Discussion doc.

In order to have a 1:1 comparison from the initial control usability tests (of the current experience) I'd like us to ensure that the protocols have parity.

This makes sense and agreed

As such, re:

How might we write this in a more abstract/less prescriptive way? Thinking: we're wanting to see how people interact with the "form" and what – if anything – they think ought to be written in each field. As it's currently written, I worry we're leading people too strongly to fill in the topic body thereby distracting them from considering whether they ought to write something in the subject field.

I don't see any tasks here that speaks directly to the topic, but please feel free to propose an edit to the protocol for me to review in regards to this point.

I think you're right; I don't see any mention of specific UI components (e.g. "Add topic") within the test.

In T243249#6641812, @iamjessklein wrote:

TEST 3: TEAHOUSE (SR)

• prep url page
• @ppelberg to confirm draft test instructions

See "resulting question" above.

Sr. Contributor Usability Test

In T243249#6643654, @ppelberg wrote:

• @iamjessklein: can you please point me to where the draft test instructions for the Teahouse are? I did not see them in the Usability test of New Discussion doc.

Today, Jess confirmed the "...draft test instructions..." she was alluding to in T243249#6641812 are represented in T243249#6597381.

As such, I will finalize this language when we get ready to posting the prototype on mediawiki.org for senior contributors to test. This will happen in: T268702.

Meta
@iamjessklein: can you please update the task description per the below? In drafting the comment above, I noticed this was still open.

In T243249#6604857, @ppelberg wrote:

• Are you able to articulate what you see as the objectives of these usability tests in this task's description? I appreciate this is something we know and have talked about. Tho, I want to make sure we make it explicit.

The tests are up and running for both the junior contributor tests (article page and user talk page). Will update here after reviewing.

Synthesized Results for both the junior contributor tests

Adding a new discussion
✅5/5 Article Talk Page testers were able to successfully add a new discussion
✅5/5 User Talk Page testers were able to successfully add a new discussion

Confidence in Publishing their New Topic
✅All participants ranked their confidence level after publish as “Extremely Confident” on a Likert Scale of “1 -Not Confident” to “ 5- Extremely Confident”

Locating discussion in the Talk page
✅5/5 Article Talk Page testers were able to successfully locate their new discussion within the context of the page after publishing
✅5/5 User Talk Page testers were able to successfully locate their new discussion within the context of the page after publishing

Notes

1. Deleting or Editing messages after posting: When test participants posted their new discussion and realized they had a change they wanted to make, they were confronted with a completely different editor and were thrown off. Additionally, when testers attempted to delete their post after it was published, they struggled.

2. Language selector keyboard: 2/10 test participants saw the keyboard for language selection and thought that would help them format content, before realizing they were in the wrong section.

3. Feedback (from the UI) after publishing: 4/10 participants missed the feedback cues (the yellow highlight and success message) which leads me to believe that we should extend the time that the messages are visible.

4. Source Toolbar: Several test participants were thrown off by the lack of a toolbar in Source mode. One participant actually left the test and went into Google to search for how to bold text using CSS and then returned to the test to format. “It would be nice if [the toolbar] was available in both [modes]” [AT-4]

5. Overall Experience:Participants used words such as "straightforward," and "easy" to describe the experience. "Being that i'm familiar with wikipedia pages, this all feels familiar. the experience is similar to what i'm used to. I haven't edited since Spring this year... i like that it was intuitive" [AT-3] "I think it was pretty straightforward what I needed to do and i was able to do it in a timely manner.”[UT-2] It was quite easy for me to figure everything out. [AT-2]] "i was not aware you were able to do these kinds of interactions on wikipedia. it is a nice way to get help from experts on how to write articles" [UT-5]

6. Look and Feel of page: These tests validated that it was challenging for participants to locate the Add Topic button, citing that the button copy and button placement was confusing. “If I were a new editor, I might be confused with how to add a new question, or with the label being called Topic.” [AT-3] And there were some outlier comments about the look and feel such as "I really like this interface. It's simple. It's very clean even though it's old school. I kind of like that aesthetic."[UT-1]

Recommendations for upcoming release:

• Extend the time that highlight and post publish success message appear on the page : T268994
• Clean up the warning message for adding a Title to a message (stylistically)

Recommendations for the future:

We should consider:

• how to make editing messages feel contained within the rest of the experience of the Talk Page.
• making it more obvious how to delete posted comments
• changing the location of the language selector
• integrating a toolbar into the source mode experience
• making it possible for contributors to perform keyboard shortcuts in source mode for formatting content
• the terminology used throughout the interface
• making the affordance for starting a new discussion more obvious

cc @ppelberg @Esanders @DLynch @dchan @matmarex

Next steps
Quick notes from the conversation @iamjessklein had today:

• 1. Jess is going to update the ticket with the number of testers who described the workflow using "positive" terms
• 2. Peter to link any existing tickets for the issues Jess observed in the test
• 3. Jess to file new tickets for issues she observed in the test that are not in Phabricator yet

In T243249#6684112, @iamjessklein wrote:

• making it more obvious how to delete posted comments

While making this, I think it’s important to stress that even deleted comments are visible for everyone in the talk page history. An option to delete something makes me feel that deleted content becomes unavailable (except for probably the “moderators”, i.e. admins).

1. Jess is going to update the ticket with the number of testers who described the workflow using "positive" terms

@ppelberg I updated this in the ticket description (with the highlights of the results)

While making this, I think it’s important to stress that even deleted comments are visible for everyone in the talk page history. An option to delete something makes me feel that deleted content becomes unavailable (except for probably the “moderators”, i.e. admins).

@Tacsipacsi I agree. This is the challenge about this whole interface: we need to communicate to the contributor that their comment is in itself an artifact.

Jess, as I was going to close this out, I saw that we still had some action items open: for me to document what tickets exist for the issues you identified through the usability testing and for you to file new tickets for issues that do not yet have them. More details below.

In T243249#6685395, @ppelberg wrote:

Next steps
Quick notes from the conversation @iamjessklein had today:

• 1. Jess is going to update the ticket with the number of testers who described the workflow using "positive" terms

See: T243249#6684112.

• 2. Peter to link any existing tickets for the issues Jess observed in the test

See below.

• 3. Jess to file new tickets for issues she observed in the test that are not in Phabricator yet

@iamjessklein: this is still needed. The specific tickets needing to be filed are marked - [ ] Ticket needed. Assigning this over to you to finish this bit up.

In T243249#6684112, @iamjessklein wrote:

Recommendations for upcoming release:

• Clean up the warning message for adding a Title to a message (stylistically)

I think the issue above is captured in T269285; @iamjessklein can you please confirm whether this is accurate or not?

Recommendations for the future:

We should consider:

• how to make editing messages feel contained within the rest of the experience of the Talk Page.

See: T245225 and T267287.

• making it more obvious how to delete posted comments

• Ticket needed.

• changing the location of the language selector

• Ticket needed. For reference, here is the ticket where we last talked about the ULS IME icon: T255191.

• integrating a toolbar into the source mode experience

See: T257391.

• making it possible for contributors to perform keyboard shortcuts in source mode for formatting content

See: T257391.

• the terminology used throughout the interface

See: T241403.

• making the affordance for starting a new discussion more obvious

See: T267444.

I created T243249 to document the issues with the language selector.

I created T271846 to document the pain point of deleting posted comments.