Tests conducted for a presentation at Iceweb 2006, Reykjavík, 2006.04.28.
Volunteers were directed to a test Basecamp site and asked to add an item to a To-Do list, edit text in the Writeboard, and delete a project milestone.
| Screen reader | Browser | Task 1: Add | Task 2: Edit | Task 4: Delete |
|---|---|---|---|---|
| Jaws 7.0 | Firefox 1.5 |
I found the page difficult to proofread after I had entered the additional item. I wanted to double-check to make sure that I had entered the additional address, and I found the layout of the page difficult to follow. The list of addresses did not seem to directly follow a header. I had to read through a number of items on the page that did not seem to be directly relate to what I was looking for. |
This was very easy to do. Nothing special or extra had to be done. |
Although it was easy to find the item, deleting the item was difficult. I had a very hard time finding the “delete” button. I also found it difficult to understand how I was navigating through the item that needed to be deleted. |
| Jaws 4.51 with upgrade | IE 6.0.290 | No problem getting to the To-Do tab and adding a new address, although I wasn’t quite sure what to do with the combo box, so I added my name. I did have problems after I clicked the Add button. It didn’t seem to be doing anything so I clicked three times looking at my cursor to see if anything was happening and there was nothing, so I clicked Cancel and the next thing I know my entry had been added. |
Perhaps I got ahead of myself, but I didn’t find the page very intuitive and the Edit button was a tad confusing. The first thing I did is use my heading tab to find content and had to backtrack to get to the edit button. Once I got to the edit boxes it wasn’t hard to change the text. |
I had to use the back button to get to the home page and found the Milestones button no problem. Once on the Milestones page, I had one heck of a time trying to find the right item to delete Once I clicked the right check box I’m not sure what happened. Jaws went into forms mode and just hung. When I finally got it back the checkbox had been checked. The check box was not properly associated with the text though. I clicked on the edit button (I was actually looking for a delete button) and made my way to the proper page and clicked the delete button. I really don’t know if I had done it properly as there was no confirmation. Overall I did not find this site very easy to navigate or understand. I don’t know if it was my connection, but there was a lot of lag time between page loading and being able to navigate. |
| Jaws 7.0 | Internet Explorer 6.0 SP 2 |
While adding the item, I was a little unclear what to do so the first time I tried to add without any text in the To-do item text field. This resulted in the validation popping up a dialog. Unfortunately, there is a Jaws bug that sometimes occurs where Jaws will not speak these dialogs. One must restart Jaws (generally not running as a Windows service) and this usually solves the problem. I decided to try again once Jaws was restarted just to make sure the validation would speak. It did. I then added the street address. I wasn’t entirely certain the To-do item text was the right field in which to enter the address but after checking, I realized that my entry showed up in the list. |
While doing the edit, I wondered whether to uncheck the box signifying a minor edit but decided to leave it at its default which was apparently the right way to go about it. |
While trying to delete the item, I decided that the way to do that might be to check the box and then locate a delete link or button. When I checked the box, the item was marked as complete. I would know for next time if I needed to use this often. I then unchecked the box and surmised that the way to delete was to first edit. Sure enough, I was able to delete the milestone once I edited it. As with many Web-based applications, it can sometimes seem unclear how to use features the first time through due to the linear presentation of screen readers. It is easy enough once one figures it out. |
| Jaws 7.1 beta | Internet Explorer 6 |
Locating the To-Do tab/link was easy, via a brief search using the Jaws search functionality (Ctrl-f). Activating the To-Do link was accomplished in the usual manner (Enter). Locating the To-Do Lists heading was easy, accomplished via the Jaws shortcut key for cycling through headings (h). Locating the Housing List was accomplished in the same fashion, although the undisciplined ordering of headings made it uncertain whether to following the Housing Lists link inside the heading, or to carry on down the page. Another simple search revealed the Add an Item link, which was again easy to follow. The form was easy to locate, but the desired information was not so clearly stated. Backtracking through the existing list of houses suggested that the edit field was where I should enter an address, and only exploration of the drop-down box provided a clue about what should be selected there. Activating the form was again straightforward, but having used the Add button once, nothing appeared to happen. A bit of exploration around the page suggested nothing much had changed, but using the Add button a second time appeared to do the trick. Saving the addition [according to] the instructions did not seem to be possible. A fairly thorough rummage through the page content, first by search, then by links, didn’t reveal a save option, so it’s assumed that the Add button performed this function already. |
Locating the Writeboard tab/link, activating the link, and locating the appropriate headings on the subsequent page were all accomplished in the same way as with Task 1. On the basis that no form field was apparent on the page (searched for with the Jaws shortcut key for cycling through form fields [f]), locating the link to the Writeboard was easy with a little cursoring around. Immediately searching for an edit field on the subsequent page didn’t reveal much, but some swift tabbing through links in the immediate vicinity quickly revealed the Edit link. Locating the form field, using the aforementioned Jaws functionality, was simple. Entering forms mode, editing and saving the text were text book straightforward. |
The requirement to return to the Housing Tasks page, before locating a Milestones link, was unexpected in the context of this site, but not within general Web use. As before, locating the Milestones link/tab, activating it, and locating the appropriate heading on the page were straightforward. A simple search found the required address and the only applicable link appeared to be the Edit link. Never one for wasting time, following that link provided the Delete button after some fairly brief exploration of the page using a combination of the h, f and down-arrow keys. |
| Voiceover 1.0 on Mac OS X 10.4.6 | Safari 2.0.3 |
This task presented no problems, although I did have to explore the page a little bit to find the form that I would use to add an item, as the To-Do list remains onscreen during the process. With Group Items turned on, the page is laid out similar to the way a person would see it. The form is in the lower right, according to Voiceover, in case you were curious. I should note that it didn’t take long to find the form, just that I had to look around a bit. |
This was easier than adding once I figured out the interface to the Writeboard itself. It has a different interface than the rest of the page. That has nothing to do with the screen reader, however. Editing presented no problems. |
This one was the hardest, though not because of the screen reader. It is not immediately obvious how to delete a milestone – I had to find it, click edit, then click Delete. I spent a little while looking for a Delete button right there, as that is, to me, the logical place for it especially when there are checkboxes next to the milestones. I did find it and delete mine, however. Conclusion: I do not see any accessibility problems in the tasks I was assigned. The issues I had were a result of the page’s interface, not the screen reader and thus not specific to those who need one. For future Voiceover users of the site, I would strongly advise turning on the Group Web Page Items option – without it, Web pages have no logical layout and it would take forever to find anything. This is true of all pages, however. |
| Voiceover on Mac OS X 10.4.6 | Safari |
This was very easy, requiring no special effort. The page reads very easily, overall. Thee one thing I wasn’t quite sure about was the checkboxes to the left of each entry. I assume one uses these to reorganize the list or move things around, but their function wasn’t immediately apparent to me. At any rate, adding the entry was easy and in line with filling out most forms using VO and Safari. Interacting with form controls and finding the appropriate button to add the record were easy and straightforward. I assume here that adding the record also saves the update, as there was no separate button for saving that I saw straight off, but I did see my addition. |
Again, super easy, and no special problems. I was gratified to see my changes were immediately shown. |
This was, ironically, the operation that took me the longest to get through, although having said that, it didn’t really take any significant length of time. I thought that perhaps checking the box next to the item would mark it somehow, but then the page refreshed and I found that the entry had moved to the bottom of the list. Selecting the edit link netted the desired result, although I found this somewhat non-intuitive. Once I was in the edit form, deleting wasn’t an issue, but as I say, putting the delete option under edit wasn’t intuitive to my mind. This is, of course, not necessarily a screen reader issue so much as a design issue. As far as screen reader issues using VO, I didn’t have any. VO worked extremely well with this application. It was fast and responsive and easy to navigate and understand, apart from the aforementioned couple of small issues, which as I say I think were more form-design issues than screen reader issues. I think I’d make the function of the checkboxes more apparent and put the delete function on its own button. No issues with the page itself, though; about all that VO seems to really have a problem with are some very large pages and, most especially, pages with Flash content. |
Additional blind or deaf-blind users contributed, with not quite enough information to be given in the table above, but with enough to cite here.
Jaws 4.11 + IE + Braille display (deaf-blind tester)
There is not always a clear ID of check boxes. I guess it is for the line above but wasn’t always sure.
I found I had to really search for the [item] to delete it. I saw it, but that takes so much time with one line at a time.
I couldn’t see what I was typing for when I added an address, but saw only when I reviewed it. Not sure I actually had added it, should give an indication something has been added.
Those links at top of page go to text links or skip navigation links are so handy for doing work on a page, always a plus to have.
Jaws 7.1 + IE
I was able to add an item to the To-Do list without a problem, but I can’t figure out how to delete milestones and edit the Writeboard.... I didn’t really have to do anything special to add an item, when I clicked on the link to add an item an edit field came up and I added the item. I couldn’t see a similar delete link or edit link in the other two pages, and Jaws wouldn’t announce the text as editable or go into forms mode.
VoiceOver and Safari
On the Add test, I had trouble figuring out where to enter the actual text. Using VO plus the arrow keys seemed to skip past the text area down to the Who’s Responsible pop-up button. I didn’t have any major problems other than those I would associate with an unfamiliar Web page. I did notice a possible bug. If you go to the Who’s Responsible pop-up button using VO + arrow keys, press the button with VO + space and then navigate with VO + up-/downarrow, you can move out of the button area.
On the Edit test, the only problem I had was with the button label. Edit did not immediately occur to me as the button to use. so I tried to use Add Comment.
No problem on the delete test.
Brothercake, that is, James Edwards (photos), ran some further tests with other combinations of adaptive technology.
IBM Home Page Reader 3.04
Test 1 (Add)
Easy to get to the To-Do tab using the Tab key.
On the housing page, navigated using the right-arrow key “next item” to move through the page. Found the housing list and could easily make sense of that, then got to the “add an item” link (which also reported as having an event). Activating the link with the Enter key as normal did nothing at all – no response. Tabbing from there took me to the checkboxes either side of the add an item link, so I went back to add item link again, and hit it a few times – nothing. I then pressed Shift-Enter and activated the onclick event, and still nothing apparently happened, but this time when i tabbed away I was in a textarea, although it had no label (I already know by now what it is, but if not I could have oriented myself by hearing the other form elements, as I did for the Window-Eyes test).
I entered some text, activated the “save” button and nothing seemed to happen, but there was a single click sound (of the kind HPR makes in quick succession as a loading progress sound). When I tabbed from there I was still inside the form, and tabbing backwards took me to the list, in which the item I had just added was now present.
Overall: Doable
Task 2 (Edit)
Got to the Writeboards tab easily, by pressing “home” to go to the top of the page, then tabbing through the links.
Got to the main writeboards heading on that page in headings mode – although this was slightly confusing because it’s also an
H1as well as the main heading – then tabbed to the item and activated it.There was a loading pause, then HPR said “please wait while we,” then it immediately said “an internal error has occurred,” then the whiteboard page loaded successfully.
I found the buttons at the top of the whiteboard the arrow keys again – recognizing the edit button as the one that says “b edit,” and activated it. The resulting page was an easy forms interface, where it was simple to find the applicable fields, edit and save. On the resulting page my edits were read out so it was clear it had been a success.
Overall: That error message was disorienting, but otherwise no problem
Task 3 (Delete)
Got to the milestones page easily, by clicking “go back to housing tasks” then tabbing to the “milestones” link.
I still don’t know what I’m doing here, but I’ve become quite familiar with the page. In HPR it’s no problem to orient and make sense of it, because all text is easily navigable with the tab or arrow keys, and each item in the list was read out as part of (i.e., clearly associated with) each checkbox and edit link.
Checking a checkbox in the normal way just reloads the page – with no other information about what’s happening. Activating its event with Shift-Enter is just the same. Activating the “edit” link loaded a new page with the expected form in it, and then activating the “delete this item” produced a JS dialog asking for confirmation of the delete action, and then reloaded the page on clicking okay. Navigating through the new list, my item had been deleted successfully (or at least – it wasn’t in that list anymore, so I assume so!).
Overall: Interface works successfully, but I still don’t know if I’m doing the right things!
Window-Eyes 5.0
Task 1 (Add)
Easy to get to the To-Do tab using the Tab key.
On the housing list, I initially tried to find the “add item” link using headings mode, and this was confusing because the headings were improperly ordered. (e.g., there were two
H1elements). So I then tabbed through the page manually, but the Housing list was very confusing because the details were not read out – it just said “checkbox unchecked” for each item, with no label text or other information.After that I found the “add item” link and activated it; this loaded the form but WE though it was also a page load, so it said “page load, found previous line...” and then started reading the form, and then the rest of the page. I tabbed around the form to orient myself and found that I was already in a textarea, which I assumed was for item text based on the contextual information of the other form elements, whose labels were read out; however, the textarea itself did not have a spoken label.
I pressed “enter” to get into MSAA mode, entered some text, then tabbed to and activated the “add this item” button. Nothing apparently happened – the reader just paused then said “To-Do item text” – the label text of the textarea, spoken for the first time! The area itself was now empty of what I just typed. So I then tabbed backwards to get back to the list, and this time it read out every item properly – its full text and the state of its checkbox (including the one I just added).
Overall: Very weird
Task 2 (Edit)
Tabbed back up to the top, found and activated the Writeboards tab easily.
Found the specific item by using headings mode, which got to a Writeboard heading then “create a new whiteboard” then it ran out, so I tabbed from there and immediately found the item, and activated it.
There was a long pause with no feedback, then the whiteboard loaded. I tabbed and found a link called “b edit” which I assumed was the edit button, activated it and was sent to the whiteboard. Navigating, editing and pressing save was all very easy using forms navigation. After save it reloaded the whiteboard page and re-read, so it was clear to see that my changes had been saved.
Overall: Easy
Task 3 (Delete)
Easy to get back to housing tasks using the link at the very top (which is where the tab went straight to after loading the final page from task 2), then to find and activate the “milestones” tab.
[...I]t was very difficult to find anything coherent. Headings mode produced the same core structural headings as on the other pages, plus a series of date headings like “5 days ago, Friday 21st April,” while tabbing through produced a series of undescribed checkboxes and links that all say “edit.”
So I pressed tab from a date heading, and that went to a checkbox, which i activated and it said “moving to completed, just a moment,” and then appeared to reload the page, and began reading it from the top. I couldn’t tell whether the data was different, because all I can get from the list is dates, and many of them are the same. I went back to a date heading, tabbed past the checkbox and found an “edit” link, and activated that. That loaded a new page and appeared to put the focus in a form. I tabbed around that and found a link “delete this milestone,” which I activated and it produced a dialog with no data – it just said “Alert. OK/Cancel/Close.” I pressed cancel and went no further. I’m fundamentally failing to grok this interface – I don’t know what’s going on!
Overall: WTF? [Note that I read out that acronym’s full expansion onstage in Iceland]
Opera 8.5 with Voice
Task 1 (Add)
Easy to get to the To-Do tab using “move” commands then “open link.”
On the Housing Tasks page it was easy to get to the “add an item” link using “next link” commands; however, once at the link it’s impossible to add an item (because it’s not a proper link; it just has
#for thehref, and when followed it does nothing. The link is using a JS click handler, but it’s not possible to activate that with voice – you can only use the “click” command on form widgets).Overall: Impossible
Task 2 (Edit)
Getting to the whiteboard, then the specific example, was equally easy using the same core navigation commands as before.
On the whiteboard itself, I initially tried to get to the “edit this page” button using “next widget” commands – since it looks like a button, I assumed it would be. However once I realized it was actually a link, it was easy to get to and activate, which successfully loaded the edit area.
Editing the text is done by hand (opera has no voice commands for editing, only navigating). But once changed, I then used voice again to navigate to the checkbox and activate that, then navigate to and activate the “save” button. All good.
Overall: Easy
Task 3 (Delete)
[Results not reliable because my instructions were unreliable]
Derek Featherstone in Ottawa, and his blind friend-cum-employee, have carried out informal tests of Basecamp over the preceding months. He reports:
Previously we had problems with simple things such as file uploads. That mechanism has now been changed and is much more straightforward and easy to use (for all users, I suspect).
One of the new features, Time Tracking, relies heavily on scripting to work. Basic log entries are added via a table-based form with a summary of the existing entries beneath. When clicking the Add to Log button, there is no confirmation that anything is happening. Entries are added to the table summary below with visual confirmation that it has been added ([via] the yellow fade technique), but there is nothing that confirms this for a screen reader. Interestingly, the table row is recognized when added, but the focus stays on the “Add to Log” button and there is no actual traditional “submit.” This lead to pressing the “Add to Log” button again which subsequently brings up an alert dialog box – “You must specify the number of hours spent on this task.” But wait – didn’t I already do that?
Editing existing records is another issue – once an Edit link is activated, Basecamp uses Ajax to get a form from the server and make that record editable. The problem is that the screen reader doesn’t recognize those as proper form fields – the screen reader treats the row as if it were still a normal table row without the editable data.
Ajax and source order
Maintaining and/or creating a logical source order is still paramount – from what I can tell there is nothing all that different than static pages. The key is that you have to help a screen reader user move around – an Ajax experience is fundamentally nonlinear with things possibly changing everywhere in the page. The key for a screen reader is to take that non-linear experience and make it linear – using JS to move them around to the next logical place they should go. Instead of using links that use
href="#"using links withhref="#nextlogicalplace"in combination with JS techniques will allow you to move around the page and make that non-linear experience more linear.More test results soon – we’ve been looking at Google Calendar and Campfire.
Thanks go out to my volunteer testers. Everyone who participated is listed here, though not everyone’s data was sufficient to cite in this report.
Thanks also to Brothercake and Derek Featherstone.
You were here: Home → Iceweb 2006 → Build Half a Product → Test results
Posted: 2006.05.04