(Archived) How Evernote sample/test new functionality

[Metrodon 2,184]

This is pretty interesting and probably explains quite a lot to those of us who are heavy users of the service.

 

http://www.heraldonline.com/2013/02/25/4644375/evernote-increases-user-retention.html

[BurgersNFries 2,407]

Interesting read!

[gazumped 11,666]

Thanks for the link

[Martin Packer 162]

Yes, interesting.

 

This is more about usability glitches and less about brand new features. On this bulletin board a lot of us are in "WIBNI Mode". :-)

[Metrodon 2,184]

Usability testing (especially with noobs) tends to lead to big icons/buttons amongst other things - stuff that can drive regular users mental as they find they can't get enough on their screen as their notebooks icon is slightly bigger than their house.

[jefito 5,589]

So this is what we can pin the Mac 5 and iOS 5 "whoopsies" on?

[Metrodon 2,184]

Sounds like it - 

 

"I can't find my Notebooks"

 

"Don't worry, we'll make you a giant icon so you can find it easily the first time"

 

"But won't it be in the way after that?"

 

"Don't worry, we'll test it on another bunch of noobs and they'll find it useful too"

[GrumpyMonkey 4,319]

I think I did testing for Evernote through this service. Seriously. We had video, audio, and a bunch of questions. Maybe I am to blame for the gray on gray! I never thought about that possibility

[Metrodon 2,184]

Grumps, you need to go and do 100 laps of the field as punishment.

[Martin Packer 162]

@GrumpyMonkey, was it something like "my eyes hurt"? :-)

[GrumpyMonkey 4,319]

@GrumpyMonkey, was it something like "my eyes hurt"? :-)

 

LOL. As I recall, I think I had trouble finding one of my notes on a search. I was supposed to search for something I wanted, so I picked a note from last year, and I couldn't find it. I forget why I flubbed up the kind of stuff I do every day! But, maybe that was interpreted as "He must be blinded by the white screen, and I think he would have more luck with gray on gray"

 

Seriously, though, if you film yourself (doing anything), I think you learn a lot about how you "really" do stuff. It is often kind of depressing to see how much time I waste. When Google Glass comes along, I am sure I will be posting a bunch about the endless videos of me trying to work through my emails for the day or whatever mundane task it is that eats up all my time!

[gazumped 11,666]

All due respect to the companies and people involved here,  and no doubt this process has the benefit of tons of research and unending stats to confirm its efficacy,  but isn't the step of designing software based on a random bunch of people's work methods a bit like trying to derive Shakespeare from a lengthy survey of monkey typists?

 

I can get the concept of perfecting screen colours (that went so well, by all accounts..) and - to a degree - screen layouts based on a bunch of people who haven't used the product before.  What concerns and confuses them is worth looking at. 

 

But most people don't have a 'perfect' system for keeping contact details,  or itemising their stamp collection - what they want to do is largely controlled by their prejudices and what they always used to do with their 'old' system. 

 

End users aren't the best people to come up with use definitions.  Someone who has the experience and skill to 1) do the job in question,  2) use a PC effectively,  and 3) thoroughly understands how Evernote can be used needs to determine the process,  then your standard user should find it works pretty well.

 

I'm sure Evernote considers all of that in analysing the feedback they receive,  and I'm kind of reassured that in a recent interview I saw Phil confiding that Evernote have received (and ignored) death threats for something as apparently harmless as changing a font.  Apparently we users tend to react badly to change,  whatever it might be,  but gradually get used to it and eventually the complaints die down.  At least they're not afraid to stick to their guns once having made a decision.

 

On the other hand if you make a decision and the users continue to hate it... 

[Metrodon 2,184]

Apart from Skitch Gaz, when they back-peddled quicker than the french army. Or with Mac v5 where they've added back a bunch of stuff that had been removed.

[gazumped 11,666]

Well they use the product internally too,  so I guess there are some voices they just have to listen to..  but if the testing process didn't throw up the likely problems with those changes, then I rest my case - your basic users don't know enough of the answers to ask a valid question!

[BurgersNFries 2,407]

From the article: " For Evernote, it was key to get their app in front of users who had never seen it."

Although I agree it's good to make an app easy to use, I don't think catering to noobs is good either. Especially for long term use. If it's too simplified, once users become acquainted with the product, they can outgrow its usefulness. I remember when I initially got into authoring my own DVDs. I tried (and regretfully bought) Nero. It was easy to use. But after making my second one, I realized I wanted more control over my work & Nero was geared towards people who just wanted to drop a video in & burn a DVD . I ended up buying DVD Lab Pro. It's got a relatively high price tag, too. But it's so much better, IMO. Seven years later, I'm *still* using DVD Lab Pro to author my DVDs. This is not an ad for DVD Lab Pro. But rather an argument about who you want your target audience/customer to be.

[Metrodon 2,184]

Do they ever publish a number of current users? (I guess you need to define "current" somehow - hitting the service once a day maybe?).

 

The total number of 40m+ becomes a lot less interesting if the user base is only the 30 of us that post on here.

 

It's not just about attracting new users, it's about keeping them...

[BurgersNFries 2,407]

Do they ever publish a number of current users? (I guess you need to define "current" somehow - hitting the service once a day maybe?).

The total number of 40m+ becomes a lot less interesting if the user base is only the 30 of us that post on here.

It's not just about attracting new users, it's about keeping them...

Or those of us who have multiple accounts... Since I have only four in total, I guess the other 29 of you have a bunch.

[Martin Packer 162]

I still think a "contact group" of experienced users to bounce stuff off would've saved them a lot of grief and moved the products along nicely. Not to ignore nubes (somehow I prefer that to "noobs" :-) ) though.

 

Having just spent part of today in such a "leading customer" regular meeting of ours I'm reminded how such experienced users would take it seriously, perhaps challenge a fair bit, but in the end contribute greatly to evolving the state of the art. Unrepresentative sample that might be.

 

I would assume the "Dirty 30" :-) of us ARE typical of Premium users. THAT ought to be significant.

[gazumped 11,666]

Dirty 30

 

:D

[jefito 5,589]

Dirty 30

 

:D

Need a badge for that, I think...