(Archived) Text Formatting Troubles and AppleScript Updates?

[Veritrope 3]

Hey Phil, Andrew, Dave, and other members of the Mac Development Team:

As much as I'd love for you guys to devote your full efforts into making an iPad client ready for a "Day-and-Date Launch" , I've been seeing a number of posts on the forum from Mac users -- especially about formatting-related issues -- that are of growing concern to me.

Now, I suspect that improving the way that Evernote for Mac handles imported text, code, documents, etc. is already at the very top of your priority list (especially now that the Windows update has been launched!), but I thought this might be a good moment to both touch base / get a sense of where things stand.... and offer some encouragement not to take your eye off this particular ball.

The Ball:

Something that Scotty321 said in a recent thread really hit home with me:

So for now, I am forced to stick with SOHO Notes, which also syncs with the iPhone, and actually SURPASSES Evernote in dozens & dozens of different ways. While I was initially drawn to Evernote because of its web syncing ability, that one compelling feature is by no means an important enough feature to get me to RETYPE all 250 of my notes from scratch.

Over the past twelve months, the landscape of Note-Taking software aimed at Mac users has grown far more competitive: Both Yojimbo and DEVONthink Pro have seen substantial upgrades, as well as other great offerings like SOHO, Instapaper, Simplenote, Notational Velocity.... each different, but all really solid, customizable, Mac-friendly apps.

Granted -- none of your competitors are (so far) attempting what I think is a core strength of the Evernote offering: Building a truly multi-platform, partner-friendly, client-based architecture for all of your information... and making it available everywhere! I can only imagine the complexity of all of the priorities and technical issues that you guys are trying to balance among all of your offerings but, as much as I love the scope and the "WOW!" factor of things like, say, the growing number of integrations with apps like reQall, Curio, and JotNot... I make my living as a consultant who helps Mac users "digitize their lives".

It makes it hard for me to keep my clients happy when the fundamentals aren't there.

Things like having their text come out of a program the way that it goes in.

Or getting it to go in correctly in the first place.

Or being able to customize a workflow for them to easily use Apple Mail if they want to.

• [*:1cpfuehz]Could you lay out a more specific timetable for when these long-standing improvements and fixes will be rolled out?
  [*:1cpfuehz]Is there anything that we in the Mac community can do to help you? Submit samples of files / use cases where things didn't go as expected?

AppleScript Improvements:

I've also been seeing a number of posts on the forum from Mac users about issues that I've felt could be quickly resolved (or temporary workarounds provided) if they had a couple of additional AppleScript hooks to utilize. As you might remember, I suggested a few enhancements to AppleScript in this post from October --

http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=12604&p=52299&hilit=AppleScript#p52299

Here are excerpts from a handful of recent posts ...

Import/Export:

...it doesn't actually import the RTF files correctly at the current moment. It truncates all blank lines together, so an entire document with spaces between the paragraphs no longer has any spaces between the paragraphs anymore... once you bring them into Evernote. Every single line in your RTF file becomes "smushed together" within Evernote. (http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=15041)

Mail Issues:

Unfortunately, highlighting everything, copying and then pasting to Evernote does not include attachments... which is one of the main thing I'm looking for. (http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=14919)

This is a feature request for a "one button capture" (similar to Evernote button in Safari) that would allow for a direct batch import to Evernote of all selected messages in Mail.app, with the ability to tag (and/or select notebook)... [and to] keep attachments in place... which is a serious issue if Evernote is to become my "universal inbox" and digital filing system. (http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=14726)

Bulk Edits:

I have been unable to find an equivalent to the Tags Assignment dialog available in the (3.1) Windows client. This allows me to easily assign multiple existing tags to a note with checkboxes. Does an equivalent exist in the Mac version? (http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=13470)

I'd like to store my document templates (usually pdf expenses forms etc) in evernote, create a duplicate, edit the file in pdf pen and save it back to evernote as expenses August 09 or similar. At the moment this is laborious - I have to open the file in evernote, open it in pdf pen, edit it, save it to finder and then reimport to evernote and add the original tags.

Can we have a 'duplicate note' function please? (http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=11942&start=0&hilit=duplicate

A Non-Steve-Jobsian "One More Thing...":

Oh...and there seems to be a general clamor for:

1.) Direct, URI-type linking to notes.

2.) Customizable Hierarchies of Notebooks/Tags

Anything that you could do to communicate a concrete road-map to improving both "the fundamentals" and the other items above would be greatly appreciated!

[engberg 89]

Thanks again for the feedback, and all of the great work you've done with the AppleScript interfaces.

I think we're in agreement with the bulk of what you have here. We try to avoid giving specific timelines for future releases, since we don't like to look like jerks when things don't go as planned. But we've been doing work towards a lot of this lately.

For example, a lot of the text formatting/editing complaints are indirect results that our "editor" is actually the WebKit engine in "edit mode", which produces funky HTML that we then sanitize to an internal, cross-platform subset. This means that some of the odd editor behavior (e.g. around lists) stems from the 20-year history of HTML behavior, which we try to work around with a cascade of tricks.

For example, I'm 99% sure that this report is an artifact of how WebKit handles lists in "edit mode": viewtopic.php?f=38&t=15084

I bet if you tried to do the same thing in raw WebKit, you'd get the same results.

We're changing our internal representation to loosen the constraints on the HTML we produce to permit things like inline CSS 'style' attributes on elements, which should allow both a more natural editing transformation, and also permit future features that can't be expressed in our current representation (e.g. highlighting of text). For gory details:

viewtopic.php?f=43&t=14812#p57922

RTF falls into the same boat ... we're taking RTF, handing it to WebKit, and getting strange results, so need to clean that up.

I know that Geech took your "AppleScript To Do" post from a couple months ago and basically pasted that into our bug tracking system.

I think the biggest issue on the Mac for the last few months has been that we've had more bugs and features than manpower to implement them. A new engineer just started yesterday, and we're shifting one of our other senior developers onto the Mac full time, so the pace of fixes and improvements can accelerate.

Thanks

[csgrad818 2]

Thank you for this. I have a much better understanding of the problems I've been seeing.

I look forward to EN fudging its strict XHTML subset used for internal representation, to more accurately capture a variety of outputs (including the output of webkit in editor mode). I now understand these as something more than a series of disconnected problems associated with a bad editor, and understand the scope of changes required, and how this type of change might cause issues with people using the EN API.

And I hope EN gets it all together, with enough time to make a really wonderful iPad client.

[timcrawf 0]

I have no giant troubles, and your explanation of webkit use probably explains it.

That said, I would love to have real tabs, instead of just spaces.

One other great feature I would like. Forms. If we had the ability to create a re-usable form, e.g. checklist, customer info, etc. It would be great.

Wait, while I am at it, why not ask for a preference as well. I like the command Paste without formatting, but I would like that to be the default. Almost all of the time, I want the pasted text to match the note format, and not the other way around.

[Veritrope 3]

It seems like there isn't enough time in the day to reply on all of the Evernote forum threads that I'd like to... so I can only imagine how you feel, Dave!

Seriously -- I appreciate you taking the time to respond.

For the sake of clarity, I'll try to comment "in line":

1.)

I started this thread about the same time that I was writing a short piece about the iPad -- which ended up expanding into some of my recent thoughts about Evernote as well. (http://veritrope.com/opinion/the-poetry-in-not-pleasing-everyone). Maybe it was just interesting timing, but by juxtaposing the two, I really had to stop and consider the idea that the technology we use makes certain "core promises" to us as users.

In that vein, I am glad to hear that Evernote is re-committing to delivering on the fundamentals -- text editing, open import/export, etc. -- that any good note-taking program must have. My suspicion is that the Evernote team is "eating it's own dogfood" quite a bit and that, as heavy Evernote users yourselves, these current short-comings annoy you as much as anyone!

2.)

"We try to avoid giving specific timelines for future releases, since we don't like to look like jerks when things don't go as planned. But we've been doing work towards a lot of this lately."

Anybody that reads this forum knows that you are not a jerk.... (QUESTION FOR DAVE'S CO-WORKERS AT EVERNOTE: IS THIS ACTUALLY TRUE?)

You shared specifics here of what the team is working on to resolve some of these problems... and your users thanked you for it! In fact, I think the responses so far on this thread have pointed to something that would really be helpful and appreciated -- a Development Status page!

Cultured Code does it for THINGS on a special page (http://culturedcode.com/status/), but even a Sticky at the top of the Mac forum with a thread of "Stuff We're Working On For The Next Release..." would probably work.

3.)

"For example, a lot of the text formatting/editing complaints are indirect results that our "editor" is actually the WebKit engine in "edit mode", which produces funky HTML that we then sanitize to an internal, cross-platform subset. This means that some of the odd editor behavior (e.g. around lists) stems from the 20-year history of HTML behavior, which we try to work around with a cascade of tricks. For example, I'm 99% sure that this report is an artifact of how WebKit handles lists in "edit mode". I bet if you tried to do the same thing in raw WebKit, you'd get the same results.

We're changing our internal representation to loosen the constraints on the HTML we produce to permit things like inline CSS 'style' attributes on elements, which should allow both a more natural editing transformation, and also permit future features that can't be expressed in our current representation (e.g. highlighting of text). RTF falls into the same boat ... we're taking RTF, handing it to WebKit, and getting strange results, so need to clean that up."

Yeah...I've run into this type of WebKit nonsense before... recently, in fact, when I was writing that AppleScript for Scotty321 to convert his notes!

But as I mentioned in his thread (http://forum.evernote.com/phpbb/viewtopic.php?f=38&t=15041&p=59931#p59931), I really feel like this draws a line under the importance of AppleScript.

Hopefully, it's not too weird to quote myself from that thread....

The fact of the matter is that, in this context, the AppleScripts that I write for import/export are like "taking an aspirin" for minor Evernote pain, whereas people like Dave and Geech are trying to provide a long-term, lasting cure for your workflow headaches.

(Is this stretching the metaphor too far?)

But aspirin has it's place -- which is why I urge anyone who cares enough about Evernote to read through forum posts like these to ask that additional AppleScript support be made a priority. Robust AppleScript implementation means that users like us can hack together some stop-gap solutions for workflow problems while the Developers work on the deeper fixes. It would also help everyday users develop workflows and contribute integrations as a community. For the Evernote Development Team and other API Developers, this could also really help shine a light on how people actually use (or would like to use) Evernote.

Seems like a "WIN/WIN" to me!

Again, here is my not-so-subtle pitch for some additional AppleScript development....

Because of the existing AppleScript support, I was able to quickly hack something together for Scott that at least was a "Band Aid" for his problem. Another example -- before Evernote had native HTML export, I was able to hack together an HTML Exporter with AppleScript (and a little Termnal-fu). It wasn't exactly the prettiest code in the world, but it worked well as a stop-gap solution until you guys rolled out the official, baked-in feature.

Similarly, there are several complaints that I've seen here where users could at least temporarily ameliorate their own problems while you guys are working on the "big fix" -- if only there were a slightly more robust AppleScript implementation.

I think you get what I'm saying here.... Anyone else feel the same?

[scotty321 0]

Wait, while I am at it, why not ask for a preference as well. I like the command Paste without formatting, but I would like that to be the default. Almost all of the time, I want the pasted text to match the note format, and not the other way around.

You can actually change this in your Mac's system preferences. Go to System Preferences>Keyboard>Keyboard Shortcuts, and you can reassign command-V to "Paste Without Formatting".

[scotty321 0]

Thanks for this thread. My main concern at this point is import/export, and I also agree with everything else that was said here. Thanks to Veritrope's efforts, I was finally able to move my RTF notes into Evernote.

[engberg 89]

We definitely agree, and I know that "AppleScript improvements" was on my short list in a recent planning meeting.

The Mac client currently has several hundred bugs in the "important" or "very important" priority levels in our bug database. Here's a random example that Geech fixed this week:

There are a few evil PDFs on the Internet that break the OS X PDF libraries. You can open them in Preview, but if you try to search for text, the whole app hangs, and you need to Force Quit Preview.

If you put one of those into Evernote, we invoke the OS X libraries to extract the text so we can index it, and you can search for the PDF. The OS X library hangs for the same reason that Preview hangs. Your entire client hangs, and you can't use the app.

If you restart, the client picks up where it left off, and hangs again.

The only workaround was to manually delete the note on the web, empty the Trash there, and then blow away your local Mac DB and rebuild from scratch. This was time consuming and frustrating for the people affected, and they could never fix the problem themselves without Support intervention.

So Geech spent ~2 days moving all of the PDF processing code into a separate process, which he then watches carefully, and kills if it goes crazy.

Then he moves on to the next similar problem on the list ...

This is why we've added another senior developer to the Mac project, so we can work through more of these problems that completely break the client for users and get to more of the "cool stuff" like AppleScript improvements.

Thanks again

[timcrawf 0]

You can actually change this in your Mac's system preferences. Go to System Preferences>Keyboard>Keyboard Shortcuts, and you can reassign command-V to "Paste Without Formatting".

This I knew, but thanks for the reminder. I just like it better when the program has the option.

[nickgrossman 0]

Hi, I'm new to Evernote -- came here primarily for the live syncing; considering a full switch from yojimbo.

It's apparent that you guys have a lot on your plate, but I wanted to throw my chips in for "real tabs." This is the only issue that's really blocking my full adoption of Evernote. I dont' know enough about webkit to know if this is possible, but boy, it would be so, so nice. These days, I do most of my text editing in either yojimbo or etherpad, both of which have much more satisfying experiences for basic text editing (lists, tabs, etc).

Of course I'm just one guy with one set of needs....

Thanks!