TaskPaper searching, trade some potential power for more transparency?

Posted July 2, 2008 by jesse

And gain some power as a result?

A cool thing about TaskPaper is that it turns your plain old text file into an actual model that you can query against in complex ways. Because of this I decided to base my search system on NSPredicate, which allows you to create arbitrarily complex searches, all in a nice iTunes style predicate editor. The problem with this approach is that it isn't very transparent, and it's hard to present the searches in a compact form that is humanly readable. Predicate searches do have string representations, but they look like this:

The search that defines (focus on inbox) looks like this:

topLevelProject.content == "Inbox"

The search that says find all entries that are tagged with "priority" and have a value of "1" looks like this:

SUBQUERY(tags, $tag, $tag.name == "priority" AND $tag.value == "1").@count != 0

So doing search this way gives the ultimate in power, but it's not really transparent how things are working. The searches don't have an easily understood text form. (an area that you would expect taskpaper to be strong in). They also are not very hackable, the only way to create or edit a search of any complexity is through the visual query builder.

The more I think about it the more I think I might have headed down the wrong path with TaskPaper's search. Jordan's TodoPaper app takes a different approach that is starting to look more appealing to me. Instead if using a visual builder for complex searches he's defining a TaskPaper specific query language that currently works something like this:

find this - finds all entries that contain the text "find this"

@today - finds all the entries that contain the text "@today".

@today NOT @done - finds all entries that contain the text @today and that do not contain @done

<Inbox> - finds all entries that are in the Inbox project.

<Inbox> find this - finds all entries in the Inbox project that contain the text "find this".

I'm not sure about TodoPaper's exact syntax right now, but the approach seems to be more in the TaskPaper spirit. Besides "text based" and hackable, another big benefit is that all searches would be "Visible". Currently TaskPaper makes it clear that you are in search mode, but it doesn't have a good way to let you know what you are actually searching for.

The two drawbacks would be:

  • Potentially harder for new users.
  • Potentially less powerful.

It would be harder for new users because instead of a nice visual builder they would have to figure out the search syntax. But I'm not so worried about this for two reasons. First TaskPaper users are really smart! :) And second I think that it's a fairly small group of TaskPaper users who are building up complex searches themselves. Most common TaskPaper searches (such as focus on project) will still be built automatically for the user when the user clicks on a project link, or clicks on a tag.

Potentially less powerful is more important. For theming to be useful we need the query language to be powerful enough so that it can describe what we want to theme. Because of that I think that TodoPaper's query language needs to be modified (or I need to understand it better) so that it's powerful enough to describe the themes that we want to build. Off the top of my head lots of our current themes need to match against entry type (project, task, note). Etc.

So what do people think? I'll talk with Jordan more about TodoPaper's query language, but if anyone has suggestions for making it more powerful (and also human readable) please send in some comments!

reply | edit

Matthew Crider - July 2, 2008 9:38 AM

Why does it have to be one or the other? Spotlight searching is analogous: one can build Boolean searches on one line or one can build a search in the query builder. Why not do both? It's just a translation from one language to another. Why not have the choice to use Jordan's syntax or to use the query builder?

Currently, you could probably translate your search structure as is and have it identify a TaskPaper search with Jordan's syntax.

Matt

reply | edit

Jordan Sherer - July 2, 2008 9:52 AM

I think it will be very beneficial to define a search syntax that is consistent within both apps. This way, Mac and PC users won't have to learn two different ways of interacting with their to-do list. Plus, I feel that it is much easier to learn:

@today NOT @done

Than opening, and building predicate searches with:

All of the following are true:
Line contains @today
None of the following are true:
    Line contains @done

What do you think?

reply | edit

Jordan Sherer - July 2, 2008 9:57 AM

One thing that confused me a little bit is how you said you are using the search syntax (i.e., the predicate builder) to define the themes. How does this work in the backend TaskPaper styling code?

reply | edit

jesse - July 2, 2008 10:39 AM

The theme UI allows the user to associate text attributes with a predicate. Then taskpaper uses that to color the document. Anytime text is changed in taskpaper:

  1. All text attributes are set to defaults:
  2. Each predicate in the them is run against the changed entry
  3. For each theme predicate that matches the changed entry, the associated text attributes in that theme are applied to the entry.

reply | edit

jesse - July 2, 2008 9:54 AM

Long term it doesn't have to be either or. But short/medium term it does, because the effort of rebuilding the current query builder for a custom language is to much.

reply | edit

Matthew Crider - July 2, 2008 9:58 AM

I'm not opposed to the change, if it includes complex Boolean and if it includes symbols for AND, NOT, and OR - similar to the way Google allows in their searches. I would hate to have to build a search using all caps for the Boolean terms.

Matt

reply | edit

Jordan Sherer - July 2, 2008 10:18 AM

Currently TodoPaper uses the word and/or a symbol:

AND => &
OR => |
NOT =>  !

So you can search like this:

@today | @tomorrow ! @done

Which translates into:

@today OR @tomorrow NOT @done

However, if you use the lowercase word, i.e., @today not @done, it doesn't register as a boolean term. This is so you can search for lines containing those actual words in a query.

reply | edit

Matthew Crider - July 2, 2008 10:28 AM

And with parentheses to add complexity? If so, then I'm all in favor of the change.

Matt

reply | edit

Jordan Sherer - July 2, 2008 10:53 AM

Currently parens are unsupported, but definitely can be added.

reply | edit

Matthew Crider - July 2, 2008 10:57 AM

Great! Because what's Boolean, if you can't make it complex?

For example: ((term1 & term2 & term3) | (term4 & term5 & term6)) ! term7

Matt

reply | edit

Maccer - July 6, 2008 5:55 PM

Yes, in my opinion parentheses are a must. Even though I have programmed a lot I have never learnt the precedence of boolean operators. Without parentheses it's hard to know whether

@today OR @tomorrow NOT @done should be read as

"tasks due today or tomorrow that are not done" (@today OR @tomorrow) NOT @done

or

"tasks due today (@done or not @done) plus tasks due tomorrow that are not done" @today OR (@tomorrow NOT @done)

reply | edit

jesse - July 2, 2008 11:46 AM

See this query language:

http://zing.z3950.org/cql/intro.html

It's solution to "and" being a boolean operator, or a search for the world "and" is that if your search contains keywords you just need to quote it. If not quotes are optional.

So:

"and or not" - is just a normal contains text search

one and two - while this is a search for contains both "one" and "two".

"one" and "two" - same as above, with optional, but unneeded in this case, quotes.

This seems a little nicer then forcing uppercase words or special symbols.

reply | edit

Matthew Crider - July 2, 2008 11:56 AM

But, special symbols are easier. They are less to type. And, anyone who Googles a lot already knows these symbols.

Matt

reply | edit

jesse - July 2, 2008 12:19 PM

I'm not sure if they are easier. Holding down the shift key while typing AND, OR, NOT is always a bit of a pain. Also, to my eye upper case terms like that look "scarry", looks like I'm trying to type SQL, something that always frightens me :)

Also the set of special symbols will likely grow beyond the basic boolean operators. For example you guys were just talking about adding parenthese support for the boolean operators... but then how to you search for the text "(find me)". Or eventually we'll want to allow for >, <, etc so that you can search against a tags value. There may be ways to work through all those cases in the search format, but to me the simple rule of encase in quotes if it contains symbols seems like a really easy and strait forward approach.

reply | edit

Matthew Crider - July 2, 2008 1:55 PM

Jesse,

I'm not sure if they are easier.

Well, they're at least as easy.

For example you guys were just talking about adding parenthese support for the boolean operators... but then how to you search for the text "(find me)".

How? Just exactly as you typed it:
(("(find me)" | "or here is another phrase") & term1) ! term2

There may be ways to work through all those cases in the search format, but to me the simple rule of encase in quotes if it contains symbols seems like a really easy and strait forward approach.

I'm not sure how this statement goes with the rest of your argument, because I agree: Use symbols and if you're looking for the symbol, put it in quotes.

Matt

reply | edit

jesse - July 2, 2008 2:57 PM

Maybe we are actually in agreement.

My point was that I don't want to force people to type in uppercase. But I also don't want their only other option to be using &,|,!. I think they should also be able to use lowercase and, or, and not as operators. I'm happy with allowing all three choices, I just don't want to disallow lowercase operators.

I think the normal reason for forcing uppercase AND, OR, and NOT is so that they won't conflict with actual text that is being typed. But since that's easy enough to put in quotes, I don't think it's a problem.

reply | edit

Matthew Crider - July 2, 2008 3:14 PM

Jesse,

We're in complete agreement. I think uppercase AND, OR, and NOT are the least attractive alternative.

Now, if we can just get that collapsing going! :-)

reply | edit

Rob Cope - July 2, 2008 10:25 PM

See this query language: http://zing.z3950.org/cql/intro.html

Works for me.

Rob

reply | edit

Rob Cope - July 2, 2008 10:23 PM

So doing search this way gives the ultimate in power, but it's not really transparent how things are working. The searches don't have an easily understood text form. (an area that you would expect taskpaper to be strong in). They also are not very hackable, the only way to create or edit a search of any complexity is through the visual query builder.

I'm all for hackable text searches. One thing I wouldn't want to lose from the current search system, however, is the ability to name a search. In most cases, and especially for a complex query, I would much rather see a descriptive name than have to stare at the query and parse out what it was doing.

Rob

reply | edit

James King - July 10, 2008 8:32 AM

I have just upgraded from 1.0 to the development version 53. Lots of nice new features but I am confused with the way that the search functionality has been implemented in the development version. It seems to be counter to the plain-text simplicity of version 1.0.

Previously, if I want to search for a tag named "bob" then I could type "@bob". This seems perfectly in-line with the fact that I was searching through a plain text document. The search syntax matched the document syntax perfectly, because they are the exact same thing.

Fast forward to development version 53 and now I have two ways of searching for a tag:

1) line contains "@bob"

2) any tag name is "bob"

These two queries are equivalent, but Taskpaper isn't aware of this. This leads to two problems:

  • TaskPaper treats each of the above queries differently. This is most apparent when looking at the search field after the query is made. For query 1, it shows "@bob", for query 2 it remains blank! This is very confusing!

  • The simplicity and elegance of TaskPaper is muddled, making it less trustworthy.

This leads me to think that the predicate based search isn't a good fit for Taskpaper. A plain-text task management application needs plain-text search functionality!

I realise that some users would find it necessary to build up complex boolean searches and it should be possible (although personally, I have never needed to do this). For this use I would much prefer a text-based search syntax with a firm basis in the Taskpaper syntax.

On separate note, I think the predicate-based system works really well for defining the rules for themes.

reply | edit

jesse - July 10, 2008 8:53 AM

Just an update here... I've pretty much decided that the current 2.0 searching needs to be scrapped and replaced with a text based method. Just need to implement it which is going to take a bit, but I think it will be a good investment of time overall. Also just one little note. "line contains @work" and "entry has tag @work" don't always give the same answer, because the line could be "look at this email, jesse@work". In that case "@work" is in the line, but it isn't parsed as a tag.

reply | edit

Jordan Sherer - July 10, 2008 9:10 AM

Sounds great Jesse. I am looking forward to working on a new query language with you. I have some more ideas about the CQL that I'd like to share with you sometime.

reply | edit

James King - July 10, 2008 9:23 AM

Also just one little note. "line contains @work" and "entry has tag @work" don't always give the same answer, because the line could be "look at this email, jesse@work". In that case "@work" is in the line, but it isn't parsed as a tag.

Good point. This stuff is complicated isn't it!

reply | edit

James King - July 10, 2008 11:30 AM

I have thought a lot about this today. I don't think scrapping the current search is the answer and I don't think creating a complicated query language is either. I think the problem I complained about in my previous comment is actually a simple usability issue that could be quite easy to address.

Perhaps the current search function is convoluted because it is trying to do two similar, but distinct things:

  • Search for things in a quick and dirty way. These are searches that you would make once and would be unlikely to use regularly. This is the sort of simple, no-frills, searching you could do with any text editor, but TaskPaper kindly provides the results as a nicely formatted list organised by project or tag. Just like Taskpaper 1.0's search, its transparent but dumb. Example searches:

    • "jesse@hogbay"
    • "@bob"
    • "toothpaste"

  • Filtering/Focusing the view. These are searches (or "views") that you use regularly for specific purposes. They are extensible and customisable. They needn't be particularly transparent as they are for more advanced users who have a specific purpose in mind. They could be exportable and shared with other people.

    • Next Actions
    • Not Done
    • Tagged with @waiting and @Bob
    • Tagged with @pending and dated "2007"

Rather than scrapping the current search system, perhaps you could provide two distinct interface elements for these two distinct purposes:

  • For the first purpose I would imagine a search box that I can type into quickly and that I know is doing nothing more than matching what I type to the raw text in the document.
  • For the second purpose there is a drop-down list of pre-defined "Views" (or "Focuses" perhaps) with names like "Next actions", "Not Done" and "Waiting for Bob". There is no text-based syntax for this type of search. The predicate system would be perfect.

I think this would be very similar to what you have already built in v53. The only difference is to separate the interface elements, and disambiguate the two types of search, even if its just a case of calling them something different.

reply | edit

Topic's comments