NotDeft

First download the source code with the command

git clone https://github.com/hasu/notdeft.git

Go to the “notdeft” directory, and prepare the Emacs Lisp code for use with the command

make

Add the directory containing those files to the Emacs search path by adding

(add-to-list 'load-path "/path/to/repo/of/notdeft")

to your Emacs startup file (e.g., “~/.emacs”). Also add

(require 'notdeft-autoloads)

to the Emacs startup file, to make NotDeft available for on-demand loading.

With the above setup work done, NotDeft is available for launching from within Emacs with the command

M-x notdeft

While the above commands acquire, build, and set up NotDeft's Emacs Lisp code, they do not build and configure the C++-based Xapian backend; see Building the Xapian Backend and Configuring the Xapian Backend.

The straight.el package manager is able to install NotDeft as a package directly from its source repository. If you have that manager correctly installed, then you can install NotDeft with the command

(straight-use-package
 '(notdeft
   :type git :host github :repo "hasu/notdeft"
   :files ("*.el" "xapian")))

which should download NotDeft, generate its autoloads, and handle Emacs Lisp file byte-compilation.

While that command downloads and unpacks the Xapian backend source code, it does not build it or configure it; see Building the Xapian Backend and Configuring the Xapian Backend.

Installing from Git is recommended where you wish to be sure that you are installing the most recent available version. Still, installation from a downloadable package file is also an option.

To install NotDeft as a package, first download the (chosen version's) package, and then install the downloaded TAR file with

M-x package-install-file

You can check whether the package has been installed by evaluating

(package-installed-p 'notdeft)

If so, information about the installation can be shown with

(describe-package 'notdeft)

No documentation is shown by that command, but it does show the location of the package's files, allowing navigation to the documentation.

One might also implement a command for opening something in the package. For example, the readme file can be opened with

(defun notdeft-open-readme ()
  (interactive)
  (find-file
   (expand-file-name
    "README.org"
    (package-desc-dir
     (cadr (assq 'notdeft package-alist))))))

While installing the package does unpack the Xapian backend source code, it does not build it or configure it; see Building the Xapian Backend and Configuring the Xapian Backend.

To enable Xapian search queries, you should build the notdeft-xapian C++ program in the “xapian” directory. On some systems simply going into that directory and typing

make

should do the trick, provided that the required tools and libraries have already been installed. Other systems may require more work not only on satisfying the dependencies, but also in finding the right C++ compiler incantation for building the program. (On some systems building it may not be feasible at all, and NotDeft's functionality will be more limited.)

Once a working compiler invocation command has been found, and the necessary C++ libraries have been installed, it is also possible to build the C++ program from within Emacs by using the included notdeft-xapian-make Emacs Lisp feature. To use it, set the variable notdeft-xapian-program-compile-command-format with the appropriate format string for the compilation command. In that format string the path of the executable comes first, and the path of the source file comes second. For example:

(setq notdeft-xapian-program-compile-command-format "g++ -o %s %s -std=c++11 -Wall `pkg-config --cflags --libs tclap` `xapian-config --cxxflags --libs`")

With the feature appropriately configured you can then try issuing the command

M-x notdeft-xapian-compile-program

which should display any build errors if the executable cannot be built.

In a simple case you would have a single directory (tree) of note file, specified by the notdeft-directories configuration variable, which you can configure with the command

M-x customize-variable RET notdeft-directories RET

For example:

(setq notdeft-directories '("~/all-my-notes"))

You can have multiple directories, which makes NotDeft use a bit harder, as you may at times get asked for a target directory for some file operations.

(setq notdeft-directories '("~/some-notes" "~/some-more"))

If your notes are not in a fixed directory, but you'd rather discover the directories programmatically, it may be convenient to set notdeft-directories in your startup file. For example:

(setq notdeft-directories (cons "~/notes" (file-expand-wildcards "~/*/notes")))

The default is to have the note filename notdeft-extension set to "org" to indicate the Org format. If you prefer some other note format, you should change that setting, which can be done with

M-x customize-variable RET notdeft-extension RET

The configured notdeft-extension is used by default when creating new notes, but a note collection can also use other extensions. There are none by default, but you can define such secondary extensions with

M-x customize-variable RET notdeft-secondary-extensions RET

For example, one might set these as

(setq notdeft-extension "txt")
(setq notdeft-secondary-extensions '("md" "org" "scrbl"))

When creating a note file with the notdeft-new-file-named command, NotDeft automatically derives a name for the file based on the title that is provided for the note. The configuration option notdeft-notename-function determines how the name is derived.

The default setting is to use the notdeft-default-title-to-notename function to translate the title to a file basename. For example, the title “Rust (programming language)” translates into

rust-programming-language

The default implementation is suitable for titles with ASCII letters, and you probably want to pick a different implementation if your titles do not tend to use the English alphabet.

Where necessary, the configuration option notdeft-new-file-data-function provides even more control over the naming (and content) of new notes. Such a function could, for example, add some metadata to every new note's file name and/or content.

To have NotDeft use the notdeft-xapian program you've built, you will have to specify its full path in the notdeft-xapian-program variable. You could use M-x customize-variable to set it, or simply

(setq notdeft-xapian-program "/path/to/notdeft-xapian")

Set the variable to the program's full absolute path, without any shorthands, as no shell expansion is performed on the path name—you may explicitly expand it using Emacs' expand-file-name function instead.

If you installed as a package, and built the notdeft-xapian executable in that location, then the appropriate setting may be

(setq notdeft-xapian-program
      (expand-file-name
       "xapian/notdeft-xapian"
       (package-desc-dir
        (cadr (assq 'notdeft package-alist)))))

Such code must appear after

(package-initialize)

See the other notdeft-xapian-* customization variables for configuring the Xapian indexing and searching behavior. Most notably:

• The configuration variable notdeft-xapian-max-results controls the maximum number of files to list in a notdeft-mode buffer. You may set it to 0 to always have all results displayed.
• The default is to order the results so that most recently edited files are listed first, but you may change this behavior by setting notdeft-xapian-order-by-time to nil, in which case Xapian's ranking mechanism is used instead.

The filter string “emacs org mode” narrows a *NotDeft* buffer file list down only to the files that contain all of the substrings “emacs”, “org”, and “mode”. To see each of the matching positions within those files, consider entering the command C-c g (or M-x notdeft-grep-for-filter) to display the matching strings with highlighting. That command invokes the shell command grep (through the Emacs command grep), and displays the results in a separate buffer. This may fail to work if you system does not have a compatible grep executable on the search path.

NotDeft allows multiple notdeft-mode buffers to exist at once, which may be useful if one wants to explore multiple sets of search results at once. Each NotDeft buffer has its own state, including a search query, filter string, default directory for creating new notes, etc.

Normally, executing the notdeft command only creates a new *NotDeft* buffer if one does not already exist—otherwise the command merely switches to an existing *NotDeft* buffer. It is possible to have the command always create a new notdeft-mode buffer by invoking it with a prefix argument, i.e., C-u M-x notdeft.

The notdeft-open-query command also accepts a prefix argument, to arrange for the search results to be listed in a new buffer. This behavior can also be made the default for that command by setting the configuration parameter notdeft-open-query-in-new-buffer to t. With that parameter set, the prefix argument's meaning is inverted, so that C-u M-x notdeft-open-query does not create an additional buffer.

The question of whether to create a new buffer does not apply to other search commands. Within a NotDeft buffer, the commands notdeft-query-edit and notdeft-query-clear merely replace the buffer's search result set, whereas the commands notdeft-lucky-find-file and notdeft-query-select-find-file do not use a NotDeft buffer for displaying their results.

For dealing with existing notdeft-mode buffers, there is a notdeft-switch-to-buffer command for interactively selecting a buffer and switching to it. It presents a choice list of buffer names in the minibuffer, and shows any query and filter strings associated with those buffers for better informed selection.

As for closing NotDeft buffers, the notdeft-quit command is bound to C-c C-q, and it can be invoked in three ways:

1. Without a prefix argument, it buries the current buffer.
2. With one prefix argument, it kills the current buffer.
3. With two prefix arguments, it kills all notdeft-mode buffers.

By default, NotDeft does not show any note directory or file names in its list view, but this behavior can be controlled by specifying a notdeft-file-display-function.

For example, we can display the name of each note's containing NotDeft (root) directory, with abbreviations for long directory names:

(setq notdeft-file-display-function
        (lambda (file w)
          (when (> w 30)
            (let* ((s (file-name-nondirectory
                       (directory-file-name
                        (notdeft-dir-of-file file))))
                   (s (pcase s
                        ("bibliography-notes" "bib")
                        ("homepage-notes" "hp")
                        (_ s)))
                   (s (if (> (string-width s) 12)
                          (truncate-string-to-width s 12)
                        s)))
              (concat " " s)))))

We refrain from displaying any directory information in cases where the Emacs window is very narrow (as indicated by the w argument), as otherwise there will be little space left for the note titles.

The simple approach is to always enable notdeft-note-mode for the major mode(s) that you use for editing notes. For example:

(add-hook 'org-mode-hook 'notdeft-note-mode)

This approach should be safe in that changes to files not residing in notdeft-directories get ignored by NotDeft. Still, the approach has the disadvantage that the minor mode indicator “¬D” does not tell you whether a note is actually a NotDeft note.

Another solution is to try enabling notdeft-note-mode for every NotDeft directory in terms of per-directory local variables. For example, have your “.dir-locals.el” file state

((org-mode . ((mode . org)
              (mode . notdeft-note))))

This way of declaring both a major and minor mode appears to work at least in some versions of Emacs, although it may rely on undefined behavior.

If enabling notdeft-note-mode directly in “.dir-locals.el” does not work or appeal to you, then it's possible to do the same thing indirectly, by using an actual per-directory local variable to indicate if the minor mode should be enabled. That is, you can have the “.dir-locals.el” file contain

((nil . ((notdeft-note-mode-auto-enable . t))))

The variable can be declared as

(defcustom notdeft-note-mode-auto-enable nil
  "Whether to enable NotDeft note mode for a buffer."
  :type 'boolean
  :safe 'booleanp)
(make-variable-buffer-local 'notdeft-note-mode-auto-enable)

To set that variable for a note directory, we can use the Emacs command

M-x add-dir-local-variable RET nil RET notdeft-note-mode-auto-enable RET t RET

Or, if we want to programmatically set the variable for all our notdeft-directories, we can use the code

(dolist (dir notdeft-directories)
  (let ((default-directory dir))
    (add-dir-local-variable nil 'notdeft-note-mode-auto-enable t)))

Defining and setting the variable alone does not enable the mode, which we want to do only for specific file types, reflecting our notdeft-extension and notdeft-secondary-extensions configuration. If we only supported org-mode files, we would like to say something like

(add-hook
 'org-mode-local-variables-hook
 (lambda ()
   (when notdeft-note-mode-auto-enable
     (notdeft-note-mode 1))))

We cannot just use org-mode-hook, as directory locals are not yet set at the time when the mode is enabled. What is needed is a later hook, which in the above is called org-mode-local-variables-hook.

We also have to get such hooks to run. Borrowing code from “phils” at Stack Overflow, we can get our org-mode-local-variables-hook run by defining and registering a new kind of hook as

(defun run-local-variables-mode-hooks ()
  "Run hooks for `major-mode' with locals set.
Like `run-mode-hooks', but run later, with any buffer and
directory local variables set."
  (run-hooks (intern (concat (symbol-name major-mode)
                             "-local-variables-hook"))))
(add-hook 'hack-local-variables-hook 'run-local-variables-mode-hooks)

The above solution gives us a “proper” way to enable the NotDeft note minor mode, and to do it only within directories that have a persistent NotDeft “signature” (in a “.dir-locals.el” file), and only for our chosen note-editing major modes.

Some of NotDeft's commands have specific support for use from within NotDeft note buffers. For example, the notdeft-rename-file command can be useful for renaming a note file that was perhaps created without a proper name (e.g., by using C-c C-n). Having written a note in a current buffer, issue the command

M-x notdeft-rename-file

to enter a new basename for the file of that buffer. Any C-u prefix causes the default value to be derived from the title of the note, as extracted from the buffer contents. (The same command also works in a *NotDeft* buffer, affecting the currently selected file.)

You might also implement additional commands in terms of the globally accessible commands and Emacs Lisp functions, for example for quickly listing documents tagged in a certain way:

(defun my-open-todo-notes ()
  (interactive)
  (notdeft-open-query "tag:todo"))

An intended use case for NotDeft is to support other applications that wish to locate files in terms of search queries instead of path names. For example, suppose we are using an org-contacts command to look for contacts by name, and that command expects the org-contacts-files list to be set. In that scenario we might set that variable for it based on a suitable NotDeft search query:

(setq org-contacts-files
      (notdeft-list-files-by-query
       "!all ext:Org AND Email"))
(org-contacts name)

Similarly, we might use org-agenda's org-todo-list command to list to-do entries, but resolving the org-agenda-files list on demand by looking for the “TODO” and “DONE” keywords in any Org files in our collection:

(setq org-agenda-files
      (notdeft-list-files-by-query
       "!all ext:Org AND (Todo OR Done)"))
(org-todo-list)

No special markup is necessarily required:

this is a title

This is body text.

Comments can be included, and they are ignored when searching:

# this is a comment
this is a title

This is body text.

Org mode's #+TITLE syntax is supported:

# this is a comment
#+TITLE: this is a title
# this is a comment

This is body text.

A note can be tagged, e.g., with the tags “some” and “tags”:

#+TITLE: this is a title
#+KEYWORDS: some tags

This is body text.

Instead of the #+KEYWORDS syntax, we can use the Org standard #+FILETAGS syntax:

#+FILETAGS: :some:tags:
this is a title

This is body text.

Stemming is used also on tags, and so the query “tag:tag” will find these two notes (assuming English stemming—see notdeft-xapian-language).

Whitespace is considered as a separator for tags, as are the delimiters “:”, “;”, and “,”. This means that the keyword declaration

#+KEYWORDS: helsinki-vantaa places

is not matched by the search phrase “tag:"vantaa places"”. However, a hyphen still separates words, so that “tag:helsinki” and “tag:vantaa” and “tag:helsinki-vantaa” all match the first tag, which is semantically appropriate at least in this case.

The following prefixes are supported by NotDeft:

file:

Indicates that the search term must appear in the (non-directory, non-extension) filename.

ext:

Indicates the string that must be the filename extension of the file (without the ".").

path:

Indicates that the search term must appear in the non-directory part of the file pathname, where the pathname is relative to the user's home directory.

title:

Indicates that the search term must appear in the title.

• Title is specified either as the first non-empty non-comment line, or as the file property (or Org mode “in-buffer setting”) #+TITLE. (Multiple #+TITLE lines are not supported.)

tag:

Indicates that the search term must appear among the tags given to the document.

• The tags for a note are specified either with the standard Org file property #+FILETAGS, or the custom file property #+KEYWORDS. (Org headline tags do not qualify.)

The following custom query syntax is supported:

!time

Prefix a query with !time to have the results sorted by decreasing file modification time, even if the notdeft-xapian-order-by-time configuration option is disabled.

!rank

Prefix a query with !rank to have the results sorted by decreasing relevance, regardless of the notdeft-xapian-order-by-time setting.

!file

Prefix a query with !file to have results sorted by (non-directory) file name, alphabetically, in decreasing order. Overrides all of the other sorting settings and modifiers.

!all

Prefix a query with !all to show all matching results. Note that unless you specify this modifier, the contents of a query result list may differ depending on how the results are sorted, since less highly ranked notes may get excluded.

A space character must be used to separate the above keywords from the rest of the query string.

The !file modifier might be useful for instance when you have file names such as “2017-01-01-0001.txt” and “2017-09-19-0123.txt”, and you would like to see them in chronological order by “creation time”, even if some of the files have been edited, and consequently have had their modification times changed.

It is simple to find all notes containing both the words Emacs and Org:

Emacs AND Org

If you have a lot of notes about Org mode, and few about other Emacs matters, it may be interesting to use

Emacs AND NOT Org

which works if the notdeft-xapian-pure-not option is set.

While you're often likely to be more interested in recent (or best maintained) notes, sorting by relevance can be useful particularly when there are multiple search terms: you may be more interested in seeing notes that contain all the terms instead of just one of them. You may use “!rank” to enable relevance-based ranking for a specific query:

!rank Emacs Org Deft

If, on the other hand, you use a single, common search term, and have a lot of documents, you may run into your notdeft-xapian-max-results limit, and miss out on some documents. In this case, you might use

!all Emacs

to list all documents mentioning Emacs.

If, unlike in the above case, you just want to see all documents that are about Emacs specifically, you may get more useful results with the query

title:Emacs

to only find documents whose title indicates that they concern Emacs. Or, to be more thorough, you might want to make sure you also find notes with the word Emacs in the filename:

title:Emacs OR file:Emacs

You can combine prefixes and “bracketed subexpressions”:

title:(Ayn AND Rand)

which will match both “Ayn Rand” and “Rand, Ayn” in a title.

Phrase searches are allowed for tags, and

tag:helsinki-vantaa
tag:"helsinki vantaa"
tag:(helsinki AND vantaa)

all match the tag “helsinki-vantaa”.

Filename extensions can be capitalized to avoid any stemming. For example, to find all “.org” documents that may contain open to-do entries, we might query with

!all ext:Org AND TODO

It is “deft:” links in particular that allow NotDeft to be used as a desktop wiki, linking documents by topic, where a topic is named by the non-directory name of a note file. For “deft:” links to consistently resolve to the same note, you should name your note files uniquely.

For example, when following the link

[[deft:notdeft.org]]

NotDeft will look for a “notdeft.org” file anywhere in the note collection, and open the first match.

A benefit of that “deft:” link semantics is that using the command

M-x notdeft-move-file

to move a note file into a different directory does not cause any “deft:” link to break, whereas regular “file:” links may break.

To conveniently create a dedicated note for a given topic in an Org-mode buffer, and also link to that note at the same time, highlight the title (and link description) of that topic so that it becomes the active region, and then issue the command

M-x notdeft-link-new-file

For example, if you've highlighted the text “desktop wikis”, the command will offer to create a note of the same title, derive a filename for it based on the title, and replace the region with a “deft:” link to it. (The command is defined by the notdeft-org feature.)

To configure Firefox to support the org-protocol: scheme, first open about:config, and add a boolean property

network.protocol-handler.expose.org-protocol false

Then craft an HTML file such as

<html>
  <body>
    <a href="org-protocol://store-link?url=URL&title=TITLE">link</a>
  </body>
</html>

and open that file in Firefox, and click the link, after which a “Launch Application” dialog is presented. “Choose other Application”, tick the box “Remember my choice for org-protocol links”, and specify emacsclient as the executable.

That application selection can later be modified from Firefox “Preferences” / “Applications”. If required, the “Content Type” should be removable at least by editing the “mimeTypes.rdf” file in the Firefox profile.

There is nothing NotDeft specific about the store-link Org protocol, as it merely stores a link to the Emacs kill-ring for yanking. To configure Firefox to support the protocol, just add a suitable bookmarklet (e.g., to the “Bookmarks Toolbar”). The bookmark “Location” can be specified as

javascript:location.href='org-protocol://store-link?url='+encodeURIComponent(document.location)+'&title='+encodeURIComponent(document.title);void(0);

By selecting that bookmark a link to the current page can be sent to Emacs. Its URL can then be inserted in Emacs with C-y. A full Org link in turn can be inserted with

M-x org-insert-link

which is bound to C-c C-l in Org.

The capture protocol, in turn, allows for web page content and metadata to be captured from Firefox into Emacs. Configuring the capture protocol for use with NotDeft is slightly more involved than in the case of store-link, as we must choose what page data to store, and where in our NotDeft note collection to store it.

Suppose we wish to store any currently selected text, along with the URL of the containing page, and a capture timestamp. Suppose also that we wish to store it into a file whose name is derived from the page title, so that if we capture multiple times from the same page, then all of the captured text snippets will end up in the same note file.

In that case the Firefox bookmarklet for sending over the required information can for example be

javascript:location.href='org-protocol://capture?template=w&url='+encodeURIComponent(document.location)+'&title='+encodeURIComponent(document.title)+'&body='+encodeURIComponent(window.getSelection());void(0);

where we have given the name “w” for the Org capture template.

We must also define that template as one of our org-capture-templates, and the definition can be

(require 'org-protocol)

(setq org-capture-templates
      '(("w" "capture selection into NotDeft" plain
         (file (lambda ()
                 (notdeft-switch-to-file-named
                   (plist-get org-store-link-plist :description))))
         "%l\non %u\n\n%i"
         :empty-lines-before 1)))

This definition assumes that the link :description is available from org-store-link-plist, and that it corresponds to the document.title; this may be undocumented functionality, but works in Org mode 9.1.1. The notdeft-switch-to-file-named derives a filename from the description, creates that file if it doesn't yet exist, and returns the complete file name.

Try doing the following in a *NotDeft* buffer:

1. Press TAB (or M-x notdeft-query-edit) to be prompted for a Xapian query.
2. If nothing happens when you press TAB, then you have probably not configured a value for notdeft-xapian-program. Assign a value to that variable.
3. Having pressed TAB, enter a query string at the prompt, one that should match some notes, and press RET.
4. If that reports "Found no notes", or an unexpected set of notes, then your search index may not be up-to-date, perhaps due to filesystem changes outside of NotDeft. Invoke the command M-x notdeft-refresh (i.e., C-c C-x g) to refresh the search index.
5. If you suspect that your search index may be corrupt or incompatible in some way, you may invoke the command M-x notdeft-reindex (i.e., C-c C-x r) to fully rebuild the search index, instead of just refreshing it.

6. If you see unexpected behavior after setting a search query, ensure that the notdeft-xapian-program variable names the complete and fully expanded path of a working executable. It may be worth trying to run the program directly, and seeing what it says. For example:

   /path/to/notdeft-xapian search -q 'Emacs OR Vi' ~/.deft
   

7. If your search query includes prefix terms such as “title:Emacs”, and you do not get all the expected matches, then make sure that any lines before any #+TITLE (or, #+KEYWORDS, etc.) are either whitespace only or begin with “#”. While the Org markup language allows in-buffer settings to appear anywhere in a file, NotDeft only scans the beginning of each file for such settings.
8. If all else fails, a tool such as xapian-delve may be used to inspect the contents of the search index to see which terms it actually contains.