Javascript and Emacs Lisp

Warning: if you don't subscribe to the One True Religion, this probably won't interest you.

With my recent affair with Elisp and the development in GNOME where some influential people are dropping C in favour of Javascript for GUI programming (we may wonder how many kilolines of boiler-plate C code they've churned out to reach that conclusion), I couldn't help thinking about replacing Elisp with Javascript in Emacs.

First a bit of introduction. Emacs is an incredibly old and incredibly smart text editor with a smallish C core and most of the intelligent stuff written in a Lisp dialect called Emacs Lisp. Lisp is a very flexible language that intermingles data and code. Making a system built on it extensible is relatively straight-forward. This is probably the main reason behind the success of Emacs, the editor has been extended in literally thousand of directions over the years. If you read my previous blog post, I'm using what nowadays is dubbed aspect-oriented programming to hack a built-in Emacs Lisp module.

It's my impression that even within the Emacs community there are few people who think Emacs Lisp is a great language. It's not Common Lisp, it has some weird idioms and as a small Lisp dialect it still lacks the simplicity of Scheme.

However, weird idioms aside, in my humble opinion the big problem with Emacs Lisp is momentum. When Emacs was started, Lisp was hot. People were thinking that Lisp would be the future. Back in those days, they even had hardware that ran Lisp! The mind boggles. Lisp was an obvious choice for a dynamic extension language.



Today much has changed and few people are using Lisp. The obvious choice for a dynamic extension language today is something else. Basic, Python, Lua, etc. Or Javascript.

A die-hard Lisp fan would think the solution is another Lisp, but the official GNU Scheme embeddable language Guile looks pretty dead to me. While learning Lisp is a worthwhile goal that will definitely teach you some lessons, I think there's good reason why most programmers aren't writing parenthesized lists all day long.

By switching to Javascript, Emacs would once again be ahead of the curve. And the Emacs maintainers would be relieved of having to maintain a language interpreter with its related problems. For instance, and I don't know if things have changed recently, the garbage collector in Emacs used be pretty basic. Probably because when it just sort of works, nobody wants to mess with it anymore, people naturally want to focus on writing the best editor possible.

I don't think the idea would fly, however, for several reasons. There is an enormous body of Lisp code out there for Emacs. And since Lisp is the extension language for Emacs, everybody contributing to it is a Lisp hacker. It's hard to see how this actually pretty large group of people making up the community would accept anything else than a Lisp. And in the past, even proposals to reform Emacs Lisp or switch to other Lisp dialects have all failed.

To add to that, Javascript has a bad reputation, especially among people who haven't done any client-side web development lately.

Thus I think the only way Javascript in Emacs could happen meaningfully is by targeting the interpreter: replace the Emacs Lisp interpreter with a Javascript engine like Spidermonkey or V8 and a bridge that translates the Lisp and makes Lisp symbols available to Javascript and vice versa. So all Lisp code would run unaltered, and everyone can continue writing Lisp as they do today.

The aim would be less code to maintain in Emacs and the benefit of a well-maintainted optimizing engine that is getting faster every day, with several independent free ones to choose from.

I think you could maybe sell that idea to the Emacs maintainers. A project like CEDET that replicates the code parsing engines of modern IDEs would definitely benefit from a speedup.

As a side-effect, it would then be possible to extend Emacs with pure Javascript. And a growing horde of web developers will suddenly find it much easier to extend Emacs.

I had a very brief look at the C source of Emacs, and it looks like you would have to rewrite and port maybe 10-20.000 lines of C code. So I think it's doable.

Of course, there are several open questions. How hard is it to do the translation? Can a Javascript engine actually run Lisp-translated-to-Javascript code efficiently? Is it a good idea at all or just a futile exercise in chasing the latest fad of the day? I am not sure. Javascript has been around for a decade and the web is still growing as a platform, so I don't think it qualifies as a fad. And GNOME is currently betting its future on it.

Another option would be to just embed the Javascript engine on top of the current core. As far as I know, that has been tried already with Python, and didn't work out. I don't think people will use it unless it is bundled with Emacs and sitting right in the core. After all, most customizations start out with a little hack building on existing code in Emacs.

Django shell in Emacs

Yet another recipe. One of the neat things with Python is its REPL, the interactive shell. When working with Django models, you need to setup a couple of things before you can use the Python shell. Django provides a shell command for doing this, but it makes it difficult to use the built-in Emacs Python REPL.

I was talking to Anders about it the other day when it occurred to me that it could be a quite powerful environment.

The main problem with the Python shell is that it lacks easy reload support. So you start it up, import the module you're writing and run a couple of functions to test it. Discover an error, fix it with Emacs, save, and then you're in trouble because you can't reload the module in the shell. You have to quit the shell, restart it and reimport the module. History helps but it's still silly (well, actually it's a blatantly stupid oversight, maybe in the top 3 of things to fix in Python to speed up development time).

But with a shell integrated in Emacs, I could solve this problem, right? So I set out to write the necessary Elisp. Dump the following in your .emacs, and it should magically work:

;; run Django shell when editing Django Python code

(defun get-file-in-upstream-dir (location filename)
  (let* ((dir (file-name-directory location))
         (path (concat dir filename)))
    (if (file-exists-p path)
        path
      (if (not (equal dir "/"))
        (get-file-in-upstream-dir (expand-file-name (concat dir "../")) filename)))))

(defadvice run-python (before possibly-setup-django-project-environment)
  (let* ((settings-py (get-file-in-upstream-dir buffer-file-name "settings.py"))
         (project-dir (file-name-directory settings-py)))
    (if settings-py
        (progn
          (setenv "DJANGO_SETTINGS_MODULE" "settings")
          (setenv "PYTHONPATH" project-dir)))))

When the Python interpreter is started, the hook above looks for a settings.py file in the parent directories. If one is found, it sets up Django and also sets the PYTHONPATH to the project toplevel.

Use the code with C-c C-z to show interpreter window, C-c C-c to evaluate buffer, C-c C-r to evaluate region, etc. (they're in the Python menu at top).

You'll probably want the following snippet too, it disables the warning that a Python process is still active when you quit Emacs:

(add-hook 'inferior-python-mode-hook
          (lambda ()
            (set-process-query-on-exit-flag (get-process "Python") nil)))

I think python-mode should do this by itself, but in the end I didn't have the energy to report a bug and fight with the Emacs maintainers.

So did it fix the import problem in Python? Of course not. The shell inside Emacs still can't reimport a module. Sigh. But at least it's easier to test the current module because you can easily evaluate the whole buffer (which works fine).

Flot 0.6 released!

I just released Flot 0.6! Highlights are a new plugin system, or at least the beginnings of it, lots of new features (most of them in plugins) and bug fixes, including fixes for IE 8. Check out the complete list of changes. Note that there are some API changes.

There were still some things I would have liked to do before the release, but it has been dragging along for far too long.

Otherwise the future is looking bright. The plugin system is not complete yet, but I think the general direction is OK, and will allow third parties to extend Flot considerably without going through me.

On the technological front, canvas support is gaining traction in the browsers and even support for what used to be advanced stuff like text rendering and export to image is getting more common place. I suspect we will soon have to think hard about whether to continue with the current labels-are-HTML approach or switch to rendering them directly on the canvas.

Completely unrelated to Flot: I haven't blogged for some time. It's entirely due to lack of time, I have been preoccupied, or gone crazy is probably closer to the truth, with another hobby project which is still some effort away from being useful.

Using CSS3 box-shadow with IE

At YayArt, we show a lot of images. An important part of the presentation is subtle drop shadows.

My original solution to that problem was a Javascript hack that inserts a couple of 1 pixel wide divs at the edge of any element with a shadow class. However, although I have tweaked it several times, it has always been a bit slow. For instance, for the shop front page we're currently rendering more than 60 elements with shadows which takes a couple of seconds with Firefox and Firebug on my old trusty PIII 950 MHz laptop. I'll note that this is actually an order of magnitude faster than the (more general) code I based it one from somewhere else. The really tricky part was decent-speed IE 6 support because that actually required measuring the DOM elements.

However, Firefox 3.5 was released about a month ago, and one of the good news was support for the box-shadow CSS3 property. It's a relatively simple property for adding drop shadows. So given that Safari has it, and has had for some time, and most Firefox users will soon have it, it starts getting interesting. For reference, about half of our visitors are Firefox + Safari (+ Chrome which uses the same rendering engine as Safari).

If you search for box-shadow, some are already using it to add a bit of extra embellishment. However, on YayArt, it's an integrated part of the design. So I had to figure out how to make it work on all browsers.

I ended up with a three-way solution. For recent Firefox/Safari/... I use box-shadow. For non-box-shadow supporting browsers, I fallback to the old hack. And for Internet Explorer, I'm using a new hack based on the proprietary filter stuff in IE.

Here's a quick sketch. I put this in the general stylesheet:

.shadowed {
  -moz-box-shadow: 2px 2px 3px #969696;
  -webkit-box-shadow: 2px 2px 3px #969696;
}

This makes it work with browsers based on the Gecko and Webkit engines. Some people recommend adding a plain box-shadow line, but I need full control over which browsers are responding, and besides it seems to defeat the purpose of the browsers using the -engine- prefix. Here's a demo:

If your browser is recent enough, you can see a shadow here.

Now for the fallback, I need to discover in Javascript whether the browser is using the CSS:

function supportsBoxShadow() {
  var s = document.body.style;
  return s.WebkitBoxShadow !== undefined || s.MozBoxShadow !== undefined;
}

The assumption is that the symbols will be defined only if the browser actually draws the shadow. This seems to be about right, the only exception I've found so far is the Webkit port to GTK (there's a bug report open).

Final point is Internet Explorer. As it turns out, there is a DropShadow filter. It's not terrible useful for this purpose, however. There's no soft edges, and it shadows the content rather than the containing box (it simply redraws the content with an offset in another color). So for my purpose I need to set a background color to ensure it's drawing a rectangle and not a text shadow. You could probably hack the lack of blurring with another filter, but in the end I went with a simple three-one-pixel-wide-lines approach that looks like my old hack but is much faster:

.shadowed {
  background-color: #fff;
  zoom: 1;
  filter:
    progid:DXImageTransform.Microsoft.DropShadow(color=#969696, offx=1, offy=1)
    progid:DXImageTransform.Microsoft.DropShadow(color=#C2C2C2, offx=1, offy=1)
    progid:DXImageTransform.Microsoft.DropShadow(color=#EFEFEF, offx=1, offy=1);
}

This is served for IE only. The zoom: 1 is a hack to ensure that the element gets a layout (to work around the usual IE bug). Interestingly, you only need to use an offset on 1 pixel in each of the offsets, the outermost then ends up 3 pixel out.

So there you have it. Practical drop shadows. I will have to adjust the code a little bit as the other browsers get support, but I can live with that.

Emacs 23...

... is out and has been so for over a week. Why didn't somebody tell me?

In any case, I upgraded to Emacs 23 on my Debian unstable laptop. This means a new font. I put Emacs.font: Bitstream Vera Sans Mono 9 in my .Xresources file and am enjoying it so far. It's a bit muddy so I wasn't sure at first, but after having stared at it for a while the old font is just too skinny. Note that configuring the font in .emacs is still broken (sigh), in that it causes the window to jerk when you start Emacs.

I've been lobbying for changing the defaults over to make copy-paste between Emacs and other X applications (from this century) work properly. Apparently, (setq x-select-enable-clipboard t) is now better, but still not according to the spec. And it's not default yet.

I noticed some Tramp updates (for accessing remote files) in the NEWS and decided to give it a spin for the first time. Try C-x C-f someserver:myfile.txt and with a little patience, you're editing a remote file locally! Very neat.

Feeling adventurous I also tried IDO, an update of iswitchb that works with file find (so overrides C-x C-f). I'm turning it on with

(ido-mode 1)
(setq ido-enable-flex-matching t)
(setq ido-use-filename-at-point t)

replacing iswitchb and ffap settings. We'll see how it goes.

Finally, Anders mentioned a new Javascript mode, js2-mode, which I'm definitely going to try out.

Using X11 over high-latency ADSL and an AltGr NX solution

X11 has been bad at high-latency links for a long time. Supposedly, the Xlib replacement XCB would fix some of that, but so far no-one seems to have picked it up. I found some work on GTK+ in 2006 but nothing since then. Sigh.

So there's this other proprietary solution, Nomachine NX with a free companion FreeNX. I decided to give it a try today, and after spending some time contemplating how to get it installed on Debian, it's not in the main repository because the basic architecture is broken (it's including it's own copy of X and SSH and some other stuff), I finally realized that the easiest setup was probably just downloading and installing the .debs on Nomachine's web site.

I was hoping to get ssh somehost then emacs & working, but it's a bit weirder than that. You have to start up a graphical connection thing, in which you can fortunately select a console. This gives you a remote xterm. The rest is simple.

I must say I'm very impressed with the performance. Of course, I'm only using Emacs but it works really well. It's like being connected through a local network.

Of course, once I started coding, the first time I needed a curly brace, which happens soon enough with Javascript, it was bam, back to start. AltGr sends an arrow-key left. After some digging and hair-pulling, it turns out I needed to run setxkbmap [layout] where layout is "dk" or "de" or similar. Except setxkbmap wasn't installed. So I installed it. Then it was missing it's data. I installed that too. Now everything is fine.

If someone would now just port this kind of thing to the default X.org setup, then I would be a happy man. No more console Emacs sessions.

Safe truncation of HTML

Another recipe, this time for solving the problem of truncating a piece of HTML, i.e. turning "<p>Blah blah blah</p>" into "<p>Blah ...</p>". Google didn't really turn anything useful up, except for a suggestion of using a full-blown HTML parser and then simplifying the result, so I thought I would post the snippet here for Google to pick up.

The code never splits a valid tag or character entity. It should be able to cope with invalid HTML too, but note that it won't sanitize it. So for instance, if there's an unbalanced <a> in the source string, it won't fix it. Character entities are dealt with by counting them as one character.

The basic idea in the snippet is that we just skip through the string unless we encounter an opening tag. If so, we see if we can find the corresponding end tag and save it for later. When we got enough non-HTML characters, a ... is put in and any saved but not yet used end tags are added to the output.

Here's the code in Python (it's easily turned into a Django filter), I aimed for readability rather than ultra-regexp ninja tricks:

import re

tag_end_re = re.compile(r'(\w+)[^>]*>')
entity_end_re = re.compile(r'(\w+;)')

@register.filter
def truncatehtml(string, length, ellipsis='...'):
    """Truncate HTML string, preserving tag structure and character entities."""
    output_length = 0
    i = 0
    pending_close_tags = {}
    
    while output_length < length and i < len(string):
        c = string[i]
        if c == '<':
            # probably some kind of tag
            if i in pending_close_tags:
                # just pop and skip if it's closing tag we already knew about
                i += len(pending_close_tags.pop(i))
            else:
                # else maybe add tag

                i += 1
                match = tag_end_re.match(string[i:])
                if match:
                    tag = match.groups()[0]
                    i += match.end()
  
                    # save the end tag for possible later use if there is one
                    match = re.search(r'(</' + tag + '[^>]*>)', string[i:], re.IGNORECASE)
                    if match:
                        pending_close_tags[i + match.start()] = match.groups()[0]
                else:
                    output_length += 1 # some kind of garbage, but count it in
                    
        elif c == '&':
            # possible character entity, we need to skip it
            i += 1
            match = entity_end_re.match(string[i:])
            if match:
                i += match.end()

            # this is either a weird character or just '&', both count as 1
            output_length += 1
        else:
            # plain old characters
            skip_to = string.find('<', i, i + length)
            if skip_to == -1:
                skip_to = string.find('&', i, i + length)
            if skip_to == -1:
                skip_to = i + length
                
            # clamp
            delta = min(skip_to - i,
                        length - output_length,
                        len(string) - i)

            output_length += delta
            i += delta
                        
    output = [string[:i]]
    if output_length == length:
        output.append(ellipsis)

    for k in sorted(pending_close_tags.keys()):
        output.append(pending_close_tags[k])

    return "".join(output)