add set terminal svg and capture output to svg buffer

[mfrasca]

I guess this is the first place where intervention is needed:

I'm holding a spreadsheet in an emacs buffer, not yet decided whether to use ses or org., in both cases I have to go through gnuplot and in particular org-plot makes use of gnuplot-mode.

I'd like to grab the graph in a buffer, in svg format.
the svg format allows me further editing of the graph still using emacs.

my workflows using ses:

• one ses cell is reserved to a lisp expression evaluating to a string looking like a pair of vectors that I paste in the calculator buffer
• have the calculator generate the svg code in the *gnuplot trail* buffer
• collect the svg code in a svg buffer

the workflow using org-plot is more complicated:

• run org-plot/gnuplot
• this will generate a file in /tmp and open a window I do not need, so I close it.
• switch to the *gnuplot* buffer and copy the commands
• use gnuplot-make-buffer to open a new gnuplot buffer
• in the new buffer execute set terminal svg
• then run the commands copied from *gnuplot* buffer
• collect the svg code in a svg buffer.

I do not see svg in the possible terminals offered by gnuplot-mode.

when terminal is set to svg, I would expect gnuplot-mode to do this:

• filter the svg output coming from gnuplot and put it in a svg buffer.

first time gnuplot outputs svg code, it will be almost complete (misses the closing svg tag).
subsequent times it starts immediately after the closing defs tag.

[joddie]

It might be possible to implement this as a more general feature: Emacs could capture Gnuplot's output in any image format by using temporary files, and place it in a buffer. This would effectively allow you to plot into an Emacs window, and you could use image-mode to toggle between SVG source and rendered image.

In fact, this is already mostly implemented by the gnuplot-mode code that inserts inline images into the Gnuplot comint buffer. It would mostly be a matter of inserting images into a different buffer, and enabling different image types besides "png".

The lack of a closing </svg> tag looks to me like a buffering problem, FWIW. If you do set output "blah", plot something, and then set output, the file is flushed and the closing tag appears.

[mfrasca]

thanks for the reply! this quite simplifies the workflow using gnuplot-mode.

set terminal svg
set title 'salinity'
set ylabel "mS/cm"
set xlabel "m³/d"
set output "/tmp/b.svg"
plot '/tmp/org-plot18589dFU' using 6:3 with points title ''
set output

but I've never really programmed in elisp, so I'm a bit lost in your code, looking for what to change/add in order to automatize the process.

I guess I would like to do this:
when one sets the terminal mode to an image type, the output is diverted to its corresponding image buffer (unless explicitly diverted to a file). what happens now is that it goes to the gnuplot "comint" buffer.

so I would execute set terminal png and gnuplot-mode sees this and sets the output to the buffer *gnuplot-png*.
same for set terminal svg sending the svg code to the *gnuplot-svg* buffer.

is this viable? what keywords to look for in the code?

[joddie]

I think this is viable. I have a branch with a proof-of-concept implementation which I will push shortly for you to try out, if you like. I am still working on getting the buffering issue sorted out.

I had thought of having a single *gnuplot output* buffer, but your suggestion is possible too. Is it useful to have separate buffers for separate image types?

[joddie]

Please try out the prototype implementation of this in plot-to-window branch. In a fresh Emacs, load "gnuplot.el" from that branch, then do the following:

1. M-x gnuplot-show-gnuplot-buffer
2. From the "Gnuplot" menu, choose "Display plot output > In dedicated buffer" and "Display plot output > SVG images"
3. Plot something as normal
4. You should get a new pop-up buffer called *gnuplot output* containing the generated SVG file. In my Emacs (without SVG support) it is in nxml-mode. If you have SVG support, it might show up as an image, in which case you could use C-c C-c to toggle between viewing the source and rendered.

[joddie]

Note: I've never used org-plot so I don't know how this feature should interact with it. If we can first get this working when plotting from the Gnuplot interaction buffer and gnuplot-mode buffers, that can be tackled separately.

[mfrasca]

with the current changes I can indeed get the graph in a svg buffer.

how can I visualize/log what org-plot does with gnuplot-mode?
with the present changes, org-plot does not manage to generate output consistently.

if I run org-plot/gnuplot, the possible outcomes are:

• it just works (in particular if I run it as first thing).
• it uses gnuplot-send-hiding-output to report "Selecting deleted buffer" (nothing else happens)
• it hides the output buffer, says 'Buffer "gnuplot" has a running process; kill it?' (and then it works, showing again the output buffer or it hangs, with the spinning mouse until C-g)

[joddie]

Hmm, that is strange. It may help to do M-x toggle-debug-on-error, which should give you a backtrace on errors which you could copy and paste here. However, this does not tend to work very well for filter functions, which run asynchronously.

Could you post a minimal org-mode example as a Gist, plus a recipe to use for testing?

[mfrasca]

me behind very intermittent internet connection. was not logged in when I created this:
https://gist.github.com/anonymous/697852a0ab35a15f208c
updated
https://gist.github.com/mfrasca/330b268fa62b390ec2dc

[mfrasca]

is there a different way to reset gnuplot?

(when (get-buffer "*gnuplot*") ;; reset *gnuplot* if it already running
  (with-current-buffer "*gnuplot*"
(goto-char (point-max))
(gnuplot-delchar-or-maybe-eof nil)))

this asks for user intervention.

[mfrasca]

I'm looking at the code of org-plot/gnuplot... one point I notice is that it tries to do some cleanup relying on an idle timer. the cleanup doesn't happen. do you think it could be possible for gnuplot-mode to offer callbacks, that could be called after specific events in gnuplot-mode?

[mfrasca]

is it OK to invoke gnuplot-send-buffer-to-gnuplot at the end of a with-temp-buffer block?

[joddie]

1. Thanks for the Gist recipe. I have pretty limited time this week, but I will try to look into this.

2. There is something funny going on with handling the gnuplot buffer and process.

3. Yes, it should be OK to have gnuplot-send-buffer-to-gnuplot within within-temp-buffer.

4. Adding more hooks is a good possibility, but there is another problem. I don't see how the line

(run-with-idle-timer 0.1 nil (lambda () (delete-file data-file)))

in org-plot can work without lexical-binding. The binding for data-file is long gone by the time the idle timer runs. It should probably be:

(run-with-idle-timer 0.1 nil #'delete-file data-file)

[joddie]

Once i made that change to org-plot, I seem to be able to plot your table repeatedly using C-c " g without problems. Does that also work for you?

[joddie]

Hmm, no, it's still possible to get the "Selecting deleted buffer" error. Investigating..

[mfrasca]

I suspect the buffer being deleted is *gnuplot* (if I have a window visit it, invoke org-plot/gnuplot, the window will switch to a different buffer)

[joddie]

I pushed a small commit to the plot-to-window branch which tweaks gnuplot's process handling slightly, and seems to improve things. However, I think how it works with the process needs to be improved more generally, and I don't have the time right now. In particular, it seems bad to have both gnuplot-buffer and a gnuplot-process variables, since they can get out of sync. Instead, we should keep track of the buffer and use (get-buffer-process gnuplot-buffer).

[mfrasca]

lisp doubt: how about this as a callback...

(run-with-idle-timer 0.1 nil (list #'delete-file data-file))

[joddie]

No, look at the docstring for run-with-idle-timer. The third argument has to be a function, and then any remaining arguments will be saved and passed to that function when it is called. The version I posted works.

[joddie]

If you can reproduce the "Selecting deleted buffer" error, could you please do M-x toggle-debug-on-error and paste the backtrace it produces?

[joddie]

You're right, it is probably the *gnuplot* buffer being deleted. I guess gnuplot-send-hiding-output should handle this case more carefully … I have to think about this.

[mfrasca]

I can consistently reproduce the "Selecting deleted buffer" error (every second time I run org-plot/gnuplot), but the program does not crash, so no backtrace. or am I missing something?

[mfrasca]

more info: in my /tmp dir, I have these files:

-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot20401v0t
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot204018-z
-rw------- 1 mario mario 0 Nov 1 18:54 gnuplot20401uID
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot204017SJ
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot20401IdP
-rw------- 1 mario mario 0 Nov 1 18:54 gnuplot20401VnV
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot20401ixb
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot20401v7h
-rw------- 1 mario mario 0 Nov 1 18:54 gnuplot204018Fo
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot20401JQu
-rw------- 1 mario mario 1306 Nov 1 18:54 org-plot20401Wa0
-rw------- 1 mario mario 0 Nov 1 18:54 gnuplot20401VuJ
-rw------- 1 mario mario 9186 Nov 1 18:54 gnuplot20401IkD

files seem to be generated in pairs, subsequent orgplot* files are equal. when the operation is successful, gnuplot* files are also generated in pairs, otherwise only the first, empty one, is generated.

[joddie]

What does Emacs print if you type M-: debug-on-error RET? If it is nil, please type M-x toggle-debug-on-error and then try to reproduce the Selecting deleted buffer error. Does that display a Lisp backtrace (in a *Backtrace* buffer)? If not, then the error is happening in some asynchronous process-filter function , and will be harder to debug.

[joddie]

Please try the latest revision on the plot-to-window branch. I think it fixes this bug.

I still sometimes see zombie gnuplot processes hanging around, but I'm not sure why.

You should also apply this patch to org-plot: https://gist.github.com/joddie/afa03f5c53d61da1b26a . If you'd like to report it as a bug to the org-mode maintainers, even better :)

I do think org-plot could ensure it starts with a fresh gnuplot process more cleanly than by doing (gnuplot-delchar-or-eof) -- probably it would be better to do (delete-process gnuplot-process) or similar, but I'm not sure.

[mfrasca]

in *scratch*:

debug-on-error <C-j>
t

with your edits and if I kill the gnuplot process before invoking org-plot/gnuplot, then I can consistently successfully run org-plot/gnuplot. :)

seems indeed that the way org-plot tries to get a fresh gnuplot process doesn't work properly.
but why should we actually need a fresh process. isn't reset good enough?
if I just remove the

(gnuplot-delchar-or-maybe-eof nil)

it works just as well, and without me needing to kill the process by hand.

there's still two gnuplot (not any more the org-plot) temporary files hanging in the /tmp dir.

[joddie]

gnuplot-mode uses temporary files to get image output from gnuplot, so it's expected you should see some temporary files hanging around. I suppose it might be better to explicitly clean them up in a similar way to what org-plot does.

Do you have to kill the gnuplot process every time before running org-plot/gnuplot? That does seem strange; I don't see that behavior here. If you're interested in debugging further, I recommend Edebug. Go to the definition of the function you are interested in (e.g. org-plot/gnuplot) and hit C-u C-M-x. Next time it is called you can step through with the space bar, evaluate variables with e, etc. There are some good Edebug tutorials around on the web and I think it's documented in the ELisp manual as well.

Otherwise i don't have too many more ideas what problem you might be seeing right now ...

[mfrasca]

I will try debugging, but what do you think about not killing the process and rely on reset?
it works fine here.