libpd: gem support

[davephillips]

VCV Rack 1.dev
Linux Ubuntu 18.04

Just asking. I have the Pure Data example scripts working here with the latest VCV Prototype, very nice. So, if I want GEM support for libpd, 1) is it possible and 2) how/where would I add it ?

Nice work on Prototype, very cool.

Dave Phillips

[mxa]

So far, we haven't been able to load externals other than the already included bob~, bonk~, choice, fiddle~, loop~, lrshift~, pd~, sigmund~ and pique. It is possible to either include your own externals at compile time of libpd, or load them dynamically later. For dynamic loading, we tried different directories (next to Rack or the prototype binaries) and libpd was not able to find the external binaries there.
You could try to include Gem at compile time as described in the libpd wiki
We'll investigate further, for now the libpd engine in the Prototype module is vanilla.

[davephillips]

@mxa - Thank you, that's exactly what I wanted to know. I'll try rebuilding libpd with GEM and report back. Thanks again for the work !

[mxa]

libpd needs to compiled with the HAVE_LIBDL flag: https://github.com/libpd/libpd/wiki/libpd#build-settings
Expect an update soon. Meanwhile you could try it out and report back. (or just hold tight)

[mxa]

I've added the compile flag to libpd with this commit: e54f216 please test.

[AndrewBelt]

@mxa Well it builds and runs, but I don't know what to test.

[mxa]

I've checked that it compiles and runs. However when trying to load an external I get errors/crahses like this:

./Rack: /home/max/Documents/Pd/externals/cyclone/grab.pd_linux: undefined symbol: s_: Unknown error -109873728
terminate called without an active exception
thread '<unnamed>' panicked at 'cannot access a Thread Local Storage value during or after destruction: AccessError', src/libcore/result.rs:1188:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
terminate called recursively
Aborted (core dumped)

[jamshark70]

Adding a comment, as this issue seems to concern externals in general, not only Gem.

Today I tried (and failed) to use the external [comport] to do Arduino --> Pd patch --> VCV Prototype. Stack trace pasted below. So... if the current release of Prototype includes the HAVE_LIBDL switch, it isn't sufficient to load comport. (If the switch hasn't been integrated into the release yet, then my result is not surprising.)

As a side note for some context: Pd's current maintenance approach is to ship a minimal core, and rely on externals to make the environment usable. If the usage here is restricted to core objects, that's somewhat at odds with Pd's philosophy. I understand there may be technical reasons why it could be prohibitively difficult to support the full range of externals, and I'd accept that if it's the conclusion, but it should be noted that Pd is unlikely to start shipping more core objects to help out specific clients that have trouble loading externals.

[18.397 fatal src/main.cpp:45] Fatal signal 6. Stack trace:
33: ./Rack() [0x56d2b1]
32: /lib/x86_64-linux-gnu/libc.so.6(+0x3efd0) [0x7f0700badfd0]
31: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f0700badf47]
30: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f0700baf8b1]
29: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957) [0x7f07015a2957]
28: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7f07015a8ae6]
27: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7f07015a8b21]
26: ./Rack() [0x54a335]
25: /lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f0700bb20f1]
24: /lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f0700bb21ea]
23: /lib/x86_64-linux-gnu/libc.so.6(+0x11eb59) [0x7f0700c8db59]
22: /lib/x86_64-linux-gnu/libc.so.6(error+0xfd) [0x7f0700c8dc5d]
21: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(+0x1e281c) [0x7f06e5ec081c]
20: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(sys_loadlib_iter+0x25) [0x7f06e5ec0a55]
19: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(canvas_path_iterate+0xfe) [0x7f06e5e6d32e]
18: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(sys_load_lib+0x16a) [0x7f06e5ec0cda]
17: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(new_anything+0x6e) [0x7f06e5eb3bee]
16: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(binbuf_eval+0x45e) [0x7f06e5eaefee]
15: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(canvas_obj+0xb6) [0x7f06e5ea0fd6]
14: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(binbuf_eval+0x45e) [0x7f06e5eaefee]
13: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(binbuf_evalfile+0xf6) [0x7f06e5eb0266]
12: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(glob_evalfile+0x49) [0x7f06e5e6dfa9]
11: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(libpd_openfile+0x36) [0x7f06e5e2faf6]
10: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(_ZN11LibPDEngine3runERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_+0x4ca) [0x7f06e5d457ea]
9: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(_ZN9Prototype9setScriptENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x427) [0x7f06e5d388a7]
8: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(_ZN9Prototype8loadPathEv+0x490) [0x7f06e5d38e60]
7: /home/dlm/.Rack/plugins-v1/VCV-Prototype/plugin.so(_ZZN9Prototype17appendContextMenuEPN4rack2ui4MenuEEN14LoadScriptItem8onActionERKNS0_5event6ActionE+0x2a9) [0x7f06e5d3ad99]
6: ./Rack(_ZN4rack2ui8MenuItem10onDragDropERKNS_5event8DragDropE+0xa0) [0x5731fc]
5: ./Rack(_ZN4rack5event5State12handleButtonENS_4math3VecEiii+0x343) [0x56b92d]
4: ./Rack(_glfwPlatformPollEvents+0xca1) [0x5f2ddc]
3: ./Rack(_ZN4rack6Window3runEv+0x8e) [0x565fd0]
2: ./Rack(main+0x448) [0x4e6298]
1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f0700b90b97]
0: ./Rack(_start+0x29) [0x4eb1f9]

[mxa]

As a side note for some context: Pd's current maintenance approach is to ship a minimal core, and rely on externals to make the environment usable. If the usage here is restricted to core objects, that's somewhat at odds with Pd's philosophy. I understand there may be technical reasons why it could be prohibitively difficult to support the full range of externals, and I'd accept that if it's the conclusion, but it should be noted that Pd is unlikely to start shipping more core objects to help out specific clients that have trouble loading externals.

I strongly oppose the notion that Pd is somehow incomplete and is relying on externals for it to be "usable". In the contrary, most externals merely provide a more convenient way to do what could be achieved with vanilla patching. The functionality of many externals can be replicated as vanilla abstractions. Using externals has disadvantages: It will make your patches less portable and you will need to ship and update external binaries for all platforms.

That said, external support will eventually arrive in the libpd backend of the VCVPrototype.
For your use-case and the need to use comport for the time being use a workaround by launching a standalone Pd, where you access the comport and send the information to the VCVPrototype Pd patch with the vanilla methods of sending/receiving OSC.
CC @clwe

[jamshark70]

1. I'm very glad to hear that external support is in the works, thanks.

2. Agreed about portability, though that won't be a concern in my courses.

3. I'm prepared to disagree at length with the other points (briefly, sufficiency != usability: a pointed stick is sufficient to start a campfire, but a match is usable). But an issue report on a VCV Rack module is not the right place for that, so I won't say any more here. Feel free to contact me privately or at pdpatchrepo if you feel that my views on Pd are mistaken.

[clwe]

I've checked that it compiles and runs. However when trying to load an external I get errors/crahses like this:

./Rack: /home/max/Documents/Pd/externals/cyclone/grab.pd_linux: undefined symbol: s_: Unknown error -109873728
terminate called without an active exception
thread '<unnamed>' panicked at 'cannot access a Thread Local Storage value during or after destruction: AccessError', src/libcore/result.rs:1188:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
terminate called recursively
Aborted (core dumped)

We investigated the issue in libpd and it could be resolved now:
For static libpd build, we need to add the -Wl,-export-dynamic linker flag, when linking libpd.a with VCV-Prototype.
For multi-instance libpd builds, the externals have to be compiled with -DPDINSTANCE -DPDTHREADS cpp flags.
see issue #364 in libpd