Add support for Rust ABI dynamic libraries

[buntonj]

Currently, there is not explicit stable Rust ABI for dynamic libraries, but Rust library may still be compiled as one. This can be useful if a large project wants to compile some core portion of its code as a shared object that can be dynamically linked against several different binaries. Bevy, for example, offers this capability as a way to allow customers of Bevy to tweak their own code without having to statically re-link the central Bevy engine code.

This PR exposes the Rust ABI dylib capabilities with a new rule, rust_dylib_library. For clarity, it also creates a rust_cylib_library rule, and updates methods to this name, with rust_shared_library just aliasing to the rust_cdylib_library implementation.

Building a Rust ABI dylib may cause the standard library to be linked dynamically against the generated dylib. When depended on, the dynamic standard library needs to be included to get the binary to execute. Currently, rules_rust offers this as a toolchain-level flag, but applying a toolchain transition at the dylib's interface imposes a new toolchain on its deps, which then forces any upstream rlibs, for example, to recompile.

To fix this, this PR exposes a link_std_dylib attr to rust_binary, rust_dylib_library and rust_test to allow these targets to optionally bundle themselves with the dynamically-linked stdlib.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[UebelAndre]

I'd love @krasimirgg to take a look since I think there's some google uses of experimental_link_std_dylib. The change makes sense to me but I wonder if link_std_dylib should have more impact outside of just the allocators and codegen flag. The attribute sounds like it would directly link the dylibs but I don't think there's any wiring for that. Maybe @krasimirgg can correct me if I'm wrong

[UebelAndre]

Can you also add rust_dylib_library.bzl and rust_cdylib_library.bzl files neighboring this one with those exports?

[buntonj]

Definitely!

[UebelAndre]

Maybe rename these attributes on the toolchain as well?

[buntonj]

Yeah, totally, done!

[krasimirgg]

Hey, I'm not an expert on dynamic linking, so please bear with me...
I don't see how this composes with the rest of the rust ruleset. Two high-level questions:

1. What's the expected way for clients (rust_binary) depending on a rust_dylib_library to work? Are folks running them expected to supply the .so-s manually? Would bazel run do some work to ensure the dylibs are correctly provided?
2. What happens if we have a mix of rust_dylib_library and rust_library dependencies? How do we avoid duplicating shared dependencies in dylibs, rendering them incompatible? For instance, if you have:
   For example, if I read the idea correctly, if you have:

rust_library(name = "d")
rust_dylib_library(name = "c", deps = ["d"])
rust_dylib_library(name = "b", deps = ["d"])
rust_library(name = "a", deps = ["c", "b"])

, you'll likely get rustc error: cannot satisfy dependencies so d only shows up once while compiling a. Example rustc commands I'm thinking about -- https://gist.github.com/krasimirgg/66c52fbebee7af4813e1e71af495bd4a.

Tangent 1: On a high-level, I see a way to set up rust dynamic libraries support in a way similar to how rules_cc does dynamic libraries -- the rust_library itself has actions that produce both the rlib and the dylib, and clients pick up the appropriate dependency based on the --dynamic_mode build setting. This is composable, in particular any transitive dependencies of a dylib are automatically dylibs, avoiding the case where duplicated common dependencies render intermediate dylibs incompatible. But in a world like this, having a separate rust_dylib_library rule becomes just something almost equivalent to a rust_library.

Tangent 2: The reason why I feel this is different than rust_shared_library (cdylib) is that that one is designed to act like a final artifact -- you collect all of your rust into a single such target, which acts more like a final binary; standard rust libraries cannot direclty depend on it and you just hook it up as a special dependency of the final binary.

[UebelAndre]

1. What's the expected way for clients (rust_binary) depending on a rust_dylib_library to work? Are folks running them expected to supply the .so-s manually? Would bazel run do some work to ensure the dylibs are correctly provided?

I would expect Bazel to bundle the .so files as runfiles and have appropriate loader paths to support that, similar to how rules_cc does.

2. What happens if we have a mix of rust_dylib_library and rust_library dependencies? How do we avoid duplicating shared dependencies in dylibs, rendering them incompatible? For instance, if you have:
   For example, if I read the idea correctly, if you have:

rust_library(name = "d")
rust_dylib_library(name = "c", deps = ["d"])
rust_dylib_library(name = "b", deps = ["d"])
rust_library(name = "a", deps = ["c", "b"])

, you'll likely get rustc error: cannot satisfy dependencies so d only shows up once while compiling a. Example rustc commands I'm thinking about -- https://gist.github.com/krasimirgg/66c52fbebee7af4813e1e71af495bd4a.

What makes the dylib unique in this case compared to rlib? Wouldn't the dylib be a runfile that needs to be bubbled up along with the top level rlib?

Tangent 1: On a high-level, I see a way to set up rust dynamic libraries support in a way similar to how rules_cc does dynamic libraries -- the rust_library itself has actions that produce both the rlib and the dylib, and clients pick up the appropriate dependency based on the --dynamic_mode build setting. This is composable, in particular any transitive dependencies of a dylib are automatically dylibs, avoiding the case where duplicated common dependencies render intermediate dylibs incompatible. But in a world like this, having a separate rust_dylib_library rule becomes just something almost equivalent to a rust_library.

I'm not sure it's always desirable to turn everything into dynamic libraries globally like this. Lets say you had a rust_binary with 3 rust_library_group dependencies, would it be reasonable to turn a number of those into rust_dylib_library to avoid the more expensive linking of rust_binary? That sounds like a nice optimization on paper.

Tangent 2: The reason why I feel this is different than rust_shared_library (cdylib) is that that one is designed to act like a final artifact -- you collect all of your rust into a single such target, which acts more like a final binary; standard rust libraries cannot direclty depend on it and you just hook it up as a special dependency of the final binary.

I agree rust_cdylib_library/rust_shared_library can be thought of as an output, akin to rust_binary. I think the open question is should a conceptual new dylib (not cdylib) rule behave the same?

[buntonj]

Thanks for thinking about this @krasimirgg ! Some thoughts:

What's the expected way for clients (rust_binary) depending on a rust_dylib_library to work? Are folks running them expected to supply the .so-s manually? Would bazel run do some work to ensure the dylibs are correctly provided?

Bazel should provide the .so files for dylibs that are in deps--that's in this PR (they have CcInfo that gets collected and bundled with the runfiles).

What happens if we have a mix of rust_dylib_library and rust_library dependencies? How do we avoid duplicating shared dependencies in dylibs, rendering them incompatible?

I feel like this is a limitation of dynamic linking with rustc than something we have to solve in rules_rust...which is why for this PR, I opted to expose the tools to manually walk back through the dep graph and convert things to dylibs that must be for their target to compile.

This is definitely a possible footgun though--asking for dynamic linking at some point in the chain might imply other libraries should become dynamically linked. It's not required if no dependency diamonds like this occur, though.

Tangent 1: On a high-level, I see a way to set up rust dynamic libraries support in a way similar to how rules_cc does dynamic libraries -- the rust_library itself has actions that produce both the rlib and the dylib, and clients pick up the appropriate dependency based on the --dynamic_mode build setting. This is composable, in particular any transitive dependencies of a dylib are automatically dylibs, avoiding the case where duplicated common dependencies render intermediate dylibs incompatible. But in a world like this, having a separate rust_dylib_library rule becomes just something almost equivalent to a rust_library.

I think this is very cool, and could be something to build toward. In my use case, I wanted to avoid forcing every transitive dep to be unconditionally built as both an rlib and a dylib, when for many steps in that graph, only the rlib version was required (in my simpler use case, I had a deep dependencies with a dylib at the focal interface point). Instead, this lets me mark only the required targets as dylibs to save compilation time.