Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 103 additions & 91 deletions src/content/blog/a-peek-inside-elixir-s-parallel-compiler.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,23 +15,25 @@ The idea of the parallel compiler is very simple: for each file we want to compi

In Elixir, we could write this code as follows:

def spawn_compilers([current | files], output) do
parent = Process.self()
child = spawn_link(fn ->
:elixir_compiler.file_to_path(current, output)
send parent, { :compiled, Process.self() }
end)
receive do
{ :compiled, ^child } ->
spawn_compilers(files, output)
{ :EXIT, ^child, { reason, where } } ->
:erlang.raise(:error, reason, where)
end
end

def spawn_compilers([], _output) do
:done
end
```elixir
def spawn_compilers([current | files], output) do
parent = Process.self()
child = spawn_link(fn ->
:elixir_compiler.file_to_path(current, output)
send parent, { :compiled, Process.self() }
end)
receive do
{ :compiled, ^child } ->
spawn_compilers(files, output)
{ :EXIT, ^child, { reason, where } } ->
:erlang.raise(:error, reason, where)
end
end

def spawn_compilers([], _output) do
:done
end
```

In the first line, we define a function named `spawn_compilers` that receives two arguments, the first is a list of files to compile and the second is a string telling us where to write the compiled file. The first argument is represented as a list with head and tail (`[current | files]`) where the top of the list is assigned to `current` and the remaining items to `files`. If the list is empty, the first clause of `spawn_compilers` is not going to match, the clause `spawn_compilers([], _output)` defined at the end will instead.

Expand All @@ -53,19 +55,21 @@ With this code, we were able to compile each file inside a different process. Ho

Imagine that we have two files, `a.ex` and `b.ex`, with the following contents:

# a.ex
defmodule A do
B.define
end

# b.ex
defmodule B do
defmacro define do
quote do
def one, do: 1
end
end
```elixir
# a.ex
defmodule A do
B.define
end

# b.ex
defmodule B do
defmacro define do
quote do
def one, do: 1
end
end
end
```

In order to compile `A`, we need to ensure that `B` is already compiled and loaded so we can invoke the `define` macro. This means the file `a.ex` depends on the file `b.ex`. When compiling files in parallel, we want to be able to detect such cases and automatically handle them.

Expand All @@ -79,87 +83,95 @@ By default, Elixir (and Erlang) code is autoloaded. This means that, if we invok

As discussed in the previous section, we want to extend the error handler to actually stop the currently running process whenever a module is not found and resume the process only after we ensure the module is compiled. To do that, we can simply define our own error handler and ask Erlang to use it. Our custom error handler is defined as follows:

defmodule Elixir.ErrorHandler do
def undefined_function(module, fun, args) do
ensure_loaded(module)
:error_handler.undefined_function(module, fun, args)
end

def undefined_lambda(module, fun, args) do
ensure_loaded(module)
:error_handler.undefined_lambda(module, fun, args)
end

defp ensure_loaded(module) do
case Code.ensure_loaded(module) do
{ :module, _ } ->
[]
{ :error, _ } ->
parent = Process.get(:elixir_parent_compiler)
send parent, { :waiting, Process.self, module }
receive do
{ :release, ^parent } -> ensure_loaded(module)
end
```elixir
defmodule Elixir.ErrorHandler do
def undefined_function(module, fun, args) do
ensure_loaded(module)
:error_handler.undefined_function(module, fun, args)
end

def undefined_lambda(module, fun, args) do
ensure_loaded(module)
:error_handler.undefined_lambda(module, fun, args)
end

defp ensure_loaded(module) do
case Code.ensure_loaded(module) do
{ :module, _ } ->
[]
{ :error, _ } ->
parent = Process.get(:elixir_parent_compiler)
send parent, { :waiting, Process.self, module }
receive do
{ :release, ^parent } -> ensure_loaded(module)
end
end
end
end
end
```

Our error handler defines two public functions. Both those functions are callbacks required to be implemented by the error handler. They simply call `ensure_loaded(module)` and then delegate the remaining logic to Erlang's original `error_handler`.

The private `ensure_loaded` function calls `Code.ensure_loaded(module)` which checks if the given module is loaded and, if not, tries to load it. In case it succeeds, it returns `{ :module, _ }`, which means the module is available and we don't need to stop the current process. However, if it returns `{ :error, _ }`, it means the module cannot be found and we need to wait until it is compiled. For that, we invoke `Process.get(:elixir_parent_compiler)` to get the PID of the main process so we can notify it that we are waiting on a given module. Then we invoke the macro `receive` as a way to stop the current process until we receive a message from the parent saying new modules are available, starting the flow again.

With our error handler code in place, the first thing we need to do is to change the function given to `spawn_link` to use the new error handler:

spawn_link(fn ->
Process.put(:elixir_parent_compiler, parent)
Process.flag(:error_handler, Elixir.ErrorHandler)
```elixir
spawn_link(fn ->
Process.put(:elixir_parent_compiler, parent)
Process.flag(:error_handler, Elixir.ErrorHandler)

:elixir_compiler.file_to_path(current, output)
send parent, { :compiled, Process.self() }
end)
:elixir_compiler.file_to_path(current, output)
send parent, { :compiled, Process.self() }
end)
```

Notice that we have two small additions. First we store the `:elixir_parent_compiler` PID in the process dictionary so we are able to read it from the error handler and then we proceed to configure a flag in our process so our new error handler is invoked whenever a module or function cannot be found.

Second, our main process can now receive a new `{ :waiting, child, module }` message, so we need to extend it to account for those messages. Not only that, we need to control which PIDs we have spawned so we can notify them whenever a new module is compiled, forcing us to add a new argument to the `spawn_compilers` function. `spawn_compilers` would then be rewritten as follows:

def spawn_compilers([current | files], output, stack) do
parent = Process.self()
child = spawn_link(fn ->
:elixir_compiler.file_to_path(current, output)
send parent, { :compiled, Process.self() }
end)
wait_for_messages(files, output, [child | stack])
end

# No more files and stack is empty, we are done
def spawn_compilers([], _output, []) do
:done
end

# No more files and stack is not empty, wait for all messages
def spawn_compilers([], output, stack) do
wait_for_messages([], output, stack)
end
```elixir
def spawn_compilers([current | files], output, stack) do
parent = Process.self()
child = spawn_link(fn ->
:elixir_compiler.file_to_path(current, output)
send parent, { :compiled, Process.self() }
end)
wait_for_messages(files, output, [child | stack])
end

# No more files and stack is empty, we are done
def spawn_compilers([], _output, []) do
:done
end

# No more files and stack is not empty, wait for all messages
def spawn_compilers([], output, stack) do
wait_for_messages([], output, stack)
end
```

Notice we added an extra clause to `spawn_compilers` so we can properly handle the case where we don't have more files to spawn but we are still waiting for processes in the stack. We have also moved our `receive` logic to a new private function called `wait_for_messages`, implemented as follows:

defp wait_for_messages(files, output, stack) do
receive do
{ :compiled, child } ->
new_stack = List.delete(stack, child)
Enum.each new_stack, fn(pid) ->
send pid, { :release, Process.self }
end
spawn_compilers(files, output, new_stack)
{ :waiting, _child, _module } ->
spawn_compilers(files, output, stack)
{ :EXIT, _child, { reason, where } } ->
:erlang.raise(:error, reason, where)
after
10_000 ->
raise "dependency on nonexistent module or possible deadlock"
```elixir
defp wait_for_messages(files, output, stack) do
receive do
{ :compiled, child } ->
new_stack = List.delete(stack, child)
Enum.each new_stack, fn(pid) ->
send pid, { :release, Process.self }
end
end
spawn_compilers(files, output, new_stack)
{ :waiting, _child, _module } ->
spawn_compilers(files, output, stack)
{ :EXIT, _child, { reason, where } } ->
:erlang.raise(:error, reason, where)
after
10_000 ->
raise "dependency on nonexistent module or possible deadlock"
end
end
```

The implementation for `wait_for_messages` is now broken into 4 clauses:

Expand Down
8 changes: 6 additions & 2 deletions src/content/blog/elixir-v0-8-0-released.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,15 @@ On the last 9th January, we celebrated [two years since Elixir's first commit](h

One of the goals for the v0.8 release was better integration with OTP applications. By passing the `--sup` option to Mix, you can start a new OTP Application containing application callbacks and a supervisor:

mix new my_app --sup
```bash
mix new my_app --sup
```

And applications can be started directly from the command line as well:

elixir --app my_app
```bash
elixir --app my_app
```

We have written a whole [guide chapter about creating OTP applications, supervisors and servers](https://hexdocs.pm/elixir/supervisor-and-application.html). Give it a try!

Expand Down
8 changes: 5 additions & 3 deletions src/content/blog/elixir-v0-9-0-released.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,11 @@ $ mix new my_project --umbrella

The generated project will have the following structure:

apps/
mix.exs
README.md
```
apps/
mix.exs
README.md
```

Now, inside the `apps` directory, you can create as many applications as you want and running `mix compile` inside the umbrella project will automatically compile all applications. The [original discussion for this feature](https://github.com/elixir-lang/elixir/issues/667) contains more details about how it all works.

Expand Down
10 changes: 7 additions & 3 deletions src/content/blog/elixir-v1-10-0-released.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,9 @@ end

This approach has one big limitation: if you change the value of the application environment after the code is compiled, the value used at runtime is not going to change! For example, if you are using `mix release` and your `config/releases.exs` has:

config :my_app, :db_host, "db.production"
```elixir
config :my_app, :db_host, "db.production"
```

Because `config/releases.exs` is read after the code is compiled, the new value will have no effect as the code was compiled to connect to "db.local".

Expand Down Expand Up @@ -143,8 +145,10 @@ In Elixir v1.10, we have addressed these problems by [introducing compiler traci

Elixir itself is using the new compiler tracing to provide new functionality. One advantage of this approach is that developers can now disable undefined function warnings directly on the callsite. For example, imagine you have an optional dependency which may not be available in some cases. You can tell the compiler to skip warning on calls to optional modules with:

@compile {:no_warn_undefined, OptionalDependency}
defdelegate my_function_call(arg), to: OptionalDependency
```elixir
@compile {:no_warn_undefined, OptionalDependency}
defdelegate my_function_call(arg), to: OptionalDependency
```

Previously, this information had to be added to the overall project configuration, which was far away from where the optional call effectively happened.

Expand Down
36 changes: 21 additions & 15 deletions src/content/blog/elixir-v1-13-0-released.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,19 +82,23 @@ Those improvements came from direct feedback from the community. A special shout

Thanks to its sigils, Elixir provides the ability of embedding snippets in other languages inside its source code. One could use it to embed XML:

~X"""
<?xml version="1.0" encoding="UTF-8"?>
<text><![CDATA[Hello World]]></text>
"""
```elixir
~X"""
<?xml version="1.0" encoding="UTF-8"?>
<text><![CDATA[Hello World]]></text>
"""
```

Or even [Zig](https://ziglang.org/), [via the Zigler project](https://github.com/ityonemo/zigler):

~Z"""
/// nif: example_fun/2
fn example_fun(value1: f64, value2: f64) bool {
return value1 > value2;
}
"""
```elixir
~Z"""
/// nif: example_fun/2
fn example_fun(value1: f64, value2: f64) bool {
return value1 > value2;
}
"""
```

However, while you can format Elixir source code with [`mix format`](https://hexdocs.pm/mix/Mix.Tasks.Format.html), you could not format the code inside snippets.

Expand Down Expand Up @@ -134,11 +138,13 @@ We are looking forward to see how this new functionality will be used by communi

`SyntaxError` and `TokenMissingError` were improved to show a code snippet whenever possible:

$ elixir -e "hello + * world"
** (SyntaxError) nofile:1:9: syntax error before: '*'
|
1 | hello + * world
| ^
```bash
$ elixir -e "hello + * world"
** (SyntaxError) nofile:1:9: syntax error before: '*'
|
1 | hello + * world
| ^
```

The `Code` module has also been augmented with two functions: [`Code.string_to_quoted_with_comments/2`](https://hexdocs.pm/elixir/Code.html#string_to_quoted_with_comments/2) and [`Code.quoted_to_algebra/2`](https://hexdocs.pm/elixir/Code.html#quoted_to_algebra/2). Those functions allow someone to retrieve the Elixir AST with their original source code comments, and then convert this AST to formatted code. In other words, those functions provide a wrapper around the Elixir Code Formatter, supporting developers who wish to create tools that directly manipulate and custom format Elixir source code.

Expand Down
14 changes: 9 additions & 5 deletions src/content/blog/elixir-v1-15-0-released.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,15 +89,19 @@ sure more feedback is reported to developers before compilation is aborted.

In Elixir v1.14, an undefined function would be reported as:

** (CompileError) undefined function foo/0 (there is no such import)
my_file.exs:1
```
** (CompileError) undefined function foo/0 (there is no such import)
my_file.exs:1
```

In Elixir v1.15, the new reports will look like:

error: undefined function foo/0 (there is no such import)
my_file.exs:1
```
error: undefined function foo/0 (there is no such import)
my_file.exs:1

** (CompileError) my_file.exs: cannot compile file (errors have been logged)
** (CompileError) my_file.exs: cannot compile file (errors have been logged)
```

A new function, called `Code.with_diagnostics/2`, has been added so this
information can be leveraged by editors, allowing them to point to several
Expand Down
4 changes: 2 additions & 2 deletions src/content/blog/elixir-v1-3-0-released.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ Elixir v1.3 includes improvements to the option parser, including `OptionParser.

For example, invoking `mix test --unknown` in earlier Elixir versions would silently discard the `--unknown` option. Now `mix test` correctly reports such errors:

```
```bash
$ mix test --unknown
** (Mix) Could not invoke task "test": 1 error found!
--unknown : Unknown option
Expand Down Expand Up @@ -251,7 +251,7 @@ end

Every test inside a describe block will be tagged with the describe block name. This allows developers to run tests that belong to particular blocks, be them in the same file or across many files:

```
```bash
$ mix test --only describe:"String.capitalize/2"
```

Expand Down
Loading
Loading