Add directed hypergraph layers and Cora node classification example - #15
Add directed hypergraph layers and Cora node classification example#15shonalidixit wants to merge 3 commits into
Conversation
Extends DirectedHypergraphAttentionLayer with user defined num_heads. preserves existing single-head behaviour with num_heads = 1 adds independent parameters for each attention head concatenates head outputs and projects them back to hidden_dim updates documentation expands tests for multi-head forward passes, attention normalisation, parameter shapes, and validation Full test suite passes.
Adds additional directed hypergraph layers with tests and documentation along with an end to end Cora node classification proof of concept using Enzyme for automatic differentiation
espottesmith
left a comment
espottesmith
left a comment
There was a problem hiding this comment.
Some areas where more documentation would be helpful. Lots of code duplication. Seems like the example is unnecessarily long and probably should be broken up into multiple files. But damn, this is a huge body of work! I so greatly appreciate this contribution.
| uuid = "1aebdcae-dc77-44ee-b348-f058d326ae63" | ||
| authors = ["Evan Walter Clark Spotte-Smith", "Punna Amornvivat"] | ||
| version = "0.0.1-DEV" | ||
| authors = ["Evan Walter Clark Spotte-Smith", "Punna Amornvivat"] |
There was a problem hiding this comment.
At this point, you should absolutely list yourself as an author.
|
|
||
| This allows the two roles in a directed hyperedge to be modelled differently | ||
| throughout the complete message-passing step. | ||
|
|
There was a problem hiding this comment.
- Follow the naming convention for new files.
- You should explain how this is different from the existing message passing layer (or at least put enough detail in this docstring such that an average user can read both and see the difference).
There was a problem hiding this comment.
Oh, also, how does this treat cases where a vertex v is in both the tail and the head of a dihyperedge?
| function _asymmetric_initialise_weight( | ||
| initializer, | ||
| rng::AbstractRNG, | ||
| input_dimension::Int, | ||
| output_dimension::Int, | ||
| ) | ||
| return permutedims( | ||
| initializer( | ||
| rng, | ||
| output_dimension, | ||
| input_dimension, | ||
| ), | ||
| ) | ||
| end | ||
|
|
There was a problem hiding this comment.
This looks identical to _initialize_weight. Am I missing something?
Also, does this function belong here? Is it likely to only be used by layers in this file, or does it belong in more of a utility file elsewhere? Should it be exported?
| function _asymmetric_initialise_bias( | ||
| initializer, | ||
| rng::AbstractRNG, | ||
| output_dimension::Int, | ||
| ) | ||
| return permutedims( | ||
| initializer( | ||
| rng, | ||
| output_dimension, | ||
| 1, | ||
| ), | ||
| ) | ||
| end |
There was a problem hiding this comment.
See comment above. Is this actually providing anything new?
| function _asymmetric_unpack_input( | ||
| ::AsymmetricDirectedHypergraphLayer, | ||
| input::Tuple, | ||
| ) | ||
| throw( | ||
| ArgumentError( | ||
| "The layer expects either " * | ||
| "(X_vertex, source_matrix, target_matrix) or " * | ||
| "(X_vertex, X_hyperedge, source_matrix, target_matrix).", | ||
| ), | ||
| ) | ||
| end |
There was a problem hiding this comment.
What is the user provides something that isn't a Tuple?
| A learnable gate controls the balance between the information received | ||
| through hypergraph message passing and the transformed representation of the | ||
| vertex itself. Each hidden feature has its own gate value between zero and | ||
| one. |
There was a problem hiding this comment.
Where does this come from? Has this been reported in the literature before? This goes for all of the layers you've made. I know you said you haven't copied from other sources, but even so, if these ideas have been published, we should give credit and references so readers/users can get more information
| function _gated_initialise_weight( | ||
| initializer, | ||
| rng::AbstractRNG, | ||
| input_dimension::Int, | ||
| output_dimension::Int, | ||
| ) | ||
| return permutedims( | ||
| initializer( | ||
| rng, | ||
| output_dimension, | ||
| input_dimension, | ||
| ), | ||
| ) | ||
| end | ||
|
|
||
|
|
||
| function _gated_initialise_bias( | ||
| initializer, | ||
| rng::AbstractRNG, | ||
| output_dimension::Int, | ||
| ) | ||
| return permutedims( | ||
| initializer( | ||
| rng, | ||
| output_dimension, | ||
| 1, | ||
| ), | ||
| ) | ||
| end |
| function Lux.initialparameters( | ||
| rng::AbstractRNG, | ||
| layer::GatedDirectedHypergraphLayer, | ||
| ) |
There was a problem hiding this comment.
Some code duplication in these initialization functions, as well.
| using Lux | ||
| using Random |
There was a problem hiding this comment.
Just as a heads up: you shouldn't need to do imports in each file. Anything you globally import in the package file HyperGraphNeuralNetworks.jl should also be available to all modules within the package.
| The layer preserves the source and target roles of vertices while adding a | ||
| residual connection to the vertex update. Directed information is first |
There was a problem hiding this comment.
What does "residual connection" mean here? What does "preserv[ing] the [...] roles" mean? Kind of a vague description.
|
Oh, just a heads up: I skipped the tests for now. |
2 participants
This PR adds several additional directed hypergraph neural network layers, together with their associated tests and documentation.
It also adds an end-to-end Cora node classification proof of concept under
examples/cora_node_classification.jl. The example represents the citation network using separate tail and head incidence matrices and usesDirectedHypergraphLayerfor directed message passing.The Cora example includes:
With a 64-dimensional hidden representation and 100 training epochs, the current run achieved:
For the current proof of concept, the incidence matrices are passed directly to
DirectedHypergraphLayer. Integration between the layers andHGNNDiHypergraphcan be addressed separately.