kindly need help to for several reproduction results

[ShuaiWang97]

Discussed in #23

^{Originally posted by ShuaiWang97 December 5, 2022}
To the community,

Hope you had a great weekend.
Thank you so much for building this package! I am quite interested in hypergraph and learned a lot from the tutorial and source code. I tried to use the method and dataset from the package to reproduce several results. The performance on co-authorship dataset seems good but the performance on cocitation datasets seems a bit low. I checked the implements several times but did not find any problem. Can anyone please help me a bit?

The accuracy score on node classification of several cocitation datasets (CocitationCora,CocitationCiteseer,CocitationPubmed) by HGNN, HyperGCN, HGNN+ are followed and the code is attached. The way I change datasets and methods are just to change data and net variable. Any ideas are incredibly welcome. Thanks in advance.

import time
from copy import deepcopy

import torch
import torch.optim as optim
import torch.nn.functional as F

from dhg import Hypergraph,Graph
from dhg.data import Cooking200, CoauthorshipCora,CocitationCora,CocitationCiteseer,CoauthorshipDBLP, CocitationPubmed,\
                     Citeseer,Cora,Pubmed
from dhg.models import HGNN, HyperGCN, HGNNP
from dhg.random import set_seed
from dhg.metrics import HypergraphVertexClassificationEvaluator as Evaluator

from data import data
#from config import config



def train(net, X, A, lbls, train_idx, optimizer, epoch):
    net.train()

    st = time.time()
    optimizer.zero_grad()

    # import the data["features"] X and Graph structure G 
    outs = net(X, A)
    outs, lbls = outs[train_idx], lbls[train_idx]
    loss = F.cross_entropy(outs, lbls)
    #loss = F.nll_loss(outs, lbls) # decrease performance a lot
    loss.backward()
    optimizer.step()
    print(f"Epoch: {epoch}, Time: {time.time()-st:.5f}s, Loss: {loss.item():.5f}")
    return loss.item()


@torch.no_grad()
def infer(net, X, A, lbls, idx, test=False):
    net.eval()
    outs = net(X, A)
    outs, lbls = outs[idx], lbls[idx]
    if not test:
        res = evaluator.validate(lbls, outs)
    else:
        res = evaluator.test(lbls, outs)
    return res



if __name__ == "__main__":
    set_seed(2021)
    #args = config.parse()
    device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
    evaluator = Evaluator(["accuracy", "f1_score", {"f1_score": {"average": "micro"}}])
    
    # Load dataset of CocitationCiteseer, CocitationCora, CocitationPubmed
    #data = CocitationCora()
    data = CocitationCiteseer()

    # Build the hypergraph dataloader
    X, lbl = data["features"], data["labels"]
    HG = Hypergraph(data["num_vertices"], data["edge_list"])

    #net = HGNNP(data["dim_features"], 16, data["num_classes"], use_bn=False)
    net = HGNN(data["dim_features"], 16, data["num_classes"], use_bn=False)

    print("net is: ", net)
    optimizer = optim.Adam(net.parameters(), lr=0.01, weight_decay=0.0005)

    train_mask = data["train_mask"]
    val_mask = data["val_mask"]
    test_mask = data["test_mask"]
    print(f"length of train is : {sum(train_mask)}, length of val is: {sum(val_mask)},length of test is: {sum(test_mask)}")

    X, lbl = X.to(device), lbl.to(device)
    HG = HG.to(device)
    net = net.to(device)

    best_state = None
    best_epoch, best_val = 0, 0
    for epoch in range(200):
        # train
        train(net, X, HG, lbl, train_mask, optimizer, epoch)
        # validation
        if epoch % 10 == 0:
            with torch.no_grad():
                val_res = infer(net, X, HG, lbl, val_mask)
                print("val acc is: ",infer(net, X, HG, lbl, val_mask,test=True)["accuracy"])

                print("val_res is: ",val_res)
            if val_res > best_val:
                print(f"update best: {val_res:.5f}")
                best_epoch = epoch
                best_val = val_res
                best_state = deepcopy(net.state_dict())
    print("\ntrain finished!")
    print(f"best val: {best_val:.5f}")
    # test
    print("test...")
    net.load_state_dict(best_state)
    res = infer(net, X, HG, lbl, test_mask, test=True)
    print(f"final result: epoch: {best_epoch}")
    print(res)

Best,
Shuai

[yifanfeng97]

Thanks for your attention. I try to debug it!

[ShuaiWang97]

Thank you for the prompt response!
Most structure of the code is from the HGNN node classification example like def train, def infer and if __name__ == "__main__". I also think the several co-citation datasets are same from the ones used in HyperGCN repo . I only changed the net and data variable.
Please let me know if I can provide more information. Thanks in advance!

[bokveizen]

As I can see, some nodes never appear in the edge list. I am not sure whether that is intended or not. For example, in CocitationPubmed, the number of nodes is 19717, but there are only 3840 nodes in the edge list.

I think we need to reorder the vertices to consecutive integers.

data_list = [
    (CoauthorshipCora, "cora_coauth"),
    (CoauthorshipDBLP, "dblp_coauth"),
    (CocitationCora, "cora_cocite"),
    (CocitationCiteseer, "citeseer_cocite"),
    (CocitationPubmed, "pubmed_cocite"),
]

for data_func, data_name in data_list:
    nodes_in_edges = set()
    for edge in data_func()["edge_list"]:
        nodes_in_edges.update(edge)
    print(data_name, len(nodes_in_edges), data_func()["num_vertices"])

cora_coauth 2388 2708
dblp_coauth 41302 41302
cora_cocite 1434 2708
citeseer_cocite 1458 3312
pubmed_cocite 3840 19717

Also, for CocitationPubmed, the training set is strangely small, while the val and test sets are identical.

print(CocitationPubmed()["train_mask"].sum())
print(CocitationPubmed()["val_mask"].sum())
print(CocitationPubmed()["test_mask"].sum())
print(torch.all(CocitationPubmed()["val_mask"] == CocitationPubmed()["test_mask"]))

tensor(78)
tensor(19639)
tensor(19639)
tensor(True)