Higher-Order Attention Networks

Hajij, M., Zamzmi, G., Papamarkou, T., Miolane, N., Guzman-Saenz, A., Ramamurthy, K., N.

We introduce higher-order attention networks (HOANs), a novel class of attention-based neural networks defined on a generalized higher-order domain called a combinatorial complex (CC). Similar to hypergraphs, CCs admit arbitrary set-like relations between a collection of abstract entities. Simultaneously, CCs permit the construction of hierarchical higher-order relations analogous to those supported by cell complexes. Thus, CCs effectively generalize both hypergraphs and cell complexes and combine their desirable characteristics.