Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS

Filters

2
2
2
Reset filters

Settings
Now showing 1 - 2 of 2
  • Publication
    RoMe: A Robust Metric for Evaluating Natural Language Generation
    ( 2022-05)
    Rony, Md Rashad Al Hasan
    ;
    Kovriguina, Liubov
    ;
    Chaudhuri, Debanjan
    ;
    Usbeck, Ricardo
    ;
    Lehmann, Jens
    Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the reference’s semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks.
  • Publication
    Dihedron Algebraic Embeddings for Spatio-Temporal Knowledge Graph Completion
    ( 2022)
    Nayyeri, M.
    ;
    Vahdati, S.
    ;
    Khan, M.T.
    ;
    Alam, M.M.
    ;
    Wenige, L.
    ;
    Behrend, A.
    ;
    Lehmann, Jens
    Many knowledge graphs (KG) contain spatial and temporal information. Most KG embedding models follow triple-based representation and often neglect the simultaneous consideration of the spatial and temporal aspects. Encoding such higher dimensional knowledge necessitates the consideration of true algebraic and geometric aspects. Hypercomplex algebra provides the foundation of a well defined mathematical system among which the Dihedron algebra with its rich framework is suitable to handle multidimensional knowledge. In this paper, we propose an embedding model that uses Dihedron algebra for learning such spatial and temporal aspects. The evaluation results show that our model performs significantly better than other adapted models.