Ivan Grubišić

Ivan Grubišić earned his master’s degree at the Faculty of Electrical Engineering and Computing, University of Zagreb. He is currently a PhD candidate at the Faculty of Electrical Engineering and Computing, and an assistant at the Rudjer Boskovic Institute in the Laboratory for Machine Learning and Knowledge Representation. His research interests are in testing characteristics of deep learning models beyond their performance, such as evaluation of their reasoning capabilities, robustness, uncertainty, and fairness. He is devoted to the development of comprehensive test environments for models in various computer vision and natural language processing tasks.

Veridical benchmarks for testing the characteristics of deep learning models: Definitions Matter for GPT

In this talk we will focus on testing characteristics of large language models, especially their ability to generate definitions from examples and ability to use class definitions for fine-grained zero-shot classification. We will compare descriptive noun phrases, human-crafted definitions, introduce a new method to help the model generate definitions from examples, and propose a method to evaluate GPT-3’s understanding of the definitions. We will also demonstrate that improving definitions of class labels has a direct consequence on the downstream classification results.