Igor Mijić

Igor Mijić is a senior engineer and lecturer with a longstanding research background in various applications of deep learning, like speech processing, computer vision and natural language processing. Throughout his career he had a special interest in Affective Computing, and related to the interplay of human emotions and computers, while in his engineering work he’s passionate about dockerizing all the developmental components of the models he develops and deploys.

Traditional Deep Learning Methods in the Era of Foundation Models

The Large Language Model (or LLM) revolution has in recent years not only transformed how we process text, but it has also brought in a radical shift in how many new deep learning projects were structured. In this new landscape it wasn’t just the people developing the models that felt the shift, it also spilled over into the everyday lives of the users with rapid ideation and development of tools based on LLMs. The shift then expanded and enabled the rapid proliferation of many other technologies like Large Vision Models (LVMs), Large Speech Models (LSMs), generative AI, multimodality, etc. Soon, a term was coined: “Foundation Models”, encapsulating all large models trained on massive, diverse datasets, which enabled them to capture features that are universal to many different domains. As such, they set new standards and increased the State-of-the-Art in numerous deep learning tasks, overshadowing their smaller counterparts. Throughout this presentation, we shall however discuss why traditional deep learning methods and smaller models are still not obsolete within this era, by showcasing some of their advantages, but also how they can leverage some of the advantages and capabilities of their bigger siblings. Topics like real time performance, reliability and interpretability are just some of the ones we shall touch on within this discussion.