Because of its impressive results on a wide range of NLP tasks, Language Large Models (LLMs) like ChatGPT have garnered a lot of interest from researchers and companies alike. Using reinforcement learning from human feedback (RLHF) and extensive pre-training on massive scripts, LLM can generate greater language comprehension, generation, interaction, and reasoning abilities. The enormous potential of LLMs has sparked a plethora of new areas of study, and the opportunities resulting from the development of cutting-edge AI systems are nearly limitless.
LLM must collaborate with other models to harness its full potential and take on challenging AI jobs. Therefore, choosing appropriate middleware to establish communication channels between LLM models and AI models is crucial. To solve this problem, researchers realize that each AI model can be represented as a language by summarizing the function of the model. As a result, the researchers propose the idea that “LLMs use language as a general interface to link different AI models together.” Specifically, the LLM can be viewed as the central nervous system for managing AI models such as planning, scheduling, and collaboration because it includes model descriptions in the prompts. As a result, LLM can now use this tactic to call third-party models to complete AI-related activities. However, another difficulty arises if one wishes to integrate different AI models into LLMs: to perform many AI tasks, they need to collect many high-quality model descriptions, which requires extensive rapid engineering. Many public ML communities have a wide range of models suitable for solving specific AI tasks, including language, vision, and voice, and these models have clear and concise descriptions.
The research team proposes HuggingGPT, which can process input from many methods and solve many complex AI problems, by linking LLMs (such as ChatGPT) and the ML community (such as Hugging Face). To communicate with ChatGPT, the researchers combine the model description from the library corresponding to each AI model in Hugging Face with the router. Then, the LLM (ie ChatGPT) will act as the “brain” of the system to answer users’ queries.
🚀 Join the fastest ML Subreddit community
Researchers and developers can work together on natural language processing models and datasets with the help of HuggingFace Hub. As a bonus, it has a straightforward user interface for selecting and downloading ready-to-use templates for different NLP applications.
HuggingGPT phases
HuggingGPT can be broken down into four distinct steps:
- Task planning: using ChatGPT to interpret user requests in terms of meaning, then break them into discrete, actionable tasks with on-screen routing.
- Model selection: Based on the model descriptions, ChatGPT chooses the expert models stored on Hugging Face to complete the predefined tasks.
- Execute the task: call and run each model chosen, and then report the results back to ChatGPT.
- After integrating all models’ predictions with ChatGPT, the last step is to generate responses for users.
to examine closely –
HuggingGPT starts with a huge language model that breaks down a user’s request into separate steps. A large language model must establish task and system relationships while dealing with complex requirements. HuggingGPT uses a set of specification-based instructions and demo-based analysis in its rapid design to guide the large language model towards efficient task planning. The following paragraphs serve as an introduction to these details.
HuggingGPT must then determine the appropriate model for each task in the task list after analyzing the job list. Researchers do this by pulling expert model descriptions from the Hugging Face Hub and then using the contextual task model mapping mechanism to dynamically choose which models to apply to certain tasks. This method is more adaptable and open (describe expert models; anyone can use it step by step).
The next step after giving a model a task is to execute it, a process known as model inference. HuggingGPT uses mixed inference endpoints to speed up and ensure the computational stability of these models. Models take task arguments as input, perform the necessary calculations, and then return the inference results to the larger language model. Models without resource dependencies can be parallelized to increase inference efficiency even further. This allows simultaneous starting of many tasks with all of their dependencies satisfied.
HuggingGPT moves to the response generation step once all tasks have been executed. HuggingGPT combines the results of the previous three steps (task planning, model selection, and task execution) into one coherent report. This report details the tasks that were planned, the models that were chosen for those tasks, and the conclusions that were drawn from those models.
contributions
- It provides collaboration protocols between models to complement the benefits of large language and expert models. New approaches to creating generic AI models are made possible by separating large language models, which act as the brains for planning and decision-making, from smaller models, which act as executors for each given task.
- By connecting the Hugging Face hub to more than 400 task-specific models centered around ChatGPT, researchers can build HuggingGPT and handle broad classes of AI problems. HuggingGPT users can access reliable multimedia chat services thanks to open collaboration of models.
- Numerous experiments on multiple challenging AI tasks in language, vision, speech, and multimedia show that HuggingGPT can understand and solve complex tasks across multiple modalities and domains.
Advantages
- HuggingGPT can perform many complex AI tasks and integrate multimodal cognitive skills because its design allows it to use external models.
- In addition, HuggingGPT can continue to absorb knowledge from specialists in a specific field thanks to this pipeline, enabling expandable and scalable AI capabilities.
- HuggingGPT has integrated hundreds of Hugging Face models around ChatGPT, covering 24 tasks such as text classification, object detection, semantic segmentation, image generation, question answering, text-to-speech, and text-to-video conversion. Experimental results show that HuggingGPT can handle complex AI tasks and multimedia data.
determinants
- There will always be limitations with HuggingGPT. Efficiency is a major concern for us as it is a potential barrier to success.
- Phenomenal language model inference is a major efficiency bottleneck. HuggingGPT has to interact with the huge language model multiple times for each user request round. This happens during task planning, model selection, and response generation. These exchanges significantly lengthen response times, which reduces the quality of service for end users. The second is the maximum length of restrictions placed on contexts.
- HuggingGPT has a maximum context length limit due to the maximum number of tokens allowed in the LLM. To address this, studies focused only on the task planning phase of the dialog window and context tracking.
- The main concern is the reliability of the system as a whole. During inference, large language models can sometimes deviate from instructions, and the output format can sometimes surprise developers. An example of this is the rebellion of very large linguistic models during inference.
- There is also the Hugging Face endpoint expert model issue that needs more management. Expert models in Hugging Face may have failed during the execution phase of the task due to network latency or service status.
The source code can be found in a directory called “JARVIS”
In conclusion
Improving AI requires solving challenging problems across a variety of domains and modalities. Although there are many AI models, it should be more powerful to handle complex AI tasks. LLMs can be a controller for managing existing AI models to perform complex AI tasks. Language is a public interface because LLMs have demonstrated proficiency in language processing, generation, interaction, and logical reasoning. In keeping with this idea, the researchers introduced HuggingGPT. This framework uses LLMs (such as ChatGPT) to connect various AI models from other machine learning communities (such as Hugging Face) to complete AI-related tasks. More specifically, it uses ChatGPT to organize tasks after receiving a user request, select models based on their job descriptions in Hugging Face, run each subtask using the chosen AI model, and compile a response from the results of the runs. HuggingGPT paves the way for cutting-edge AI by leveraging ChatGPT’s superior language capabilities and Hugging Face’s wealth of AI models to perform a wide range of complex AI tasks across many modalities and domains, with amazing results in areas such as language, vision, voice, and more.
scan the paper And github. All credit for this research goes to the researchers on this project. Also, don’t forget to join 17k+ML Sub RedditAnd discord channelAnd Email newsletterwhere we share the latest AI research news, cool AI projects, and more.
Dhanshree Shenwai is a Computer Science Engineer with sound experience in FinTech companies covering Finance, Cards, Payments and Banking field with a keen interest in AI applications. She is passionate about exploring new technologies and developments in today’s evolving world making everyone’s life easy.
🔥 MUST READ – What is an AI Hallucination? What’s going wrong with intelligent chatbots? How do you discover the presence of artificial intelligence hallucinations?