Multi-agent LLMs in 2024 [+frameworks]

Large language models (LLMs) are playing a team sport in 2024. Gone are the days when a single model tried to handle everything. Now, we use a lineup of specialized LLM agents, each one focused on what it does best. This strategy aims to gather a “dream team” of multi-agent LLMs, where every player excels in their position, bringing more depth and precision to solving complex challenges.

Consider this example: Imagine one agent is dedicated to gathering all the necessary data, while another analyzes this information to detect patterns and insights. In the meantime, a third agent then uses these insights to strategize and determine the optimal course of action. Together, they operate like a well-oiled machine that can solve planning problems on different complexity levels.

This collaborative model opens up new possibilities in what language models can do. For instance, in scenarios where constant updates are needed, like monitoring climate change or managing city traffic, these multi-agent LLMs can continuously exchange fresh data and strategies, keeping the system effective and up-to-date.

In this article, we’ll explore multi-agent LLMs, how they work, their benefits over single-agent systems, and some widely favored multi-agent frameworks.

What are multi-agent LLMs?

Multi-agent LLMs are language models teamed up to work and solve complex tasks, each agent taking a unique role that it’s good at. They do better than traditional single-agent models, especially in complicated tasks and real-life uses. What makes them stand out is their teamwork, pulling together the strengths of different specialized agents.

These agents work together smoothly, either as a team or on their own, depending on the task. Even though they mostly run on their own, they still need a human to oversee their decisions and review their work. For their tasks, agents use a variety of tools to do things like search the web or process documents, all powered by the powerful language models they’re built on.

Multi-agent LLMs are trending right now, and the chart below clearly shows why. It displays the number of papers published in various categories every three months. The counts at each leaf node reveal just how many papers are being written.These impressive figures, collected over just a few months, clearly show the rising popularity of multi-agent LLMs.

With this quick overview of multi-agent LLMs, let's move on to an easy-to-follow example to see what such a system can look like in action.

Understanding multi-agent LLMs with example

Let’s see how a multi-agent application could be used in real life. Imagine having a personal assistant that could plan your entire trip from start to finish. Here's how a multi-agent system could work for travel enthusiasts.

The travel planning multi-agent team

A multi-agent system for travel planning would consist of a few specialized agents, each focusing on a specific aspect of your trip:

1. Flight agent:

Finds and books airline flights

Accesses flight search engines and airline booking tools

Expertise in optimal routes, times, and prices

2. Hotel agent:

Searches and books accommodations

Uses hotel search engines and booking platforms

Knowledgeable about ratings, amenities, and locations

3. Transportation agent:

Handles rental cars, shuttles, trains, etc.

Accesses various transport booking tools

Expert in pricing, vehicle types, and pickup/drop-off locations

4. Activity agent:

Books activities, tours, events, and restaurants

Uses activity booking platforms and local guides

Informed about popular attractions, reviews, and schedules

By breaking down the complex travel planning task into subtasks handled by specialized agents, the overall system becomes more efficient than any single agent trying to figure out all aspects of travel. The agents collaborate, share information, and sequence their efforts through the manager for an integrated solution.

How multi-agent LLMs work

Here’s what a typical workflow in a multi-agent LLM system would look like: It begins with a user who provides a high-level task or query. The system then breaks down the task into smaller subtasks and assigns them to the appropriate specialized agents based on their roles and capabilities.

Each agent uses its LLM to reason about its assigned subtask, devise a plan, and execute that plan using its available tools and memory. In this process, agents communicate and share information as needed to complete interdependent subtasks. The final output is assembled by combining the results from all the agents involved.

Single-agent vs. multi-agent LLMs

Multi-agent LLMs are often better for complex tasks because they work together and efficiently. Here's why people who use these systems think they are a good choice:

1. ‍Accuracy and LLM hallucinations: A big issue with single-agent LLMs is that they sometimes hallucinate, meaning they can generate believable but incorrect information. This is a serious problem in areas like medicine or law, and any field where accuracy is crucial. Multi-agent systems help solve this problem by letting agents check each other's work, which greatly reduces mistakes and boosts reliability. Using LLM fine-tuning techniques on these agents may also significantly improve their performance. Research shows that using multiple agents can make responses more accurate and reliable, making multi-agent systems especially valuable in critical settings.
2. Handling extended contexts: Single-agent LLMs have a drawback: their limited context windows, which only allow them to consider a small amount of text at once. This is problematic when dealing with long documents or conversations over extended periods. Multi-agent systems handle this better by dividing the work among several agents. Each agent focuses on a segment of the text and works together to maintain a clear and continuous understanding of the whole discussion. This teamwork extends their ability to manage and process information effectively.
3. Efficiency and multitasking: Single-agent LLMs operate on a single thread, meaning they process one task at a time. This can cause delays, particularly where quick responses to multiple queries are needed. Multi-agent systems improve efficiency through parallel processing, where several agents handle different tasks simultaneously. This setup not only cuts down response times but also boosts productivity, making it ideal for business environments where every second counts.
4. Collaborative capabilities: Multi-agent systems shine in situations where teamwork is key. Unlike setups with just one agent, these systems bring together the strengths and expertise of different agents. This collaboration is crucial for complex problems that need a mix of skills and viewpoints. It's valuable in areas like scientific research or strategic planning, where pooling diverse knowledge and ideas leads to better results.

Single-agent systems are good in cognitive tasks and work well independently. In contrast, multi-agent systems combine different agents that collaborate and make decisions together. This setup helps them handle more complex and dynamic tasks. Each agent in a multi-agent system has unique problem-solving methods and communicates with others to achieve common goals.

Multi-agent LLM frameworks

Here’s a list of the most popular multi-agent LLM frameworks:

1. AutoGen: Microsoft's AutoGen is like a playground for AI agents. It lets you create chatty AI assistants that can work together, use tools, and also loop in humans when needed. It's pretty flexible, allowing for all sorts of conversation patterns. It has a very active and growing community, which is great for developers needing support and collaboration.

2. ‍LangChain: Think of LangChain as a LEGO set for AI applications. It gives you building blocks to connect different AI components, making it easier to create complex AI-powered apps. It's great for developers who want to mix and match various AI capabilities.‍
3. LangGraph: This new kid on the block is part of the LangChain family. LangGraph aims to enable the creation of LLM workflows containing cycles better, which are a critical component of most agent runtimes. It's designed to create AI workflows that aren't just linear but can branch out and loop back. It's like giving AI agents a roadmap with multiple routes to the destination. LangGraph uses a graph representation for agent connection, offering a clear and scalable method to manage multi-agent interactions‍
4. CrewAI: This framework is also about teamwork. It lets you create a crew of AI agents, each with its own role and expertise. CrewAI is particularly useful for production-ready applications, featuring clean code and focusing on practical applications. CrewAI's CEO João Moura, also offers a course that explains the key components of multi-agent systems with practical examples using CrewAI's framework.‍
5. AutoGPT: AutoGPT is like giving an AI agent a to-do list and watching it go. It's particularly good at remembering things and understanding context, which makes it great for tasks that require a bit more persistence. It also has some cool visual tools for setting up your AI systems, making it a good choice for developers who want to use multi-agent systems for visual design tools.

Here’s a list of its features

• Internet access to search for information
• GPT-4 for text generation
• Long and short-term memory management
• Access to popular websites and platforms
• File storage and summarization with GPT-3.5
• Extensibility with plugins

6. Mindsearch: A brand-new open-source AI Search Engine Framework that performs like Perplexity.ai Pro. You can easily set it up as your own search engine using either closed-source LLMs like GPT and Claude or open-source models like InternLM2.5-7b-chat. It's designed to answer any question by browsing hundreds of web pages, providing in-depth responses and showing detailed paths to those answers. MindSearch also offers a variety of user interfaces such as React, Gradio, Streamlit, and Terminal to suit different needs.
7. Hierarchical multi-agent RL: Hierarchical multi-agent reinforcement learning (RL) framework allows agents to learn at multiple levels of hierarchy simultaneously. The key advantage of this framework is its ability to use the hierarchical structure of tasks to learn coordination strategies more effectively. It extends established single-agent HRL methods, such as hierarchies of abstract machines (HAMs), options, and MAXQ, particularly the MAXQ value function decomposition, to a cooperative multi-agent environment.
8. ‍Haystack: Haystack is your go-to if you want to use AI to dig through your own data. It's known for being stable and having great documentation, which is always a plus. It's particularly useful for projects involving question-answering or semantic search.

Multi-agent LLM applications

Here are two real-life applications that use different multi-agent frameworks.

GPT-newspaper

GPT-newspaper creates personalized newspapers tailored to user preferences. It has 6 main agents working under the hood, the two main being the “planner” and “execution” agents. The planner generates questions to research, and execution seeks the most related information based on each generated research question. Finally, the planner filters and aggregates the related information and creates a research report.

Example with CrewAI, LanchChain, and LangGraph

Joao Moura (CEO at CrewAI) developed an example of how CrewAI, combined with LangChain and LangGraph, can be used to automate the process of checking emails and drafting responses. CrewAI manages autonomous AI agents that work together to solve tasks efficiently.

Below is a graph showing how this works.

Multi-agent LLM challenges and limitations

Multi-agent LLMs face several hurdles in allocating roles and tasks, managing memory and time, and more detailed:

1. Task allocation: It's tricky to efficiently divide complex tasks among different agents. It's like assigning roles in a team project but for AI.
2. Coordinating reasoning: Getting agents to debate and reason together effectively isn’t simple. Imagine trying to get a group of people to solve a puzzle collaboratively—it's similar to AI agents.
3. Managing context: Keeping track of all the information and conversations between agents can be overwhelming. It's like trying to remember everything said in a long group chat.
4. Time and cost: Having multiple agents interact takes more time and computational resources, which can be expensive.

Closing remarks

As we wrap up, it's clear that multi-agent LLMs are making a big impact. These systems handle complex tasks by working together, much like teams in the real world. This approach is improving how well AI can perform in areas like healthcare, law, and customer service. Looking ahead, multi-agent LLMs will play a crucial role in making AI smarter and more helpful in our daily lives. This is just the start, and plenty more exciting progress will come.