Enhancing AI Systems with Agentic Workflows

Patterns in Large Language Model

Aditi Singh Abul Ehtesham Saket Kumar Tala Talaei Khoei
Department of Computer The Davey Tree Expert The Mathworks Inc Khoury College of Computer
Science Company [email protected] Science
Cleveland State University [email protected] Roux Institute at Northeastern
[email protected] University
[email protected]

Abstract— This paper explores the significant shift feedback and improvement. Much like a team working
towards agentic workflows in the application of Large collaboratively to refine a project, the agent iteratively
2024 IEEE World AI IoT Congress (AIIoT) | 979-8-3503-8780-3/24/$31.00 ©2024 IEEE | DOI: 10.1109/AIIoT61789.2024.10578990

Language Models (LLMs), moving away from traditional, revises, tests, and optimizes its responses. This approach not
linear interactions between users and AI. Through a case only provides more flexibility but also holds the potential for
study analysis, we highlight the effectiveness of agentic achieving higher quality outcomes, as the LLM agent takes
workflows, which facilitate a more dynamic and iterative on a more dynamic role in seeking ways to improve its
engagement, in improving outcomes in tasks such as question performance based on the initial prompt from the user.
answering, code generation or stock analysis. Central to the The agentic workflow operates on four fundamental
agentic workflow are four foundational design patterns: principles: reflection, tools, planning, and multi-agent
reflection, planning, multi-agent collaboration, and tool collaboration. This paper delves into these four pillars,
utilization. These components are crucial for boosting LLM highlighting the potential of agentic workflows. Specifically,
productivity and enhancing performance. The study we explore the application of reflection patterns in Q/A
demonstrates how agentic workflows, by promoting an scenarios, trip planning through multi-agent collaboration,
iterative and reflective process, can serve as a crucial step and the strategic use of tools and planning.
towards achieving Artificial General Intelligence (AGI).

Keywords — Agentic Workflows, Agentic Patterns, Large

Language Models, LLM Agent, AI Planning, Reflective AI, Multi-
agent, Tools, Agent Collaboration.

I. INTRODUCTION
The research community is currently captivated by the
capabilities Large Language Models (LLMs) withholds by
Fig. 1. Non-Agentic vs Agentic Workflow
using a zero-shot manner of prompting, LLM generate
outputs that are almost indistinguishable from those produced II. BACKGROUND
by human experts, showcasing their remarkable versatility
and potential as studied by Singh [1] and in investigations of Recent advancements in artificial intelligence have
AI generation models from textual descriptions [2]. This emphasized the importance of multi-agent reasoning design
advancement opens up exciting avenues for innovation, patterns, integrating capabilities like reflection, tool use,
though it also encourages ongoing dialogue about responsible planning, and multi-agent collaboration. These components
development and application. are essential for developing systems capable of sophisticated
decision-making and problem-solving.
In the process of experimenting with zero-shot
prompting, the journey often starts with users typing their Reflection is a crucial aspect of agentic reasoning,
initial query, which kicks off a sequence of iterations that are allowing systems to analyze and improve their outputs
critically dependent on user feedback for any subsequent iteratively. Madaan et al. introduced Self-Refine, a method
improvements. Each round of interaction unfolds in a linear that enhances LLMs' initial outputs through iterative
manner, where the quality and direction of the LLM's next feedback and refinement without additional training data or
response are directly influenced by the user's preceding reinforcement learning, demonstrating significant
feedback. This iterative exchange, characterized by a performance improvements across diverse tasks [3].
continuous back-and-forth between user and agent, is crucial Similarly, Shinn et al. proposed Reflexion, a framework that
for refining the content. Importantly, the LLM's capacity to uses linguistic feedback for language agents to learn from
enhance its output in the following query is contingent upon trial-and-error efficiently, achieving notable results in coding
receiving specific feedback from the user. This engagement and reasoning tasks [4] as illustrated in Fig. 2. Gou et al.
method, marked by its reliance on ongoing user input, is developed CRITIC, a framework enabling LLMs to self-
commonly referred to as a non-agentic workflow. correct through interaction with external tools, enhancing
their performance in various domains [5]. The reflection
In an agentic workflow, the process begins with the user process can be described as:
initiating the first query. Before delivering the final output,
the LLM agent enters a proactive series of questions and 𝑝𝑖+1 = 𝑟𝑒𝑓𝑖𝑛𝑒(𝑝𝑖 , 𝑓𝑒𝑒𝑑𝑏𝑎𝑐𝑘(𝑝𝑖 , 𝑡𝑎𝑠𝑘𝑐𝑜𝑛𝑡𝑒𝑥𝑡 )); 𝑖 = 0,1,2, … 𝑛 (1)
explorations on its own. This signifies a departure from It continuously improves an output pi through iterative
simply reacting to user inputs; instead, the agent actively refinements using task-specific feedback, until optimal
seeks to enhance the outcome through a self-guided cycle of results are achieved.

979-8-3503-8780-3/24/$31.00 ©2024 IEEE

0527
Authorized licensed use limited to: Dr. D. Y. Patil Educational Complex Akurdi. Downloaded on July 23,2024 at 05:51:07 UTC from IEEE Xplore. Restrictions apply.
 selection, accommodation, local attractions, and logistics.
The Multi-Agent Travel Planning System is an innovative
Agent solution developed using Langchain [13] and its LangGraph
for multi-agent workflows (as shown in Fig. 3). This
Self-reflection advanced system incorporates the agentic workflow patterns
(LM)
of reflection, tool use, planning, and multi-agent
collaboration. It adheres to best practices such as self-
Evaluator (LM) examination for continuous improvement, leveraging
(long-term memory) specialized tools for task execution, strategic multi-step
planning, and collaborative problem-solving among agents to
enhance the vacation planning process.
(short-term memory)
A. Langchain Tools Utilized by Agents:
• SearchTools: (Google SERPER: https://serper.dev/):
Empowers agents with internet search capabilities to
source travel data.
• Scrape and summarize website ( browserless.io):
Fig. 2. Schematic of Reflection [2] Enables agents to scrape and summarize web content,
yielding concise data outputs.
Tool use has been explored to extend LLMs' capabilities
• CalculatorTools: Assists agents in conducting
beyond language processing. Patil et al. released Gorilla, a
various mathematical calculations, crucial for
model that effectively utilizes APIs, addressing the
financial aspects of travel planning.
challenges of accurate input generation and hallucination [6].
Yang et al. proposed MM-REACT, integrating ChatGPT B. Agents and Their Roles
with vision experts for advanced visual intelligence tasks, • Data Collector Agent: The primary task of Data
demonstrating the feasibility of multimodal reasoning and collector agent is to collects user input regarding
action [7]. The tool use process can be defined as: travel preferences, including origin, desired
𝑜𝑢𝑡𝑝𝑢𝑡 = 𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒(𝑎𝑝𝑖_𝑐𝑎𝑙𝑙(𝑖𝑛𝑝𝑢𝑡, 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠), 𝑚𝑜𝑑𝑒𝑙_𝑐𝑜𝑛𝑡𝑒𝑥𝑡) (2) destinations, travel dates, and interests. The user
input is captured and passed to the Destination Finder
This illustrates the enhancement of LLM outputs through agent to initiate the destination selection process.
the integration of external API responses, combining these • Destination Finder Agent (City Selection
with the model's context to improve functionality and Expert): The role of Destination finder agent is to
precision. selects the best city for the traveler based on weather,
seasonality, and prices. It analyzes travel data to
Planning involves the strategic formulation of actions to
compare multiple cities, considering factors like
achieve specific goals. Wei et al. highlighted the
current weather conditions, cultural events, and
effectiveness of chain-of-thought prompting in eliciting travel expenses, resulting in a detailed report on the
reasoning in LLMs, significantly improving their chosen city.
performance on complex tasks [8]. Shen et al. introduced • Local Expert Agent: This agent provides in-depth
HuggingGPT, leveraging LLMs as controllers to manage AI insights about the selected city, emphasizing local
models across different domains, showcasing the potential attractions. It compiles comprehensive city guide,
for solving sophisticated AI tasks [9]. The planning process including information about key attractions, special
can be defined as: events, and recommended daily activities,
𝑝𝑙𝑎𝑛 = 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑒(𝑎𝑐𝑡𝑖𝑜𝑛𝑠(𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒), 𝑒𝑣𝑎𝑙𝑢𝑎𝑡𝑒(𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒, 𝑔𝑜𝑎𝑙𝑠)); (3) highlighting the unique characteristics that appeal to
This outlines the strategic planning process where a the traveler's interests.
sequence of actions is generated and evaluated against • Itinerary Expert Agent: This agent creates detailed
travel itineraries with suggestions for
specific goals, aligning the model's actions with intended
accommodations, dining, and activities, including
outcomes.
budget and packing recommendations.
Multi-agent collaboration is pivotal for systems to
perform in concert towards common objectives. Qian et al. C. Data Flow within the System
presented ChatDev, a paradigm that employs communicative The Data Collector Agent initiates the flow by gathering
agents throughout the software development process, initial user data such as the origin, potential cities, travel
demonstrating its efficiency and cost-effectiveness [10]. Wu dates, and interests. The Destination Finder Agent receives
et al. developed AutoGen, a framework for building LLM this data and engages in a two-fold process: internet searching
applications through conversable agents, enabling diverse and content summarization, leading to the selection of the
applications across several domains [11]. Additionally, Du optimal city. The selected city information is then passed on
and Ding provided an extensive overview of multi-agent deep to the Local Destination Expert Agent, who crafts a detailed
reinforcement learning, underscoring the significance of local guide, enriched with cultural and practical insights.
collaborative efforts in AI research [12]. With the local guide as its foundation, the Itinerary Expert
Agent develops a complete travel itinerary, providing a day-
III. AGENTIC WORKFLOW to-day plan along with budget and packing advice.
Vacation or Trip planning is a complex process that
requires considering multiple factors such as destination

0528
Authorized licensed use limited to: Dr. D. Y. Patil Educational Complex Akurdi. Downloaded on July 23,2024 at 05:51:07 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3. Overall Architecture

Throughout the process, the Trip Critic Agent monitors National Historical Park. This guide (illustrated in Fig. 5) is
the quality of outputs from each agent, offering iterative transformed into a detailed itinerary by the Itinerary Expert
feedback to refine the recommendations until they meet a Agent, ensuring a balance of activities tailored to the
satisfactory standard. The agents use Langchain ChatOpenAI traveler’s interests.
with model='gpt-4' underlies the entire system, driving the
intelligent interaction and decision-making processes of all Throughout the planning, the Trip Critic Agent provides
the specialized agent’s analysis. feedback to refine the itinerary, ensuring it meets the
traveler's expectations for a memorable Atlanta experience.
D. Agentic Workflow Patterns Utilised in Vacation Planner This concise scenario highlights the system’s ability to
The agentic workflow patterns have four pillars [14]: seamlessly tailor a travel plan, demonstrating its efficiency in
enhancing the travel planning process from New York to
• Reflection: The Trip Critic Agent implements Atlanta as shown in Fig. 6.
reflective analysis, enhancing the itinerary based on
a continuous feedback loop.
• Tool Use: LLMs are equipped with an array of
tools that assist in tasks such as data retrieval,
processing, and action execution.
• Planning: Each LLM follows a strategic planning
approach, methodically working through the steps
from user engagement to the delivery of the final
product.
• Multi-Agent Collaboration: The system's agents
work in concert, sharing tasks and insights to
formulate a travel plan that surpasses the capabilities
of individual agents
IV. CASE STUDY: AGENTIC WORKFLOWS IN LLMS
Let’s take an example of Alex, a traveler planning a
weekend in Atlanta from New York utilizes the Multi-Agent
Travel Planning System for an optimal journey. The system
kicks off with the Data Collector Agent, which meticulously
gathers the traveler's preferences as depicted in Fig. 4. This
foundational information is crucial in tailoring the journey to
the traveler's specific interests.
The Destination Finder Agent then identifies the best
travel dates and arrangements, considering weather and
events. The Local Destination Expert Agent crafts a Fig. 4. Interaction between user and multi-agent system
personalized guide featuring Atlanta’s jazz scenes, culinary
hotspots, and historical sites like the Martin Luther King Jr.

0529
Authorized licensed use limited to: Dr. D. Y. Patil Educational Complex Akurdi. Downloaded on July 23,2024 at 05:51:07 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5. Intermediate Interaction of agent with tools for internet search and web scraping

0530
Authorized licensed use limited to: Dr. D. Y. Patil Educational Complex Akurdi. Downloaded on July 23,2024 at 05:51:07 UTC from IEEE Xplore. Restrictions apply.
 Fig. 6. Final Vacation Plan Output by Multiagent system

The system consists of four major components: Planner, the output for the tasks is sent to the user. Overall, our system
Multiagent, Critic Agent, and Tools. The user provides effectively employs agentic design patterns to facilitate
information about their trip preferences, including their intelligent interaction between users and agents, enabling
origin, destination city options, travel dates, and interests via personalized and comprehensive trip planning assistance.
interactive conversation with the data collector agent. Based
on these inputs, the planner creates a series of tasks like There is always a risk of mislabeling or misidentification
gathering information and creating the travel plan. The of text that are comprehensively edited by humans. This is
multiagent utilizes its tools to gather data from various because modifications made by humans can change the
sources, such as internet searches, website scraping, and features that the detector looks for, making it harder to detect
calculations, to fulfill the task requirements. Each agent in machine-generated text. As AI models become more
this multiagent ecosystem has specialized roles and advanced and produce text that is more similar to human
responsibilities guiding their behaviors and interactions writing, it will become increasingly difficult for detectors to
within the system. The system incorporates a feedback loop distinguish between machine-generated and human-
where it scrutinizes the response from the multiagent and generated text.
based on the confidence score, a task can be re-executed for Therefore, it's important for users to be aware of these
that agent multiple times until the critic score reaches a limitations and to use these tools in combination with other
satisfactory threshold. Once a satisfactory score is reached, methods of verification, such as manual inspections, to

0531
Authorized licensed use limited to: Dr. D. Y. Patil Educational Complex Akurdi. Downloaded on July 23,2024 at 05:51:07 UTC from IEEE Xplore. Restrictions apply.
ensure accuracy. These tools have different use cases We conclude that the adoption of agentic workflows is crucial
depending up on the specific purpose and requirement in for unlocking the full potential of LLMs, facilitating the
various industries. For instance, these have application in development of AI systems that are not only more robust and
language translation, social media posts, customer reviews, adaptable but also significantly more intelligent. As the quest
educational institutions, legal contexts, text reuse, content for Artificial General Intelligence (AGI) continues, it
creation or marketing or advertising and detecting plagiarism. becomes clear that agentic workflows may provide the
essential, incremental steps needed to achieve this ambitious
V. CONCLUSION goal. The future significance of LLMs lies not just in their
text generation capabilities but in their ability to engage in
The shift towards agentic workflows marks a significant
self-dialogue, pushing the boundaries of AI towards
paradigm change in the deployment of Large Language
unprecedented levels of sophistication and utility. By
Models (LLMs). Our research indicates that these workflows
embracing this methodology, we pave the way for a future
yield superior results and endow LLMs with a closer
where LLMs can address complex challenges in artificial
approximation to human-like problem-solving capabilities.
intelligence, heralding a major leap forward for the field.

[7] Z. Yang et al., "MM-REACT: Prompting ChatGPT for Multimodal

REFERENCES Reasoning and Action," 2023, arXiv:2303.11381.
[1] A. Singh, "Exploring Language Models: A Comprehensive Survey and [8] J. Wei et al., "Chain of thought prompting elicits reasoning in large
Analysis," 2023 International Conference on Research Methodologies language models," ArXiv, vol. abs/2201.11903, 2022.
in Knowledge Management, Artificial Intelligence and
Telecommunication Engineering (RMKMATE), Chennai, India, 2023, [9] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, "HuggingGPT:
pp. 1-4. Solving AI tasks with ChatGPT and its friends in HuggingFace," in
Advances in Neural Information Processing Systems, vol. 36, 2024.
[2] A. Singh, "A Survey of AI Text-to-Image and AI Text-to-Video
Generators," 2023 4th International Conference on Artificial [10] C. Qian et al., "Communicative Agents for Software Development,"
Intelligence, Robotics and Control (AIRC), Cairo, Egypt, 2023, pp. 32- 2023, arXiv:2307.07924.
36 . [11] Q. Wu et al., "AutoGen: Enabling Next-Gen LLM Applications via
[3] A. Madaan et al., “Self-Refine: Iterative Refinement with Self- Multi-Agent Conversation," 2023, arXiv:2308.08155.
Feedback,” in Advances in Neural Information Processing Systems, A. [12] W. Du and S. A. Ding, "A survey on multi-agent deep reinforcement
Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, learning: from the perspective of challenges and applications,"
Eds., vol. 36, Curran Associates, Inc., 2023, pp. 46534-46594. Artificial Intelligence Review, vol. 54, pp. 3215–3238, 2021.
[4] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, [13] A. Singh, A. Ehtesham, S. Mahmud and J. -H. Kim, "Revolutionizing
"Reflexion: Language agents with verbal reinforcement learning," in Mental Health Care through LangChain: A Journey with a Large
Advances in Neural Information Processing Systems, A. Oh, T. Language Model," 2024 IEEE 14th Annual Computing and
Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., Communication Workshop and Conference (CCWC), Las Vegas, NV,
vol. 36, Curran Associates, Inc., 2023, pp. 8634-8652. USA, 2024, pp. 0073-0078,
[5] Z. Gou et al., "CRITIC: Large language models can self-correct with [14] A. Ng, "Agentic Design Pattern," [Online]. Available:
tool-interactive critiquing," ArXiv, vol. abs/2305.11738, 2023 https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-
[6] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, "Gorilla: Large performance/?ref=dl-staging-website.ghost.io. [Accessed: April 8,
language model connected with massive APIs," arXiv preprint 2024].
arXiv:2305.15334, 2023.

0532
Authorized licensed use limited to: Dr. D. Y. Patil Educational Complex Akurdi. Downloaded on July 23,2024 at 05:51:07 UTC from IEEE Xplore. Restrictions apply.