### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

### Example Code

```
from langchain.memory import ConversationBufferWindowMemory
from langchain_core.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
from langchain_community.utilities import BingSearchAPIWrapper
from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser
from langchain.agents.format_scratchpad.openai_tools import (
    format_to_openai_tool_messages,
)
from langchain_openai import AzureChatOpenAI
from langchain.chains import LLMMathChain
from langchain.agents import AgentExecutor
from langchain.agents import tool

memory = ConversationBufferWindowMemory(memory_key="chat_history", return_messages=True,k=12, verbose=True, output_key="output")

messages_data = [
    "What is 2+2",
    "The answer to 2+2 is 4",
]

memory.save_context({"input":messages_data[0]},{"output": messages_data[1]})



system = '''Your name is TemperaAI.
You are a data-driven Marketing Assistant designed to help with a wide range of tasks, from answering simple questions to providing in-depth plans.
YOU MUST FOLLOW THESE INSTRUCTIONS:
1. Add a citation next to every fact with the file path within brackets. For example: [//home/docs/file.txt]. You can only skip this if your answer has no citations.
2. Always include the subject matter in the search query when calling a retriever tool to ensure relevance.
3. If the tool response is not useful, try a different query.'''

human = '''

{input}

{agent_scratchpad}

'''

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        MessagesPlaceholder("chat_history", optional=True),
        ("human", human),
    ]
)

llm = AzureChatOpenAI(
  openai_api_version="2024-03-01-preview",
  azure_deployment="chat",
)

llm_math = LLMMathChain.from_llm(llm=llm)
@tool
def math_tool(query: str) -> str:
  '''Use it for performing calculations.'''
  chain = llm_math
  return chain.invoke(query)

def agent(llm,tools,question,memory,prompt):
    result = {}

    llm_with_tools = llm.bind_tools(tools)

    agent = (
        {
            "input": lambda x: x["input"],
            "agent_scratchpad": lambda x: format_to_openai_tool_messages(
                x["intermediate_steps"]
            ),
        }
        | prompt
        | llm_with_tools
        | OpenAIToolsAgentOutputParser()
    )

    agent_executor = AgentExecutor(agent=agent, tools=tools, memory=memory, verbose=True, return_intermediate_steps=True, max_iterations=5, trim_intermediate_steps=1)

    chat_history = memory.buffer_as_messages

    result = agent_executor.invoke({"input": question,"chat_history": chat_history,})

    return result

tools = [math_tool]

question = "Can you please summarize this conversation?"
  
response = agent(llm,tools,question,memory,prompt)
  
print("\nQUESTION",response['input'])
print("\n\nRESPONSE: ",response['output'])

```

### Error Message and Stack Trace (if applicable)

> Entering new AgentExecutor chain...
I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?

> Finished chain.

QUESTION Can you please summarize this conversation?


RESPONSE:  I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?

### Description

I'm trying to use the ConversationBufferWindowMemory from Langchain on a Agent with Tools.
I create the agent and load the memory but when I ask a question about a past message or ask for a summary it does not respond well.

I implemented a solution to add the messages manually in the prompt and it gave me this response.
Expected Response: The conversation so far has been brief. The user asked a simple math question, "What is 2+2?". I provided the answer, which is 4. There hasn't been any further discussion or questions.

But using Langchain ConversationBufferWindowMemory I responds.
Response: "I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?"

Memory Content:
Memory content [HumanMessage(content='What is 2+2'), AIMessage(content='The answer to 2+2 is 4'), HumanMessage(content='Can you please summarize this conversation?'), AIMessage(content="I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?")]

Have any of you found this issue before? How can I make the Agent use the Memory?

### System Info

!pip install -qU langchain \ langchain-community \ langchain-core \langchain-openai \ numexpr \ hub \ langgraph \ langchainhub \ azure-cosmos \ azure-identity

Platform: Google Colab

ConversationBufferWindowMemory
This is a GitHub Issue
repo:langchain
owner:hwchase17
Title : ConversationBufferWindowMemory
Issue date:

--- start body ---
### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

### Example Code

```
from langchain.memory import ConversationBufferWindowMemory
from langchain_core.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
from langchain_community.utilities import BingSearchAPIWrapper
from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser
from langchain.agents.format_scratchpad.openai_tools import (
    format_to_openai_tool_messages,
)
from langchain_openai import AzureChatOpenAI
from langchain.chains import LLMMathChain
from langchain.agents import AgentExecutor
from langchain.agents import tool

memory = ConversationBufferWindowMemory(memory_key="chat_history", return_messages=True,k=12, verbose=True, output_key="output")

messages_data = [
    "What is 2+2",
    "The answer to 2+2 is 4",
]

memory.save_context({"input":messages_data[0]},{"output": messages_data[1]})



system = '''Your name is TemperaAI.
You are a data-driven Marketing Assistant designed to help with a wide range of tasks, from answering simple questions to providing in-depth plans.
YOU MUST FOLLOW THESE INSTRUCTIONS:
1. Add a citation next to every fact with the file path within brackets. For example: [//home/docs/file.txt]. You can only skip this if your answer has no citations.
2. Always include the subject matter in the search query when calling a retriever tool to ensure relevance.
3. If the tool response is not useful, try a different query.'''

human = '''

{input}

{agent_scratchpad}

'''

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        MessagesPlaceholder("chat_history", optional=True),
        ("human", human),
    ]
)

llm = AzureChatOpenAI(
  openai_api_version="2024-03-01-preview",
  azure_deployment="chat",
)

llm_math = LLMMathChain.from_llm(llm=llm)
@tool
def math_tool(query: str) -> str:
  '''Use it for performing calculations.'''
  chain = llm_math
  return chain.invoke(query)

def agent(llm,tools,question,memory,prompt):
    result = {}

    llm_with_tools = llm.bind_tools(tools)

    agent = (
        {
            "input": lambda x: x["input"],
            "agent_scratchpad": lambda x: format_to_openai_tool_messages(
                x["intermediate_steps"]
            ),
        }
        | prompt
        | llm_with_tools
        | OpenAIToolsAgentOutputParser()
    )

    agent_executor = AgentExecutor(agent=agent, tools=tools, memory=memory, verbose=True, return_intermediate_steps=True, max_iterations=5, trim_intermediate_steps=1)

    chat_history = memory.buffer_as_messages

    result = agent_executor.invoke({"input": question,"chat_history": chat_history,})

    return result

tools = [math_tool]

question = "Can you please summarize this conversation?"
  
response = agent(llm,tools,question,memory,prompt)
  
print("\nQUESTION",response['input'])
print("\n\nRESPONSE: ",response['output'])

```

### Error Message and Stack Trace (if applicable)

> Entering new AgentExecutor chain...
I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?

> Finished chain.

QUESTION Can you please summarize this conversation?


RESPONSE:  I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?

### Description

I'm trying to use the ConversationBufferWindowMemory from Langchain on a Agent with Tools.
I create the agent and load the memory but when I ask a question about a past message or ask for a summary it does not respond well.

I implemented a solution to add the messages manually in the prompt and it gave me this response.
Expected Response: The conversation so far has been brief. The user asked a simple math question, "What is 2+2?". I provided the answer, which is 4. There hasn't been any further discussion or questions.

But using Langchain ConversationBufferWindowMemory I responds.
Response: "I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?"

Memory Content:
Memory content [HumanMessage(content='What is 2+2'), AIMessage(content='The answer to 2+2 is 4'), HumanMessage(content='Can you please summarize this conversation?'), AIMessage(content="I'm sorry, but as an AI, I don't have the ability to summarize the current conversation. However, I can assist you with any specific questions or tasks you have. How can I help you today?")]

Have any of you found this issue before? How can I make the Agent use the Memory?

### System Info

!pip install -qU langchain \ langchain-community \ langchain-core \langchain-openai \ numexpr \ hub \ langgraph \ langchainhub \ azure-cosmos \ azure-identity

Platform: Google Colab

--- end body ---
