GitHub Copilot Agent: A Disappointing Review

Posted on In Views:

Discover the fatal flaws that make Copilot’s agent mode unusable for serious tasks

GitHub Copilot was one of the earliest providers when the concept of vibe coding emerged. However, this glory was short-lived, as commercial software like Cursor and open-source software like Cline quickly overshadowed GitHub Copilot.

The fundamental reason is that Copilot’s AI functionality remained at the most basic level, namely auto-completion. However, Cursor and Cline had already entered the realm of Agentic AI, demonstrating how effortless it could be to delegate tasks to an agent for execution.

GitHub Copilot thus nearly disappeared from the battlefield. Until the latter half of 2025, Copilot finally introduced Agent mode, but by then it was already too late.

However, keeping the spirit of trying everything, I recently spent some time testing it out. After a week of use, my impression is that it’s really not good.

Major Problems

I will list two serious issues that I believe are most critical.

As for whether the effect is good or bad, or whether AI is intelligent, I believe that is no longer the point.

The worst issue, in my opinion, is that Copilot Agent is “almost” unable to obtain the results of executed commands.

I say “almost” rather than “completely” because for commands with extremely fast response times, such as ls, it works. However, for commands that take longer to execute, it is completely impossible to obtain the results, as it simply won’t wait for them.

In fact, this is not an isolated case; people are discussing this serious flaw in Copilot on various forums.

Just to list a few, but there are actually many more.

This means that you cannot rely on Copilot to perform many important tasks. These include, but are not limited to, the following common examples.

  1. Installing dependencies, such as pip install or npm install. These installations can take a significant amount of time, and sometimes we ask the Agent to analyze the output to help resolve dependency conflicts.
  2. Unit testing. This should also be easy to understand. Unit testing takes time unless you run them one by one. If Copilot can’t help us run unit tests, how can it help us fix bugs?
  3. Remote operations. This is a bit abstract but also common, such as asking the Agent to connect to a remote machine via SSH or to a container via Docker to execute commands and troubleshoot issues based on the results.

There are many other scenarios where the Agent’s inability to wait imposes limitations.

This is a fatal flaw.

We expect the Agent to be capable of anything, so we guide it through tools like MCP and other methods. But if Copilot can’t even handle the basic task of waiting for results, that’s truly tragic.

Task functionality missing

A task refers to breaking down a complex instruction into smaller, executable tasks and executing them in sequence like a to-do list, then summarizing the results.

Whether this functionality is explicit or not, it should be an essential feature. As I mentioned earlier, numerous papers have demonstrated that when LLMs are presented with overly large tasks, they can collapse, and even with countless tokens, progress cannot be advanced.

Therefore, whether it’s Cursor, Claude Code, or the open-source Cline, they have all incorporated this functionality, though the names may vary, the capabilities are similar.

Copilot, however, is still struggling to catch up as it slowly enters the world of agentic AI.

This directly results in Copilot performing poorly when faced with complex tasks, necessitating the use of various prompt engineering techniques to adjust the workflow. This undoubtedly increases the learning curve for users.

Some Mitigations

Is there a way to resolve Copilot’s mentioned challenges?

Yes, but the effectiveness is limited.

I made an attempt by using the LiteLLM Gateway to treat Copilot as a model provider and allow open-source frameworks like Cline or Kilo Code to directly leverage it.

Is it better? It is indeed more helpful.

These open-source frameworks can solve the two biggest problems mentioned above, but the downside is that they are prone to crashing. When dealing with specific tasks, they often fail to perform smoothly, especially when calling tools or using MCP.

After all, Copilot was not originally designed to be a model provider, and this means that even if the API format can be converted to OpenAI Compatible Format, its behavior still struggles in various scenarios.

Wrap Up

I have conducted in-depth research on various products, as I am an architect focused on AI implementation in enterprises. Therefore, I will conduct in-depth testing using the daily tasks of software engineers, and the insights gained should be quite accurate.

I spent a significant amount of time testing Copilot, and I can only say that Copilot is usable but not outstanding.

If your company provides Copilot as a resource, you can consider it a powerful programmer. When it comes to auto-completion, Copilot does have some capabilities. It can also provide real-time suggestions, and in small-scale scenarios, Copilot’s capabilities are definitely reliable.

However, if you want to fully integrate Copilot into the AI development workflow, Copilot is far from capable of handling that.

I will also provide my personal ranking. The following is based solely on commercial products; open-source software is not included due to different evaluation criteria. The ranking is based on my actual testing experience and is for reference only.

Kiro > Claude Code > Cursor = Rovo Dev >>> Codex >>> Gemini Cli > Github Copilot

From this ranking, it is clear that the top-ranked ones are all Claude’s models. As for the ones behind them, the results are really disappointing, especially the ones at the bottom, which are in a completely different league from the ones at the top.

Gemini CLI does not have the two serious flaws of Copilot, but Gemini CLI also has its own issues, which we may discuss another time.

Originally published on Medium