The preview release of GitHub Copilot Agents this week ushers in a whole new world of AI development, and if you are smart about it, increased productivity. However, your success with GitHub Copilot can be often driven by the prompting and other contextual information that you provide the agent. In this post, we explore the reasoning behind why we need to educate, what we need to share, and how we might go about doing it.
Why Education is Important
Although this may seem simple to many, it is important to understand that we need to provide instructions to the AI Bot if we want to get the best results. If you try to assign items to a Copilot agent, but don't give it any context your results may not be the best, and long-term, you might end up consuming excessive amounts of credits, which can add up to a costly monthly bill. By taking some time BEFORE you explore with Copilot to provide proper instructions, you can greatly improve the overall effective nature of the bots, and long-term, if you notice common issues you can use the same concepts to provide context to train away the bad behaviors.
Organization Level Configurations
I'm going to start by looking at configuration options that will apply across ALL repositories that you interact with inside of your organization. You access this configuration prompt via "Settings" -> "Copilot" -> "Custom Instructions" within your organization. In this location, you are given the ability to supply up to 600 characters of instruction. The instructions provided by GitHub state, "Teach Copilot your coding standards, languages, and frameworks. Keep instructions high-level, not about specific workflows. Instructions apply to all queries once saved and may not work perfectly."
Within the editor, they give helpful hints for the following:
- Code with the base prompt detail of "Prefer writing <language> if no language is specified."
- Dependencies with the base prompt detail of " Use <package manager> for <language> dependencies"
- Knowledge Base with the base prompt detail of "Prioritize <knowledge base> when asking about <topic>."
- Responses with the base prompt detail of "Respond with <bullet points/minimal preamble>"
We can see from these items that we get a high-level idea of what kinds of things we MIGHT want to do, but what the heck are knowledge bases, you might ask? Well, let's take a look.
Knowledge Bases
Github supports the utilization of repositories as knowledge bases to provide high-level grounding information that is utilized for all interactions with AI. GitHub will index all Markdown files within the selected repositories to create the high-level knowledge base. When we look at being efficient, I believe it is critical for organizations to create knowledge bases with development standards, practices, and other elements, as it makes it easy to prompt both at an organization, or repository basis to ground against a set of rules.
In my case, we already had a "development standards" repository, so it was easy to mark that as a Knowledge Base and then add it to our prompt. If you don't have any of these types of documents, you may want to prioritize getting them written, You might even be able to use Copilot to help you create the initial standards if you desire!
Building our Organization Instructions
With our knowledge base created, at the highest level, I made the following instructions within my organization, which seems to have helped greatly in our overall strategy.
- Prioritize Internal Development Standards when asking about best practices for ASP.NET MVC UI implementations
- Respond with bullet point details, and succint communications when you respeond.
In my case, we have a single knowledge base called "Internal Development Standards," which only has detailed information for ASP.NET MVC, so I provided the context and limited it to the language. This allows for short but effective communication, with many characters to spare.
Repository Specific Configurations
With our organization-specific items created, it is time to shift the focus to our repositories. This is where we can provide individual context to the specific project, and providing detailed structure information here can really help. You create these using a custom markdown file at the path '.github/copilot-instructions.md'. The existing primary documentation on this feature is very limited, but there is another Agent-specific set of recommendations that provides a lot more details.
I have found, in my limited, although fairly extensive testing, that it is critical to structure your repository-specific instructions following the format listed for the agent, especially focusing on a few key areas.
- Required Before Each Commit
- This was a critical step in my findings. Without setting a standard here, such as running all tests, the AI would blindly make the changes and submit PRs that did not work. When adding a simple instruction here to tell it to run tests first, it resulted in far better results.
- Development Flow
- Tell the AI how you want it to work, be specific, I provided the exact examples of the "dotnet build" and similar commands, which seemed to be great.
- Repository Structure
- I found that this was worthwhile to add, as it helped to ensure that the files added were placed in the right location. It also allowed me to interact better in a more human language by referring to things by their location.
- Key Guidelines
- I see this in conjunction with the organization-level items and as something that helps to add a bit of final improvements.
By thinking of these requirements and being specific, thinking of writing this just like how you would tell a new developer to interact, you can get great results with minimal effort!
My Results & Next Steps
This is all VERY early, the public preview of the GitHub Copilot Agents only came out on Monday. However, my results to date are game-changing and very successful. I have been able to assign complex tasks to the Agent and get mergeable PR's that meet our development standards with little, if any, edits, clearing a backlog of tasks that already had issues that were waiting for development help.
I encourage you to give it a try, share your findings, and be sure to share your feedback with the team @ GitHub as well.