It’s no secret that ChatGPT and other tools utilizing large language models (LLMs) are in the headlines every day. With the advancement of this technology, questions have emerged from the public and private sectors over AI assistants, with the full range of opinions on its impact: from a panacea for society’s problems all the way to a sign of the end times.
The truth, unsurprisingly, is more mundane. Like any tool, the effectiveness of AI assistants (or chatbots) largely depends on their usage and how well these applications match up with the tasks they’re designated to complete. One such task is AI-coding assistants or the usage of these models in tools focused on writing and understanding computer code. As LLM-based tools become more commonplace, it’s expected that dev teams will explore this tech, attempting to find productivity boosts that keep them in the IDE and not on Stack Overflow.
Like any new tool, there are risks and unknowns. Risks must be understood, and mitigations considered by security teams to help maximize potential while also prioritizing safety and security.
Common Misconceptions & Biases
As mentioned, the discourse around LLMs has been characterized by alternating levels of hype and hysteria. Just as we’ve seen internet pundits claim that since ChatGPT can pass the Bar Exam now, lawyers are in jeopardy, so too have AI Coding assistants been touted as a way to replace traditional development. The job of a lawyer isn’t sitting around answering Bar questions, just like the job of a developer isn’t purely smashing out lines of code. The reality is far more nuanced and complex.
These tools cannot perform tasks such as generating whole large programs or making contextual decisions based on a large, interconnected codebase. This is because the prompts constructed to interact with these tools are limited to a certain number of tokens. Only so much information from the current context can be included in the request.
In addition, the tools today are not capable of parsing large documents, inferring requirements, and generating code based on these parsed requirements. There are quite a few challenges and complexities that would need to be worked out before this becomes viable with an acceptable number of errors. There’s no doubt that this is a goal, but today, from a practical sense, this is still out of reach.
Risks
In practice, several risks are involved in using automated coding assistants. Just like traditional LLMs applied to human language provide surprising and puzzling results, the same can be said of LLMs applied to code. Dev teams must exercise caution and evaluate LLM-generated solutions to avoid issues.
One major obstacle is automation bias, where individuals defer to suggestions created by an automated system, even when it might run counter to intuition or logic. In this sense, having devs implicitly trust the LLM’s output can create a series of issues with implementing potentially buggy code and impacting the company’s security posture. Even though the code is functional, there are no guarantees that the output of the tool is safe.
Another issue is architectural challenges. LLMs don’t have contextual knowledge of the larger software system being built or individual goals. In that sense, they only have limited visibility into previously written code. Suppose developers are working on a large code base with interconnected components. In that case, the assistant won’t understand that context and may make one-off suggestions leading to bloated code and causing maintenance issues.
Finally, data leakage. Current AI coding assistants are delivered from various suppliers in an “as a service” model. As such, the information sent in a prompt can include sensitive information like keystrokes and other metadata from the project, including chunks of source code, comments, and telemetry data to these 3rd party hosts. This data can be seen when prompts are constructed and captured before being sent to the API endpoint.
Functionally, you do not have control over how this data is collected, stored, or used after the fact. These organizations can use your data for additional training and improvements. It may also be the case that humans review the submissions of the tool as well, so not all of the training is done through an automated process. For many organizations, this represents a serious risk and one that should be considered at length.
Mitigating Challenges
Even though there are technical risks, there’s a need for additional caution. The idea that companies should outright “ban” these tools by default can lead to issues in and of themselves. For one, coders will continue to use the tools at their disposal and not notify security teams. This is something we witnessed in the early days of cloud computing. Two, as these technologies mature, they have the potential to assist in creating more streamlined development. As with any new technology, thoughtful approaches based on real risk are necessary.
Instead, security teams must work with developers to introduce these tools into their environment, albeit in a manner that prioritizes safety and security, ensuring developers are aware of the potential risks.
Although automated coding assistants are relatively new, many existing tools and processes can address these new risks. No amount of tooling or processes will reduce it to zero, but implementing an effective strategy can reduce risks to an acceptable level while still letting your developers use the tools they’d like to use. It’s a good idea to expect issues from these tools, and the quality level of the output should be considered alpha or beta-quality as of the time of this writing.
A recommended sequence to reduce risks is implementing a process akin to the following:
Given the risks and potential for errors, it’s best to think about code contributions from these tools akin to being contributed by a 3rd party contributor. They may function perfectly well and not have obvious issues, but you should put additional scrutiny on the security aspects.
In the end, the true impact of transformer-based LLMs remains to be seen. Technology has a way of surprising us, and surprises aren’t always good for security. On the positive side, these tools may provide a different way of working that assists developers and enhances productivity in a positive way. We are in the beginning stages. However,
these tools use inputs created by countless people worldwide, a majority of whom haven’t consented to their data being used for these purposes. We’ve already seen legal challenges, and in the future, we may see even more legal issues. However, the truth is that these tools are here now, and they’re being used every day, so it’s imperative to understand them and create a firm organizational stance on their deployment as soon as possible.
##
ABOUT THE AUTHOR
Nathan Hamiel, Senior Director of Research at Kudelski Security
Nathan leads the Fundamental and Applied Research team. To support the innovation of future products and services, his team focuses mainly on privacy, advanced cryptography, emerging technologies, and on internal and external collaboration. Nathan is a regular public speaker at global security events, including Black Hat, DEF CON, HOPE, ShmooCon, SecTor, ToorCon, and many others. He is also a veteran member of the Black Hat review board, where he serves as the track lead for AI, ML, and data science.
