AI Weekly: The promise and limitations of machine programming tools

Machine programming, which automates the development and maintenance of software, is becoming supercharged by AI. During its Build developer conference in May, Microsoft detailed a new feature in Power Apps that taps OpenAI’s GPT-3 language model to assist people in choosing formulas. Intel’s ControlFlag can autonomously detect errors in code. And Facebook’s TransCoder converts code from one programming language into another.

The applications of computer programming are vast in scope. And as computers become ubiquitous, the demand for quality code draws an ever-growing number of aspiring programmers to the profession. After years of study to become proficient at coding, experts learn to convert abstracts into concrete, executable programs. But they spend the majority of their work hours not programming. According to a study from the University of Cambridge, at least half of developers’ efforts are spent debugging, which costs the software industry an estimated $312 billion per year.

AI-powered code suggestion and review tools promise to cut development costs substantially while allowing coders to focus on more creative, less repetitive tasks, according to Justin Gottschlich, principal AI scientist at Intel’s machine programming division. Gottschlich is spearheading the work on ControlFlag, which fuses machine learning, formal methods, programming languages, and compilers to detect normal coding patterns, identifying abnormalities in code that are likely to cause a bug.

“Prior to machine learning- or AI-based programming systems, programmers had dozens — perhaps hundreds — of tools to help them be more productive, produce code with fewer logic errors, improve the software’s performance, and so on. However, nearly all of these systems were ‘rules-based,’” Gottschlich told VentureBeat via email. “While useful, rules-based systems are inherently limited in scope by the rules that they have been programmed into them. As such, if new kinds of things occur, the systems would need to be updated by humans. Moreover, these rules-based systems have always been prone to human error in creating the rules encoded in them. For example, programmers may accidentally create a rule to find a certain type of bug, but incorrectly define the rules to find it. This hidden bug in the rules system could go undetected forever.”

Gottschlich asserts that AI-based systems offer benefits over the rules-based systems of yesteryear because AI can learn on its own in an unsupervised fashion, enabling it to draw on massive code databases. With unsupervised learning, an algorithm is fed “unknown” data for which no previously defined labels exist. The system must teach itself to classify the data by processing it to learn from its structure.

For example, ControlFlag was trained on over 1 billion unlabeled lines of code to identify stylistic variations in programming language. As for TransCoder, it learned to translate between C++, Java, and Python by analyzing a GitHub corpus containing over 2.8 million repositories. Microsoft trained a bug-spotting program on a dataset of 13 million work items and bugs from 47,000 developers across AzureDevOps and GitHub repositories. And code review platform DeepCode’s algorithms were taught using billions of lines of code captured from public open source projects.

Code generation versus augmentation

There’s a difference between AI-powered coding tools that can generate code from whole cloth versus augment a programmer’s workflow, of course. The latter is more common. Startups such as Tabine (formerly Codota) are developing platforms that suggest and autocomplete scripts in Python, C, HTML, Java, Scala, Kotlin, and JavaScript. Ponicode taps AI to check the accuracy of code. Intel’s Machine Inferred Code Similarity engine can determine when two pieces of code perform similar tasks, even when they use different structures and algorithms. And DeepCode offers a machine learning-powered system for whole-app code reviews — as does Amazon.

“Currently, we see a lot of AI-powered assistants, enabling software engineers to gain velocity and accuracy in their work. And the reason for the availability of more assistant tools than automation tools is that AI-powered automation has simply not yet reached the level of accuracy required,” Ponicode CEO Patrick Joubert told VentureBeat. “Our industry is still young, and even though we can already see the potential of automation with AI based code generators, we have to acknowledge that automatically generated code is still pretty unmaintainable and the overall quality is not meeting the right standards yet. While some engineers are working on the future of AI powered automation, my team and I, along with many other stakeholders, are dedicated to creating tools that can be used today. Within a few years I believe there will be enough tools to cover all steps of the development lifecycle.”

For Joubert, the most intriguing categories of machine programming tools today are autocompletion and code analysis. Autocompletion systems like Tabnine and Kite employ AI to analyze semantics and make sense of code, autocompleting functions with a sense of the code’s semantic content and purpose. As for code analysis tools like Snyk and DeepCode, they’re dedicated to finding vulnerabilities in the code and suggesting actions to resolve them — often with surprising speed and precision.

“When we see the numerous leaks and bugs from any software, including the ones built by leading multinationals, we can agree that [the software] industry has not yet matured. AI-powered coding tools are mostly meant to enhance the developer experience and empower them, thanks to greater velocity and greater efficiency,” Joubert added. “Behind these developer-focused benefits, I believe we are on the way to allowing software engineers to build industrial-grade software, where quality, innovation, and speed are reached systematically … Autocompletion [in particular is] enabling software engineers to focus on the most complex part of their codebase and removing the burden of manually writing long strings of code.”

Limitations

Despite their potential, both AI-powered code generators and coding assistance tools have their limitations. For example, while GitHub has over 250 million code repositories alone, most of the data is unannotated. There’s only a few examples that describe precisely what the code does, posing a particular challenge for any system that can’t learn from unlabeled data.

In an effort to address this, IBM recently released CodeNet, a 14-million-sample labeled dataset with 500 million lines of code written in 55 programming languages. The company claims that the rich annotations added to CodeNet make it suitable for a diverse set of tasks as opposed to other datasets specialized for specific programming tasks. Already, researchers at IBM have conducted several experiments with CodeNet, including code classification, code similarity evaluation, and code completion.

“It is my speculation that in the next decade, code semantics understanding systems are likely to be one of the most important areas of machine programming in the coming decade,” Joubert said. “It depends on the domain the machine programming system is being applied to. For small programs, such as unit tests or regression tests, full program synthesizers are a reality today. Yet, for larger programs, it’s currently computationally intractable for machine programming systems to generate the potential thousands or millions of lines of code without the assistance of a programmer.”

Boris Paskalev, the cofounder and CEO of DeepCode, calls creating a couple of lines of code with AI “more of a toy than a productivity breakthrough.” While techniques like natural language processing work well with text because there’s fixed limits on the words and syntax that need to be understood, code isn’t the same, he argues.

“Since there are no formal rules for software development, [programming] is an art that requires a complete understanding of code and a developer’s intentions to produce something that works as expected without bugs,” Paskalev told VentureBeat. “As far as we’ve come in using machine learning and neural networks for code, we’re still only in the ‘invention of the wheel’ phase …  machine learning is already proving to be very useful for code, but only after it goes through a semantic machine learning-representation of the code: making sure all semantic facts, variables, transitions, and logical interrelations are clearly represented and considered by the learning model.”

To Paskalev’s point, recent studies suggest that AI has a ways to go before it can reliably generate code. In June, a team of researchers at the University of California at Berkeley, Cornell, the University of Chicago, and the University of Illinois at Urbana-Champaign released APPS, a benchmark for code generation from natural language specifications. The team tested several types of models on APPS, including OpenAI’s GPT-2, GPT-3, and an open source version of GPT-3 called GPT-Neo. In experiments, they discovered that the models could learn to generate code that solves easier problems — but not without syntax errors. Approximately 59% of GPT-3’s solutions for introductory problems had errors, while the best-performing model — GPT-Neo — attained only 10.15% accuracy.

“When generating code from whole cloth, there are typically challenges around both specifying the intent and consuming the results,” Tabine CEO Dror Weiss told VentureBeat. “User intent can be specified in natural language by providing examples, writing code in a higher-level language, or in other means. But in most cases, this intent does not provide a full specification of the desired behavior. Also, the generated code may be following different route than what the developer had in mind. As such, it may be challenging for the developer to judge whether the code performs the desired operation exactly.”

Facebook AI researchers Baptiste Rozière  and Marie-Anne Lachaux, who worked on TransCoder, agree with Tabine’s assessment. “It is inherently difficult to generate correct code from unspecific natural language problem descriptions that could correspond to several different code snippets. An easier task would be to generate code from an input that is more specific and closer to the output code, like pseudo-code or code written in a different language,” they told VentureBeat. “A huge obstacle to the adoption of … methods generating large amounts of code without human supervision is that they would need to be extremely reliable to be used easily. Even a tool that could generate methods with 99% accuracy would fail to generate a working codebase of hundreds of functions. It could speedup the code generation process but would still require human testing and intervention.”

Rozière and Lachaux also point out that tasks around code generation are generally much harder than classification tasks because the model has a lot of freedom and can create many different outputs, making it hard to control the correctness of the generation. Moreover, compared with natural languages, programming languages are very sensitive to small errors. A one-character difference can change the semantics of the code and make the output faulty.

“Current machine learning algorithms may not be able to generalize well enough to different problems to match human performance for coding interviews without larger datasets or much better unsupervised pre-training methods,” Rozière and Lachaux said.

Potential benefits

Paskalev thinks it’ll be at least five to ten years until natural language processing enables developers to create “meaningful components” or even entire apps from a simple description. But Gottschlich is more optimistic. He notes that AI-powered coding tools aren’t just valuable in writing code, but also when it comes to lower-hanging fruit like upgrading existing code. Migrating an existing codebase to a modern or more efficient language like Java or C++, for example, requires expertise in both the source and target languages — and it’s often costly. The Commonwealth Bank of Australia spent around $750 million over the course of five years to convert its platform from COBOL to Java.

“Deep learning already enables us to cover the smaller tasks, the repetitive and redundant ones which clutter a software engineers’ routine. Today, AI can free software engineers from tedious tasks slowing them down and decreasing their creativity,” Gottschlich said. “The human mind remains far superior when it comes to creation, innovation, and designing the most complex parts of our softwares. Enabling them to increase velocity in these exciting, high added value parts of their work is, I believe, the most interesting way to leverage the power of machine learning today.”

Joubert and Weiss say that the potential business value of machine programming also can’t be ignored. An estimated 19% to 23% of software development projects fail, with that statistic holding steady for the past couple of decades. Standish Group found that “challenged” projects — i.e., those that fail to meet scope, time, or budget expectations — account for about 52% of software projects. Often, a lack of user involvement and clear requirements are to blame for missed benchmarks.

“We see a great number of new tools using AI to enhance legacy code and help existing assets reach industrial-grade standards. We can elevate developer legacy code management workflows and be part of reducing the hefty level of technical debt built up over the past 50 years in the software industry,” Joubert said. “The days when developers had to write and read code line by line are gone. I’m excited to see how the other steps in the software development lifecycle are going to be transformed and how tools will reach the same level that Kite or Snyk have attained. Leveraging AI to build efficient, one-purpose, tested, secure, and documented code effortlessly is going to profoundly change the way software companies can create incremental value and innovation.”

From Weiss’ perspective, AI-powered coding tools can reduce “costly” interactions between developers like Q&A sessions and repetitive code review feedback while shortening the project onboarding process. “[These] tools make all developers in the enterprise better. They take the collective code intelligence of the organization and make it available, during development time, to all developers. This allows any developer on the team to punch above their weight,” he said.

For AI coverage, send news tips to Kyle Wiggers — and be sure to subscribe to the AI Weekly newsletter and bookmark our AI channel, The Machine.

Thanks for reading,

Kyle Wiggers

AI Staff Writer