Compiled by Yue Jia, Ke Mao, and Mark Harman, Heart of machine.
Please note: this is not a reason to write a Bug.
Debugging code can be boring, but SapFix, a new AI hybrid tool created by Facebook engineers, can save engineers a lot of time and speed up the rollout of new software. The tool automatically generates fixes for specific bugs and then submits them to engineers for approval and deployment into production.
SapFix is being used to speed up the transfer of stable code updates to millions of devices running Facebook’s Android app, the first use of ai testing and debugging tools in such mass production. Facebook intends to share SapFix with the engineering community because it is the next step in the evolution of automated debugging, potentially driving the productivity and stability of new code for many companies and research institutions.
SapFix is designed to be a standalone tool that runs with or without Sapienz, Facebook’s intelligent automation software testing tool that was released at F8 and deployed in production. In its current proof-of-concept state, SapFix focuses on fixing bugs found by Sapienz before deploying to build. The process starts with Sapienz, which, along with Facebook’s inferential static analysis tool, helps locate bugs in your code that need to be fixed. Once Sapienz discovers and deduces a specific piece of code that is relevant to a Bug, it can pass that information to SapFix, which automatically references some policies and generates a patch.
How is SapFix debugged
To address high-trigger bugs, SapFix creates patches that restore the original code completely or partially. For more complex problems, SapFix generates patches by extracting material from the template fix collection. These templates are automatically retrieved from templates created by human engineers based on past fix libraries.
When previously used templates designed by humans don’t work, SapFix will attempt to fix them based on mutations. To do this, it performs a small code fix to the abstract syntax tree (AST) of the statement that caused the crash, tweaking the patch until a possible solution is found.
Independent verification and manual approval
Even if SapFix finds a specific patch, its work is far from over. The tool generates multiple potential fixes for each Bug and evaluates the quality of the fixes through three issues. Is there a compilation error, does the program still crash, and does the patch introduce a new conflict?
To address the last two issues, SapFix runs existing, developer-written tests on the fix, along with tests created by Sapienz. As with the previous patch generation steps, the validation process is automated and separated from the larger code base implementation. SapFix is replicating the debugging work that is currently done manually, but it is not designed to deploy patches into production code.
When the auto-generated patches are fully tested, SapFix sends them to a human reviewer for approval. This is similar to manually generated reports that need to be reviewed and approved by other developers, and in addition to automatically tracking reviewers’ feedback, the system accepts approved patches and cleans up other unapproved patches. In some cases, SapFix can select the best solution from multiple candidate patches and make recommendations to engineers.
If the fix is rejected, it discards the patch and, if accepted, loads it. Because the underlying technology of SapFix is very powerful and automatic running saves a lot of time and effort, SapFix cannot implement its proposed patches. Engineers are usually professional, and the tool relies on their expertise to determine whether the proposed patch should be deployed.
Because SapFix is still in development, it can’t be used as widely as Sapienz, which now generates hundreds of Bug reports a month, pinpointing the lines of code with bugs, It reviews code related to apps like Facebook, Instagram, Workplace and Messenger on Android devices. About three-quarters of the bugs Sapienz reported were fixed by developers. In August, Facebook began testing SapFix, a tool that can successfully generate patches that can be accepted by human engineers and go into production.
Lays the foundation for fully automated debugging
According to Facebook, this marks the first time machine-generated patches — automated end-to-end testing and patching — have been deployed to a Facebook-sized code base. This is an important milestone in AI technology and further proof that search-based software engineering can reduce problems in software development. Facebook developed SapFix to deal with different kinds of bugs and software, so the tool has the potential to change the speed and quality of code generation. This is true not just for large-scale businesses, but for almost any individual who writes code. Whether used by organizations or individuals, SapFix and Sapienz help developers spend less time on debugging and more time on generating more code.
Even with this work, Facebook wants to encourage the community to continue working on auto-tuning and improving the code. There has been a lot of exciting research in this area, including empirical studies of technologies, open problems and challenges for community research, and recent investigations into automated code improvement. As the first tool deployed on a Facebook scale, SapFix will provide new momentum and energy for challenging research.
Facebook said it will open source Sapienz and SapFix after completing additional engineering sections. Feedback on these tools will help both the researchers involved and the broader AI community improve the task of automatically finding and fixing code bugs. Facebook is currently focusing on how SapFix automatically stops code crashes, but longer-term applications should include ways to make software faster and more responsive. These systems provide great baseline benefits, and the effects of their use are varied and broad.