Literature Review Synthesizer

AI-Assisted Coding Tools, Developer Productivity, Code Quality and Skill Development: A Literature Review

Scope: 25 sources. Six are credible preprints from academic, industrial or research-institute teams that were not peer-reviewed as of September 2026. They are marked "[preprint]" and weighted with that in mind. Every study was checked against its publisher, arXiv or official project page. One conflict of interest: Shen & Tamkin (2026) was written by researchers at Anthropic, which makes the model producing this review. It is included because it is one of the few randomized studies of skill formation, not because of that link.

1. Overview

Research on AI-assisted coding moved from early studies of Copilot's correctness and usability (2021–2022) to large field experiments and causal studies of AI agents (2024–2026). The evidence now supports two broad statements. First, AI assistance can speed up well-specified, bounded tasks. Second, that gain shrinks, disappears or reverses on complex work in familiar codebases, and it often costs something in code quality, security or learning. The central disagreement is no longer whether these tools change developer work. It is for whom, on which tasks, and over what time horizon the net effect is positive.

2. Thematic Synthesis

Theme 1: The productivity paradox — large lab gains, mixed field effects

Intellectual background. Code-completion research began with statistical models that exploit how predictable code is (Hindle et al., 2012). The pair-programming literature was its behavioral forerunner. Williams et al. (2000) argued that pairing improves quality at small time cost. Hannay et al.'s (2009) meta-analysis gave a more cautious picture: a small positive effect on quality, a medium positive effect on duration, and a medium negative effect on effort, with benefits depending on task complexity. That "it depends on the task" result foreshadows the current AI literature almost exactly, although AI studies rarely cite it.

Evidence of large gains. The most cited result is Peng et al.'s (2023) [preprint] controlled experiment. Developers with Copilot finished a JavaScript HTTP-server task 55.8% faster than controls. The task was isolated, greenfield and well-specified, which is the setting where completion tools should do best. Cui et al. (2025) moved to the field. They pooled three randomized trials at Microsoft, Accenture and a Fortune 100 firm (N = 4,867) and estimated a 26.08% increase in completed tasks (SE = 10.3%). Paradis et al. (2024) [preprint] ran an RCT with 96 Google engineers on an enterprise-grade task and estimated a speedup of about 21%, while noting a wide confidence interval. Ziegler et al. (2024) took a telemetry approach. They found that acceptance rate of suggestions predicted developers' perceived productivity better than more detailed usage measures. That is a finding about perception, not output.

Evidence of null or negative effects. Several studies push back. In Xu et al. (2022), an early in-IDE natural-language-to-code plugin did not significantly improve productivity or accuracy. Vaithilingam et al. (2022) found that Copilot "did not necessarily improve" completion time or success rate, yet most participants preferred to use it. The strongest counter-result is Becker et al. (2025) [preprint]. In an RCT with 16 experienced open-source maintainers on 246 real tasks in repositories they knew well (about five years' experience each), allowing AI tools (mainly Cursor with Claude 3.5/3.7 Sonnet) increased completion time by 19%. Developers had predicted a 24% speedup, and even afterward they believed they had been about 20% faster.

Synthesis. These studies agree more than the headline numbers suggest once you account for three moderating factors:

1. Task specification and novelty. Gains are largest on self-contained greenfield tasks (Peng et al., 2023) and smallest on work inside complex, familiar codebases (Becker et al., 2025).
2. Developer familiarity. Experts who already know the codebase have the least to gain. Cui et al. (2025) found larger gains and higher adoption among less experienced developers, and Peng et al. (2023) reported similar heterogeneity.
3. Outcome measure. "Tasks completed," "time on task" and "lines added" measure different things, and none captures downstream rework (see Theme 2).

The pattern in Becker et al. (2025) and Vaithilingam et al. (2022), where perceived and measured productivity diverge, is a warning about the many self-report studies (e.g., Liang et al., 2024). This is the most contested area of the field. No consensus effect size is warranted.

Theme 2: Code quality — correctness, maintainability and the quality debt of speed

Benchmark-style evaluations. Early work tested tools in isolation. Nguyen and Nadi (2022) found Copilot's correctness on LeetCode problems ranged from 57% (Java) to 27% (JavaScript), with generally low complexity but some calls to undefined helper methods. Moradi Dakhel et al. (2023) found that human solutions were correct more often than Copilot's, but Copilot's buggy solutions were easier to repair. They concluded Copilot is "an asset for experts" but a possible "liability" for novices who may miss errors. Yetistiren et al. (2023) [preprint] compared tools on HumanEval: correctness was 65.2% for ChatGPT, 46.3% for Copilot and 31.1% for CodeWhisperer, and technical-debt estimates were modest.

These studies agree that generated code is often plausible but wrong. Their shared limitation is ecological validity. Algorithmic benchmark problems are not representative of production code, and the tool versions tested are now several generations old.

Project-level effects. The more consequential evidence comes from the field. He et al. (2025) [preprint] used a difference-in-differences design with matched GitHub projects. Cursor adoption produced a large but transient velocity increase (3–5× lines added in the first month, fading after about two months). It was followed by persistent increases in static-analysis warnings (+30%) and code complexity (+41%), and the authors found these increases contributed to later velocity slowdowns. This is the clearest evidence yet linking short-term productivity to long-term quality debt. However, it covers one tool, open-source projects, and proxy measures of quality.

The downstream-maintainability gap. Almost no study measures whether AI-assisted code is harder for other developers to maintain later. Borg et al. (2024) registered a two-phase experiment to test exactly this. Its Stage 1 protocol was accepted at ICSME 2024, but no results have been published. Evidence on long-term maintainability is therefore thin.

Theme 3: Security — insecure suggestions versus insecure users

This theme has the most internally consistent methods and the most instructive disagreement.

Tool-centric evidence. Pearce et al. (2022) prompted Copilot with scenarios based on MITRE's high-risk CWE weaknesses and found it often suggested vulnerable code. Fu et al. (2025) looked at Copilot-, CodeWhisperer- and Codeium-attributed code in real GitHub projects. They identified 733 snippets likely to contain security weaknesses, including 29.5% of the Python and 24.2% of the JavaScript snippets.

User-centric evidence conflicts. Perry et al. (2023) found that participants using a Codex-based assistant wrote significantly less secure code, and were more confident that their code was secure. Sandoval et al. (2023) studied C programming with 58 student participants and found AI-assisted users introduced critical security bugs at a rate no more than 10% higher than controls. They concluded that LLM assistance did not introduce new security risk in that setting.

Reconciling them. The two user studies differ in task type (security-critical API use versus a linked-list implementation), language, and participant population. Taken together, they suggest the risk depends on the task. It is greatest where secure usage depends on knowledge the developer does not have, and where overconfidence stops them from checking. Perry et al.'s overconfidence finding also links this theme to the perception-reality gap in Theme 1.

Theme 4: Interaction patterns — acceleration, exploration and the cost of verification

HCI and programming-languages research explains the mechanisms behind the mixed outcomes. Barke et al. (2023) developed a grounded theory of two modes of use. In acceleration mode, the programmer knows what they want and uses the tool to get there faster. In exploration mode, they are unsure and use the tool to survey options. Productivity gains plausibly come mostly from acceleration mode, which requires existing competence. That fits the expertise effects in Themes 1 and 5.

Mozannar et al. (2024) had 21 programmers label their own Copilot sessions using the CUPS taxonomy. They found programmers spent more time verifying suggestions than writing code. Verification is a hidden cost that acceptance-rate measures (Ziegler et al., 2024) do not capture. Liang et al.'s (2024) survey of 410 developers found people use assistants mainly to cut keystrokes, finish tasks quickly and recall syntax, and much less for brainstorming. They also identified usability problems in controlling tools and aligning their output with developers' intentions.

Together, these studies suggest that the cost-benefit balance depends on how cheap verification is. That would explain why experts in complex, high-standard codebases (Becker et al., 2025) may gain little: verifying a suggestion against implicit project conventions is expensive.

Theme 5: Skill development — novices, experts and the "widening gap"

The expertise-moderation question is where the evidence is most divided.

Arguments that novices benefit. Kazemitabaar et al. (2023) ran a controlled study with 69 learners aged 10–17. Codex access did not hurt learning and improved outcomes for learners with prior block-based programming experience. Cui et al. (2025) and Peng et al. (2023) found larger productivity gains among less experienced developers, suggesting AI may flatten the experience curve.

Arguments that novices are harmed. Moradi Dakhel et al. (2023) warned that novices may not detect buggy suggestions. Prather et al. (2024) observed novices using AI and found that stronger students used it to speed up code they already intended to write, while struggling students' metacognitive difficulties got worse and they often ended with an "illusion of competence." The authors argue this widens the gap between stronger and weaker learners. Shen and Tamkin (2026) [preprint; Anthropic authors] ran randomized experiments on developers learning a new asynchronous programming library. AI use impaired conceptual understanding, code reading and debugging without significant average efficiency gains. Participants who fully delegated the work gained some speed but learned less. They also identified interaction patterns involving cognitive engagement that preserved learning.

Reconciling them. The apparent conflict eases once you separate performance with the tool from competence without it. Productivity studies measure the first. Learning studies (Prather et al., 2024; Shen & Tamkin, 2026) measure the second. Kazemitabaar et al. (2023) is the exception because it tested retention, and it found no harm. That may reflect its structured learning environment, where each AI-assisted task was followed by a manual modification task. Taken together, the current evidence suggests that how learners engage with AI (delegating versus staying cognitively engaged) matters more than whether they have access to it. The evidence is recent, includes a preprint, and comes from short-term studies. No longitudinal study of professional skill atrophy has been published among the sources reviewed.

3. Methodological Landscape

Cross-cutting concerns:

1. Construct validity of "productivity." Speed, completed tasks, lines added and acceptance rate are not interchangeable. Few studies use a multidimensional framework.
2. Obsolescence. Tools change faster than the peer-review cycle, so findings about 2021 Codex may not hold for 2025 agentic tools. This helps explain why so much recent evidence is in preprints.
3. Conflicts of interest. Much of the headline productivity evidence was authored or funded by tool vendors (GitHub/Microsoft, Google). The skill-formation RCT comes from Anthropic researchers.
4. Theory. Few studies draw on established theory. Pair-programming results (Hannay et al., 2009) and the cognitive and metacognitive frameworks used in education research (Prather et al., 2024) are rarely brought into SE productivity studies.

4. Research Gaps and Future Directions (prioritized)

1. Long-term quality and maintainability outcomes. He et al. (2025) suggest speed now may create quality debt later, but that finding is a preprint about one tool. Pre-registered longitudinal studies, such as the design Borg et al. (2024) proposed, are urgently needed to test whether AI-assisted code costs more to maintain.
2. Longitudinal skill development and deskilling. No study among those reviewed tracks professional developers' unassisted competence over months or years. The short-term harms found by Shen and Tamkin (2026) and Prather et al. (2024) need replication in the workplace.
3. Explaining the expert slowdown. Becker et al. (2025) conflicts with the field RCTs (Cui et al., 2025; Paradis et al., 2024). Studies are needed that deliberately vary codebase familiarity, task complexity and verification cost to test the mechanisms Mozannar et al. (2024) and Barke et al. (2023) suggest.
4. Perception-reality divergence. Self-reported productivity is systematically inflated in at least one RCT (Becker et al., 2025), and users overrate the security of their code (Perry et al., 2023). Tool evaluations and adoption decisions need calibrated, objective measures.
5. Interventions that preserve learning. Shen and Tamkin (2026) and Kazemitabaar et al. (2023) suggest that cognitive engagement protects learning. Designs such as forced explanation or delayed suggestions should be tested experimentally in university courses and in developer onboarding.
6. Independent replication across tools and contexts. Replications without vendor involvement are needed, especially for non-Copilot tools (e.g., CodeWhisperer, now rebranded, has little human-subject evidence), for languages other than Python and JavaScript, and for non-Western and non-enterprise settings.
7. Team- and organization-level effects. Almost all studies measure individuals. Effects on code review load, knowledge sharing and team skill mix are largely unstudied among these sources.

5. Reference List (APA 7th)

Barke, S., James, M. B., & Polikarpova, N. (2023). Grounded Copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1), 85–111. https://doi.org/10.1145/3586030

Becker, J., Rush, N., Barnes, B., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity (arXiv:2507.09089) [Preprint]. METR. https://arxiv.org/abs/2507.09089

Borg, M., Hewett, D., Graham, D., Couderc, N., Söderberg, E., Church, L., & Farley, D. (2024). Does co-development with AI assistants lead to more maintainable code? A registered report (arXiv:2408.10758) [Stage 1 registered report, ICSME 2024]. https://arxiv.org/abs/2408.10758

Cui, Z. K., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2025). The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Management Science. Advance online publication. https://doi.org/10.1287/mnsc.2025.00535

Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025). Security weaknesses of Copilot-generated code in GitHub projects: An empirical study. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3716848

Hannay, J. E., Dybå, T., Arisholm, E., & Sjøberg, D. I. K. (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51(7), 1110–1122. https://doi.org/10.1016/j.infsof.2009.02.001

He, H., Miller, C., Agarwal, S., Kästner, C., & Vasilescu, B. (2025). Does AI-assisted coding deliver? A difference-in-differences study of Cursor's impact on software projects (arXiv:2511.04427) [Preprint]. https://arxiv.org/abs/2511.04427

Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012). On the naturalness of software. In Proceedings of the 34th International Conference on Software Engineering (ICSE). IEEE.

Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023). Studying the effect of AI code generators on supporting novice learners in introductory programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3544548.3580919

Liang, J. T., Yang, C., & Myers, B. A. (2024). A large-scale survey on the usability of AI programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE). ACM. https://doi.org/10.1145/3597503.3608128

Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (2023). GitHub Copilot AI pair programmer: Asset or liability? Journal of Systems and Software, 203, Article 111734. https://doi.org/10.1016/j.jss.2023.111734

Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024). Reading between the lines: Modeling user behavior and costs in AI-assisted programming. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3613904.3641936

Nguyen, N., & Nadi, S. (2022). An empirical evaluation of GitHub Copilot's code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR). ACM. https://doi.org/10.1145/3524842.3528470

Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024). How much does AI impact development speed? An enterprise-based randomized controlled trial (arXiv:2410.12944) [Preprint]. https://arxiv.org/abs/2410.12944

Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022). Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP) (pp. 754–768). IEEE. https://doi.org/10.1109/SP46214.2022.9833571

Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot (arXiv:2302.06590) [Preprint]. https://arxiv.org/abs/2302.06590

Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). Do users write more insecure code with AI assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM. https://doi.org/10.1145/3576915.3623157

Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024). The widening gap: The benefits and harms of generative AI for novice programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research (ICER). ACM. https://doi.org/10.1145/3632620.3671116

Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2023). Lost at C: A user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium. USENIX Association. https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval

Shen, J. H., & Tamkin, A. (2026). How AI impacts skill formation (arXiv:2601.20245) [Preprint]. https://arxiv.org/abs/2601.20245

Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022). Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (Article 332). ACM. https://doi.org/10.1145/3491101.3519665

Williams, L., Kessler, R. R., Cunningham, W., & Jeffries, R. (2000). Strengthening the case for pair programming. IEEE Software, 17(4), 19–25. https://doi.org/10.1109/52.854064

Xu, F. F., Vasilescu, B., & Neubig, G. (2022). In-IDE code generation from natural language: Promise and challenges. ACM Transactions on Software Engineering and Methodology, 31(2), Article 29. https://doi.org/10.1145/3487569

Yetiştiren, B., Özsoy, I., Ayerdem, M., & Tüzün, E. (2023). Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT (arXiv:2304.10778) [Preprint]. https://arxiv.org/abs/2304.10778

Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024). Measuring GitHub Copilot's impact on productivity. Communications of the ACM, 67(3), 54–63. https://doi.org/10.1145/3633453

Things to check before publishing:

• Cui et al. Management Science lists the article online, but I couldn't confirm a final volume or issue, so it's cited as advance online publication.
• Paradis et al. I cited the arXiv version. It may also have appeared at ICSE-SEIP 2025, which I couldn't confirm.
• Missing details. A few author lists (Kazemitabaar, Paradis, Ziegler) and pages for conference papers came from memory or index records. Check them against the DOIs.
• Borg et al. It is cited only as a published study protocol. One search summary claimed it had results, but the arXiv record shows only the Stage 1 design.
• Swaps if you want only peer-reviewed sources. You could replace the preprints with Weber et al. (2024, PACMHCI EICS) or the TOSEM systematic review of LLM-assistant productivity (https://dl.acm.org/doi/10.1145/3809494). I found both but couldn't open their full text to confirm their findings, so I left them out.

Sources

• Paradis et al.: https://arxiv.org/abs/2410.12944
• Becker et al. (METR): https://arxiv.org/abs/2507.09089 · https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
• Shen & Tamkin: https://arxiv.org/abs/2601.20245
• Cui et al.: https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535 · https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566
• Prather et al.: https://dl.acm.org/doi/10.1145/3632620.3671116
• Fu et al.: https://dl.acm.org/doi/10.1145/3716848
• Xu et al.: https://dl.acm.org/doi/10.1145/3487569
• Perry et al.: https://dl.acm.org/doi/10.1145/3576915.3623157
• Kazemitabaar et al.: https://dl.acm.org/doi/full/10.1145/3544548.3580919
• Yetiştiren et al.: https://arxiv.org/abs/2304.10778
• He et al.: https://arxiv.org/abs/2511.04427v2
• Sandoval et al.: https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval
• Mozannar et al.: https://dl.acm.org/doi/abs/10.1145/3613904.3641936
• Ziegler et al.: https://dl.acm.org/doi/10.1145/3633453
• Liang et al.: https://conf.researchr.org/details/icse-2024/icse-2024-research-track/3/A-Large-Scale-Survey-on-the-Usability-of-AI-Programming-Assistants-Successes-and-Cha
• Barke et al.: https://dl.acm.org/doi/10.1145/3586030
• Moradi Dakhel et al.: https://www.sciencedirect.com/science/article/abs/pii/S0164121223001292
• Peng et al.: https://arxiv.org/abs/2302.06590
• Pearce et al.: https://ieeexplore.ieee.org/document/9833571
• Vaithilingam et al.: https://dl.acm.org/doi/10.1145/3491101.3519665
• Hannay et al.: https://www.sciencedirect.com/science/article/abs/pii/S0950584909000123
• Williams et al.: https://dl.acm.org/doi/10.1109/52.854064
• Nguyen & Nadi: https://dl.acm.org/doi/10.1145/3524842.3528470
• Borg et al.: https://arxiv.org/abs/2408.10758
• Hindle et al.: https://softwareprocess.es/homepage/papers/2012-hindle12012icse/