32 lines
6.5 KiB
Markdown
32 lines
6.5 KiB
Markdown
type:: [[REVIEWS]]
|
||
tags::
|
||
year:: 2024
|
||
venue:: [[TOSEM]]
|
||
full-title:: Rethinking Software Engineering in the Foundation Model Era: From Task-Driven AI Copilots to Goal-Driven AI Pair Programmers
|
||
date-start:: [[25-04-2024]] - 19:01
|
||
date-submitted::
|
||
external-links:: [https://mc.manuscriptcentral.com/tosem?URL_MASK=92d33ad6901845ad99b2a200e3e90ab6](https://mc.manuscriptcentral.com/tosem?URL_MASK=92d33ad6901845ad99b2a200e3e90ab6)
|
||
status:: [[DONE]]
|
||
deadline-submission:: [[26-04-2024]]
|
||
file:: [[@Rethinking Software Engineering in the Foundation Model Era: From Task-Driven AI Copilots to Goal-Driven AI Pair Programmers]]
|
||
parent::
|
||
todoist:: https://app.todoist.com/app/task/tosem-2024-0255-reviewer-agreed-7913839489
|
||
|
||
- ### [[Highlights]]
|
||
-
|
||
- ### [[Comments]]
|
||
- **Summary**: The paper discusses the envisioned transformation of software engineering by integrating AI-based pair programmers, shifting from task-driven to goal-driven approaches. The discussed AI pair programmers are envisioned as collaborative, context-aware partners to human developers, aiming to enhance productivity and software quality. The paper highlights the limitations of current AI copilots that primarily offer code completion, advocating for a more holistic approach where AI acts as both a coder and a mentor. The paper also introduces relevant challenges, i.e., improving human-AI interaction, developing technologies for automatic prompt generation starting from human inquiry, enhancing the efficiency of AI models for code, and expanding the mentoring capabilities of AI.
|
||
- **Comments**: The paper presents interesting discussion points concerning the shortcomings of current AI copilot technologies. It advocates for a transition from task-driven to goal-driven AI pair programming. However, the identified challenges are not directly tied to the envisioned paradigm shift. For a more coherent integration, the concept of a 'goal' in software engineering needs a precise definition and should be contextualized within the various stages of SE processes, such as requirement elicitation, software development, deployment, and testing. In other words, the four critical challenges should be better linked to the foundational concept of goal-driven support.
|
||
-
|
||
- ### [[REVIEWS/Notes]]
|
||
- The advent of Foundation Models (FMs) and AI-powered copilots has transformed the landscape of software development, offering unprecedented code completion capabilities and enhancing developer productivity. However, the current task-driven nature of these copilots falls short in addressing the broader goals and complexities inherent in software engineering (SE). In this paper, we propose a paradigm shift towards goal-driven AI-powered pair programmers that collaborate with human developers in a more holistic and context-aware manner. We envision AI pair programmers that are goal-driven, human partners, SE-aware, and self-learning. These AI partners engage in iterative, conversation-driven development processes, aligning closely with human goals and facilitating informed decision-making. We discuss the desired attributes of such AI pair programmers and outline key challenges that must be addressed to realize this vision. Ultimately, our work represents a shift from AI-augmented SE to AI-transformed SE by replacing code completion with a collaborative partnership between humans and AI that enhances both productivity and software quality.
|
||
- While code completion is definitely useful in the acceleration of mundane tasks and exploration of new ideas [3], maintaining a large legacy system cannot be done on the basis of adding more and more code. Bootstrapping developers’ productivity while simultaneously creating high-quality software requires a more shift from AI-augmented SE to AI-transformed SE. In this paper, we propose the concept of an AI pair programmer that is goal-driven, a human partner (not replacer), SE-aware, and a self-learner. We also discuss key challenges that must be overcome to realize our vision of AI pair programmers. Fully tackling those challenges requires the design of novel technologies and techniques that touch on several knowledge areas (e.g., interaction design, software engineering, cognition, telemetry, and multi-agent collaboration). The list of challenges discussed in this paper is nonexhaustive and will be expanded in the future.
|
||
- **Challenge 1: Speeding up human-AI alignment**
|
||
- First, an AI pair programmer must be have excellent conversation skills (e.g., by using more powerful FMs) that can help humans refine their goals. Next, the AI must also ask humans for clarification when needed. Yet, finding the balance between asking too many clarifying questions and not asking enough is extremely hard. To mitigate this problem, AI pair programmers should have the ability to develop a theory of the mind of the human with whom they are interacting
|
||
- **Challenge 2: From prompts to natural inquiries**
|
||
- In our vision, the burden of crafting an effective prompt should be on the AI (instead of humans). Although techniques such as Automatic Prompt Engineer [53], PromptBreeder [19] and DSPy [32] exist, they still require manual intervention (setup and/or programming). There is a need for a prompt transpiler technology that seamlessly takes a human inquiry and transforms it into an optimal prompt for a given model.
|
||
- **Challenge 3: Cheaper and smarter code models**
|
||
- AI Pair Programmers should leverage Large Language Models for code (a.k.a., code LLMs). These models are specifically trained with source code, aiming to better capture code semantics compared to generalist LLMs. Code LLMs can be seen as contextualized models, in which special focus is given to the training data. The recent work of Lozhkov et al. [35] on StarCoder v2 show that curating the training data (e.g., selecting high quality sources, adhering to licenses) and applying careful preprocessing (e.g., ordering source code files per project and using an LLVM representation) can generate significantly smaller models with a performance that rivals that of much bigger models
|
||
- **Challenge 4: Leveraging mentoring potential**
|
||
- In our vision, an AI pair programmer should not only code like a senior human engineer (or even better!), but also mentor like one. While research effort has focused on the former, we believe that the latter is equally important. After all, great software engineers are much more than good coders. More specifically, we believe that AI pair programmers should behave like emphatic team players that care about the technical growth and development of their peers
|
||
- |