Towards Providing On-Demand Expert Support for Software Developers

– Alright, hey everyone,
thanks for coming. Today I’m gonna talk about
this project where we propose a new space of support systems that provides expert support
for software developers. And used two studies to
motivate this new space and we then informed the
system designs in the future. So why we need an on-demand
programming support? We defined on-demand
programming support as providing answers to questions about code when they are needed. This questions may be things
like how to use an API library or how to structure program and so on. And on demand programming
support is great because a third party resources
can provide answers that help programmers to write better code. Let’s first take a look at the
current programming support. So with the current support
resources, developer will use built-in IDE support
to save the effort of remembering all the syntax details. They will look for code
snippets, examples, from online community support
such as Stack Overflow or they’ll check other
developers code repositories to find examples with tools like Github. Or they find online tutorials
from sites like W3school. Although they can provide
support like this, these resources have a
variety of limitations. For example, the results
from the web are generally lack of contextual information which makes the integration process
inefficient for developers. Or the results are often not personalized which requires the
developers to have high level of expertise to modify them. Additionally, the query often
needs to contain sufficient context so that the results
will be more accurate. This is difficult and time
consuming for developers. For many limitations like
this, the best support resource is other developers
who have the expertise in the domain. Finding a colleague who is
familiar with the context and nearby or hiring
freelancers with the expertise and available from online
platforms like Opwork can achieve this goal. But colleagues are not
always available on demand or not to be able to use
support frequent questions. Also, hiring reliable
workers who have sufficient knowledge of a project
usually takes very long time and requires certain level
of evaluation skills. Recognizing these issues,
within past few years, companies start offering online one-on-one mentoring service such as
Hackhands or Codemantra. These services allow the
clients request developers to create requests and
connect them with the experts. The services also provide
a shared code editor and text or voice communication channel. These services represent
the state of the art, the state of the art
for seeking remote help from expert programmers
and use synchronous one-on-one communication model. Although the mentorship model
has solved most of issues we mentioned before from online resources, it also has a few limitations. First, there is a
coordination cost of finding an expert who is available to help. If the first expert does not
have the sufficient expertise, which they cannot know until they connect, there’s a further cost
in both time and money to find a new expert. One-on-one mentoring also
requires developers who watch what the other person is
doing for the whole time which prevents them from
moving onto other tasks while connecting. In addition, this model is
impractical during a long programming session as the developer has to pay the expert to remain
standby between questions. Although it has all the
limitations, the mentor service is a great model and we
wanna find more about it. To better understand how well
this model of mentor support can help programmers, we
conducted in-lab studies to simulate this one-on-one
support synchronous model. So we look at two research questions. Which questions do developers want to ask an on-demand support agent? And what are the design
needs of providing remote developer support on-demand? To better understand these two
aspects of the design space will allow us better
reason about the challenges that we need to be overcome
and create fast usable and efficient assistant tools
for software developers. Now I’m gonna talk about
the two studies that we conducted in details to
answer these two questions. Because the type of queries
developers will make are influenced by the tools that they use, in the first study we
wanted better understand the type of questions
that a software developer would want to ask an intelligent system in the best case scenario,
if there’s no limitations on his time, compensation,
and the capability. We asked five participant to
work on their own project, programming project,
on their computer while asked them to imagine that
they have an intelligent human assistant nearby who
is capable of answering any questions that they verbally express. Participants could use
their hypothetical assistant to answer questions or perform a variety of types of work. The hypothetical assistant
serve as the conceptual prop that participants could make requests to. And what we found is that
the high level of strategic guidance, shows in this
rectangle, is the type of request that participants
asked for the most. A typical request is like
wondering if there’s a way to ensure they understand
if the radio button in HTML can be color coded. This type of request needs
to have personalized support which cannot be provided from
web but other developers. We also observed that
participants made some requests that can be easily find
and answered on the web. What surprised us is that four out of five participants asked for effort
saving type of request. For example, one participant
asked the assistant to do some unitasks for him. This type of request is
beyond what we usually understand as a question. And our study shows that
this is a popular need for developers to
program more efficiently. However, this type of request is not cannot be automated. Additionally, we found that
participants often phrase questions that requires
contextual information of their code base to understand. Only 18% of participants
questions were self contained meaning they do not need
any context information to be answered. And lastly, every participant
indicated that the timely responses would be
crucial to their adoption of such a system. So now we know the type
of request that developers would need and then we
follow that up with another study to see how well this
model of mentor support can help programmers. In this study we explore
the communication needs and breakdowns between
experts and the developers. This time we’re focusing on
the communication challenges such as uploading
challenges, contextual needs, and interaction difficulties. And here we setup a study in this way. We paired up 24 participants
with half being requesters and the other half being helpers. They were assigned their
roles based on the scores from the pre survey. Each pair was physically
separated and the requesters were given programming tasks and then they asked for support from helpers via Skype which are connected for the session. During the session, requester
sshared their screen with their helper and
they were asked to program in JSBin, an online shared editor, to save configuration after. To better simulate the
model of mentor support, we told helpers to be
reactive, meaning they only respond to queries and do not proactively propose solutions and approaches. This allows users to
initiate help interaction just like the mentor service support does. And from our observation
and interviews we found that there are six major needs
that requesters care about. The red rectangle shows
that the requesters from every session needs immediate response which means the requesters
want to communicate with the helpers without
noticeable delays. One interesting finding
that we were not expected before was that almost all
the helpers were interested in knowing the developers
coding background. For example, a helper said: if I knew where she was
coming from, it made it more efficient because I
won’t be asked you know, what this is, how do I make this thing? Another interesting
finding is that although we provide a state of the art
tools to support developers, we found evidence of
inefficient communication from 11 out of 12 sessions. Here’s a common conversation. A helper said just bring the
cursor after the curly brace and the requester say,
after the curly brace here? And helper say yeah and
then put a comma and put a string call JSON and the quotes. While the requester’s
typing, the helper is saying no no, the quotes. Another requester said,
the code she send me can run on her side but I’m not
able to run it on my side. It’d be great to have a, if she can share it in some
way like do a comparison of the output that would be great. After finding all those
needs from these studies we took one, we took the
first step in this space by prototyping some features and add, to address needs. We first asked participants
what they wish they could have to address them. For example, a helper
said, I wish I could have something like Google Doc for code. So after we find all
those needs and feedback, we want to prototype, we want to propose a different model of support systems. From the first study we
found that many types of requests cannot be solved
without a human support. So we suggest that you use human power to provide help for programmers. Also we noticed that the
importance of context. As we mentioned, 82% of the
requests are not self-contained. We suggest that this
model will automatically capture all the context
such as the code reference, the developer’s background,
and pack all those and send them over to the helpers. To instantiate this
space, we prototyped six system features as listed here. And I’m gonna focusing
on the text and the voice feature for now. To make the request making more efficient, we also provide, we
prototyped this feature that enables both text and
voice modality for requesters to verbally describe their
requests in the code editor. Plus they can select
their content as well. And um, oh sorry, and then we, we went back to 11 of our 24 participants and asked their feedback
and we want them to, we want them to, to have one more run of design iteration. We went back to 11 participants and showed them this feature and
get their feedback on how it fits their use cases. We found that some types of request text is more effective while
for others voice is. For example, one participant said, I prefer text because a
helper can send me example but text will be very
slow if I want to explain what I’m doing. Voice can do that
interaction very quickly. As an ongoing work, we
are working on the system to instantiate the rest of the features. To recap what we did, we
conducted a study to find out what questions developers would ask an on-demand support agent
when they are programming and we found seven types. We then conducted a remote
pair programming study to better understand how
well the model of mentor support can help programmers and we found six design needs in that particular space. Lastly, we draw design
takeaways and recommend and prototype features
for systems that aim to achieve the goal of providing
on-demand programming support that we mention at
the beginning of the talk. All these pieces motivate us to inform a broad space of support
systems and present it a first glimpse into the
challenges and opportunities of on-demand expert support for a software developer in text. Additionally, we are
planning to explore more on this new space from other perspectives. We saw constantly that
the pair has the same kind of interaction which would
be great if we could automate some of those process. And we also planned to
allow developers to better control their information
that it will not share with the helpers in order to
reduce the privacy concerns. Another interesting open
questions is that how to best motivate and compensate
experts for their help. And we look at different
cost structures for on-demand support for our
real deployment instead of in-lab studies which
might change the way that participants view the cost. And thanks for listening,
happy to take questions. (clapping) – [Programmer] Hi, I’m
Konchan from Fujisa Lab. So why your strategy of
class change programs ask for on-demand help, did
you analyze the profiles of the developers, like
their background, expertise? My question is, different people might have different questions. – Right, yeah, so you’re talking about the first study, right,
just asking hypothetical. We did collect their
background but we didn’t actually try to focusing on
analyzing those background and they’re coming from,
actually using different, even different programming language. So, but it’s interesting
question and it’s definitely programmers have dynamic background. And what we are looking
for is that this type of system will support
any type of background. Any programmers will have
different background. – [Staff] Are there any further questions? So, I actually had one. The whole time I was
thinking crowd sourcing, crowd surfing, expert crowd sourcing. You briefly mentioned in
the end, can you speak in a little bit more detail about what you’re thinking on that front? – Uh, this space or–
– In this space, yeah. – Well, so, we are thinking that crowd, crowd sourcing, we can use, basically we are proposing
to use expert crowd to support end users and So– – [Staff] What type of
person would do this? – Right, so, as we mentioned before, a couple of ways we are trying
to design the system for or like, either team based,
we can outsource questions to other colleagues or
we can outsource this question to freelancers but yeah, the crowdsourcing support here is not, not in particular, like, platform, more like general
many-to-one type of support.

Leave a Reply

Your email address will not be published. Required fields are marked *