X
Group Project / UX Research / Sponsored
* Based on company confidentiality, sponsors requested that the project be more targeted at the entire U.S. technology industry.
Amazon Web Services Onboarding System with AI for New Software Engineers
The challenges of implementing high-quality onboarding programs are significant, particularly due to the stress and burnout experienced by HR management leaders. Many organizations struggle with the resources and time required to develop effective onboarding, which can exacerbate HR workload and negate the benefits of such programs. Despite these difficulties, the importance of well-structured onboarding cannot be understated, as it impacts new hires, their teams, and the entire organization. Amazon Web Services (AWS) has recognized these challenges and is seeking innovative solutions, including the potential use of generative AI, to enhance the onboarding process for new software engineers (SWEs).
How Might WE...
better support SWEs/developers onboard to a new team by leveraging generative AI as a core component?
Target Audience
Introduction
Context of use
-
Onboarding processes, including tasks, access, and meetings, in both in-person and remote settings.
-
Integration with existing development environments and relevant tools for software development.
-
Assistance with work tasks such as troubleshooting, accessing resources, documentation, and code generation.
Behavior of interest
-
Onboarding efficiency: How quickly SWEs adapt to new tools and processes during onboarding.
-
Engagement with AI systems: How SWEs use AI tools and their comfort with them in daily tasks.
-
Tools and systems used: Key development tools and platforms SWEs rely on.
-
Troubleshooting and problem-solving: How SWEs resolve technical issues, whether through documentation, peer support, or tools.
Business or market needs
-
The tech industry’s talent competition needs efficient onboarding to increase productivity and boost retention.
-
Scalable solutions are needed to efficiently onboard large groups of new graduates.
-
Generative AI offers personalized, real-time onboarding support that traditional methods struggle to scale.
Industry current status
The overall onboarding process has historically been viewed as the responsibility of HR professionals, and lack of personalization and commitment to the individual employee is a common issue.
“During the rest of the new hire’s first year–a make-or-break period in an employee’s tenure–firms leave it to overwhelmed and under-prepared hiring managers to address informally the critical needs that all new hires face."
(Stein and Christiansen, 2010, p. 2)
Primary Research Methods
1. Survey with stakeholders
Information Goals:
Understand stakeholders' perspective on experience with onboarding, and satisfaction with current processes to identify areas where AI could improve efficiency. Additionally, understand common onboarding challenges and methods, and the potential for AI adoption while gauging preferences for tool features and integration.
Why did we choose that method?
Surveys are efficient means of collecting large amounts of quantitative data which we can use for pattern analysis. Survey formats also allow us to more easily engage with our target stakeholders.
Method Details:
We asked stakeholders about their attitudes about onboarding SWEs, satisfaction with current processes, and common challenges. We also asked about preferences for AI adoption, familiarity with Gen AI, and how they would rank the value of specific Gen AI-based features.
Insights
-
Most companies never or rarely update onboarding process, and new employees are still using outdated onboarding processes or documents
-
New employees have difficulty integrating into the existing team for some reason
-
New SWEs face onboarding challenges with unfamiliar internal tools and new technologies
Structured Interview with 2 target users & Semi-Structured Interview with 6 target users
2.
Information Goals:
2 structured pre-interviews are for narrowing down the user base and getting familiar with SWE work content and environment.
Semi- Structure Interviews are for exploring and deeply understanding the process and problems of new SWE employees during onboarding. Discover the challenges they are facing their comfort level with AI autonomy and the impact AI can have on making the onboarding process more efficient and personal.
Core Interview Questions
Affinity Map（Data Synthesis）
Why did we choose that method?
Interviews provide strong qualitative data that can help support or connect to the quantitative data from surveys and offer a different medium/perspective of qualitative information than diary studies. It can go deeper into participant thinking than surveys can.
Method Details:
Asked SWEs, SDEs, ML engineers/scientists, and an SDE manager about current onboarding processes, their biggest issues with onboarding, their opinions/use cases of generative AI, and how they believed AI could fit into the onboarding process.
Interviewees Profile
-
U1: Apple ML Intern (Pre-interview)
-
U2: Workday New Grad (Pre-interview)
-
U3: Technical Lead at Hubspot
-
U4: ML Engineer at Apple (was Google)
-
U5: Software Engineer II at Amazon
-
U6: ML Engineer at Apple
-
U7: SDE at Microsoft (was Amazon)
-
U8: SDE at AWS
Insights
-
SWEs feel existing solutions did not cater to their existing skill set and background
-
New SWEs often need to reach out to other team members or external groups for assistance
-
SWEs value hands-on learning and work-relevant topics in their onboarding programs
-
Some SWEs are concerned with AI performance monitoring
3. Diary Study with 1 target user
Information Goals: Understand participants' daily work tasks and challenges they encounter deeply with the actual example (none-confidential), and assess how they attempt to solve these issues. We also looked for insights into problem-solving strategies, satisfaction with solutions, time spent addressing challenges, and areas of inefficiency or frustration in their workflow.
Section Insights
-
Documentation is a key factor in the onboarding and troubleshooting process, but it is often outdated, irrelevant, or scattered across multiple platforms.
-
SWE needs to deal with some trivial and repetitive yet unavoidable tasks at work.
Why did we choose that method?
Effective for capturing accurate real-time reflections on participants' experiences, challenges, and workflows, providing in-depth qualitative data that reveals insights into their daily tasks.
Method Detail:
Duration: 1 week
Profile: New Grad SWE at Workday
Created a brief Qualtrics questionnaire that the participant would answer at the end of each workday for one week.
Challenges participant meet
Difficulty navigating complex internal systems
Limited support from AI for context-specific issues.
Where did participant spend onboarding time?
Opportunities for Improvement
Streamlining access to documentation and resources
Enhancing AI to handle company-specific queries
Secondary Methods
Website Walkthrough
Understand how these platforms handle onboarding, user assistance, and overall design to identify strengths and gaps that could inform improvements for our own system.
Competitive Analysis
Compare onboarding platforms to identify market positions and uncover strategic opportunities for improvement, focusing on generative AI.
SWOT Analysis of Amazon Q Developer
Measure AWS's existing AI systems from the perspectives of strength, weakness, opportunity, and threat. Try to find the user needs that Amazon Q does not cover yet, and think about how to differentiate our system. Introduce by sponsors.
Case Study – Agentic AI
Understand how AI systems can act as autonomous agents, taking initiative and making decisions within predefined parameters, especially in complex environments like onboarding and task automation. Recommended by sponsors.
Task Analysis
Streamline the onboarding process, align training with role-specific needs, and explore automation opportunities to reduce manual effort.
Key of Journey Map
Action
Learning to work in a new team and getting to know colleagues
Adapting to a new code environment
Trying to search how to use the tools and the technique problem
The First Week
After 1 month
After 3 month
The Second Week
　Frist day at work
Painpoint
Overwhelming to reach out to multiple people
Unfamiliarity with new tools and internal technologies/ language
Finding the right guides or information are tough since everything is scattered
Design Details
User Needs & Design Implications
User Needs
Design Implication
Concept Generation Process
Prioritization Martix: Prioritize different user needs
Information Architecture:
The concepts classification & Hierarchy
Crazy 8: Collect 24 ideas
Brainstorm: Select 10 ideas from 24 of them
SCAMPER:
Eliminate 2 uncomprehensive thoughts
Final Concepts
Concept Sketchs
Dashboard - to do
Dashboard with progress tracker
Documentation Linker & AI Chatbot
SME Outreach
Lofi Prototype
Final Prototype Explanation
Neoboard
AI-Driven Onboarding System: Quick and Easy Adaptation for New Software Engineers
Onboarding
Before users land on the dashboard, we offer a personalization opportunity to customize the features they would like to see in their dashboard.
Dashboard
Users can quickly see the most recent tasks that need to be completed, the specific progress of a certain task, and quick reading pages that can help users access them.
Documentation Liner
Offer other related documents as optional reading
After toggle the "Linker View" option, which guides users to read related documents in sequence.
it will pick up information from related documents to provide them with an AI-generated summary
Task List
Based on the feedback from SWEs who have joined the company in the past, Gen AI will provide guides for fixed onboarding tasks to help them adapt to their new jobs more easily and quickly.
Some tasks could generated by AI, so we cannot guarantee 100% accuracy. The user's leader will leave a message on it if there is anything that needs attention.
Generally, in the early stage of employment, people will not be assigned many formal tasks, but many people are full of enthusiasm. It will recommend tasks to new SWEs by Gen AI to improve themselves or manually add tasks.
Task Progress
Sometimes, employees have to multitask, and all their task processes connected to the system will be automatically recorded.
Neo Report:
Generate reports and give suggestions based on the time spent on completed tasks, knowledge gaps, and common errors. As the number of tasks completed by the user increases, the AI database will gain a better understanding of the user's work quality and level.
Summary of AI-generated recommendations, Focus on the entire task rather than each step.
AI Chatbot
If the answer is not good enough, the chatbot also maps relevant Experts or colleagues that the user can reach out for helping.
The user can ask any doubts or queries to the AI chatbot.
We will collect all the questions from users into the database to facilitate our continuous updating
Usability Testing
User Feedback Survey
Participants
4 SWEs and 2 MS-CS students with prior experience as SWEs
Satisfaction of Features
The majority of users agreed (40%) or strongly agreed (40%) with the satisfaction of features
Interface Usability
62.5% of users agreed, with 25% strongly agreeing on usability
Meeting User Expectations
Most users agreed (55.56%) or strongly agreed (27.78%) that the product met expectations
Expert Cognitive Walkthrough & Heuristic Evaluation
Participants
1 ML Engineer and 2 MS-HCI students with UX expertise
Cognitive Walkthrough
-
Task 1: Personalize your onboarding dashboard
-
Task 2: Set up each feature
-
Subtasks for each feature
-
Heuristic Evaluation
-
Following Nielsen’s 10 Usability Heuristics for evaluation criteria
-
Tracking experts’ ratings, issues, and recommendations
Usability and accessibility concerns
Overall size of UI elements
Discrepancy between user perception and intention of icon
Overall Findings
-
More integration between features
-
More instruction/guidance in setup
-
Consistency in visual elements and information​​
​
From User Research to Evaluation Findings
-
Personalization and customization: more intuitive process
-
Quickly find relevant information: desire for more user control
-
Track work progress: more visualization
-
Facilitate communication and collaboration: expanded functionality
