Leveraging AI in User Research

Grand Studio has a long history of working with various AI technologies and tools (including a chatbot for the underbanked and using AI to help scale the quick-service restaurant industry). We’ve created our own Human-Centered AI Framework to guide our work and our clients to design a future that is AI-powered and human-led and that builds on human knowledge and skills to make organizations run better and unlock greater capabilities for people. When ChatGPT hit the scene, we started experimenting right away with how it could improve our processes and make our work both more efficient and more robust. 

Given our experience with what AI is good at doing (and what it’s not), we knew we could use ChatGPT to help us distill and synthesize a large amount of qualitative data in a recent large-scale discovery and ideation project for a global client. 

Here are some takeaways for teams hoping to do something similar: 

1. Don’t skip the clean-up. As they say: garbage in, garbage out. Generative AI (GenAI) tools can only make sense of what you give them – they can’t necessarily decipher acronyms, shorthand, typos, or other research input errors. Spend the time to clean up your data and your algorithmic synthesis buddy will thank you. This can also include standardized formats, so if you think you may want to go this route, consider how you can standardize note-taking in your upfront research prep.

2. Protect your – and your client’s – data. While ChatGPT doesn’t currently claim any ownership or copyright over the information you put in, it will train on your data unless you make a specific privacy request . If you’re working with sensitive or private company data, do your due diligence and make sure you’ve cleaned up important or easily identifiable data first. Data safety should always be your top priority.

3. Be specific with what you need to know. ChatGPT can only do so much. If you don’t know what your research goals are, ChatGPT isn’t going to be a silver bullet that uncovers the secrets of your data for you. In our experience, it works best with specific prompts that give it clear guidelines and output parameters. For example, you can ask something like: 

“Please synthesize the following data and create three takeaways that surface what users thought of these ideas in plain language. Use only the data set provided to create your answers. Highlight the most important things users thought regarding what they liked and didn’t like, and why. Please return your response as a bulleted list, with one bullet for each key takeaway, with sub-bullets underneath those for what they liked and didn’t like, and why.” 

Doing the upfront human-researcher work of creating high quality research plans will help you focus on the important questions at this stage.

4. It’s true, ChatGPT gets tired. As with any new technology, ChatGPT is always changing. That being said,  the 4.0 version of ChatGP that we worked with demonstrated diminishing returns the longer we used it. Even though the prompts were exactly the same from question to question, with the input of fresh data sources each time, ChatGPT’s answers got shorter and less complete. Prompts asking for three synthesized takeaways would be answered with one or two, with fewer and fewer connections to the data sets. By the end, its answers were straight up wrong. Leading us to our final takeaway:

5. Always do an audit of the answers! Large language models like ChatGPT aren’t able to discern if the answers they provide are accurate or what you were hoping to receive. It’s also incredibly confident when providing its answers, even if they’re wrong. This means you can’t blindly rely on it to give you an accurate answer. You have to go back and sift through the original data and make sure that the answers it gives you line up with what you, the researcher, also see. Unfortunately this means the process will take longer than you were probably hoping for, but the alternative is incomplete, or incorrect answers – which defeat the purpose of synthesis in the first place and could cause the client to lose trust in you. 

Outcome: Did using ChatGPT speed up our synthesis significantly? Absolutely. Could we fully rely on ChatGPT’s synthesis output without any sort of audit or gut check? Not at all. We’ll keep experimenting with ways to incorporate emerging technologies like Generative AI into our workstreams, but always with research integrity and humans at our center. 

Interested in how GenAI might work for your organization? Drop us a line – we’d love to chat!

A New Way of Understanding Sports Fans

A lot of sports organizations think about their fan base in terms of subscribership tiers. Their business strategy is largely about moving fans up those tiers, converting them to higher levels of monetization. Accordingly, they ask themselves questions like: what would it take for a fan to upgrade to a season ticket holder, an ESPN+ subscriber, or a daily reader of sports news? 

This approach makes sense. After all, a company is in the business of monetization. But to get a fan to upgrade, they must first and foremost be engaged with whatever it is you’re offering — be that a product, a team, or the game itself. To bring them up the tiers is essentially to ask them to increase their level of engagement with you. And if you want fans to engage deeply, you have to deeply understand what it is that they want.  

In other words, the better you can understand the crux of a fan’s engagement — how it is shaped, how it’s maintained, and how it grows (or stagnates…) — the better you can cultivate their inspiration to upgrade. Getting to this level of a fan identity requires an intimacy with their beliefs that goes beyond the details on an account subscription. 

A fan-centric view of sports

As mentioned, while there is clear value in analyzing subscription trends, it is inherently top-down and corporation-centric, placing fan behaviors primarily in relation to their monetary value for the company. If this is the key variable by which segments are sliced and diced, it can limit the organization’s ability to surface the most meaningful characteristics and variations that define their fan base. And, subsequently, limit the organization’s ability to serve such needs, and get the very upgrades they are after.

What many sports companies could use is a complementary bottom-up approach to segmenting and analyzing fans. This approach would start by understanding how and why a fan engages in a sport. Is it all about supporting a particular team? Is it a larger appreciation for the sport? Is it about the culture? Belonging? Nostalgia? Hometown pride? Is it about going to games because all their friends do? Starting on the ground to understand attitudinal and behavioral differences across fans can set organizations up to learn something deeper and more important about that fan than their subscribership status — things that ultimately do more to determine how they can serve each segment. 

Stories from the stadium 

In a previous project, a major sports league asked us to overhaul their mobile app. Their goal was to get fans to spend more time on the app so they could generate more ad revenue. When we started doing research on their fan base, we uncovered surprising trends that ended up influencing the league’s overall engagement strategies. For example, there was a significant portion of the fan base who were what we called “adopted fans.” Instead of inheriting a team from traditional family ties, they adopted a new team when they moved to a new state, or adopted one based on its underdog status. As newer fans, they looked to national news for sports intel. Diehard fans, on the other hand, primarily went straight to their local beat reporters for sports news. This stratification uncovered opportunities for the league to serve each group differently and personalize their experience on the app, increasing the engagement opportunity for each group. 

In another project with a major sports team, the avidity level of a fan turned out to be among the most important characteristics to analyze. We uncovered, for instance, a segment of the population we called “tag-alongs” — those who attended a sporting event because someone had invited them. Many of these tag-alongs didn’t know much about the sport to begin with, but loved the experience of going to a live game and rooting for the team. For this sub-group, the atmosphere and amenities at the stadium made a big impact on their likelihood to return. Once this group was uncovered, the team was able to do more to convert these tag-alongs into fans in their own right. 

Doing the “field” work: meeting fans where they are

Let’s assume your organization has bought into the value of uncovering the unique fan archetypes within their population. What comes next? 

One important way to research fan attitudes is, of course, going to games. Observing fans interacting with their sport or team, and also observing them in community with one another at games, is not to be overlooked. 

But fans are not only fans during sports games. They are also fans when they are reading the news, keeping up with players or stats or the league at large. They are fans when they’re out at a bar with friends and see their favorite player’s jersey on the wall. They are fans during off-season as well, even when there aren’t as many ways to show it. 

Understanding a fan means understanding the rhythm of their fandom, the ebbs and flows in addition to the moments of peak excitement and engagement. How they stay connected to their team or sport when games aren’t going on can be just as informative as how they behave during a game. There are cadences to the experiences of different sports fans, and understanding that richness of detail is key to understanding how their needs can best be met. We’ve found that understanding these harder-to-capture aspects of fandom require different research methodologies — for instance, perhaps you need diary studies to check in on fans during off-season or lulls in action. Perhaps you need to post up in a sports bar and catch people stretching out the emotion of a game by connecting over it. Perhaps you need data points from people as they read sports news throughout the week. 

Sports mean a lot to people. For some, their fandom is a key piece of how they see their own identities. Taking the time to understand these segments with multidimensional attributes with care can pay off greatly for fan satisfaction as well as overall engagement metrics.

Looking to better understand and serve your fans? We’d love to hear from you!