Why do we remember what we remember?

Part of the Memory Issue of The Highlight, our home for ambitious stories that explain our world.

Wilma Bainbridge, a cognitive neuroscientist at the University of Chicago, recently showed me eight images and asked me to guess: Which ones will I remember? For each pair, she hinted, one image would be more likely to stick in my mind.

It seemed like a trick question. None of the images seemed particularly striking. These were the kinds of mundane photos we may come across every day — two men, two women, a dining room and an empty cubicle in an office, two tropical beaches.

Maybe, I thought, I’d remember the man in the bottom row if I saw him again. He was more classically handsome with swooshy hair, deep-set eyes, and superhero-square jawline, compared to the smirking man in the top row (sorry, dude). I’d choose the dining room with the tree over the sad office cubicle. That’s a place I’d want to eat a big bowl of pasta and laugh with friends. The women seemed to have equally kind smiles, so why would I remember one more than the other? And the beaches, well, they both seemed like boring postcards.

Courtesy of Wilma Bainbridge

Bainbridge wasn’t asking about my memory, but memorability. On any given day, we’re bombarded with images — in the news, on TV, on Instagram, faces we encounter on the street, scenes we see outside a car window — and some of those images stick around in our brains. Others are just lost.

While memory is what helps us recall things we’ve encountered or thought of before, studies of memorability ask: Why do we remember what we remember? Why do we forget what we forget?

Memory still looms as a big mystery in science. No scientist is perfectly sure how the brain physically sorts and stores all the information — and all the types of information — that gets encoded into memories. But cognitive psychologists hope that asking what we remember will start to teach them how we remember. And with a better understanding of how, scientists can perhaps come up with ways to fix lapses in memory.

What’s puzzling, and a bit provocative, is that artificial intelligence is getting remarkably good at predicting which images the human brain is going to remember, even outperforming our own human intuitions. Which is making scientists wonder: Can they help engineer more memorable images — for classrooms, for maps, for the memory-impaired? Can they help design a more memorable world? So much our minds encounter eventually slips away. Maybe we can control what sticks.

In psychology, memory is a huge topic. It’s one word, but it stands for a lot of different things our brains can do. You might use working memory for holding a few digits in your head while you go to unlock a keypad; episodic memory to recall a school trip to the state fair from when you were 10; sensory memory to conjure up the smells of the funnel cakes there. You can remember historical facts like when the state fair was founded; you can remember the directions for how to get to the fairgrounds.

Exactly how all of these sources of memory work together, and exactly how they are different and the same, is a source of endless scholarship in psychology. No experiment can capture the whole of our human experience with memory and explain every instance of it. Instead, in labs, researchers can really only study it in smaller slices, and then try to figure out what it all means in the bigger picture.

The smaller slices can lead to some fascinating conclusions. In the early 2010s, researchers at the Massachusetts Institute of Technology started probing memorability by asking participants to play an experimental game. Cognitive scientist Aude Oliva and her colleagues showed study participants a series of images like the eight photos I was given, and then quizzed them later on which ones they recognized. If a participant recognized a photo they’d actually seen before, it counted as being remembered.

Previous research shows humans are really good at this sort of recognition game. We’re capable of recognizing tens of thousands of images we’ve seen before. But we’re not perfect at it. Some images stick vividly in our minds, while others may fade away even when we actively try to remember them.

The MIT memory game studies asked whether some images are inherently more likely to be remembered than others. The answer was yes. In study after study, researchers have found that some images leave a much more lasting impression. The findings hold true across different categories of images: Some faces are more memorable than others, some scenes are more memorable than others, and even some random noise images — scrambled, unrecognizable fields of light and color — are more memorable than others.

Whatever is influencing the memorability of images, it’s not something we’re consciously aware of. Some studies have asked participants to guess which images will be remembered, just as Bainbridge asked me. “Our intuitions are really bad,” Bainbridge says. Participants “perform almost at chance.”

My guesses were as bad as the next person’s. I predicted that I’d remember the handsome man with the swooshy hair, and the dining room with the tree in it. But according to Bainbridge, the images I chose weren’t actually memorable. Thirty percent more people remembered the images I didn’t choose.

“It’s crazy, because the indoor dining room with the tree is definitely more interesting and beautiful, and the two beaches look almost indistinguishable,” says Bainbridge, who started in memorability research as a PhD student in Oliva’s lab. “But something about the top images makes them better remembered by people.” (This is true regardless of what order the images are presented in.)

“It’s still a mystery,” Bainbridge says.

Phillip Isola ,Jianxiong Xiao, Antonio Torralba, Aude Oliva/CVPR 2011

Memorability turns certain common assumptions about how memory works on their head. “The traditional way of thinking about memory is, it’s about ourselves,” says Chris Baker, a National Institutes of Health neuroscientist who studies visual perception. (Baker has also collaborated with Bainbridge.) I’m outdoorsy and love nature, for example, so perhaps I’d be more likely to remember a gorgeous mountain vista and less likely to remember a boring street corner.

But memorability research considers whether there’s something predetermined about what makes it into our memories. As Baker says: “How much of what you remember is not about ourselves, but about what it is that we’re trying to remember?” If the street corner is inherently more memorable than the mountain vista, then maybe my personal preferences and interests don’t matter that much.

It’s clear from memory research that certain life events are more likely to be remembered than others. “You remember getting married, you remember the college graduation, you remember these kinds of events that we’ve deemed to be important milestones,” says Lisa Fazio, a memory researcher at Vanderbilt University. Certain words are more likely to be remembered, too. “It’s much easier to remember words that you can imagine, versus abstract things,” Fazio explains. “The ‘Liberty Bell’ is easier to remember than ‘liberty’.”

But none of that solves the mysteries of memorability. “What’s new about the research you’re talking about,” Fazio says, “is that they’re dealing with images that don’t have simple explanations why one would be more memorable than another.”

There are a lot of factors that partially explain why an image sticks in a person’s mind. For example, when it comes to faces, you might remember one that you perceive as attractive or one that looks familiar somehow. Yet these attributes, Bainbridge says, only account for about half of what makes an image memorable. “Many people think, ‘Oh, this face is attractive, I’ll remember it.’ That’s not necessarily true,” Bainbridge says.

Images with people in them tend to be more memorable than images of a blank landscape, and we remember brighter pictures with higher contrast better. Novelty — like an image of a mailbox placed in a bedroom — seems to leave a bit of an impression too, Bainbridge says. Still, there’s so much more that isn’t explained.

“We keep thinking that maybe we’ll find a set of attributes that determine what makes something memorable,” Bainbridge says. Those attributes “include things like categories of objects, functions, colors, and texture.” An image can be categorized by how much it seems like an animal or a piece of technology, or if it’s an object like a container, or if it appears to be the color red.

But all of these attributes combined can only account for around 60 percent of what makes an image memorable, Bainbridge says, based on a study (under peer review) on 26,000 images — “basically all objects in human existence,” she says. The remaining 40 percent? “It’s just this mystery.”

Rust, Mehrpour/Trends in Cognitive Sciences

Even more puzzling, Bainbridge has found that asking participants to try to remember or forget a certain face, even when incentivizing them with money, doesn’t really make a difference. “Effort isn’t really able to override this effect,” she says. Participants “still couldn’t forget the memorable ones or remember the forgettable ones, even with the reward.”

Studies on memorability haven’t been perfectly representative of all people around the world, so it’s hard to know if there are some important cultural differences in what people remember. So far, Bainbridge and her colleagues in the field haven’t found big differences between groups of people. They can even assign any given image a “memorability score” based on data derived from one group, and have that score predict how well another group of participants will remember an image.

“The fact that it’s consistent across people must mean that there’s something consistent about our brains,” Baker, the NIH neuroscientist, says.

The mechanics of memorability seem to be so fundamental in our brains that even our fellow primates experience something similar. At the University of Pennsylvania, cognitive neuroscientist Nicole Rust wanted to see if memorability scores generated by the human experiments could predict what images two rhesus monkeys remembered. The monkeys are trained to react if they come across an image they’ve seen before. “The human memorability scores were predictive of monkey performance,” Rust says. That suggests there’s something deep in the brain that transcends culture and even species.

“Memorability might indicate how our brain prioritizes information,” Bainbridge says. “So almost like a sorting algorithm that you might imagine Google uses to search.” In other words, as our eyes scan an image, certain shapes, textures, objects, and attributes are prioritized to be stored in our memories, while others are deprioritized. This seems to happen instantaneously, and it’s something we’re apparently not consciously aware of.

“The brain is able to identify what’s high-priority information,” she suggests. “Maybe this memorability property is helping our brain identify what information is important for long-term memory, because we can’t sort everything at once.”

Even though scientists don’t fully understand why some things are easier to remember, they may have the tools to manipulate memorability. “If memorability is a property of an image, it means it’s something that can be computed from an image,” Bainbridge says. That is, AI can be trained to look at an image and guess how memorable it might be.

It’s an intriguing possibility. Can we make educational slides from a lecture more likely to stick in a student’s brain? Can we redesign maps and streets to help people remember their way? There’s big potential for businesses too: Product designers could use algorithms to create the most memorable logos and ads.

In her lab, Bainbridge and colleagues have created an AI tool called Resmem that can predict how memorable an image is. “It’s based on this data of tens of thousands of people doing a memory test with tens of thousands of images,” she says. You upload an image to Resmem and the AI determines how memorable the image is, spitting out a memorability score. The higher the score, the more likely the image is to be remembered.

I recently ran Resmem on a couple of images I took from a summer vacation at the New River Gorge in West Virginia. A waterfall and natural swimming pool I photographed got a memorability score of 0.512 out of 1. The ruins of an abandoned bridge? A score of 0.485.

Brian Resnick/Vox

Brian Resnick/Vox

Basically, according to the AI, my picture of the waterfall is more likely to be remembered by other people. If I wanted to make an advertisement for West Virginia, maybe I’d choose the waterfall photo.

The applications may seem boring or limited for now: Bainbridge uses Resmem to choose photos for her lab’s website, while her lab members use it to predict the memorability of their PowerPoint presentations. (Other work has shown that even some infographic designs may be more memorable than others.) But she wants to take it further. She hopes to figure out just how engineerable memorability could be, across many areas of our lives.

“Can we generate images that are more memorable and more forgettable? Can we take an image and boost its memorability, or make it more forgettable?” she asks. “It has all these cool applications — like you can imagine for education. You want to make textbooks with pictures that kids are going to remember.”

This research won’t stop at images: “Could we create a similar model that might predict the memorability of a voice, or even a selection of music?” Imagine how much a record label or a streaming platform might pay for an algorithm that tells you whether a song will stick in your head, or whether you’ll remember a viral video.

Bainbridge wonders if more memorable art is more popular, and is working with the Art Institute of Chicago to test that hypothesis. She’s even testing the memorability of dance moves, as depicted by animated dancing stick figures — asking participants the question, “Have you seen this dance move before?”

If memorability can be engineered in this way, it could help people with memory loss and reduce confusion in the world. Baker also hopes the memorability research will help dementia researchers refine their memory tests, which can pick up early signs of conditions like Alzheimer’s.

This research could be used for less idealistic ends. Certainly marketers want the most memorable images of their products to appear in Instagram feeds. Imagine a “memorability filter” which enhances the mental stickiness of selfies. Given our poor intuition for what is memorable, we might not notice if our environments were manipulated in this way. It’s still unclear whether engineering the memorability of our world would have any meaningful impact on our behavior.

Psychologists have long documented a phenomenon called the “mere exposure” effect. “If you’ve seen things before, you like them better,” Fazio, the Vanderbilt memory researcher, explains. “For more memorable things, you’re more likely to get the mere exposure effect and then have more positive viewpoints toward it.”

Is manipulating the world in this way bad? “It’s one of those things that can be used for good or evil,” says Fazio. The NIH’s Baker is slightly more optimistic. “I don’t necessarily call it bad,” he says. “You can imagine people wanting to make images of themselves more memorable, but that’s sort of already happening. It’s just happening in a way where we’re just relying on our intuition.”

Michael Waraksa for Vox

There are some hard limits to how far we can extrapolate from all this research. For now, memorability studies have been mostly limited to studies of visual memory, and they merely ask people to indicate if they’ve seen an image before. We rely on roughly the same kind of memory when we recognize faces, objects, and places, but otherwise, “it’s not a form of memory that we use very often,” says Fazio.

But our ability to recognize images does seem to be connected to the inner workings of the brain. Rust suspects “it’s used during development to drive curiosity,” meaning that when we’re babies, we’re primed to notice things we’ve never seen before, scrutinize them, and learn more about the world.

Studies of brain activity have also undermined a common assumption that the brain responds less vigorously to something it’s seen before. “Memorability challenges that, because it turns out things we remember better produce more vigorous responses,” she says. “It’s not just through the first time we see them. It’s also true the second time we see them as well.”

By studying what we remember, scientists may be able to assemble more of the puzzle pieces that make up the memory machinery in our minds. For Rust, the ultimate goal is to understand how memory works — so that when it doesn’t, “we can fix it.”

Recently, my mom had me go through old boxes that contained schoolwork from the second grade. It was a weird experience because I remembered none of its contents. All the drawings, the composition notebooks, art projects — I had almost expected these documents to serve as memory cues, portals taking me back into the brain of some past version of myself. Nope. Nothing. They might as well have been made by a different person. We promptly threw most of the work in the garbage.

The memory researchers struggle with this, too. “It’s really unfortunate when you go on a trip, and then you come back, and you don’t remember that much about it a year later,” Bainbridge says. “Even while you’re having that experience, you’re like, ‘I want to remember this forever,’ you still can’t hold on to it. It still slips away. I feel like that’s very sad.”

Memory is a source of scientific mysteries, but also personal ones. Every day, we see and experience things we will forget. What remains, in a way, becomes part of our consciousness. It helps us tell the stories of our lives.

I used to think on this experience of forgetting with a lot of sadness. The research on memorability doesn’t perfectly speak to this experience, but it’s comforting to know that some of this forgetting is not my fault, that some things are just bound not to stick. Some of it has nothing to do with me personally, and is a fundamental part of being alive.

It is sad that our consciousnesses can’t hold onto all the treasures the world shows us on any given day. But memory, the researchers remind me, doesn’t exist to be an archive that perfectly preserves the past. Instead, it exists for the future. It helps us sort and make sense of the world, and it prepares us for what’s ahead. We can’t take everything with us wherever we go. That would be overwhelming.

At the same time, the meaning of memory is changing. In a world where a terabyte of storage might cost me a few bucks a month, I don’t have to remember everything I see. I might not even need to pay attention. I can take pictures of things with my iPhone, and an algorithm will choose and curate my “memories,” serving up recaps of photos that I took years ago, sorting them by the places and faces that might trigger actual memories.

There’s still some hope that even lost memories aren’t completely gone — that perhaps they could be reawakened. “There’s actually a pretty big debate in the memory field,” Fazio says. “Are things lost or are they just inaccessible?” Some scientists think that with the right cues, you can remember many things that aren’t top of mind. “Some people would argue, ‘Yes, our memories are infinite — nothing ever gets fully forgotten.’”

Sometimes a memory will lie dormant for decades, and then spontaneously reappear in our minds. Scientists call these experiences “involuntary autobiographical memories,” and they’re not exactly sure why they happen. I was brushing my teeth the other night and suddenly recalled a PBS cooking show recipe for chicken and figs that had aired sometime in the mid-2000s. I specifically remembered the host saying that when her kids smell the chicken and figs, a favorite dish for company, they would ask, “Who’s coming over?” Where was this memory hiding all these years? And why did it bubble up?

Learning about this research makes me think of my brain as a sieve with imperceptible holes of oddly specific sizes. While the scientists figure out how it all sorts out, more than the sadness of memory loss, I feel a sense of wonder — a sort of magic feeling — about my own memory.

Our brains can still surprise us. And that makes me smile.

Brian Resnick is a science reporter for Vox, covering social and behavioral sciences, space, medicine, and the environment.

More from the Memory Issue

Will Staehle for Vox

Will you support Vox today?

We believe that everyone deserves to understand the world that they live in. That kind of knowledge helps create better citizens, neighbors, friends, parents, and stewards of this planet. Producing deeply researched, explanatory journalism takes resources. You can support this mission by making a financial gift to Vox today. Will you join us?

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via