Stressful events in life, such as the death of a child, divorce or being fired, can age the brain by at least four years, US researchers suggests.
They looked at performance in memory and thinking tests of 1,300 people in their 50s to gauge brain health.
The study did not look at the risk of dementia and experts said there could be many different factors at play.
The findings were presented at the Alzheimer's Association International Conference in London.
Although the research could not establish any direct link between stress and an increased risk of dementia, stressful experiences are known to have an impact on brain function, which could then lead to dementia in the longer term.
The theory is that stress increases inflammation, which could increase the chances of developing dementia - and this is currently being tested by University of Southampton researchers.
This study, from the University of Wisconsin School of Medicine, found that African Americans were more at risk of stress in life than other ethnic groups. This is because they scored poorer results in the memory tests than other groups and also tended to live in poorer neighbourhoods.
Stressful experiences across all groups included educational difficulties, financial insecurity, serious health problems and psychological trauma.
Other studies presented at the conference point to growing evidence that stress in early life and where people live can be factors in an individual's risk of developing dementia.
Dr Doug Brown, director of research and development for Alzheimer's Society, said studying the role of stress was complex.
"It is hard to separate from other conditions such as anxiety and depression, which are also thought to contribute towards dementia risk.
"However, the findings do indicate that more should be done to support people from disadvantaged communities who are more likely to experience stressful life events."
There are currently around 850,000 people in the UK with dementia.
It mainly affects people over the age of 65 and, while the likelihood of developing dementia rises sharply with age, about 42,000 of those suffering from the condition are younger than that.
Dr Carol Routledge, director of research at Alzheimer's Research UK, said there could be a number of different factors involved in the link between stress and memory decline.
But she said the brain was an "incredibly intricate organ" to research.
"There is a growing realisation that events and experiences throughout life can impact the brain decades later and researchers must take a whole lifespan approach to understanding brain health in later life."
Tips for reducing the risk of dementia
Keep physically active for at least 30 minutes, five times a week.
Don't smoke, or give up if you do.
Eat a healthy balanced diet including oily fish, fruit, vegetables and low levels of red meat and sugar.
Keep your alcohol intake to a maximum of 14 units per week for men and women.
Manage other health conditions such as high blood pressure, high cholesterol and type 2 diabetes.
Keep to a healthy weight to reduce your risk of type 2 diabetes, stroke and heart disease.
Give your brain a daily workout by doing puzzles, word searches or crosswords or learning something new.
Keeping socially engaged and have a good social network.
(Source: Alzheimer's Society)