“Although our results are not significant…” (a rant)


Imagine that you are a scientist, nearing the end of a twenty-year study. Two decades ago, you thought that mothers drinking milk during pregnancy might lead to benefits for their children. When those babies were born, you weighed them and found that the milk-drinkers’ newborns were indeed a little bigger. Just wait twenty years, you said. We’ll see that those kids are taller as adults.

Now that the twenty years have gone by, you measure the grown-up tykes. As you plug the numbers into your computer, a sudden realization makes you shiver: we’ve lost almost a third of the original participants. The remaining group doesn’t have enough statistical power to give you a valid result.

It gets worse: you have detailed records on the mothers and their milk consumption during pregnancy, but that was twenty years ago. You have no records of what the children themselves were eating or drinking as they grew up, which can certainly affect their growth.

You shrug, and finish punching in the numbers. The computer spits out two things:

(1) The kids of milk-swilling moms are a teensy bit taller.

(2) The p-value is 0.19, but you were hoping for 0.05 or less.

Bzzt. Not significant.

Uh-oh. A twenty-year project down the drain. So, let me ask you…

As this scientist, would you then write a paper declaring that maternal milk consumption makes kids grow taller?

As the editor of a journal, would you publish such a paper?

As a science writer at the freaking New York Times, would you give this study any space in your esteemed publication?

Of course n–Oh. Wait.

Taller children

If you click that link, you can see the little disclaimer buried at the end of the fourth paragraph: “But these trends did not achieve statistical significance.”

Now, people often misunderstand what significance means in this sense. It doesn’t mean that the groups had very different heights, or that the result is important to know. “Significance” means that it passed a simple yes-or-no test for whether the result counts or not.

Imagine you have a skeptical friend. A skeptical owl.

Skeptical Owl

“I think something is up with these new nickels,” you tell the owl. “Watch this.” You toss the coin three times and it comes up heads every time. “They always land heads!”

Skeptical owl is skeptical. “Three heads? That’s not so unusual. Happens all the time with completely normal coins.”

You toss some more, and get ten heads in a row.

“OK.” says skeptical owl. “That would be really unlikely if the coin is fair.”

“So you believe me?” you ask the owl, hopefully.

“I’m just saying that’s really unlikely,” says the owl, ever skeptical. “Your p value is, like,  0.00097.” Owls can calculate p-values in their heads.

This is what significant means: You convince the owl that your result is “really unlikely” when the p-value gets smaller than 0.05, which the owl would describe as a 5% chance that it could have happened by accident. Computer statistics programs are all written by owls. They calculate this number for you, and you have a clear, black-and-white answer to the question of whether the result is statistically significant.

Just say no to non-significant results!

This is where the milk-hyping researchers went wrong: they wrote up their results in spite of the owl’s disapproval. It’s true that the .05 cutoff is arbitrary, but really, their p-value of 0.19 is slightly less impressive than a three-heads streak. Skeptical owl is skeptical.

This study is not the only one where researchers try to report non-significant results; it’s an epidemic of bad judgment and/or bad math. Take a look, for example, at this study, published earlier today in the American Journal of Clinical Nutrition, which should really know better. The researchers hoped that Vitamin D supplements would result in people getting fewer infections, as measured by prescriptions for antibiotics. Guess what: they had to begin their conclusion with “Although this study was a post hoc analysis and statistically nonsignificant…”

Friends, don’t fall for it. A non-significant result is one that doesn’t deserve to have lofty conclusions based on it. Not even if you really want it to be significant, if you are wishing super hard for it to be significant. If you feel the need to describe your result as “tantalisingly close to significant,” you already know the truth.

Rejection letters as gold mines

Gold mine

Gold mine photo by Cavin

Found some good stuff on the Twitters today. Andreas von Bubnoff writes about transforming rejection into great articles. He gives several examples of stories that wouldn’t die, despite rejection; their writers re-pitched them, and sometimes they evolved into different types of stories depending on new information or editors’ needs. The ending quote made me think of a few pitches I’ve held in my back pocket for too long:

To Emma [Maris], the lesson is clear: “If you get something killed and it’s not because it’s lousy, then you’d be a fool not to try to sell it somewhere else, because it really only takes ten minutes to write a pitch.”

This pairs nicely with Kelly James-Enger’s ten reasons your pitch was rejected, which is packed with related suggestions. Among them, the 24-hour rule: within a day of receiving a rejection, you send (1) a follow-up to the same editor, with a new idea; and (2) the rejected topic, to a new market.

On the topic of follow-ups, she lists not doing so, especially when the editor likes you, as freelancers’ #1 mistake. This one hit home, too. I’ve gotten some really good rejections, including one I can accurately paraphrase as “This topic is really perfect for us, too bad we’ve got something kind of similar in the works. We really loved your pitch so please send us more. Here are our guidelines including what we pay which is more than you’ve ever made writing before.”

Also of interest: her proposed query structure, ending with an “I’m so great” paragraph. Of course I already include the stuff she mentions, but do I include all of it every time? I hate checklists, which is why I recognize the power of a really pared-down checklist of absolute essentials, the kind that works for emergency surgeons and airline pilots and packing the gear bag for a marathon.

If you don’t have diarrhea, you can thank a herd of babies.

lots of babies

Only babies can be vaccinated against rotavirus; the vaccine (which is oral, not a shot, by the way) puts older kids at risk of some rare but scary side effects.

Still, a recent study shows that since the vaccine was introduced, rotavirus rates have dropped among adults and children, not just babies.  It seems that babies spread rotavirus to other family members, and vaccinating the little ones has a powerful effect similar to herd immunity. Richard Knox writes on NPR’s Shots blog that “no one appreciated that rotavirus was such a major cause of hospitalizations for diarrhea in older children and adults.”

The authors estimate that the vaccine has kept 40,000 people out of the hospital, or about 15,000 per year.

(Photo by Rick Panganiban)

Deadly maple bacon jam (with recipe!)

Earlier this week I wrote about the cronut-burger-associated food poisoning of hundreds of festival-goers in Toronto. New evidence has emerged: The culprit was neither the cronut nor the burger, but the maple bacon jam, produced at the same bakery that made the cronuts. Epic Burger is off the hook, and their bakery partner, La Dolci, is now being investigated to find out how the toxin got in the jam.

Since you can no longer get maple bacon jam at the CNE, try making it at home instead. This recipe from Martha Stewart is excellent (I’ve made it a few times), but please, store it safely.

Fetal “soap” addiction (includes nostalgia if you were Australian in the 1980s)

While researching my piece yesterday on ScienceNow (“Babies learn to recognize words in the womb”), Eino Partanen mentioned an earlier study on fetal learning that showed that babies could recognize a melody they had heard in utero.

Are you ready? The year was 1988, and the melody was this one, played daily across Australia (as well as the UK, where the study was done):

The researchers published their very small study with the title “Fetal ‘Soap’ Addiction.” They noted that babies whose mothers had watched the soap seemed to respond to the tune: two stopped crying when they heard the theme song, and six out of seven became “quiet alert” upon hearing it. (That is the state of alertness known better as “awake and not crying.” It is rare in newborns.) The control babies couldn’t care less; the four that were crying kept on crying, and only two of the eight were quiet alert.It may be small and not super rigorous, but the field of fetal learning has confirmed the basic idea. Fetuses can hear speech and music in the last few months of pregnancy, at the same time sound-processing circuits are being formed in the brain. Yesterday’s paper is the latest finding: newborns aren’t just generally recognizing voices or melodies, but can make out individual syllables and pitch changes that may figure into language learning.

A tale of two cord clampings; or, why one of my sons might have more hemoglobin than the other


(This baby is not mine. The photo is by Wickstopher on Flickr)

Research confirms what my midwives knew: you don’t have to clamp and cut the umbilical cord right after birth; waiting even a minute means the baby gets more blood, which means less chance of being iron-deficient later.

My first son was born at a hospital; we had come there from the birth center because there was meconium in his amniotic fluid. As soon as he popped out, the cord was cut and he was rushed to a table across the room to have his airway suctioned before he could take his first breath. (The worry is that meconium, which is fetus poop, would get into his airway and cause trouble breathing.)

My second son also had meconium in his fluid, but we didn’t know that until I was lying on a bed at the birth center, minutes after arriving, already beginning to push. My water broke, I heard someone say “meconium,” and I thought to myself: “too late now, suckers!”

The midwives suctioned his airway as soon as they could see his mouth; his shoulders weren’t even out yet. After another push they toweled him off and put him on my chest, umbilical cord still connected inside me (the average cord is about twenty inches long). I snuggled and nursed him for the better part of half an hour before I noticed the midwife standing by, holding the cord. She said to the nurse: “It’s still pulsing.” Normally that would be over within a few minutes, but my boy decided to take a bit longer. Which is fine. Policy at the midwife center is not to cut the cord until it stops pulsing, when they know the transfer of blood is complete.

My son’s breathing was a little snorfely (not a technical term) from the meconium, and they were watching him carefully to see if he would need to transfer to the hospital; luckily, he didn’t. The midwife and nurse agreed: keeping the cord attached is especially good for babies who need time to “transition”, as they put it, adjusting to the outside world where oxygen comes from the air you breathe instead of getting pumped into your belly button.

Reading the NY Times article about cord clamping is a little funny after that experience, because it’s so hospital-centric. “It’s a good chunk of blood the baby is going to get, if you wait a minute and a half or two minutes,” one doctor says. (Forget half an hour.) He adds that early clamping is required in cases like … aspirating meconium. I still wonder if I could have requested that my first son be suctioned at my bed, with cord still attached.

Of course, this isn’t about me. Early cord clamping may still make sense for c-section births, and may prevent jaundice. Full term babies, it seems, do fine with early cord clamping, although they don’t note whether those babies are likely to need iron supplements when their hemoglobin is tested at their nine-month checkup.

Do I know that my second son had more hemoglobin than the first? I don’t. Babies’ blood is routinely checked at their nine month doctor visit; my first son’s iron levels were slightly low. As for my second, I may never know. His blood clotted in the collection tube before it got to the lab, and I never brought him in to re-check (the nurse I spoke to agreed that there was no particular concern about his iron levels, and if I wanted to skip the test that was fine with her). Now I kind of want to know - as if that would prove anything. (Cochrane review, meet anecdotal evidence!)

Does swimming rot your teeth?

From the “found this while looking for something else” file…

Competitive swimmers spend a lot of time in chlorinated pools. Between the chlorine and stabilizing chemicals, the pH of the water can drop to “tooth decalcifying levels” that result in yellow teeth and potentially tooth decay. A pair of researchers hypothesized that even well-maintained pools, with neutral pH and standard chlorine levels, can affect the saliva that swimmers produce.

So they showed up to a University of Michigan pool. Before and after a swim team practice, members of the men’s team were asked to drool into a cup for three minutes. “The saliva collection was performed with the athletes seated, head slightly down,” they write, “and they were asked not to swallow or move the tongue or lips during the collection period.”

After practice, the swimmers made inferior saliva: there wasn’t as much to collect, and it had increased levels of calcium and fluoride. They don’t mention if dehydration could be a factor, and I don’t know enough about salivary research to really critique the study. The bottom line is a couple of inarguable statements: Swimmers should keep up with their dentist appointments, and more research is needed.

The latest poop on … oh, never mind. (Also, where to buy maggot candy.)

Remember when the FDA said to quit doing fecal transplants without an IND permit? Yeah, it was just a few weeks ago. (I wrote about it here.) Well, last week they changed their mind. The AGA comments. Since trials of FMT had to be stopped early because it was unethical to withhold the treatment from the control group, many doctors and patients said that it’s illogical to prohibit FMTs from patients who could benefit. The FDA’s announcement was quiet, but it sounds like the transplants are back on, and doctors’ only requirement is to make sure patients understand what the treatment is, and that it’s unapproved.

Untitled 2

The next big thing?[*]

Meanwhile, one of the first companies trying to commercialize poop, Rebiotix (they aim to make a somewhat standardized “intestinal preparation” to make docs’ lives easier) got Fast Track status for FDA approval. This shortens the process somewhat, but the final product will likely be many years in the making. (This company is not the only one working on such a product: another is Monarch Labs, better known as the supplier of Medical Maggots and purveyor of maggot-shaped mint candies. At least they have a sense of humor about their work.)

[*] Yes, I knitted a turd. I have a sense of humor about my work too.

You’re 35. Is your baby doomed?

The idea that a woman’s fertility declines during her thirties is based on some dubious statistics, writes Jessica Grose at Slate. Some of the oft-cited statistics on birth and conception rates come from eighteenth-century French surveys, or from “no specific published medical literature” at all.

Yet, there is a specific reason why 35 is considered a sharp cutoff for the condition of “advanced maternal age.” It’s nothing to do with fertility.

The test for Down syndrome, done during pregnancy, is risky. It involves sampling the amniotic fluid the baby floats in with a gigantic needle. There is about a 1 in 300 chance of miscarriage as a result of the test.

Fish en amniocitos

Chromosomes fluoresce in a fetal cell, in one type of amnocentesis testing. Credit: Instituto Bernabeu

The risk of Down syndrome rises with a mother’s age, from about one baby in every thousand at age 30 to ten at age 40. The rise is dramatic over that decade but smooth: there isn’t a jump at 35. Rather, 35 is the magic number because that’s when the risk of Down syndrome is equal to the risk of miscarrying from the test.

In the past, testing for Down syndrome and other chromosomal abnormalities was only offered to the 35-and-up crowd, but guidelines have changed and there are less-invasive, though less conclusive, tests available. The “advanced maternal age” moniker stuck, and is still considered useful … but nothing terrible happens at age 35.

Some papers I haven’t read this week

Antibiotics + stress = dysbiosis? (in mice)

The latest negative evidence on probiotics (the germs in yogurt don’t take up residence in you, which isn’t to say they do nothing.) I asked Rob Knight about a similar paper once, and he was all “Well, duh.” (Not his exact words.)

There are 10 kinds of people: those that know binary and those that don’t. BA DUM BUM Alright, let me start over: there are two kinds of people, those with more Prevotella than Bacteroidetes, and the other way around. Not nearly as catchy. (Also, arguable.)  Anyway, enterotype may influence how your gut flora respond to diet.

Gut microbes seem to be necessary for the immune response to necrosis.

WordPress Themes