(index page)
Farewell CDL!
A little over two years ago, after an exhausting day of packing up our apartment in Brooklyn, I turned to my partner and said “Hey, remember when said I wasn’t going to do a postdoc?”.
This was a joke, intended to offset the anxiety we were both feeling about our impending move across the country. But, after deciding to not pursue the “traditional” academic path (graduate school → postdoctoral fellowships → faculty position) and shifting from working in cognitive neuroscience labs to working in academic libraries, I had long assumed that my window into the liminal space occupied by postdocs had closed. That is, until I learned about the CLIR Postdoctoral Fellowship Program and saw an opportunity to dive headfirst into the wider world of scholarly communications and open science with the UC3 team at California Digital Library.
Today is my last day in the office at CDL and so much has happened in the world and for me personally and professionally over the course of my fellowship that I’m not sure anything I could write here would ever do it all justice. I suppose I could assess my time at CDL in terms of the number of posters, papers, and presentations I helped put together. I could mention my involvement with groups like BITSS and RDA. I could add up all the hours I’ve spent talking on Skype and Zoom or all the words I’ve written (and rewritten) in Slack and Google Docs. But really the most meaningful metric of my time at CDL would be the number of new colleagues, collaborators, and friends I’ve gained as a postdoc. I came to CDL because I wanted to become part of the broad community of folks working on research data in academic libraries. And now, as I’m about to move into a new position as the Data Services Librarian at Lane Medical Library, I can say that has happened more than I would have thought possible.
Looking back on the last two years, there are about a million people I owe a heartfelt thanks. If you’re out there and you don’t get an email from me, it’s almost definitely because I wrote something, decided it was completely insufficient, wrote something else, decided that was completely insufficient, and then got completely overwhelmed by the number of drafts in my mailbox. But seriously, thanks to everyone on the UC3 team, at CDL and the UC libraries, and beyond for everything you’ve done for me and for everything you’ve helped me do.
Looking forward to what comes next, I have about a million ideas for new projects. Some of are extensions of work I started during my fellowship while others are the product of the connections, insights, or interests I developed while at CDL. But, since this is my last blog post as a postdoc, I also want to devote some space one last UC3 project update.
Support Your Data
If there is a common thread that ties together all of the work I’ve done at CDL it is that I really want to bridge the communication gap that exists between researchers and data librarians. The most explicit manifestation of this has been the Support Your Data project.
If you’ve missed all my blog posts, posters, and presentations on the topic, the goal of the Support Your Data project is to create tools for researchers to assess and advance their own data management practices. With an immense amount of help from the UC3 team, I drafted a rubric that describes activities related to data management and sharing in a framework that we hope is familiar and useful to researchers. Complementing this rubric, we also created a series of short guides that give actionable advice on topics such as data management planning, data organization and storage, documentation, and data publishing. Because we assumed that different research communities (e.g. researchers in different disciplines, researchers at different institutions) have different data-related needs and access to different data-related resources, all of these materials were designed with an eye towards easy customization.
A full rundown of the Support Your Data project will be given in a forthcoming project report. The short version is that, now that the majority of the content has been drafted, the next step is to work on design and adoption. We want researchers and librarians to use these tools so we want to make sure the final products don’t look like something I’ve been working on in a series of Google spreadsheets. Though I will no longer be leading the project, this work will continue at CDL. That said, I have a lot of ideas about using the Support Your Data materials as they currently exist as a jumping off point for future projects.
Data Management Practices in Neuroscience
I’m still surprised I convinced a library to let me do a neuroimaging project. I mean, I’m not that surprised, I can be pretty convincing when I start arguing that neuroimaging is a perfect case study for studying how researchers actually manage their data. But I think it says a lot about the UC3 team that they fully supported me as I dove deep into the literature describing fMRI data analysis workflows, charted the history of data sharing in cognitive neuroscience, and wrangled all manner of acronyms (ahem, BIDS, BIDS).
As I outlined in a previous blog post, the idea to survey neuroimaging researchers literally started with a tweet. But, before too long, it became a full fledged collaborative research project. As a former imaging researcher, I am still marveling over the fact that my collaborator Ana Van Gulick- another neuroscientist turned research data in libraries person- and I managed to collect data from over 140 participants so quickly. Our principle aim was to provide valuable insights to be both the neuroimaging and data curation community, but this project also gave us the opportunity to practice what we preach and apply open science practices to our own work. A paper describing the results of our survey of the data management practices of MRI researchers is currently through the peer review process, but we’ve already published a preprint and made our materials and data openly available.
We definitely hope to continue working with the neuroimaging community, but we also plan to do follow-up surveys of other research communities. Given the growing emphasis on transparency and open science practices in the field, what do data management practices look like in psychology? We hope to find out soon!
Exploring Researcher Needs and Values Related to Software
One of the principle aims of my fellowship was to explore issues around software curation. Spoiler alert: Though the majority of my projects touched on the subject of research software in some way, I’m still not sure I’ve come up with a comprehensive definition of what “software curation” actually means in practice. Shoutout to my fellow software curation fellows who continue to bring their array of perspectives and high levels of expertise to this issue (and thanks for not rolling your eyes at the cognitive neuroscientist trying to understand how computers work).
Before I started at CDL I knew that I would be working with Yasmin AlNoamany, my counterpart at the UC Berkeley library, on a project involving research software. To extend previous work done by the UC3 around issues related to data publishing, we eventually decided to survey researchers on how their use, share, and value their software tools. Our results, which we hope will help libraries and other research support groups shape their service offerings, are described in this preprint. We’ve also made our materials and data openly available.
There is still a lot of work to be done defining the problems and solutions of software curation. Though we currently don’t have plans to do any follow-up studies, we have another paper in the works describing the rest of our results and our survey will definitely inform how I plan to organize software-related training and outreach in the future. The UC3 team will also be continuing to work in this area, through their involvement with The Carpentries.
But wait, there’s more
Earlier this week, after another exhausting day of packing up our apartment outside of Berkeley, I keep remarking to my partner “Hey, remember when I thought I’d never get a job at Stanford.”
This is a joke too. We’re not moving across the country this time, but the move feels just as significant. Two years ago I was sad to leave New York, but ultimately decided I needed to take a step forward in my career. Now, as I’m about to take another step, I’m very sad to leave CDL. I’ve very excited about what comes next, of course. But I will always be grateful for CLIR and the UC3 team giving me to opportunity to learn so much and connect with so many amazing friends, collaborators, and colleagues.
Thanks everyone!
Neuroimaging as a case study in research data management: Part 2
Part 2: On practicing what we preach
A few weeks ago I described the results of a project investigating the data management practices of neuroimaging researchers. The main goal of this work is to help inform efforts to address rigor and reproducibility in both the brain imaging (neuroimaging) and academic library communities. But, as we were developing our materials, a second goal emerged- practice what we preach and actually apply the open science methods and tools we in the library community have been recommending to researchers
Wait, what? Open science methods and tools
Before jumping into my experience of using open science tools as part of a project that involves investigating open sciences practices, it’s probably worth taking a step back and defining what the term actually means. It turns out this isn’t exactly easy. Even researchers working in the same field understand and apply open science in different ways. To make things simpler for ourselves when developing our materials, we used “open science” broadly to refer to the application of methods and tools that make the processes and products of the research enterprise available for examination, evaluation, use, and re-purposing by others. This definition doesn’t address the (admittedly fuzzy) distinctions between related movements such open access, open data, open peer review, and open source, but we couldn’t exactly tackle all of that in a 75 question survey.
From programming languages used for data analysis like Python and R to collaboration platforms like the Github and the Open Science Framework (OSF) to writing tools like LaTex and Zotero to data sharing tools like Dash, figshare, and Zenodo, there are A LOT of different methods and tools that fall under the category of open science. Some of them worked for our project, some of them didn’t.
Data Analysis Tools
As both an undergraduate and graduate student, all of my research methods and statistics courses involved analyzing data with SPSS. Even putting aside the considerable (and recurrent) cost of an SPSS licence, I wanted to go a different direction in order to get some first-hand experience with the breadth of analysis tools that have been developed and popularized over the last few years.
I thought about trying my hand at a Jupyter notebook, which would have allowed us to share all of our data and analyses in one go. However, I also didn’t want to delay things as I taught myself how to work within a new analysis environment. From there, I tried a few “SPSS-like” applications like PSPP and Jamovi and would recommend both to anyone who has a background like mine and isn’t quite ready to start writing code. I ultimately settled on JASP because, after taking a cursory look through our data using Excel (I know, I know), it was actually being used by the participants in our sample. It turns out that’s probably because it’s really intuitive and easy to use. Now that I’m not in the middle of analyzing data, I’m going to spend some time learning other tools. But, while I do that, I’m going to to keep using and recommending JASP.
From the very beginning, we planned on making our data open. Though I wasn’t necessarily thinking about it at the time, this turned out to be another good reason to try something other than SPSS. Though there are workarounds, .sav is not exactly an open file format. But our plan to make the data open not only affected the choice of analysis tools, it also affected how I felt while running the various statistical tests. One one hand, knowing that other researchers would be able to dive deep into our data amplified my normal anxiety about checking and re-checking (and statcheck-ing) the analyses. On the other hand, it also greatly reduced my anxiety about inadvertently relegating an interesting finding to the proverbial file-drawer.
Collaboration Tools
When we first started, it seemed sensible to create a repository on the Open Science Framework in order to keep our various files and tools organized. However, since our collaboration is between just two people and there really aren’t that many files and tools involved, it became easier to just use services that were already incorporated in our day-to-day work- namely e-mail, Skype, Google Drive, and Box. Though I see how it could be potentially useful for a project with more moving parts, for our purposes it mostly just added an unnecessary extra step.
Writing Tools
This is where I restrain myself from complaining too much about LaTeX. Personally, I find it a less than awesome platform for doing any kind of collaborative writing. Since we weren’t writing chunks of code, I also couldn’t find an excuse to write the paper in R Markdown. Almost all of the collaborative writing I’ve done since graduate school has been in Google docs and this project was no exception. It’s not exactly the best when it comes to formatting text or integrating with tables and figures, I haven’t found a better tool for working on a text with other people.
We used a Mendeley folder to share papers and keep our citations organized. Zotero has the same functionality, but I personally find Mendeley slightly easier to use. In retrospect, we could also have used something like the F1000 Workspace that has a more direct integration with Google docs.
This project is actually the first time I’ve published on a preprint. Like making our data open, this was the plan all along. The formatting was done in Overleaf, mostly because it was a (relatively) user friendly way to use LaTeX and I was worried our tables and figures would break the various MS Word bioRxiv templates that are floating around. Similar making our data open, planning to publish a preprint had a impact on the writing process. I’ve since notices a typo or two, but knowing that people would be reading our preprint only days after its submission made me especially anxious to check the spelling, grammar, and the general flow of our paper. On the other hand, it was a relief to know that the community would be able to read the results of a project that started at the very beginning of my postdoc before it’s conclusion.
Data Sharing Tools
Our survey and data are both available via figshare. More specifically, we submitted our materials to Kilthub, Carnegie Mellon’s instance of figshare for institutions. For those of you out there currently raising an eyebrow, we didn’t submit to Dash, UC3’s data publication platform, because of an agreement outlined when we were going through the IRB process. Overall, the submission was relatively straightforward, through the curation process definitely made me consider how difficult it is to balance adding proper metadata and documentation to a project with the desire (or need) to just get material out there quickly.
A few more thoughts on working openly
More than once over the course of this project I joked to myself, my collaborator, or really to anyone that would listen that “This would probably be easier or quicker if we could just do it the old way.”. However, now that we’re at a point where we’ve submitted our paper (to an open access journal, of course), it’s been useful to look back on what it has been like to use these different open science methods and tools. My main takeaways are that there are a lot of ways to work openly and that what works for one researcher may not necessarily work for another. Most of the work I’ve done as a postdoc has been about meeting researchers where they are and this process has reinforced my desire to do so when talking about open science, even when the researcher in question is myself.
Like our study participants, who largely reported that their data management practices are motived and limited by immediate practical concerns, a lot of our decisions about open which open science methods and tools to apply were heavily influenced by the need to keep our project moving forward. As much as I may have wanted to, I couldn’t pause everything to completely change how I analyze data or write papers. We committed ourselves to working openly, but we also wanted to make sure we had something to show for ourselves.
Additional Reading
Borghi, J. A., & Van Gulick, A. E. (2018). Data management and sharing in neuroimaging: Practices and perceptions of MRI researchers. bioRxiv.
Neuroimaging as a case study in research data management: Part 1
Part 1: What we did and what we found
This post was originally posted on Medium.
How do brain imaging researchers manage and share their data? This question, posed rather flippantly on Twitter a year and a half ago, prompted a collaborative research project. To celebrate the recent publication of a bioRxiv preprint, here is an overview of what we did, what we found, and what we’re looking to do next.
What we did and why
Magnetic resonance imaging (MRI) is a widely-used and powerful tool for studying the structure and function of the brain. Because of the complexity of the underlying signal, the iterative and flexible nature of analytical pipelines, and the cost (measured in terms of both grant funding and person hours) of collecting, saving, organizing, and analyzing such large and diverse datasets, effective research data management (RDM) is essential in research projects involving MRI. However, while the field of neuroimaging has recently grappled with a number of issues related to the rigor and reproducibility of its methods, information about how researchers manage their data within the laboratory remains mostly anecdotal.
Within and beyond the field of neuroimaging, efforts to address rigor and reproducibility often focus on problems such as publication bias and sub-optimal methodological practices and solutions such as the open sharing of research data. While it doesn’t make for particularly splashy headlines (unlike, say, this), RDM is also an important component of establishing rigor and reproducibility. If experimental results to be verified and repurposed, the underlying data must be properly saved and organized. Said another way, even openly shared data isn’t particularly useful if you can’t make sense of it. Therefore, in an effort to inform the ongoing conversation about reproducibility in neuroimaging, myself and Ana Van Gulick set out to survey the RDM practices and perceptions of the active MRI research community.
https://twitter.com/JohnBorghi/status/758030771097636869
With input from several active neuroimaging researchers, we designed and distributed a survey that described RDM-related topics using language and terminology familiar to researchers who use MRI. Questions inquired about the type(s) of data collected, the use analytical tools, procedures for transferring and saving data, and the degree to which RDM practices and procedures were standardized within laboratories or research groups. Building on my work to develop an RDM guide for researchers, we also asked participants to rate the maturity of both their own RDM practices and those of the field as a whole. Throughout the survey, we were careful to note that our intention was not to judge researchers with different styles of data management and that RDM maturity is largely orthogonal to the sophistication of data collection and analysis techniques.
Wait, what? A brief introduction to MRI and RDM.
Magnetic resonance imaging (MRI) is a medical imaging technique that uses magnetic fields and radio waves to create detailed images of organs and tissues. Widely used in medical settings, MRI has also become important tool for neuroscience researchers especially since the development of functional MRI (fMRI) in the early 1990’s. By detecting changes in blood flow that are associated with changes in brain activity, fMRI allows researchers to non-invasively study the structure and function of the living brain.
Because there are so many perspectives involved, it is difficult to give a single comprehensive definition of research data management (RDM). But, basically, the term covers activities related to how data is handled over the course of a research project. These activities include, but are certainly not limited to, those related to how data is organized and saved, how procedures and decisions are documented, and how research outputs are stored are shared. Many academic libraries have begun to offer services related to RDM.
Neuroimaging research involving MRI presented something of an ideal case study for us to study RDM among active researchers. The last few years have seen a rapid proliferation of standards, tools, and best practice recommendations related to the management and sharing of MRI data. Neuroimaging research also crosses many topics relevant to RDM support providers such as data sharing and publication, the handling of sensitive data, and the use and curation of research software. Finally, as neuroimaging researchers who now work in academic libraries, we are uniquely positioned to work across the two communities.
What we found
After developing our survey and receiving the appropriate IRB approvals, we solicited responses to our survey during Summer 2017. A total of 144 neuroimaging researchers participated and their responses revealed several trends that we hope will be informative for both neuroimaging researchers and also data support providers in a academic libraries.
As shown below, our participants indicated that their RDM practices throughout the course of a research project were largely motivated by immediate practical concerns such as preventing the loss of data and the ensuring access to everyone within a lab or research group and limited by a lack of time and discipline-specific best practices.
We were relatively unsurprised to see that neuroimaging researchers use a wide array of software tools analyze their often heterogeneous sets of data. What did surprise us somewhat was the different responses from trainees (graduate students and postdocs) and faculty on questions related to the consistency of RDM practices within their labs. Trainees were significantly less likely to say that practices related to backing up, organizing, and documenting datas were standardized within their lab than faculty, which we think highlights the need for better communication about how RDM is an essential component of ensuring that research is rigorous and reproducible.
Analysis of RDM maturity ratings revealed that our sample generally rated their own RDM practices as more mature than the field as a whole and practices during the data collection and analysis phases of a project as significantly more mature than those during the data sharing phase. There are several interpretations of the former result, but the later is consistent with the low level of data sharing in the field. Though these ratings provide an interesting insight into the perceptions of the active research community, we believe there is substantial room for improvement in establishing proper RDM across every phase of a project, not just after after the data has already been analyzed.
For a complete overview of our results, including an analysis of how the field of neuroimaging is at a major point of transition when it comes to the adoption of practices including open access publishing, preregistration, replication, check out our preprint now on bioRxiv. While you’re at it, feel free to peruse, reuse, or remix our survey and data, both of which are available on figshare.
Is this unique to MRI research?
Definitely not. Just as the consequences of sub-optimal methodological practices and publication biases have been discussed throughout the biomedical and behavioral sciences for decades, we suspect that the RDM-related practices and perceptions observed in our survey are not limited to neuroimaging research involving MRI.
To paraphrase and reiterate a point made in the preprint, this work was intended to be descriptive not prescriptive. We also very consciously have not provided best practice recommendations because we believe that such recommendations would be most valuable (and actionable) if developed in collaboration with active researchers. Moving forward, we hope to continue to engage with the neuroimaging community on issues related to RDM and also expand the scope of our survey to other research communities such as psychology and biomedical science.
Additional Reading
Our preprint, one more time:
- Borghi, J. A., & Van Gulick, A. E. (2018). Data management and sharing in neuroimaging: Practices and perceptions of MRI researchers. bioRxiv.
For a primer on functional magnetic resonance imaging:
- Soares, J. M., Magalhães, R., Moreira, P. S., Sousa, A., Ganz, E., Sampaio, A., … Sousa, N. (2016). A hitchhiker’s guide to functional magnetic resonance imaging. Frontiers in Neuroscience, 10, 1–35.
For more on rigor, reproducibility, and neuroimaging:
- Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke M., … Yeo, B. T. T. (2017). Best practices in data analysis and sharing in neuroimaging using MRI. Nature Neuroscience, 20(3), 299–303. (Preprint)
- Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., … Yarkoni, T. (2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18(2), 115–126. (Preprint)
Support your Data
Building an RDM Maturity Model: Part 4
By John Borghi
Researchers are faced with rapidly evolving expectations about how they should manage and share their data, code, and other research products. These expectations come from a variety of sources, including funding agencies and academic publishers. As part of our effort to help researchers meet these expectations, the UC3 team spent much of last year investigating current practices. We studied how neuroimaging researchers handle their data, examined how researchers use, share, and value software, and conducted interviews and focus groups with researchers across the UC system. All of this has reaffirmed our that perception that researchers and other data stakeholders often think and talk about data in very different ways.
Such differences are central to another project, which we’ve referred to alternately as an RDM maturity model and an RDM guide for researchers. Since its inception, the goal of this project has been to give researchers tools to self assess their data-related practices and access the skills and experience of data service providers within their institutional libraries. Drawing upon tools with convergent aims, including maturity-based frameworks and visualizations like the research data lifecycle, we’ve worked to ensure that our tools are user friendly, free of jargon, and adaptable enough to meet the needs of a range of stakeholders, including different research, service provider, and institutional communities. To this end, we’ve renamed this project yet again to “Support your Data”.
What’s in a name?
Because our tools are intended to be accessible to a people with a broad range of perceptions, practices, and priorities, coming up with a name that encompasses complex concepts like “openness” and “reproducibility” proved to be quite difficult. We also wanted to capture the spirit of terms like “capability maturity” and “research data management (RDM)” without referencing them directly. After spending a lot of time trying to come up with something clever, we decided that the name of our tools should describe their function. Since the goal is to support researchers as they manage and share data (in ways potentially influenced by expectations related to openness and reproducibility), why not just use that?
Recent Developments
In addition to thinking through the name, we’ve also refined the content of our tools. The central element, a rubric that allows researchers to quickly benchmark their data-related practices, is shown below. As before, it highlights how the management of research data is an active and iterative process that occurs throughout the different phases of a project. Activities in different phases represented in different rows. Proceeding left to right, a series of declarative statements describe specific activities within each phase in order of how well they are designed to foster access to and use of data in the future.
The four levels “ad hoc”, “one-time”, “active and informative” and “optimized for re-use”, are intended to be descriptive rather than prescriptive.
- Ad hoc — Refers to circumstances in which practices are neither standardized or documented. Every time a researcher has to manage their data they have to design new practices and procedures from scratch.
- One time — Refers to circumstances in which data management occurs only when it is necessary, such as in direct response to a mandate from a funder or publisher. Practices or procedures implemented at one phase of a project are not designed with later phases in mind.
- Active and informative — Refers to circumstances in which data management is a regular part of the research process. Practices and procedures are standardized, well documented, and well integrated with those implemented at other phases.
- Optimized for re-use — Refers to circumstances in which data management activities are designed to facilitate the re-use of data in the future
Each row of the rubric is tied to a one page guide that provides specific information about how to advance practices as desired or required. Development of the content of the guides has proceeded sequentially. During the autumn and winter of 2017, members of the UC3 team met to discuss issues relevant to each phase, reduce the use of jargon, and identify how content could be localized to meet the needs of different research and institutional communities. We are currently working on revising the content based suggestions made during these meetings.
Next Steps
Now that we have scoped out the content, we’ve begun to focus on the design aspect of our tools. Working with CDL’s UX team, we’ve begun to think through the presentation of both the rubric and the guides in physical media and online.
As always, we welcome any and all feedback about content and application of our tools.
The Significance of Managing Research Data
Some of the most influential research tools of the last century were created to ensure the quality of beer and extrapolate the results of agriculture experiments conducted in the English countryside. Though ostensibly about the placement of a decimal point, an ongoing debate about the application of these tools also provides a window for understanding what it actually means to manage research data.
The p-value: A very quick introduction
Though now ubiquitous in experiment-based research, statistical techniques for extending inferences from small sample (e.g. the participants in a research study) to larger populations are actually a relatively recent invention. The t-test, an early and still widely used example of “small sample” statistics was developed by William Sealy Gossett in the early 20th century as an economical way of ensuring the quality of stout. Several years later, while assisting with long-term experiments on wheat and grass at Rothamsted Experimental Station, Ronald Fisher would build on the work of Gosset and others to develop a statistical framework based around the idea of comparing observations to the null hypothesis- the position that there is no significant difference between two or more specified sets of observations.
In Fisher’s significance testing framework, devices like t-tests are tests of the null hypothesis. The results of these tests indicate the likelihood of observing a result when the null hypothesis is true. The logic is a little tricky, but the core idea is that these tests give researchers a way of understanding the likelihood that their data is the result of sampling or experimental error. In quantitative terms, this likelihood is known as a p-value. In his highly influential 1925 book, Statistical Methods for Research Workers, Fisher would introduce an informal threshold for rejecting the null hypothesis: p < 0.05.
Despite the vehement objections of all three, Fisher’s work would later be synthesized with that of statisticians Jerzy Neyman and Egon Pearson into a suite of tools that are still widely used in many fields of research. In practice, p < 0.05 has since become a one-size-fits-all indicator of success. For decades it has been acknowledged that work that meets this criterion is generally more likely to be reported in the scholarly literature while work that doesn’t is generally relegated the proverbial file drawer.
Beyond p < 0.05
The p < 0.05 threshold has become a flashpoint the ongoing conversation about research practices, reproducibility, and replicability. Heated conversations about the use and misuse of p-values have been ongoing for decades, but over the summer a group of 72 influential researchers proposed a seemingly simple step forward- change the threshold from 0.05 to 0.005. According to the authors, “Reducing the p-value threshold for claims of new discoveries to 0.005 is an actionable step that will immediately improve reproducibility.”.
As of this writing, two responses have been published. Both weigh the pros and cons of p < 0.005 and argue that the placement of a decimal point is less of a problem than the uncritical use of a single one-size-fits-all threshold across many different circumstances and fields of research. Both end on calls for greater transparency and stronger justifications for how decisions related to research design and statistical practice are made. If the initial paper proposed changing the answer from p < 0.05 to 0.005, both responses highlight the necessity of changing the question from one that is focused on statistics to one that incorporates research data management (RDM).
Ensuring that data can be used and evaluated in the future is one of the primary goals of RDM. For example, the RDM guide we’re developing does not have a space for assessing p-values. Instead, its focus is assessing and advancing practices related to planning for, saving, and documenting data and other research products. Such practices come with their own nuance, learning curves, and jargon, but are important elements to any effort to ensure that research decisions are transparent and justified.
Resources and Additional Reading
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., … & Cesarini, D. (2017). Redefine statistical significance. Nature Human Behaviour. doi: 10.1038/s41562-017-0189-z
Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., … Zwaan, R. A. (2017). Justify your alpha: A response to “Redefine statistical significance”. PsyArxiv preprint. doi: 10.17605/OSF.IO/9S3Y6
McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2017). Abandon statistical significance. arXiv preprint. arXiv: 1709.07588.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54(285), 30-34. doi: 10.1080/01621459.1959.10501497
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. doi: 10.1037/0033-2909.86.3.638
Managing the new NIH requirements for clinical trials
As part of an effort to enhance transparency in biomedical research, the National Institutes of Health (NIH) have, over the last few years, announced a series of policy changes related to clinical trials. Though there is still a great deal of uncertainty about which studies do and do not qualify, these changes may have significant consequences for researchers who may not necessarily consider their work to be clinical or part of a trial.
Last September, the NIH announced a series of requirements for studies that meet the agency’s revised and expanded definition of a clinical trials. Soon after, it was revealed that many of these requirements may apply to large swaths of NIH-funded behavioral, social science, and neuroscience research that, historically, have not been considered to be clinical in nature. This was affirmed several weeks ago when the agency released a list of case studies that included a brain imaging study in which healthy participants completed a memory task as an example of a clinical trial.
What exactly constitutes a clinical trial now?
Because many investigators doing behavioral, social science, and neuroscience research consider their work to be basic research and not a part of a clinical trial, it is worth taking a step back to consider how NIH now defines the term.
According to the NIH, clinical trials are “studies involving human participants assigned to an intervention in which the study is designed to evaluate the effect(s) of the intervention on the participant and the effect being evaluated is a health-related biomedical or behavioral outcome.”, In an NIH context, intervention refers to “a manipulation of the subject or subject’s environment for the purpose of modifying one or more health-related biomedical or behavioral processes and/or endpoints.”. Because the agency considers all of the studies it funds that investigate biomedical or behavioral outcomes to be health-related, this definition includes mechanistic or exploratory work that does not have direct clinical implications.
Basically, if you are working on an NIH-funded study that involves biomedical or behavioral variables, you should be paying attention to the new requirements about clinical trials.
What do I need to do now that my study is considered a clinical trial?
If you think your work may be reclassified as a clinical trial, it’s probably worth getting a head start on meeting the new requirements. Here is some practical advice about getting started.
Applying for Funding
NIH has specified new requirements about how research involving clinical trials can be funded. For example, NIH will soon require that any application involving a clinical trial be submitted in response to a funding opportunity announcement (FOA) or request for proposal (RFP) that explicitly states that it will accept a clinical trial. This means, that if you are a researcher whose work involves biomedical or behavioral measures, you may have to apply to funding mechanisms that your peers have argued are not necessarily optimal or appropriate. Get in touch with your program officer and watch this space.
Grant applications will also feature a new form that consolidates the human subjects and clinical trial information previously collected across multiple forms into one structured form. For a walkthrough of the new form, check out this video.
Human Subjects Training
Investigators involved in a clinical trial must complete Good Clinical Practice (GCP) training. GCP training addresses elements related to the design, conduct, and reporting of clinical trials and can be completed via a class or course, academic training program, or certification from a recognized clinical research professional organization.
In practice, if you have already completed human subjects training (e.g. via CITI) and believe your research may soon be classified as a clinical trials, you may want to get proactive about completing those couple additional modules.
Getting IRB Approval
Good news if you work on a multi-site study, NIH now expects that you will use a single Institutional Review Board (sIRB) for ethical review. This should help streamline the review process, since it will no longer be necessary to submit an application to each site’s individual IRB. This requirement also applies to studies that are not clinical trials.
Registration and Reporting
NIH-funded projects involving clinical trials must be registered on Clinicaltrials.gov. In practice, this means that the primary investigator or grant awardee is responsible for registering the trial no later than 21 days after the enrollment of the first participant and is required to submit results information no later than a year after the study’s completion date. Registration involves supplying a significant amount of information about a study’s planned design and participants while results reporting involves supplying information about the participants recruited, the data collected, and the statistical tests applied. For more information about Clinicaltrials.gov, check out this paper.
If you believe your research may soon be reclassified as a clinical trial, now is probably a good time to take a hard look at how you and your lab handle research data management.The best way to relieve the administrative burden of these new requirements is to plan ahead and ensure that your materials are well organized, your data is securely saved, and your decisions are well documented. The more you think through how you’re going to manage your data and analyses now, the less you’ll have to scramble to get everything together when the report is due. If you haven’t already, now would be a good time to get in touch with the data management, scholarly communications, and research IT professionals at your institution.
What We Talk About When We Talk About Reproducibility
At the very beginning of my career in research I conducted a study which involved asking college students to smile, frown, and then answer a series of questions about their emotional experience. This procedure was based on several classic studies which posited that, while feeling happy and sad makes people smile and frown, smiling and frowning also makes people feel happy and sad. After several frustrating months of trying and failing to get this to work, I ended my experiment with no significant results. At the time, I chalked up my lack of success to inexperience. But then, almost a decade later, a registered replication report of the original work also showed a lack of significant results and I was left to wonder if I had also been caught up in what’s come to be known as psychology’s reproducibility crisis.
While I’ve since left the lab for the library, my work still often intersects with reproducibility. Earlier this year I attended a Research Transparency and Reproducibility Training session offered by the Berkeley Institute for Transparency in the Social Sciences (BITSS) and my projects involving brain imaging data, software, and research data management all invoke the term in some way. Unfortunately, though it has always has been an important part of my professional activities, it isn’t always clear to me what we’re actually talking about when we talk about reproducibility.
The term “reproducibility” has been applied to efforts to enhance or ensure the research process for at at least 25 years. However, related conversations about how research is conducted, published, and interpreted have been ongoing for more than half a century. Ronald Fisher, who popularized the p-value that lies so central to many modern reproducibility efforts, summed up the situation in 1935.
“We may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us statistically significant results.”
Putting this seemingly simple statement into action has proven to be quite complex. Some reproducibility-related efforts are aimed at how researchers share their results, others are aimed at how they define statistical significance. There is now a burgeoning body of scholarship devoted to the topic. Even putting aside terms like HARKing, QRPs, and p-hacking, seemingly mundane objects like file drawers are imbued with particular meaning in the language of reproducibility.
So what actually is reproducibility?
Well… it’s complicated.
The best place to start might be the National Science Foundation, which defines reproducibility as “The ability of a researcher to duplicate the results of a prior study using the same materials and procedures used by the original investigator.”. According the NSF, reproducibility is one of three qualities that ensure research is robust. The other two, replicability and generalizability, are defined as “The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected.” and “Whether the results of a study apply in other contexts or populations that differ from the original one.” respectively. The difference between these terms is in the degree of separation from the original research, but all three converge on the quality of research. Good research is reproducible, replicable, and generalizable and , at least in the context of the NSF, a researcher invested in ensuring the reproducibility of their work would deposit their research materials and data in a manner and location where they could be accessed and used by others.
Unfortunately, defining reproducibility isn’t always so simple. For example, according to the NSF’s terminology, the various iterations of the Reproducibility Project are actually replicability projects (muddying the waters further, the Reproducibility Project: Psychology was preceded by the Many Labs Replication Project). However, the complexity of defining reproducibility is perhaps best illustrated by comparing the NSF definition to that of the National Institutes of Health.
Like the NSF, NIH invokes reproducibility in the context of addressing the quality of research. However, unlike the NSF, the NIH does not provide an explicit definition of the term. Instead NIH grant applicants are asked to address rigor and reproducibility across four areas of focus: scientific premise, scientific rigor (design), biological variables, and authentication. Unlike the definition supplied by the NSF, NIH’s conception of reproducibility appears to apply to an extremely broad set of circumstances and encompasses both replicability and generalizability. In the context of the NIH, a researcher invested in reproducibility must critically evaluate every aspect of their research program to ensure that any conclusions drawn from it are well supported.
Beyond the NSF and NIH, there have been numerous attempts to clarify what reproducibility actually means. For example, a paper out of the Meta-Research Innovation Center at Stanford (METRICS) distinguishes between “methods reproducibility”, “results reproducibility”, and “inferential reproducibility”. Methods and results reproducibility map onto the NSF definitions of reproducibility and replicability, while inferential reproducibility includes the NSF definition of generalizability and also the notion of different researchers reaching the same conclusion following reanalysis of the original study materials. Other approaches focus on methods by distinguishing between empirical, statistical, and computational reproducibility or specifying that replications can be direct or conceptual.
No really, what actually is reproducibility?
It’s everything.
The deeper we dive into defining “reproducibility”, the muddier the waters become. In some contexts, the term refers to very specific practices related to authenticating the results of a single experiment. In other contexts, it describes a range of interrelated issues related to how research is conducted, published, and interpreted. For this reason, I’ve started to move away from explicitly invoking the term when I talk to researchers. Instead, I’ve tried to frame my various research and outreach projects in terms of how they relate to fostering good research practice.
To me, “reproducibility” is about problems. Some of these problems are technical or methodological and will evolve with the development of new techniques and methods. Some of these problems are more systemic and necessitate taking a critical look at how research is disseminated, evaluated, and incentivized. But fostering good research practice is central to addressing all of these problems.
Especially in my current role, I am not particularly well equipped to speak to if a researcher should define statistical significance as p < 0.05, p < 0.005, or K > 3. What I am equipped to do is to help a researcher manage their research materials so they can be used, shared, and evaluated over time. It’s not that I think the term is not useful, but the problems conjured by reproducibility are so complex and context dependent that I’d rather just talk about solutions.
Resources for understanding reproducibility and improving research practice
Goodman A., Pepe A, Blocker A. W., Borgman C. L., Cranmer K., et al. (2014) Ten simple rules for the care and feeding of scientific data. PLOS Computational Biology 10(4): e1003542.
Ioannidis J. P. A. (2005) Why most published research findings are false. PLOS Medicine 2(8): e124.
Kitzes, J., Turek, D., & Deniz, F. (Eds.). (2017). The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences. Oakland, CA: University of California Press.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
Wilson Gl, Bryan J., Cranston K., Kitzes J., Nederbragt L., et al. (2017) Good enough practices in scientific computing. PLOS Computational Biology 13(6): e1005510.
From Brain Blobs to Research Data Management
If you spend some time browsing the science section of a publication like the New York Times you’ll likely run across an image that looks something like the one below: A cross section of a brain covered in colored blobs. These images are often used to visualize the results of studies using a technique called functional magnetic resonance imaging (fMRI), a non-invasive method for measuring brain activity (or, more accurately, a correlate of brain activity) over time. Researchers who use fMRI are often interested in measuring the activity associated with a particular mental process or clinical condition.
Because of the size and complexity of the datasets involved, research data management (RDM) is incredibly important in fMRI research. In addition to the brain images, a typical fMRI study involves the collection of questionnaire data, behavioral measures, and sensitive medical information. Analyzing all this data often requires the development of custom code or scripts. This analysis is also iterative and cumulative, meaning that a researcher’s decisions at each step along the way can have significant effects on both the subsequent steps and what is ultimately reported in a presentation, poster, or journal article. Those blobby brain images may look cool, but they aren’t particularly useful in the absence of information about the underlying data and analyses.
In terms of both the financial investment and researcher hours involved, fMRI research is quite expensive. Throughout fMRI’s relatively short history, data sharing has been proposed multiple times times as a method for maximizing the value of individual datasets and for overcoming the field’s ongoing methodological issues. Unfortunately, a very practical issue has hampered efforts to foster the open sharing of fMRI data- researchers have historically organized, documented, and saved their data (and code) in very different ways.
What we are doing and why
Recently, following concerns about sub-optimal statistical practices and long-standing software errors, fMRI researchers have begun to cohere around a set of standards regarding how data should be collected, analyzed, and reported. From a research data management perspective, it’s also very exciting to see that there is also an emerging standard regarding how data should be organized and described. But, even with these emerging standards, our understanding of the data-related practices actually employed by fMRI in the lab and how those practices relate to data sharing and other open science-related activities remains mostly anecdotal.
To help fill this knowledge gap and hopefully advance some best practices related to data management and sharing, Dr. Ana Van Gulick and I are conducting a survey of fMRI researchers. Developed in consultation with members of the open and reproducible neuroscience communities, our survey asks researchers about their own data-related practices, how they view the field as a whole, their interactions with RDM service providers, and the degree to which they’ve embraced developments like registrations and pre-prints. Our hope is that our results will be useful for both the community of researchers who use fMRI but and for data service providers looking to engage with researchers on their own terms.
If you are a researcher who uses fMRI and would like to complete our survey, please follow this link. We estimate that the survey should take between 10 and 20 minutes.
If you are a data service provider and would like to chat with us about what we’re doing and why, please feel free to either leave a comment or contact me directly.
Building a Community: Three months of Library Carpentry.
Back in May, almost 30 librarians, researchers, and faculty members got together in Portland Oregon to learn how to teach lessons from Software, Data, and Library Carpentry. After spending two days learning the ins and outs of Carpentry pedagogy and live coding, we all returned to our home institutions, as part of the burgeoning Library Carpentry community.
Library Carpentry didn’t begin in Portland, of course. It began in 2014 when the community began developing a group of lessons at the British Library. Since then, dozens of Library Carpentry workshops have been held across four continents. But the Portland event, hosted by California Digital Library, was the first Library Carpentry-themed instructor training session. Attendees not only joined the Library Carpentry community, but took their first step in getting certified as Software and Data Carpentry instructors. If Library Carpentry was born in London, it went through a massive growth spurt in Portland.
Together, the carpentries are a global movement focused on teaching people computing skills like navigating the Unix Shell, doing version control with Git, and programming with Python. While Software and Data Carpentry are focused on researchers, Library Carpentry is by and for Librarians. Library Carpentry lessons include an introduction to data for librarians, Open Refine, and many more. Many attendees of the Portland instructor training contributed to these lessons during the Mozilla Global Sprint in June. After more than 850 Github events (pull requests, forks, issues, etc), Library Carpentry ended up as far and away the most active part of the global sprint. We even had a five month old get in on the act!
Since the instructor training and the subsequent sprint, a number of Portland attendees have completed their instructor certification. We are on track to have 10 certified instructors in the UC system alone. Congratulations, everyone!
Building an RDM Guide for Researchers – An (Overdue) Update
It has been a little while since I last wrote about the work we’re doing to develop a research data management (RDM) guide for researchers. Since then, we’ve thought a lot about the goals of this project and settled on a concrete plan for building out our materials. Because we will soon be proactively seeking feedback on the different elements of this project, I wanted to provide an update on what we’re doing and why.
Communication Barriers and Research Data Management
Several weeks ago I wrote about addressing Research Data Management (RDM) as a “wicked problem”, a problem that is difficult to solve because different stakeholders define and address it in different ways. My own experience as a researcher and library postdoc bears this out. Researchers and librarians often think and talk about data in very different ways! But as researchers face changing expectations from funding agencies, academic publishers, their own peers, and other RDM stakeholders about how they should manage and share their data, overcoming such communication barriers becomes increasingly important.
From visualizations like the ubiquitous research data lifecycle to instruments like the Data Curation Profiles, there are a wide variety of excellent tools that can be used to facilitate communication between different RDM stakeholders. Likewise, there are also discipline-specific best practice guidelines and tools like the Research Infrastructure Self Evaluation Framework (RISE) that allow researchers and organizations to assess and advance their RDM activities. What’s missing is a tool that combines these two elements that enables researchers the means to easily self-assess where they are in regards to RDM and allows data service providers to provide easily customizable guidance about how to advance their data-related practices.
Enter our RDM guide for researchers.
Our RDM Guide for Researchers
What I want to emphasize most about our RDM guide is that it is, first and foremost, designed to be a communication tool. The research and library communities both have a tremendous amount of knowledge and expertise related to data management. Our guide is not intended to supplant tools developed by either, but to assist in overcoming communication barriers in a way that removes confusion, grows confidence, and helps people in both communities find direction.
While the shape of RDM guide has not changed significantly since my last post, we have refined its basic structure and have begun filling in the details.
The latest iteration of our guide consists of two main elements:
- A RDM rubric which allows researchers to self-assess their data-related practices using language and terminology with which they are familiar.
- A series of one page guides that provide information about how to advance data-related practices as necessary, appropriate, or desired.
The rubric is similar to the “maturity model” described in my earlier blog posts. In this iteration, it consists of a grid containing three columns and a number of rows. The leftmost column contains descriptions of different phases of the research process. At present, the rubric contains four such phases: Planning, Collection, Analysis, and Sharing. These research data lifecycle-esque terms are in place to provide a framing familiar to data service providers in the library and elsewhere.
The next column includes phrases that describe specific research activities using language and terminology familiar to researchers. The language in this column is, in part, derived from the unofficial survey we conducted to understand how researchers describe the research process. By placing these activities beside those drawn from the research data lifecycle, we hope to ground our model in terms that both researchers and RDM service providers can relate to.
The rightmost column then contains a series of declarative statements which a researcher can use to identify their individual practices in terms of the degree to which they are defined, communicated, and forward thinking.
Each element of the rubric is designed to be customizable. We understand that RDM service providers at different institutions may wish to emphasize different services toggled to different parts data lifecycle and that researchers in different disciplines may have different ways of describing their data-related activities. For example, while we are working on refining the language of the declarative statements, I have left them out of the diagram above because they are likely the rubric that will remain most open for customization.
Each row within the rubric will be complemented by a one page guide that will provide researchers with concrete information about data-related best practices. If the purpose of the rubric is to allow researchers to orient themselves in RDM landscape, the purpose of these guides is to help them move forward.
Generating Outputs
Now that we’ve refined the basic structure of our model, it’s time to start creating some outputs. Throughout the remainder of the summer and into the autumn, members of the UC3 team will be meeting regularly to review the content of the first set of one page guides. This process will inform our continual refinement of the RDM rubric which will, in turn, shape the writing of a formal paper.
Moving forward, I hope to workshop this project with as many interested parties as I can, both to receive feedback on what we’ve done so far and to potentially crowdsource some of the content. Over the next few weeks I’ll be soliciting feedback on various aspects of the RDM rubric. If you’d like to provide feedback, please either click through the links below (more to be added in the coming weeks) or contact me directly.
Provide feedback on our guide!
More coming soon!