2
$\begingroup$

I have to reproduce simulations of scientific articles in Matlab.

Do you have any tips for programming scientific articles? What are your methods when you have to reproduce simulations of scientific articles?

$\endgroup$
  • $\begingroup$ I suggest to take a look at IEEE's new site, codeocean.com $\endgroup$ – MimSaad Apr 8 '18 at 17:04
  • $\begingroup$ Can I please ask if this was resolved? $\endgroup$ – A_A Apr 30 '18 at 8:15
3
$\begingroup$

Unfortunately, there is accumulating evidence on lack of reproducibality of results across different fields (for example, for another example, for yet another example).

As a response to that, professional bodies and research teams are publishing guidelines for reproducible research and you might have to search for something like that in your field. It would be great to cite something along the lines of "...in our work we have tried to take on board the Reproducible Research guidelines published by...[citation] wherever it was possible".

Some of those are as follows:

And of course, on the wake of these guidelines, there are projects like the Open Science Framework which try to provide implementations to guidelines and propagate best practices.

While you can read lots of articles, about these things, the point is to create such conditions that demonstrate clearly how you arrived to your results. And this has to be done in such a way that it does not require your input. You don't have to be there to explain anything (this is also what the scientific papers are supposed to be about).

This is very powerful for three reasons:

  1. Obviously, if there is any doubt by anyone about your results, they can simply try to repeat everything themselves or, if possible, simply download a virtual machine with "the whole lab" already set up and go through your whole process.

  2. If you have made some sort of mistake, it is easy for someone to point it out and it is also a way to demonstrate that this research was performed with the best knowledge at the time it was conducted.

    • If you have made some sort of mistake and someone else spots it everybody wins.
  3. If you cannot reproduce the results then you simply say "Here is the whole setup, here are all of my workflows and data, here is how the experiments were setup (if you cannot share the data directly). I have tried my best to reproduce these results but so far I have not been able to do so".

This last point is probably more important for what you are trying to do because sometimes, there is a lot of frustration as to why "our numbers don't satisfactorily match XYZ's numbers".

Hope this helps.

$\endgroup$
2
$\begingroup$

Reproduction is a complicated issue. I recently came across the interesting Basic Statistical Issues for Reproducibility: Models, Variability, Extensions.

Main factors are:

  • knowledge on the premises for reproducibility by the authors (I personally have never been taught about that on my studies)
  • access to data and its pre-processing
  • sufficiently detailed algorithmic implementation
  • algorithm parameter determination
  • computational power (difficult to reproduce a super-computer implementation on a laptop)
  • performance metrics
  • mistaking or cheating: some paper results are flawed, and you cannot reproduce them

Steps would be:

  • Check whether the code and data are openly available
  • If not, ask the authors whether the would share them
  • If not possible, build your code as if you were the author of the paper and wanted to share it
  • Test your code to the authors. That could make them more prone to share theirs
  • Share it, with due reference, and you can benefit from others testing it.
$\endgroup$
  • 1
    $\begingroup$ Generally, especially for simulations, I always wonder why authors would not share their code right away - reproducibility is key to the scientific method. $\endgroup$ – Marcus Müller Apr 8 '18 at 1:50
  • $\begingroup$ One reason might be the conditions and funding under which the research has been made. A second one is the competition. I hope that sheer dishonesty is not the third reason $\endgroup$ – Laurent Duval Apr 8 '18 at 9:26
  • 1
    $\begingroup$ I guess the first one is probably rather hurtful, emotionally, for most researchers. The second: Can't really say it's a scientific publication if you don't let the others in your field actually know what you know! The third: for most parts, I don't think that is the reason. What I can imagine is that code written for a one-off simulation ... Might not be the best example of high-quality code, so that researchers are afraid to show weaknesses $\endgroup$ – Marcus Müller Apr 8 '18 at 11:54
  • 1
    $\begingroup$ This discussion could be rather interesting. I do have shared codes, some I need time to recode to share unashamed, codes I can't for contractual reasons, codes I've lost. Same for the data. With emotion and will, you DO raise an important issue $\endgroup$ – Laurent Duval Apr 8 '18 at 12:03
0
$\begingroup$

That's a great issue both for the readership of any paper, but also for it's authors, allowing for a better validation and dissemination of results.

My tips are:

  • use jupyter notebooks to display all results. These are available for many different languages and may render on any OS that I know off, check out https://jupyter.org/. My strategy is to have one notebook per figure. In addition, I like to show variants of the figure showing off the things that were unnecessary to pout in the paper itself. These notebooks go in one folder notebooks, for instance in this paper or that one

  • put the main scripts in one separate folder with installation instructions. In general, I test the whole installation on a clean and empty virtual machine to be sure not a single bit of code or exotic dependency could be missing to people wanting to reproduce the code.

  • use version control on a public webserver for posterity. For instance, a popular choice is https://github.com/ : this makes the code accessible to the widest community (using git, zip, ...). In addition, github renders notebooks, such that you do not have to run the code if you just need a preview, see github rendering a notebook.

  • advertise your work : tweet it, blog it, send e-mails. The more people will read your code, the more bullet-proof it will become.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.