An example of one moment in that experimental process that recently surprised me comes from an experiment we are running to see how sensitive babies are to patterns in their environments. We show them two different sequences of coloured shapes that appear one at a time on a computer screen, and see which one they find more boring. In one case the order of the shapes is random. In the other, there is some pattern to the sequence (a red square always follows the blue triangle, the green circle always comes after the pink diamond, etc.) Babies seem sensitive to the fact that there is a difference between the two type of sequence.. whereas, surprisingly, to an naive adult there is no obvious difference. At least anecdotally.
We realised that we would also have to test the adults. But we cannot test them in the same way as babies, just showing them sequences of shapes and seeing when they get bored. The experiment needs to change somewhat and they need to be given some instructions. The trouble is that any instruction we give them will influence what they actually do. If we tell them to look for patterns they have more information than the babies and approach the task far too rationally. If we asked them which would a baby prefer, they bring their preconceptions about babies. We considered all kinds of presentations and possible instructions but each one could tip the participants off as to what we wanted them to do.
After considering many very elaborate testing procedures we settled on presenting the shapes almost identically to how the babies see them and giving the participants as little information as we possibly can. We hope that this final design ensures that we are measuring the same underlying psychological quantity in both adults and babies. But already know that these two experiments alone cannot prove this. Similarly, in designing the adult version of the experiment we realised that we needed to reassess and broaden our own assumptions about what the babies were doing in the original experiment.
I think this illustrates that no experiment ever exists in isolation and no experimental result can ever provide certainty. Experimental science is a continual process