I don't think you actually lose any science if you transmit stuff back. Based on what I'm seeing, you've got a number that's the total amount of science a given part can give you for a given location.
Example(numbers might be off slightly, doing it from memory, but the theory is correct as I've watched it happen in several places):
Using the Materials Lab in orbit of the Mun can get you 100 science total. You can use the lab and then return it to Kerbin for all 100 science, or your can transmit for 20% of that, which would be 20 science. The other 80 isn't lost, it's still there since you can transmit again for another 20% of that 80 which gives you 16 science and leaves 64. Transmit again for 20% of that and you get 12.8 science, leaving you 51.2 science. You can keep doing this, shaving 20% off the remaining science each time, for as long as you have power available. Now, if you keep doing it it will inevitably leave you with a small remnant of science, since you can never collect it ALL if you just keep shaving 20% off of it, but if you transmit enough, you can end up with a number of remaining science left that's so small it'd be thrown away by the game anyway when rounding numbers.
It is tedious, requires a lot of clicking and requires some sort of power source to keep your batteries charged through all this inefficient transmitting but you never REALLY lose anything and it allows you to collect the science from multiple places with one vehicle. I just sent a probe to the Mun which got high mun orbit, low mun orbit and landed in a crater on the far side and researched that. With both the materials lab and the goo canister, it was very profitable science wise.
EDIT: I should have mentioned, I have strong suspicions that whatever science you DON"T transmit back in a given mission, can simply be grabbed in a later mission either through return samples or transmission. More tests will have to be done on this though.
If that were the way it worked, then why did I get more science on my second mun flyby with the material bay? I should have gotten all of the science available, no?
I returned the first one to KSP for 100% of the science but 2nd trip out I got more.
I haven't figured out the exact math, nor do I expect to on my own before someone else does, but it seems as if on your second and possibly even third flights through an area after cleaning the science out, it loads the area up with just a little bit of science to give you a chance to get something. Once you've cleaned it out multiple times(guess is 3, but number might be different or dependent on something else) you stop getting science. And I'm not sure if it's a pure cleanout rather than a situation where if the science in an area is lower than some threshold, add more science once. If it drops again, add again, but less this time. Again, no serious studies done on this, just guesses from observations.
In other news, I accidentally my whole spaceship and Jeb ended up on the surface of the Mun without one. About 10km up from a landing I hit space one too many times and ended up with a free falling capsule. For a moment I panicked and figured my lead pilot was dead. After regaining my senses I realized I still had a chance with the EVA suit thrusters. It wasn't pretty and I probably could have done it more efficiently but I managed to get Jeb down from 10km and 200 m/s of velocity to a nice soft landing on a crater rim with a whole 2.5% of my fuel left.
This... oddly enough matches up well with a problem I had before. I'd flown another nameless Kerbal to the Mun first, landed successfully, but he managed to kill himself playing around with his own EVA thrusters. So now I have a Kerbal without a rocket, and a rocket without a Kerbal. They just happen to be 1/4 of the moon apart in two completely different large craters.
Lesson learned from this: EVA thrusters are surprisingly powerful? or Don't let Jeb fly alone.
EDIT: Some nice sounding facts and numbers from the official forums.
It seems that the reduction of science on repeated experiments is actually different depending on which part is used.
For example 5/9 of the science (recovered or transmitted doesn't matter) you get from the mystery goo gets substracted from the amount you will receive for the same experiment the next time. I shot a capsule with six goo canisters into space an used them all at around the dame time. The science i got from this was: 10.0/4.4/2.0/0.9/0.4/0.2.
As you can see, you can calculate the amount of science for each canister by multiplying the amount from the previous one with 1-5/9=4/9 (numbers in KSP are rounded to one decimal).
Other numbers I found so far for reduction per one point of science:
Crew Report: 5/8
EVA Report: 3/4
Soil Sample: 4/5
Material Studies/Science Jr.: 5/7
Author
Topic: Kerbal Space Program: Now Hiring Optimistic Astronauts for Dangerous Munission (Read 1507439 times)