Just my thoughts on the issue :-
Fiction : "The Mote in God's Eye" has a millions year old technological civilization with specialized tech castes. Sort of like Orks without the stupid. A must-read for anyone looking for ideas for this sort of scenario (I would describe more relevant details, but it's a great read so no spoilerz 4 u)
Effects of Inbreeding: This does not have quite the effect that you think, excessive in-breeding actual weeds-out the harmful recessive genes. Consider the scenario below for a single gene location (A = good gene copy, B = bad gene copy)
First Generation (M = male, F - female)
M : AB, F : AA ; % gene B = 25%
Second Generation : Each child has 50% chance of getting gene B, say they have 4 kids :
M1 : AB, M2 : AA , F1 : AB, F2 : AA ; % gene B = 25%
Third Generation : A worst case scenario of M1 breeding with F1 (C = Couple)
C1(M1,F1) : Children AA, 2xAB, BB ; % gene B = 50%
C2(M2,F2) : Children 4xAA ; % gene B = 0%
Having 2 pairs of gene B will be fatal/guarantee non-breeding. "BB" will be weeded out of the gene pool, thus reducing the % of the population with gene 'B' below the 25% baseline. Basically this shows that 25% of the population carrying the same fatal recessive trait is not sustainable. Those traits are rare for a reason, because they are harmful. A recessive trait which is beneficial will come to dominate the population over time (harmful, dominant traits get weeded out very quickly).
Harmful recessive genes survive longer in the gene pool when they are not expressed. They could be increased due to random fluctuations (which will be more pronounced for bottleneck populations), but as likely, if not more so, to be reduced under those same circumstances.
You can extend the idea to the 30-or-so harmful/recessive traits that each of us carries (thankfully there are billions of gene loci, so the odds of marrying someone with matching bad genes is rare, then it's still only a 1/4 chance of getting that gene from both parents).
With inbreeding, and 30+ bad traits, it'd be hard to avoid any children in the third generation from a single source-couple having some defective traits, which would seem to back up the idea of racial genetic weakness stemming from that source. But the problem with that is that there's nothing to stop future mutations from changing the defective gene back to one of the better versions. And as the defect is a recessive gene, those mutations back to the better dominant gene will show up immediately, giving the individual with the mutant gene a strong survival advantage over it's siblings. So the bad traits may become 'standard' amongst a population for a time, but they're very unstable.
Your best bet for this sort of thing is something like sickle-cell anaemia, where carriers of one copy of the gene gain some advantage over those carrying no copies, but carrying one of two copies of the gene causes some other harm (it still has to convey an over-all increased chance of having offspring). e.g. greater physical strength or say protection from vacuum vs reduced brain size / intellgence.