In future, look at the errorlog.txt file that's put in your base DF folder (with the executable). If it doesn't help you fix stuff, including it should help other people fix stuff.
For the sizes, I think the reason the RELSIZE isn't working is because you've added the claws by layering the claw tissue ontop of the hands, rather than adding the claws as a separate body part (and I believe RELSIZE only works on body parts).
To fix this, you could add a custom body part definition - make a file called body_custom.txt in your objects folder and put this inside it:
body_custom
[OBJECT:BODY]
[BODY:3HAND_CLAW][
[BP:3HAND_CLAW:claw:STP][CON_CAT:HAND][SMALL][SKELETON][CATEGORY:CLAW][NUMBER:3]
[DEFAULT_RELSIZE:300]
(or whatever RELSIZE size you want it to be)
Then, remove this from your creature file:
[TISSUE_LAYER:BY_CATEGORY:HAND:CLAW:FRONT]
And then add :3HAND_CLAW to the end of the body definition.
Then, get rid of the old claw attack and replace it with this:
[ATTACK:SCRATCH:BODYPART:BY_CATEGORY:CLAW]
[ATTACK_SKILL:GRASP_STRIKE]
[ATTACK_VERB:scratch:scratches]
[ATTACK_CONTACT_PERC:100]
[ATTACK_PENETRATION_PERC:100]
[ATTACK_FLAG_EDGE]
[ATTACK_PREPARE_AND_RECOVER:3:3]
[ATTACK_PRIORITY:MAIN]
[ATTACK_FLAG_CANLATCH]
Theoretically this should work but I'm basically as new as you so don't put too much credence to what I say