Prime = divisible by exactly two numbers, itself and 1. 1 is only divisible by one number. This is useful when you need to find the factors of a number: You keep dividing out the primes. If 1 was prime, you'd keep dividing the number by 1, and you'd never be done.
And yeah, there's a ton we could gain out of communications if we restrict ourselves to math. Once you establish a unary and then binary system through primes or fibonacci or something, you can establish symbols for mathematical operations (I've seen people who already did that exercise with specific regard to SETI). Then you start firing off interesting facts at the other guy, starting with 1+1=2, going through Euler's identity (easy if you establish symbols, and easy enough to establish, say Pi: (unique symbol) (equals) (first ten digits of pi in binary) (symbol for 'continues'))
Once you get that far, start flinging physics or chem facts. A list of all the atomic weights would be a good one. See how far you can go! Who has made the highest atomic numbered element? If the other guys are up to the 200s, but they haven't made element 162, I'm sure someone can derive some interesting facts there.
You can start encoding time, and specifically, mathematical constants WRT time. Humanity establishes a time symbol for "In our understanding, the time that the universe started"; alien species does the same, but a different symbol. (Time isn't hard to represent; you'd be dealing with it anyway once you said anything involving frequencies and that would be a very early thing.) Humanity says "We think that 0.5 nanoseconds after the start of the universe, this constant was this" and the aliens respond "We think that 0.5 nanoseconds after the start of the universe, that constant was THIS", and maybe they introduce some equations to prove their points. With the signal lag, you can't have much of a conversation...but stream-of-consciousness would really work great for this.
Besides, engineers are good at understanding constants. If you say to a US engineer "What comes to mind when I say...60", he'll probably say "Hertz", and once you make a few other references, it's easy to understand what symbols are involved. You fling enough facts that are interconnected, and even with a 40 year time lag, you'll be able to make sense of things even with no time lag. And you keep updating your facts and sending them when you learn new stuff. There's a WHOLE LOT of constants that we have in common, some of them part of the universe, some of the part of the signal we send. And if you think defining 'one second' is hard, just say "9,192,631,770 (symbol 1) x Cesium = (symbol 2). 456 x (symbol 2). 456 x (symbol 2)." And pretty soon they notice "Hey, if symbol 1 refers to the period of state transitions of an atom, and symbol 2 is a quantity of time, then 456 x that symbol 2 is the frequency we're receiving on! Symbol 2 must be a constant they use."
The first thing you talk about will be science, clearly. But even then, you can start sending SOME culture. Remember, it's easiest to learn a language when you are fully immersed in it. Just start encoding some black-and-white movies or something, specifically designed to illustrate our life, and festooned with special symbols EVERYWHERE. Give them 'a week in the life of a human'. They will quickly figure out "Oh, they seem to have some activity cycle along the lines of 24 hours, that adds up with the rotation of their planet we noticed from their transmissions, that makes sense. Oh, they seem to have a basic family structure, and we can assume that there are analogues like ~hive-mother~. What small families." Establish cultural words through picture books, then start sending them more cultural information with some kind of set notation. "Okay. We keep seeing A and B together. And when we see them together, we see lots of symbol C things, too. This says that symbol D contains one A, one B, one or more C. Hey neat, family unit (oversimplified but we kind of assumed that). A and B seem to have these body parts with their own symbols. Hey, now they're breaking out the chemistry symbols. This body part on A catalyzed with this body part on B produces C. Oh hey, maybe that's reproduction."
The only hard part is going to be establishing a symbol for "fiction". VERY IMPORTANT. Probably the most time consuming part because you need to wait for a response. You want to make sure they understand what fiction is very, very well before you start telling them stories. But once you've done that, you start with Dick & Jane and you just go from there.