Okay first of all, ed boy, your reasoning and estimated score result are false, because the proportion of As touching each B is not equal to the global ratio of As to Bs. Easiest example:
AAAAA
BBBBB
AAAAA
BBBBB
AAAAA
If you expand this pattern, it has a 1:1 ratio of As to Bs, but every B touches six As instead of only four, so it gives a larger score than your S=I+J*(1+8*I/N).
Here's an easier reasoning to get a perfectly accurate result:
You get exactly the same score if you assign points like this: One point for each letter, plus one point for each pair of touching A's and B's. So what we want to do is maximize the number of heterosymbolic relationships. The problem is that we have lots of "love triangles" of the form
XY
Z
which guarantee that there is at least one homosymbolic relationship inside. There are also "love quadrangles" of the form
XY
ZW
which have at least two homosymbolic relationships inside. Our goal is to find an upper bound on the number of possible heterosymbolic relationships, and we can do this by finding a lower bound on the number of homosymbolic relationships. To do this, we'll have a look at this image:
Here I've placed some love triangles and quadrangles onto the board (in this case 4x4), and I've called them "special". Every special love n-drangle is disjoint from all others, which means that any homosymbolic relationship can only be part of at most one special n-drangle. This means that there are at least one HR per special triangle and two HRs per special quadrangle, and since there are (X-1)(X-2) special triangles and (X-1) special quadrangles, we find that we have at least X(X-1) homosymbolic relationships, regardless of our distribution of As and Bs.
Now since there are exactly (X-1)(4X-2) total relationships, we find that at most (X-1)(3X-2) relationships may be heterosymbolic at any time, so the total score of a character distribution is at most (X-1)(3X-2) + X².
To show that this score is actually possible, we simply give a distribution of As and Bs that has this score:
AAAAAA...
BBBBBB...
AAAAAA...
BBBBBB...
AAAAAA...
BBBBBB...
...
...
And that's a complete proof that (X-1)(3X-2) + X² is the maximum possible score for an X-by-X grid. This, fellas, is real mathematics.
Edit: If I may ask, Bouchart, where did you get this problem from?