Okay, I get what the problem is. Would it be possible to keep track of the previous state in a smaller format using some form of CRC or something?
The short answer is no, unfortunately. A CRC (or another hash function) will, after scanning both data sets, tell you if the new data and old data are different. BUT, you have to scan the new data first, which defeats the purpose for us. Furthermore, if there were changes, it doesn't tell you what changed, only that changes occurred.
That's more useful than it might sound, even if it doesn't solve the problem we're talking about. It's often faster do this than to directly compare everything in each of the data sets. More importantly, it gives you a quick and dirty way of making sure any sized amounts of data come through a network intact -- you send the data's hash along with the data, then have the receiving computer recompute the hash value on the data it receives. If the computed hash matches the hash you sent, the receiver can be confident that the data came through uncorrupted. If it doesn't match, the receiver knows some corruption occurred, and can send a message back asking the sender to send the data again. This is useful for individual network packets all the way up to entire files. Since hash values are very small, the scheme provides a very bandwidth-efficient way of making networking a lot less frustrating than it would be if computers weren't constantly doing this. There are other uses for hash functions besides network verification, but I'm already going way off topic.