This is, what I believe to be, an accurate explanation of the so called "511 Glitch" on the NTSC version of Star Fox 64. This explanation may not be 100% accurate in every aspect, and further inspection into the programming of the game may reveal further answers, but this explanation should be sufficient for the time being. So without further ado, let us venture into the mysterious realm of the 511 Glitch.
To understand what I am trying to say here, you need to first understand how a game is programmed. Basic computer programming is in binary, which is a base-2 number system that is used to program computers. For example, 1 in binary is 1, 10 in binary is 2, 11 in binary is 3, 100 in binary is 4, 101 in binary is 5, etc. In computer programming, each of these digits is called a "bit". Each bit's value can be either 0 or 1. A byte is a set of 8 bits that form one "chain" of numbers. Typically these bytes are counted separately from other bytes and divided by spaces.
However, programming with binary can be extremely tedious, and can take quite a bit of time. Because of this, a different system has been created to make programming in binary simpler: Hexadecimal. Basically, hexadecimal is a base-16 system that contains the digits 0-9, A, B, C, D, E, and F. The letter A represents 10, B represents 11, C represents 12, D represents 13, E represents 14, and F represents 15. Therefore, the number 10 in hexadecimal is actually the number 16. Hexadecimal simplifies things by making each set of 2 digits a byte, and each digit representing 4 bits of binary code.
In video games, different "limits" of how many bits can be used have been formed. Originally, each value in the game could only be stored in a single byte. These games were known as "8-bit" games, as each value in the game could only be stored on 8 bits of data (meaning the maximum possible value for each variable could only be 255 since 11111111 in binary is 255 in decimal). However, in years following these limits have been reduced, and the current limit is 128 bits per variable, used by systems such as the Sega Dreamcast, Nintendo Gamecube and Nintendo Wii, PS2 and PS3, and the XBOX and XBOX360 consoles. Since Star Fox 64 is a 64-bit game, each address can store data up to 64 bits (meaning a possible maximum of 18446744073709551616 in decimal).
On Star Fox 64, the counter used for points is likely a 16-bit value (meaning that, in actuality, the possible maximum for the hit counter would be 65535 if it didn't max out at 999). However, if you go over 511 points on any level on Star Fox 64, it passes a limit that is caused by the 64-bit processor of the game. The number 256 in hexadecimal is 100, and since 256 + 255 = 511, adding 0xFF (255 in hex) to 0x100 (256 in hex) will get you 1FF, which is 511 in decimal. Now, here's where things get complicated. The number 511 in binary is 00000001 11111111 (expressed using 2 bytes). Now, since the most Star Fox 64 can go up to is 64 bits, each level's score on the rankings screen needs to be expressed in only 9 bits, because 63 / 7 = 9 (since 64 is not divisible by 9 it needs to be converted to 9 bits for each level with a remainder of 1 bit that is not used at all). Now, if you go over 511, the score counter goes to 512, which in binary is 00000010 00000000, which uses 10 bits. Since each level can only use 9 bits for the rankings screen, it needs to drop one bit. Basically, what it does is, it takes away the 10th bit (counting from right to left), and replaces the 9th bit (again counting from right to left) with it. This changes the value into 00000001 00000000, which is 256. If you were to get, say, 655 in area 6, the binary value would be 00000010 10001111. The game would drop the 10th bit from the right, and replace the 9th bit from the right with it, changing the value to 00000001 00001111, or 399. Now, if you were to get 768 in area 6 (only possible with tool-assistance), the binary value would be 00000011 00000000. The 10th bit from the right would again be dropped at the rankings screen, and the 9th bit from the right would be replaced with it, changing the value to 00000001 00000000, or 256. This explains why it subtracts 256 until the number is below 512.
Now, if you were to get an absurdly high score of over 1023 (impossible due to the 999 limit but sometimes possible with a gameshark code), the binary value would be 11 bits (00000100 00000000). When this happens, the game drops the 10th and 11th bits from the right, and replaces the 9th bit from the right with the 11th bit from the right. This would change the 1024 value (00000100 00000000) to 00000001 00000000, or 256. This, along with the 768+ scenario from the above paragraph, explain why the game will always subtract 256 until the value is below 512. In later versions of Star Fox 64, the score counter was forced to cap out at 511 so that this subtraction would not need to happen.
I hope that those of you who haven't suffered an aneurism due to the immense amount of information in this essay have enjoyed it, and have found it interesting to learn about this mysterious glitch that has plagued NTSC players since the time of the game's release.
-Steven Brown }