Vertex Fog is a setting in the RSP that can be used to fade polygons into the background. It works by gradually blending the fog color (or transparency) into the geometry until it looks like it completely faded away. This effect can be used along with lowering the draw distance to reduce lag.

Do note that you cannot use vertex alpha when fog is enabled. This is because when the SHADE alpha value is used in the color combiner, it will get the fog alpha instead of the vertex alpha.

To enable fog you first need to make sure that the RDP is set to 2 cycle mode:

Fast3D / Fast3DEX
BA 00 14 02 00 10 00 00

Fast3DEX2 / Fast3DZEX 
E3 00 14 02 00 10 00 00

C/C++
gsDPSetCycleType(G_CYC_2CYCLE)

Next we need to blend the fog in AA ZB mode. (see: http://n64devkit.square7.ch/n64man/gdp/gDPSetRenderMode.htm)

Fast3D / Fast3DEX
B9 00 03 1D C8 11 20 78 (If surface is fully opaque)
B9 00 03 1D C8 11 30 78 (If surface has transparency)

Fast3DEX2 / Fast3DZEX 
E2 00 03 1D C8 11 20 78 (If surface is fully opaque)
E2 00 03 1D C8 11 30 78 (If surface has transparency)

C/C++
gsDPSetRenderMode(G_RM_FOG_SHADE_A, G_RM_AA_ZB_OPA_SURF2) // If surface is fully opaque
gsDPSetRenderMode(G_RM_FOG_SHADE_A, G_RM_AA_ZB_XLU_SURF2) // If surface has transparency

After that we now need to tell the RDP we want to use fog, so we enable the geometry mode flag for fog:

Fast3D / Fast3DEX
B7 00 00 00 00 01 00 00

Fast3DEX2 / Fast3DZEX 
D9 00 00 00 00 01 00 00

C/C++
gsSPSetGeometryMode(G_FOG)

The fog itself works in a range from a minimum to a maximum. Min can go as low as 0 (at near clip plane), and max can go as high as 1000 (at far clip plane). You can adjust these values to get your desired effect.

Fast3D / Fast3DEX
BC 00 00 08 AA AA BB BB

AAAA = 128000 / (max - min)
BBBB = (500 - min) * 256 / (max - min)

Fast3DEX2 / Fast3DZEX 
DB 00 00 08 AA AA BB BB

AAAA = 128000 / (max - min)
BBBB = (500 - min) * 256 / (max - min)

C/C++
gsSPFogPosition(min, max)

For those who have used the SM64Editor tool, here are the corresponding (min, max) values for each fog setting:

SUBTLE_1     = (980, 1000)
SUBTLE_2     = (948, 976)*
MODERATE_1   = (965, 1000)
MODERATE_2   = (960, 1000)
MODERATE_3   = (950, 1000)
MODERATE_4   = (940, 1000)
INTENSE      = (930, 1000)
VERY_INTENSE = (900, 1000)
HARDCORE     = (716, 932)*

* = Not exact, but its as close as I could get.

The fog color is a simple RGBA value that you define in the display list. Generally, this should be the same color as the background. If the background is complicated (clouds, mountains, gradients, etc), then you should try making the objects fade by transparency rather than by color.

Fast3D / Fast3DEX / Fast3DEX2 / Fast3DZEX 
F8 00 00 00 RR GG BB AA

C/C++
gsDPSetFogColor(RED, GREEN, BLUE, ALPHA)

To disable fog, you simply have to do the reverse of what you did to enable it. This can vary based on what the default settings are in each game. For custom Super Mario 64 levels, we do the following:

1.) Set the RDP back into 1 cycle mode.

Fast3D / Fast3DEX
BA 00 14 02 00 00 00 00

Fast3DEX2 / Fast3DZEX 
E3 00 14 02 00 00 00 00

C/C++
gsDPSetCycleType(G_CYC_1CYCLE)

2.) Next we set the render mode back to normal. (see: http://n64devkit.square7.ch/n64man/gdp/gDPSetRenderMode.htm)

Fast3D / Fast3DEX
B9 00 03 1D 00 44 30 78

Fast3DEX2 / Fast3DZEX 
E2 00 03 1D 00 44 30 78

C/C++
gsDPSetRenderMode(G_RM_AA_ZB_OPA_SURF, G_RM_NOOP2) // If surface is fully opaque
gsDPSetRenderMode(G_RM_AA_ZB_XLU_SURF, G_RM_NOOP2) // If surface has transparency

3.) Finally we clear the fog flag in the geometry mode.

Fast3D / Fast3DEX
B6 00 00 00 00 01 00 00

Fast3DEX2 / Fast3DZEX 
D9 01 00 00 00 00 00 00

C/C++
gsSPClearGeometryMode(G_FOG)

• http://ultra64.ca/files/documentation/online-manuals/man/pro-man/pro11/index11.8.html
• http://ultra64.ca/files/documentation/online-manuals/man/n64man/gdp/gDPSetFogColor.html
• http://n64devkit.square7.ch/n64man/gdp/gDPSetRenderMode.htm