Fog (N64) [Hack64 Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======== Fog (N64) ========

{{ https://i.imgur.com/Hh1JQVH.png?280}}

**Vertex Fog** is a setting in the RSP that can be used to fade polygons into the background. It works by gradually blending the fog color (or transparency) into the geometry until it looks like it completely faded away. This effect can be used along with lowering the draw distance to reduce lag.

Do note that you cannot use vertex alpha when fog is enabled. This is because when the SHADE alpha value is used in the color combiner, it will get the fog alpha instead of the vertex alpha.

----

===== Enabling Fog =====

To enable fog you first need to make sure that the RDP is set to 2 cycle mode:

  Fast3D / Fast3DEX
  BA 00 14 02 00 10 00 00

  Fast3DEX2 / Fast3DZEX 
  E3 00 14 02 00 10 00 00

  C/C++
  gsDPSetCycleType(G_CYC_2CYCLE)
  
Next we need to blend the fog in AA ZB mode. (see: http://n64devkit.square7.ch/n64man/gdp/gDPSetRenderMode.htm)

  Fast3D / Fast3DEX
  B9 00 03 1D C8 11 20 78 (If surface is fully opaque)
  B9 00 03 1D C8 11 30 78 (If surface has transparency)

  Fast3DEX2 / Fast3DZEX 
  E2 00 03 1D C8 11 20 78 (If surface is fully opaque)
  E2 00 03 1D C8 11 30 78 (If surface has transparency)

  C/C++
  gsDPSetRenderMode(G_RM_FOG_SHADE_A, G_RM_AA_ZB_OPA_SURF2) // If surface is fully opaque
  gsDPSetRenderMode(G_RM_FOG_SHADE_A, G_RM_AA_ZB_XLU_SURF2) // If surface has transparency
  
After that we now need to tell the RDP we want to use fog, so we enable the geometry mode flag for fog:

  Fast3D / Fast3DEX
  B7 00 00 00 00 01 00 00

  Fast3DEX2 / Fast3DZEX 
  D9 00 00 00 00 01 00 00

  C/C++
  gsSPSetGeometryMode(G_FOG)

----

===== Setting Fog Distance =====

The fog itself works in a range from a minimum to a maximum. Min can go as low as 0 (at near clip plane), and max can go as high as 1000 (at far clip plane). You can adjust these values to get your desired effect.

  Fast3D / Fast3DEX
  BC 00 00 08 AA AA BB BB
  
  AAAA = 128000 / (max - min)
  BBBB = (500 - min) * 256 / (max - min)

  Fast3DEX2 / Fast3DZEX 
  DB 00 00 08 AA AA BB BB
  
  AAAA = 128000 / (max - min)
  BBBB = (500 - min) * 256 / (max - min)

  C/C++
  gsSPFogPosition(min, max)
  
For those who have used the SM64Editor tool, here are the corresponding (min, max) values for each fog setting:
  SUBTLE_1     = (980, 1000)
  SUBTLE_2     = (948, 976)*
  MODERATE_1   = (965, 1000)
  MODERATE_2   = (960, 1000)
  MODERATE_3   = (950, 1000)
  MODERATE_4   = (940, 1000)
  INTENSE      = (930, 1000)
  VERY_INTENSE = (900, 1000)
  HARDCORE     = (716, 932)*
  
  * = Not exact, but its as close as I could get.

----

===== Setting Fog Color =====

The fog color is a simple RGBA value that you define in the display list. Generally, this should be the same color as the background. If the background is complicated (clouds, mountains, gradients, etc), [[http://ultra64.ca/files/documentation/online-manuals/man/pro-man/pro11/index11.8.html|then you should try making the objects fade by transparency rather than by color]].

  Fast3D / Fast3DEX / Fast3DEX2 / Fast3DZEX 
  F8 00 00 00 RR GG BB AA

  C/C++
  gsDPSetFogColor(RED, GREEN, BLUE, ALPHA)

----

===== Disabling Fog =====

To disable fog, you simply have to do the reverse of what you did to enable it. This can vary based on what the default settings are in each game. For custom Super Mario 64 levels, we do the following:

1.) Set the RDP back into 1 cycle mode.

  Fast3D / Fast3DEX
  BA 00 14 02 00 00 00 00

  Fast3DEX2 / Fast3DZEX 
  E3 00 14 02 00 00 00 00

  C/C++
  gsDPSetCycleType(G_CYC_1CYCLE)
  

2.) Next we set the render mode back to normal. (see: http://n64devkit.square7.ch/n64man/gdp/gDPSetRenderMode.htm)

  Fast3D / Fast3DEX
  B9 00 03 1D 00 44 30 78

  Fast3DEX2 / Fast3DZEX 
  E2 00 03 1D 00 44 30 78

  C/C++
  gsDPSetRenderMode(G_RM_AA_ZB_OPA_SURF, G_RM_NOOP2) // If surface is fully opaque
  gsDPSetRenderMode(G_RM_AA_ZB_XLU_SURF, G_RM_NOOP2) // If surface has transparency

3.) Finally we clear the fog flag in the geometry mode.
  Fast3D / Fast3DEX
  B6 00 00 00 00 01 00 00

  Fast3DEX2 / Fast3DZEX 
  D9 01 00 00 00 00 00 00

  C/C++
  gsSPClearGeometryMode(G_FOG)

----

===== References =====
  * http://ultra64.ca/files/documentation/online-manuals/man/pro-man/pro11/index11.8.html
  * http://ultra64.ca/files/documentation/online-manuals/man/n64man/gdp/gDPSetFogColor.html
  * http://n64devkit.square7.ch/n64man/gdp/gDPSetRenderMode.htm