The APM uses channel 5 for the flight modes. Six mixes will need to be added, one for each custom switch created. Set each mix as follows, adjusting the 'weight' and 'switch' appropriately.
Adjust the weight of each of the 6 mixes on channel 5 for each flight mode. For the APM 2.6, these are the values you should aim for when setting each mode: 992 (something significantly lower than 1230 will do), 1296, 1426, 1556, 1685, 2017 (something significantly over 1750 will do). The PPM value can be monitored in the radio calibration window in the APM Planner software.