To avoid connection errors, it's important to distinguish between insert and aux effects. Their operation is different, so their integration in the audio path is different too. That's why it's important to distinguish inserts from auxiliaries to process a signal.
A mixing desk is like a road network. Users (audio signals) on this network travel either with their own vehicle (audio channels) or with public transport (audio bus). As a consequence, a channel hosts only one signal while a bus can transport many signals at once.
When a signal leaves its channel, it is sent to a bus (subgroup or master) where it is mixed with other signals. By analogy, when a traveler leaves his own vehicle, he travels on by taking public transport (bus). So an audio bus is an audio channel that hosts several signals coming from other channels.
Insert: a detour
Channel inserts are usually between the preamp stage and the EQ section in the input channels of an analog mixer. The same applies in digital desks and virtual mixers. An insert diverts the channel signal (insert send) to a processor or a chain of processors. This means the audio signal flow in the channel is interrupted. When it comes out of the last processor in the chain, the signal is brought back into the mixing channel (insert return) at the exact same point where it left. And the signal then keeps its normal flow all the way through the audio channel.
Important note for later: audio buses in mixing desks are equipped with inserts that work exactly the same as channel inserts.
Aux: an alternate route
As we have already seen, an insert changes the course of a channel signal but it also brings it back to the channel afterwards. On the other hand, auxiliary buses come after the inserts on the signal path and they provide an alternate route that the signal can take while staying on the normal channel path at the same time. So the signal takes two paths simultaneously: the audio channel and the aux.
An aux is a bus, which means that it can host signals coming from several audio channels. It is fed by the Aux or FX knobs on the channels, which control the level of the channel signal sent to the aux bus. The aux bus is equipped with an insert where you can connect one or several processors to affect all signals in the bus.
Finally, the normal route (channel) and the alternate route (aux) usually come together at the output bus of the mixer (master or main). This means the channel and effect signal coming from the aux bus are summed there.
The police regulates the traffic…
As you already know, inserts are a detour from the channel, so they have a direct impact on the signal. That’s why inserts are usually used for effects that change the waveform of the audio signal, like dynamic processors (gate/expander, compressor/limiter), saturation/distortion effects, bitcrushers, de-essers, filters (EQs, wah effects, etc.), audio restoration tools, and psycho-acoustic processors (harmonic generators, stereo width processors, etc.).
We’ve also mentioned that an aux adds an effect without actually changing the source signal, which makes them ideal for effects that are mixed with the original signal. We are talking about “acoustic generators” (reverb, delay, echo, etc.), pitch effects (harmonizer, octaver, pitch-shifter, etc.) and modulation effects (chorus, flanger, phaser, etc.).
A special remark: if you have the choice, use a post-fader aux so that the effect amount (ratio between effect signal and direct signal) stays the same, regardless of the fader position in the channel.
And also note how things are well done: Aux are nothing more than a bus, and this is a good thing because they often host effects like reverbs, delays, etc. Moreover, inserts are placed in the input channels, and this is also a good thing since they are usually used to process a single signal. And, if needed, the bus inserts (subgroups and master) allow you to process several signals simultaneously with a dedicated insert (a drum kit, guitar tracks, backing vocals, room mics, etc.).
… users reinvent the road
The distinction between insert effects and aux effects is not always that obvious: instead of inserting a distortion (like you would with an electric guitar), you can connect your distortion effect to an aux to keep the original signal and add a distorted signal, for example for a bass guitar, e-piano, snare, etc.
And to increase your possibilities, most aux effects feature a dry/wet control to mix the original and processed signals. So why not try to connect an aux effect to an insert in a channel or subgroup and adjust the effect amount with the dry/wet controls? However, keep in mind that inserting a processor, especially a hardware one, may color the sound, even if the signal isn’t actually being processed.
Fans of parallel compression will acknowledge that sometimes it’s a good idea to break the rules. For example you can put a compressor in an aux (why not?) to see how the mix of direct and compressed signals sounds.