It took me a long time to figure out the settings to get a nice looking lettuce hem on knits using my serger. So I hope this can be useful to someone else as well.
I set up my serger for a 3-threads stitch using the right needle.
Tensions: 4 - 5 - 7
Stitch lenght: R
Stitch width: 7
I find that turning up the hem allowance and sewing on the fold produces the best result (much more similar to rtw). In the photo you can see the results of the stitch on a single layer of fabric letting the machine cut off the hem allowance (top) and on the fold (bottom). Both look pretty good, but I definitely prefer the second.
I also find that pulling the fabric just slightly from the back only (and not front and back) results in a more regular stitch.