cumquat and other citrus

In spring almost two years ago I planted the spindly cumquat tree in view of the house.  I imagined the plant to be fuller, sagging with golden fruits and a pleasure to view from the house.  It is still spindly but the thin branches are sagging with golden fruits. I used to make marmalade and pickles from the cumquat in my previous garden but these fruits will remain on the tree to add colour to the garden.
My lime and lemon tree have fruit buds but I don't know if I'll be getting any fruits at all as they seem to drop off after reaching pea size.  The leaves look healthy and I wonder if there are grubs gnawing away at the tiny fruits.  I found a huge caterpillar making a big hole in one of the lime leaves.  The butterflies and moth are wonderful to look at as they flit from flower to flower but you just do not know where their eggs are and when the hungry caterpillars will be feasting on your leaves.
I don't use spray and will check the different plants that are most conducive to hosting caterpillars, such as the brocolli seedlings, which have more holes than leaves.  I have to make a habit of checking them daily to remove the pests if the plants are to survive.  The chores are never ending but when the sun shines through and a warm breeze blows gently, I can sit and watch the butterflies and bees doing their chores.