There’s no doubt that a generic rules engine can sometimes be a solution in search of a problem. The work to implement and manage both the rules and their interfaces with external systems can often completely dwarf any work which might be needed to implement the same behaviour in a regular programming language.
Martin Fowler has recently written about this problem in his “bliki”, and comes down in favour of always putting the effort in to work on a programming language solution alongside any exploration of a rules engine approach.
However, rather than seeing this as a reason to distrust the idea of rules engines as a whole, I prefer to see this is a problem with the implementation of traditional rules-engine technology. There is an argument, for example, that current fan favourite language Erlang is largely a rules engine.
The history or rules engine development is long, and many of the common conventions have a largely historical basis. The choice to represent rules in a text file represented as some lisp-based format (or a slightly more modern equivalent using XML), for example, is not really key to the concept of a rules engine. Likewise the heavyweight and single-threaded way that many rules engines operate is also mainly a historical implementation choice.
A light weight and efficient rules engine, well-integrated with the host language or environment is entirely possible, but despite several searches around the software universe I have found very few candidates, and none of the usual suspects, which work this way.
Maybe I ought to dust off my own implementation again 😉