[Buildroot] Library search paths and ld.so.conf

[Quotient Remainder]

Is it possible to dynamically add a library search path on the target?
I'm aware that it's possible to add to the ld.so library search path by
adding an entry into "/etc/ld.so.conf" and running ldconfig as well as
using LD_LIBRARY_PATH as is done in target-finalize but I don't want to
run ldconfig and take the time penalty it entails to generate the cache.
Can't I just tell ld.so to look in an extra directory or two if the
normal ones fail?