[ILUG] Limits of grep?

[kevin lyda]

On Tue, Sep 26, 2000 at 06:38:05AM +0000, Subba Rao wrote:
> I have a directory of 10000+ text files and would like to search for
> some strings in these files. When I tried using "grep" command with an
> asterisk, I get the error message somthing to the effect,
> 
> 	"File argument list too long"
> 
> What is the file argument limit for grep? I guess you need the grep source
> for this. I did not find any information in the man page.

the file arg list is related to the shell (or maybe kernel) and not grep.

solution?

look into one of xargs or find ("haystack" is the dir with all the files,
"needle" is the text you;re looking for):

cd haystack; /bin/ls|xargs grep needle
find haystack -type f -exec grep needle '{}' /dev/null \;

note, i use /bin/ls because some people (me for instance) alias ls to be
ls-F or ls -F --color.  on the find command i have grep look at the file
('{}') and /dev/null so that it will print the file names.  as an aside
sunos used to let you do this:

find haystack -type f -exec grep needle '{}' /dev/null \&

which either let multiple processes run at once or would append filenames
until it found all the files or reached the argument limit.  or at least
i seem to remember that.

as an additional aside this is one of the reasons i first started using
linux.  sco had a restriction of one meg for argv and environ combined.
our build environment managed to hit that - even with tricks like short paths
and env -.  add a 10 xterm limit, crashes once every two weeks, and nfs/nis
limitations.  all of it added up to a linux cross-build platform.

and one more aside, ext2 doesn't handle that many files in a single
directory well.  you might want to split them across directories
or use a different fs.  or both.  if you split them across directories
then you'll either want the find command or you can combine find and
xargs:

find haystack -type f -print | xargs grep needle

the first find command execs one grep per file, this one execs one grep
for many files.  read the xargs man page for more info on how you can
control it (number of execs at a time, number of args per exec, etc).

cheers,

kevin

-- 
kevin at suberic.net        the only problem with being a man of leisure is
fork()'ed on 37058400    that you can never stop and take a rest.
meatspace place: home    http://suberic.net/~kevin/  yank? www.votenader.com
 >> want privacy? www.gnupg.org or www.pgp.com.  encrypted mail preferred <<