Script to delete millions of files

  • Follow


We just inherited a server that has a folder which contains millions
of old log files that need to be cleaned up.
Using the GUI to even view the folder, let alone to do anything in it
obviously just hoses the server with so many file handles.

We tried deleting files from the folder using a command prompt but ran
into similar issues.

I'm thinking maybe we can write a script that uses the folder's file
collection and read in maybe 500 file names at a time into an array.
Then we could then loop through the array and delete the files one at
a time, then read in another 500 and so on. This script could run over
a weekend and maybe finish by monday morning.

Problem is I've been focused on learning CF and SQL for the past year
and my VBS skills are a bit rusty. So I thought I would just see if
anyone has something close to this already that I could use as a
starting point.

Thanks.
0
Reply Mickey 12/8/2009 3:40:58 PM

 You can try looping through the Files collection
and calling File.Delete on each one. you don't
need the intermediate step of an array. But....
maybe it'd be easier to just format and re-install.

> We just inherited a server that has a folder which contains millions
> of old log files that need to be cleaned up.
> Using the GUI to even view the folder, let alone to do anything in it
> obviously just hoses the server with so many file handles.
>
> We tried deleting files from the folder using a command prompt but ran
> into similar issues.
>
> I'm thinking maybe we can write a script that uses the folder's file
> collection and read in maybe 500 file names at a time into an array.
> Then we could then loop through the array and delete the files one at
> a time, then read in another 500 and so on. This script could run over
> a weekend and maybe finish by monday morning.
>
> Problem is I've been focused on learning CF and SQL for the past year
> and my VBS skills are a bit rusty. So I thought I would just see if
> anyone has something close to this already that I could use as a
> starting point.
>
> Thanks.


0
Reply mayayana 12/8/2009 3:49:51 PM

On Dec 8, 10:49=A0am, "mayayana" <mayaXXy...@rcXXn.com> wrote:
> =A0You can try looping through the Files collection
> and calling File.Delete on each one. you don't
> need the intermediate step of an array. But....
> maybe it'd be easier to just format and re-install.
>

Yeah I wish that was an option but this is a production web server and
it needs to stay up and running.
I guess I was just trying to find a way to limit the stress on the
master file table by deleting files in chunks. Not sure if that really
makes a difference or not. I thought maybe if I only open the files
collection long enough to grab 500 items and then close it, I might
leave more memory free to process the deletions. It's all just
theory... and very possibly overkill. I just need to avoid locking up
the server.
0
Reply Mickey 12/8/2009 4:10:41 PM

"Mickey" <xyxjunkmale@gmail.com> wrote in message 
news:163687c0-45f6-4ca6-b765-7c3219aaa175@9g2000yqa.googlegroups.com...
> We just inherited a server that has a folder which contains millions
> of old log files that need to be cleaned up.
> Using the GUI to even view the folder, let alone to do anything in it
> obviously just hoses the server with so many file handles.
>
> We tried deleting files from the folder using a command prompt but ran
> into similar issues.

Could you be a little more specific and explain what exactly you mean with 
"hosing the server", in particular when using console commands. Also - what 
console command did you use? 


0
Reply Pegasus 12/8/2009 4:23:52 PM

On Dec 8, 11:23=A0am, "Pegasus [MVP]" <n...@microsoft.com> wrote:
> "Mickey" <xyxjunkm...@gmail.com> wrote in message
>
> Could you be a little more specific and explain what exactly you mean wit=
h
> "hosing the server", in particular when using console commands. Also - wh=
at
> console command did you use?

I am repeating what someone else told me... I have not tried this
myself but was just trying to offer them a solution based on what I
was told... that using the GUI (Windows Explorer) took an impossibly
long time to enumerate the files (longer than an hour) and after any
attempted action the process started over and eventually locked up the
server requiring a reboot.

It's moot now. Someone else is cleaning up the files though I wasn't
told how. But it seems to be working.
Thanks anyway.
0
Reply Mickey 12/8/2009 5:29:57 PM

Mickey wrote:
On Dec 8, 11:23 am, "Pegasus [MVP]" <n...@microsoft.com> wrote:
> "Mickey" <xyxjunkm...@gmail.com> wrote in message
>
> Could you be a little more specific and explain what exactly you mean with
> "hosing the server", in particular when using console commands. Also -
what
> console command did you use?

I am repeating what someone else told me... I have not tried this
myself but was just trying to offer them a solution based on what I
was told... that using the GUI (Windows Explorer) took an impossibly
long time to enumerate the files (longer than an hour) and after any
attempted action the process started over and eventually locked up the
server requiring a reboot.

It's moot now. Someone else is cleaning up the files though I wasn't
told how. But it seems to be working.
Thanks anyway.
************

If they did it any way other than opening a command prompt and typing
DEL "path_to_folder\*.log" then too much effort went into it.

Although the deed is done now, if the logs needed to be retained, I was
going suggest moving the logs into folders or zip files sorted by date.

-- 
Todd Vargo
(Post questions to group only. Remove "z" to email personal messages)

0
Reply Todd 12/8/2009 9:43:31 PM

Hi Todd,

> It's moot now. Someone else is cleaning up the files though I wasn't
> told how. But it seems to be working.
> Thanks anyway.
> ************
>
> If they did it any way other than opening a command prompt and typing
> DEL "path_to_folder\*.log" then too much effort went into it.

I have had a server where I tried to do something like that (also with a routine thet kept 
generating log files every hour without me knowing about it) and kept getting errors about not 
having enough ....., don't remember what it was, might have been stack or heap memory. When you are 
talking about millions of files you will be running into strange limitations.

> Although the deed is done now, if the logs needed to be retained, I was
> going suggest moving the logs into folders or zip files sorted by date.

Might have run into the same problem. I'm not sure a standard ZIP file can handle that many 
filenames or whether the ZIP program can handle the number of files.
In my case even braking it up into a bunch of Del a*.* Del b*.* was not enough, I had to break it up 
into Del aa*.* Del ab*.* and so on. Made a batch file that did this for me and it ran over a day.

Bonno Bloksma


0
Reply Bonno 12/9/2009 6:48:21 AM

6 Replies
662 Views

(page loaded in 0.214 seconds)

10/20/2014 11:18:12 AM


Reply: