Confused by implementing an IEnumerable with iterator blocks

I wanted to write a class to encapsulate an enumeration of all the files 
in a directory. I could just use

   Directory.GetFiles(directoryPath, "*.*", 
SearchOptions.SearchOption.AllDirectories);

but when hundreds of files are involved and the directory is on a remote 
server this causes a substantial delay, so I wanted to design an 
enumeration that would browse through the root directory and its 
subdirectories incrementally as iteration took place.

So I tried this, having read up only briefly on how "yield" works:

     public class DirectoryFileList : IEnumerable<string>
     {
         public string Root { get; set; }
         public DirectoryFileList(string root)
         {
             Root = root;
         }
         public IEnumerator<string> GetEnumerator()
         {
             DirectoryFileEnumerator enumerator = new 
DirectoryFileEnumerator();
             enumerator.Root = this.Root;
         }
         System.Collections.IEnumerator 
System.Collections.IEnumerable.GetEnumerator()
         {
             throw new NotImplementedException(); // to implement
         }
     }

     public class DirectoryFileEnumerator : IEnumerator<string>
     {
         public string Root { get; private set; }
         public string Current
         {
             get
             {
                 foreach (string file in Directory.GetFiles(Root))
                 {
                     yield return file;
                 }
                 foreach (string subdirectory in 
Directory.GetDirectories(Root, "*.*", SearchOption.AllDirectories))
                 {
                     foreach (string file in 
Directory.GetFiles(subdirectory))
                     {
                         yield return file;
                     }
                 }
                 yield break;
             }
         }
     }

But this produces a compiler error, because "The body of 
'ListAspScripts.DirectoryFileEnumerator.Current.get' cannot be an 
iterator block because 'string' is not an iterator interface type". What 
I'm finding out from elsewhere is that my iterating code has to be in a 
method that returns an IEnumerable. From what I now read when I look 
more closely, this code that returns one string after another has to be 
in a method that returns IEnumerable<string>!

Well, that stinks. I can imagine wanting to retrieve all the files in a 
directory in this manner in a variety of applications, and I would like 
to encapsulate in a class, decoupled from any other class that might use 
it! Now it looks as though I have to define a method in the class that 
needs such an enumeration to return that enumeration. Am I missing 
another approach?
0
Harlan
11/13/2009 8:09:20 PM
dotnet.languages.csharp 1931 articles. 0 followers. Follow

8 Replies
1464 Views

Similar Articles

[PageSpeed] 56

Harlan Messinger wrote:
> I wanted to write a class to encapsulate an enumeration of all the 
> files in a directory. I could just use
>
>   Directory.GetFiles(directoryPath, "*.*", 
> SearchOptions.SearchOption.AllDirectories);
>
> but when hundreds of files are involved and the directory is on a 
> remote server this causes a substantial delay, so I wanted to design 
> an enumeration that would browse through the root directory and its 
> subdirectories incrementally as iteration took place.
>
> So I tried this, having read up only briefly on how "yield" works:
>
>     public class DirectoryFileList : IEnumerable<string>
>     {
>         public string Root { get; set; }
>         public DirectoryFileList(string root)
>         {
>             Root = root;
>         }
>         public IEnumerator<string> GetEnumerator()
>         {
>             DirectoryFileEnumerator enumerator = new 
> DirectoryFileEnumerator();
>             enumerator.Root = this.Root;
>         }
>         System.Collections.IEnumerator 
> System.Collections.IEnumerable.GetEnumerator()
>         {
>             throw new NotImplementedException(); // to implement
>         }
>     }
>
>     public class DirectoryFileEnumerator : IEnumerator<string>
>     {
>         public string Root { get; private set; }
>         public string Current
>         {
>             get
>             {
>                 foreach (string file in Directory.GetFiles(Root))
>                 {
>                     yield return file;
>                 }
>                 foreach (string subdirectory in 
> Directory.GetDirectories(Root, "*.*", SearchOption.AllDirectories))
>                 {
>                     foreach (string file in 
> Directory.GetFiles(subdirectory))
>                     {
>                         yield return file;
>                     }
>                 }
>                 yield break;
>             }
>         }
>     }
>
> But this produces a compiler error, because "The body of 
> 'ListAspScripts.DirectoryFileEnumerator.Current.get' cannot be an 
> iterator block because 'string' is not an iterator interface type". 
> What I'm finding out from elsewhere is that my iterating code has to 
> be in a method that returns an IEnumerable. From what I now read when 
> I look more closely, this code that returns one string after another 
> has to be in a method that returns IEnumerable<string>!
>
> Well, that stinks. I can imagine wanting to retrieve all the files in 
> a directory in this manner in a variety of applications, and I would 
> like to encapsulate in a class, decoupled from any other class that 
> might use it! Now it looks as though I have to define a method in the 
> class that needs such an enumeration to return that enumeration. Am I 
> missing another approach?
Basically move your implementation of 
DirectoryFileEnumerator.Current.get{} into 
DirectoryFileList.GetEnumerator().  Then your DirectoryFileList becomes 
your decoupled object you want.  The point of yield is you should not be 
implementing a IEnumerator object.
So, then whenever you want the file list, you just do:
DirectoryFileList fileList = new DirectoryFileList(@"c:\temp");
foreach (string file in fileList)
{
  // Do something with file
}
0
Adam
11/13/2009 9:10:11 PM
Harlan Messinger wrote:
> I wanted to write a class to encapsulate an enumeration of all the 
> files in a directory. I could just use
>
>   Directory.GetFiles(directoryPath, "*.*", 
> SearchOptions.SearchOption.AllDirectories);
>
> but when hundreds of files are involved and the directory is on a 
> remote server this causes a substantial delay, so I wanted to design 
> an enumeration that would browse through the root directory and its 
> subdirectories incrementally as iteration took place.
>
> So I tried this, having read up only briefly on how "yield" works:
>
>     public class DirectoryFileList : IEnumerable<string>
>     {
>         public string Root { get; set; }
>         public DirectoryFileList(string root)
>         {
>             Root = root;
>         }
>         public IEnumerator<string> GetEnumerator()
>         {
>             DirectoryFileEnumerator enumerator = new 
> DirectoryFileEnumerator();
>             enumerator.Root = this.Root;
>         }
>         System.Collections.IEnumerator 
> System.Collections.IEnumerable.GetEnumerator()
>         {
>             throw new NotImplementedException(); // to implement
>         }
>     }
>
>     public class DirectoryFileEnumerator : IEnumerator<string>
>     {
>         public string Root { get; private set; }
>         public string Current
>         {
>             get
>             {
>                 foreach (string file in Directory.GetFiles(Root))
>                 {
>                     yield return file;
>                 }
>                 foreach (string subdirectory in 
> Directory.GetDirectories(Root, "*.*", SearchOption.AllDirectories))
>                 {
>                     foreach (string file in 
> Directory.GetFiles(subdirectory))
>                     {
>                         yield return file;
>                     }
>                 }
>                 yield break;
>             }
>         }
>     }
>
> But this produces a compiler error, because "The body of 
> 'ListAspScripts.DirectoryFileEnumerator.Current.get' cannot be an 
> iterator block because 'string' is not an iterator interface type". 
> What I'm finding out from elsewhere is that my iterating code has to 
> be in a method that returns an IEnumerable. From what I now read when 
> I look more closely, this code that returns one string after another 
> has to be in a method that returns IEnumerable<string>!
>
> Well, that stinks. I can imagine wanting to retrieve all the files in 
> a directory in this manner in a variety of applications, and I would 
> like to encapsulate in a class, decoupled from any other class that 
> might use it! Now it looks as though I have to define a method in the 
> class that needs such an enumeration to return that enumeration. Am I 
> missing another approach?
Basically move your implementation of 
DirectoryFileEnumerator.Current.get{} into 
DirectoryFileList.GetEnumerator().  Then your DirectoryFileList becomes 
your decoupled object you want.  The point of yield is you should not be 
implementing a IEnumerator object.
So, then whenever you want the file list, you just do:
DirectoryFileList fileList = new DirectoryFileList(@"c:\temp");
foreach (string file in fileList)
{
  // Do something with file
}
0
Adam
11/13/2009 9:10:35 PM
Harlan Messinger wrote:
> I wanted to write a class to encapsulate an enumeration of all the files 
> in a directory. I could just use
> 
>   Directory.GetFiles(directoryPath, "*.*", 
> SearchOptions.SearchOption.AllDirectories);
> 
> but when hundreds of files are involved and the directory is on a remote 
> server this causes a substantial delay, so I wanted to design an 
> enumeration that would browse through the root directory and its 
> subdirectories incrementally as iteration took place.
> 
[snip]
> Well, that stinks. I can imagine wanting to retrieve all the files in a 
> directory in this manner in a variety of applications, and I would like 
> to encapsulate in a class, decoupled from any other class that might use 
> it! Now it looks as though I have to define a method in the class that 
> needs such an enumeration to return that enumeration. Am I missing 
> another approach?

Never mind--I think. Despite what I'd read earlier, I guess the 
IEnumerable's GetEnumerator method can contain the iterator block. My 
code compiles with the explicit IEnumerator-implementing class removed 
and this implementation of GetEnumerator:

         public IEnumerator<string> GetEnumerator()
         {
             foreach (string file in Directory.GetFiles(Root))
             {
                 yield return file;
             }
             foreach (string subdirectory in 
Directory.GetDirectories(Root, "*.*", SearchOption.AllDirectories))
             {
                 foreach (string file in Directory.GetFiles(subdirectory))
                 {
                     yield return file;
                 }
             }
             yield break;
         }
0
Harlan
11/13/2009 9:17:54 PM
Harlan Messinger wrote:
> Harlan Messinger wrote:
>> I wanted to write a class to encapsulate an enumeration of all the 
>> files in a directory. I could just use
>>
>>   Directory.GetFiles(directoryPath, "*.*", 
>> SearchOptions.SearchOption.AllDirectories);
>>
>> but when hundreds of files are involved and the directory is on a 
>> remote server this causes a substantial delay, so I wanted to design 
>> an enumeration that would browse through the root directory and its 
>> subdirectories incrementally as iteration took place.
>>
> [...]
> Never mind--I think. Despite what I'd read earlier, I guess the 
> IEnumerable's GetEnumerator method can contain the iterator block.

Not only can it, that's the whole point of an iterator block.  It has to 
appear in something that is returning an enumerator.

By the way, you can improve significantly on the "deferred processing" 
aspect of your enumerator with some changes (right now, you still wind 
up having .NET enumerate the entire depth and breadth of the directory 
structure under the target one, to retrieve all the directory names). 
The simplest fix would be to apply the same per-directory logic to the 
retrieval of directory paths as for the files themselves, making your 
enumerator recursive:

   public IEnumerator<string> GetEnumerator()
   {
     return GetFilesForDirectory(Root);
   }

   private IEnumerator<string> GetFilesForDirectory(string strDirectory)
   {
     foreach (string strFile in Directory.GetFiles(strDirectory))
     {
       yield return strFile;
     }

     foreach (string strDirectory in Directory.GetDirectories(strDirectory))
     {
       foreach (string strFile in GetFilesForDirectory(strDirectory))
       {
         yield return strFile;
       }
     }
   }

Some notes:

     -- An iterator block doesn't have to be part of the implementation 
for IEnumerable<T> per se.  You can use one in _any_ situation you want 
an IEnumerator<T>.

     -- There's no need to write a "yield break" unless you specifically 
need to stop enumeration in the middle of the method.  Simply exiting 
the method normally will implicitly terminate the iteration.

     -- The basic idea of the above is a method that return all of the 
files in the given directory, and then uses itself to return all of the 
files in all of the subdirectories in the given directory.  In this way, 
at worst a given iteration of the files will involve a call to retrieve 
all the directories in a specific directory, followed by a call to 
retrieve all the files in the first directory in that specific directory.

It's _possible_ that you can achieve even lower latency per-iteration by 
using p/invoke to access the Win32 file enumeration functions.  In 
particular, these functions only return one file at a time, and in some 
cases they might be able to do so without processing an entire directory 
at once.

Fortunately, someone else already asked this question in the past _and_ 
they were an MSDN subscriber, so Microsoft wrote the code for them.  You 
can find their reply here:
http://groups.google.com/group/microsoft.public.dotnet.framework/msg/ebac5afd85780bac

Note that this unmanaged code approach may actually have _lower_ 
throughput overall, because of the frequent transitions between managed 
and unmanaged code, as well as whatever overhead might exist in the 
unmanaged API by returning only one file at a time (which depends on how 
it's implemented...I don't actually know whether this is a real issue or 
not).  Presumably, reducing the latency for a given iteration is more 
important than total throughput.

And of course, if your processing of each file is relatively 
time-consuming, you can have the best of both worlds by enumerating the 
files in one thread while you process them in another, allowing the 
processing thread to start work as soon as the very first filename has 
been returned (e.g. use a queue of filenames with one thread producing 
and the other consuming).

Hope that helps!
Pete
0
Peter
11/13/2009 11:27:22 PM
Peter Duniho wrote:
> Harlan Messinger wrote:
>> Harlan Messinger wrote:
>> Never mind--I think. Despite what I'd read earlier, I guess the 
>> IEnumerable's GetEnumerator method can contain the iterator block.
> 
> Not only can it, that's the whole point of an iterator block.  It has to 
> appear in something that is returning an enumerator.

Yes, and unfortunately, I was going by the examples I'd seen on some web 
pages, all of which were methods that returned IEnumerable, without 
having noticed a sentence on one of them that read, "yield statements 
are only valid in a method/operator/property which returns one of 
IEnumerable, IEnumerable<T>, IEnumerator or IEnumerator<T>."

> By the way, you can improve significantly on the "deferred processing" 
> aspect of your enumerator with some changes (right now, you still wind 
> up having .NET enumerate the entire depth and breadth of the directory 
> structure under the target one, to retrieve all the directory names). 
> The simplest fix would be to apply the same per-directory logic to the 
> retrieval of directory paths as for the files themselves, making your 
> enumerator recursive:
> 
>   public IEnumerator<string> GetEnumerator()
>   {
>     return GetFilesForDirectory(Root);
>   }
> 
>   private IEnumerator<string> GetFilesForDirectory(string strDirectory)
>   {
>     foreach (string strFile in Directory.GetFiles(strDirectory))
>     {
>       yield return strFile;
>     }
> 
>     foreach (string strDirectory in Directory.GetDirectories(strDirectory))
>     {
>       foreach (string strFile in GetFilesForDirectory(strDirectory))
>       {
>         yield return strFile;
>       }
>     }
>   }
> 
> Some notes:
> 
[snipping]
> 
> Hope that helps!

It does, very interesting, thank you for your time! Only problem is now 
I've got the following code, which is similar to yours except that I've 
added a Mask property for the file selection, and on the inner foreach 
I'm getting the error "foreach statement cannot operate on variables of 
type 'System.Collections.Generic.IEnumerator<string>' because 
'System.Collections.Generic.IEnumerator<string>' does not contain a 
public definition for 'GetEnumerator'", and I don't understand where 
that comes from.

   public class DirectoryFileList : IEnumerable<string>
   {
     /* ... */
     public IEnumerator<string> GetEnumerator()
     {
       return GetFilesInDirectory(Root);
     }

     public IEnumerator<string> GetFilesInDirectory(string path)
     {
       foreach (string file in Directory.GetFiles(path, Mask))
       {
         yield return file;
       }
       foreach (string subdirectory in Directory.GetDirectories(path))
       {
         foreach (string file in GetFilesInDirectory(subdirectory))
         {
           yield return file;
         }
       }
     }

     System.Collections.IEnumerator 
System.Collections.IEnumerable.GetEnumerator()
     {
       return this.GetEnumerator();
     }
   }


0
Harlan
11/16/2009 8:13:17 PM
Harlan Messinger wrote:
> Peter Duniho wrote:
>> Harlan Messinger wrote:
>>> Harlan Messinger wrote:
>>> Never mind--I think. Despite what I'd read earlier, I guess the 
>>> IEnumerable's GetEnumerator method can contain the iterator block.
>>
>> Not only can it, that's the whole point of an iterator block.  It has 
>> to appear in something that is returning an enumerator.
> 
> Yes, and unfortunately, I was going by the examples I'd seen on some web 
> pages, all of which were methods that returned IEnumerable, without 
> having noticed a sentence on one of them that read, "yield statements 
> are only valid in a method/operator/property which returns one of 
> IEnumerable, IEnumerable<T>, IEnumerator or IEnumerator<T>."
> 
>> By the way, you can improve significantly on the "deferred processing" 
>> aspect of your enumerator with some changes (right now, you still wind 
>> up having .NET enumerate the entire depth and breadth of the directory 
>> structure under the target one, to retrieve all the directory names). 
>> The simplest fix would be to apply the same per-directory logic to the 
>> retrieval of directory paths as for the files themselves, making your 
>> enumerator recursive:
>>
>>   public IEnumerator<string> GetEnumerator()
>>   {
>>     return GetFilesForDirectory(Root);
>>   }
>>
>>   private IEnumerator<string> GetFilesForDirectory(string strDirectory)
>>   {
>>     foreach (string strFile in Directory.GetFiles(strDirectory))
>>     {
>>       yield return strFile;
>>     }
>>
>>     foreach (string strDirectory in 
>> Directory.GetDirectories(strDirectory))
>>     {
>>       foreach (string strFile in GetFilesForDirectory(strDirectory))
>>       {
>>         yield return strFile;
>>       }
>>     }
>>   }
>>
>> Some notes:
>>
> [snipping]
>>
>> Hope that helps!
> 
> It does, very interesting, thank you for your time! Only problem is now 
> I've got the following code, which is similar to yours except that I've 
> added a Mask property for the file selection, and on the inner foreach 
> I'm getting the error "foreach statement cannot operate on variables of 
> type 'System.Collections.Generic.IEnumerator<string>' because 
> 'System.Collections.Generic.IEnumerator<string>' does not contain a 
> public definition for 'GetEnumerator'", and I don't understand where 
> that comes from.
> 
>   public class DirectoryFileList : IEnumerable<string>
>   {
>     /* ... */
>     public IEnumerator<string> GetEnumerator()
>     {
>       return GetFilesInDirectory(Root);
>     }
> 
>     public IEnumerator<string> GetFilesInDirectory(string path)
>     {
>       foreach (string file in Directory.GetFiles(path, Mask))
>       {
>         yield return file;
>       }
>       foreach (string subdirectory in Directory.GetDirectories(path))
>       {
>         foreach (string file in GetFilesInDirectory(subdirectory))
>         {
>           yield return file;
>         }
>       }
>     }

The problem is that GetFilesInDirectory returns an IEnumerator, not an 
IEnumerable, so of course the foreach doesn't work!

This code, where GetFilesInDirectory returns an IEnumerable and my 
GetEnumerator method explicitly calls that IEnumerable's GetEnumerator 
method, does compile:

     public IEnumerator<string> GetEnumerator()
     {
       return GetFilesInDirectory(Root).GetEnumerator();
     }

     private IEnumerable<string> GetFilesInDirectory(string path)
     {
       foreach (string file in Directory.GetFiles(path, Mask))
       {
         yield return file;
       }
       foreach (string subdirectory in Directory.GetDirectories(path))
       {
         foreach (string file in GetFilesInDirectory(subdirectory))
         {
           yield return file;
         }
       }
     }

Finally: what I'd really love is a method that, before doing all this, 
could return up front the number of files at all levels under my initial 
directory, so I can initialize a progress dialog, but that appears to be 
too much to hope for, since as far as I can tell Directory and 
DirectoryInfo don't offer a file count.
0
Harlan
11/16/2009 8:22:17 PM
Harlan Messinger wrote:
> [...]
> The problem is that GetFilesInDirectory returns an IEnumerator, not an 
> IEnumerable, so of course the foreach doesn't work!

Yes, exactly.  Sorry for the typo.  I should have included my usual 
"uncompiled, untested" disclaimer.  :)

> This code, where GetFilesInDirectory returns an IEnumerable and my 
> GetEnumerator method explicitly calls that IEnumerable's GetEnumerator 
> method, does compile:
> 
>     public IEnumerator<string> GetEnumerator()
>     {
>       return GetFilesInDirectory(Root).GetEnumerator();
>     }
> 
>     private IEnumerable<string> GetFilesInDirectory(string path)
>     {
>       foreach (string file in Directory.GetFiles(path, Mask))
>       {
>         yield return file;
>       }
>       foreach (string subdirectory in Directory.GetDirectories(path))
>       {
>         foreach (string file in GetFilesInDirectory(subdirectory))
>         {
>           yield return file;
>         }
>       }
>     }

Right.  Noting, of course, that an alternative would be to not implement 
IEnumerable in your class at all, and simply make the 
GetFilesInDirectory() method a public static method in some helper class 
(i.e. unless you have some specific need to have a class that stores the 
instance of the "Root" string persistently, it would suffice to just 
provide for a method to which you can pass whatever starting directory 
you want to inspect).

> Finally: what I'd really love is a method that, before doing all this, 
> could return up front the number of files at all levels under my initial 
> directory, so I can initialize a progress dialog, but that appears to be 
> too much to hope for, since as far as I can tell Directory and 
> DirectoryInfo don't offer a file count.

Well, to count the number of files, the OS needs to access the directory 
information, which is mostly the same work you have to do just to get a 
list of the files anyway.  Any API that provided for counting files 
without enumerating them would just wind up hiding duplicated effort, 
possibly encouraging inefficient code in a non-transparent way.

As I mentioned in my previous post, if the cost of processing each file 
is significant, you can set up a producer/consumer scenario, in which 
the enumeration of the filenames happens in a separate thread from where 
you actually process each file.  Not only does that allow processing to 
start immediately, rather than having intermittent pauses in processing 
as the recursive iterator method has to retrieve more information (i.e. 
directories in a directory, files in a directory), it opens the 
possibility for more accurate progress indication.

In particular, you can accumulate the total files that will be processed 
as you enumerate them (the producer), and add to the progress value as 
you process each file (the consumer).  Initially the progress indication 
will be optimistic, as it won't have the complete count of files.  But 
the enumeration of the filenames to process will complete much earlier 
than the actual processing of them, and once that happens, you'll have 
accurate indication of progress in terms of total files to process 
versus files processed so far.

I've done this sort of thing in my own code and it works well.  In fact, 
I've even implemented it in some cases by running the enumeration of the 
files twice: once to count them, and another time to process them.  This 
alternative isn't quite as efficient in terms of disk/network i/o, but 
it does avoid the overhead and complexity of the producer/consumer 
queue.  I don't know how well it'd work on a network, but when I've used 
it for local file system access, there's enough caching built into the 
OS that the inefficiency isn't really noticeable.

So, there's at least two approaches you can use in conjunction with this 
kind of enumeration in order to provide reasonably accurate progress 
indication (you'll still have whatever inaccuracies arise due to 
differences in processing time for each file, but that's unavoidable).

Pete
0
Peter
11/16/2009 8:57:58 PM
Peter Duniho wrote:
> Harlan Messinger wrote:
>> [...]
>> The problem is that GetFilesInDirectory returns an IEnumerator, not an 
>> IEnumerable, so of course the foreach doesn't work!
> 
> Yes, exactly.  Sorry for the typo.  I should have included my usual 
> "uncompiled, untested" disclaimer.  :)
> 
>> This code, where GetFilesInDirectory returns an IEnumerable and my 
>> GetEnumerator method explicitly calls that IEnumerable's GetEnumerator 
>> method, does compile:
>>
>>     public IEnumerator<string> GetEnumerator()
>>     {
>>       return GetFilesInDirectory(Root).GetEnumerator();
>>     }
>>
>>     private IEnumerable<string> GetFilesInDirectory(string path)
>>     {
>>       foreach (string file in Directory.GetFiles(path, Mask))
>>       {
>>         yield return file;
>>       }
>>       foreach (string subdirectory in Directory.GetDirectories(path))
>>       {
>>         foreach (string file in GetFilesInDirectory(subdirectory))
>>         {
>>           yield return file;
>>         }
>>       }
>>     }
> 
> Right.  Noting, of course, that an alternative would be to not implement 
> IEnumerable in your class at all, and simply make the 
> GetFilesInDirectory() method a public static method in some helper class 
> (i.e. unless you have some specific need to have a class that stores the 
> instance of the "Root" string persistently, it would suffice to just 
> provide for a method to which you can pass whatever starting directory 
> you want to inspect).
> 
>> Finally: what I'd really love is a method that, before doing all this, 
>> could return up front the number of files at all levels under my 
>> initial directory, so I can initialize a progress dialog, but that 
>> appears to be too much to hope for, since as far as I can tell 
>> Directory and DirectoryInfo don't offer a file count.
> 
> Well, to count the number of files, the OS needs to access the directory 
> information, which is mostly the same work you have to do just to get a 
> list of the files anyway.  Any API that provided for counting files 
> without enumerating them would just wind up hiding duplicated effort, 
> possibly encouraging inefficient code in a non-transparent way.

Correct, assuming that OS directories don't keep a count of the number 
of files they contain. I was just kind of hoping that they did.
> 
> As I mentioned in my previous post, if the cost of processing each file 
> is significant, you can set up a producer/consumer scenario, in which 
> the enumeration of the filenames happens in a separate thread from where 
> you actually process each file.  Not only does that allow processing to 
> start immediately, rather than having intermittent pauses in processing 
> as the recursive iterator method has to retrieve more information (i.e. 
> directories in a directory, files in a directory), it opens the 
> possibility for more accurate progress indication.
> 
> In particular, you can accumulate the total files that will be processed 
> as you enumerate them (the producer), and add to the progress value as 
> you process each file (the consumer).  Initially the progress indication 
> will be optimistic, as it won't have the complete count of files.  But 
> the enumeration of the filenames to process will complete much earlier 
> than the actual processing of them, and once that happens, you'll have 
> accurate indication of progress in terms of total files to process 
> versus files processed so far.

That sounds like a useful approach, thanks.

> I've done this sort of thing in my own code and it works well.  In fact, 
> I've even implemented it in some cases by running the enumeration of the 
> files twice: once to count them, and another time to process them.  This 
> alternative isn't quite as efficient in terms of disk/network i/o, but 
> it does avoid the overhead and complexity of the producer/consumer 
> queue.  I don't know how well it'd work on a network, but when I've used 
> it for local file system access, there's enough caching built into the 
> OS that the inefficiency isn't really noticeable.
> 
> So, there's at least two approaches you can use in conjunction with this 
> kind of enumeration in order to provide reasonably accurate progress 
> indication (you'll still have whatever inaccuracies arise due to 
> differences in processing time for each file, but that's unavoidable).

Cool. Again, I appreciate the time you spend answering these questions!
0
Harlan
11/16/2009 9:08:27 PM
Reply:

Similar Artilces:

Blocking mail downloads by attachment size
All, Is there a way to configure Outlook 2000 to not download files from the mail server that have attachments, or that have attachments over a set file size? Mail server is Exchange 2000. Thanks in advance, Nick ...

Outlook blocked access to this potentially unsafe attachment
Does anyone know how to eliminate the message below? 'outlook blocked access to this potentially unsafe attachment' I've saved web links in here (from MS as a matter of fact). Now I cannot open them. I've found nothing in the KB. tx, RG http://www.slipstick.com/outlook/esecup/getexe.htm /neo ps - can't eliminate, but can configure. ;) "Richard Gutery" <rgutery@mentorits.com> wrote in message news:uPC6wtAaEHA.2340@TK2MSFTNGP09.phx.gbl... > Does anyone know how to eliminate the message below? > 'outlook blocked access to this potentially uns...

does browser defender block new browsers?
I currently have internet explorer 6 and need a new browser installed. When I install "any" other browser, my computer doesn't work. It won't let me open the new browser. It says " cannot connect with the server". Also, I am unable to get into the itunes store with my current browser. -- the sewer ------------------------------------------------------------------------ the sewer's Profile: http://forums.techarena.in/members/210588.htm View this thread: http://forums.techarena.in/xp-hardware/1329116.htm http://forums.techarena.in ...

Masking out #ifdef blocks
Is there a software tool or Visual Studio add-in that I can use to mask out given #ifdef blocks? I am working on a code (VC++, visual studio 6) that is supposed to work for multiple projects .. and so they've put #ifdef PROJ_NAME all over the place .. and it is very irritating to view and follow the flow of the code. I saw VS.NET having this ability to open or close code blocks .. but it doesn't seem to have one for #ifdefs. Ideally, this should either let me grey out or hide all the code within a ifdef block. Thanks, drizzy DRIZAII wrote: > Ideally, this should either let me ...

attachment blocked
If I try to open PDF file or TXT file in OWA, I get a message "Attachment Blocked You do not have permissions required to open or download e-mail attachment from this computer. Contact your server administrator for details about your user permisions." This message seems to be misleading. I get it even if I log in as an administrator. Please help!!! Antonin On Mon, 4 Dec 2006 15:18:02 +1030, "Antonin" <Antonin.Koudelka@fmc.sa.gov.au> wrote: >If I try to open PDF file or TXT file in OWA, I get a message >"Attachment Blocked >You do not have permi...

How to disable accounts being blocked by login failures retries
Hi, I'm running Exchange 5.5 on windows NT, I have an email account that gets automatically blocked every now and then, it seems the reason is that, in this Exchange Server, any account gets blocked when there are 3 or 4 failed logon tries to that account. But these failed logons are not from the mail client of the account owner, he has the account data well configured, I imagine it could be some old owner of that account, because this email address was used by other people in the past (it's the email account of a department so whenever a new boss comes in he gets it), so it could be...

app blocks on MSUI.Msg.MuiMgrDirtyUpdate message
I develop a MDI application. When I open a document window, the application is blocked for about 1 -2 seconds. During this time the document window is already visible but the application doesn't react on user inputs. Even the toolbar items are not updated. During debugging I found out that there is a MSUI.Msg.MuiMgrDirtyUpdate message processed by a CicMarshalWndClass object during this time. I hardly found information about the message and the class. What kind of message is it and how can I avoid the blocking time? ...

Yellow block down the side of my page
Hi, a faded yellow block of colour has appeared down the left side of my page when i create a new document from my saved template. It doesn't happen when i open the template itself. Any ideas how to get rid of it please? thanks It would appear that you have Track Changes enabled. The colored area is provided for the "balloons." -- Suzanne S. Barnhill Microsoft MVP (Word) Words into Type Fairhope, Alabama USA http://word.mvps.org "Angel_1" <Angel1@discussions.microsoft.com> wrote in message news:E7A33CF4-DDFB-408B-BF53-ABDEB33FA742@microsof...

Title Block as a header?
I'd like to create header with my company title block. It was very eas to do in Word, but it doesn't look like the header function works th same way in Excel. Is there a way to use the draw function and impor pictures into the header like you do in Word -- mae177 ----------------------------------------------------------------------- mae1778's Profile: http://www.excelforum.com/member.php?action=getinfo&userid=2571 View this thread: http://www.excelforum.com/showthread.php?threadid=39225 xl2002 added support for pictures in the header (and footer). Or you could put the p...

Subtable / Conditional Block of Cells
Hello Group, I would like to have a group of cells in the bottom left corner with different content depending on the results of calculations in the sheet. I tryed to split the sheet in to 3 parts (Left Top, Left bottom, Right) to scroll the left bottom to the different blocks but does not work you have to splitt to 4 parts and the right bottom scrolls with the left one. I also looked for some concept of subtable, block switching but found nothing. The calculations are different depending on some sub-result and have to be printed different but I would like to appear it in the same region of th...

Location for blocked attachments in Exchange Server 5.5
When Outlook blocks attachments that are unsafe, where are the blocked attachements located or how can I find out where they are located? Thanks ...

Problem with internet links from HTML mail (plus a question about application blocking)
I have a problem in that if I simply click links within emails in OutlookXP they open in an existing/open Internet Explorer window. Most of the time this is fine. If I want it to open in a new window shift click works fine in text emails but some HTML emails don't open the link but rather bring up a save dialogue. It isn't all HTML emails though which is a bit strange, and doesn't seem predictable. Any ideas how to get Shift/Click to correctly open a new window would be appreciated. Also I am getting the "A program is trying to access email addresses you have stored in...

Need help in blocking a certain email from a certain domain.
I'm trying to block a certain email from a certain domain and I've made some entries to block such account in three different locations on the Message Delivery System: Sender Filtering, Recepient Filtering, & Connection Filtering. But it does not seem to work. Can someone help me on this? Are the messages coming from a particular smtp address or domain? Insert the smtp address or domain in Sender Filtering, enable Sender Filtering on SMTP virtual server. Recipient Filtering is used to block messages to particular internal recipients - not relevant in this scenario unl...

DLookup Confusion
In the below code sample, Phase 1 works as it should. Phase 2 causes errors and I am not sure why the errors are being caused. I would appreciate another set of eyes on this. I have also been able to create Phase 2 by using recordset method (not shown) and also generate the same errors. Thank you. Msg1: The "Microsoft Access" type isn't an installed database type or doesn't support the operation you chose. Msg2: "Invalid file type" Private Sub Command0_Click() Dim dbPath1 As String Dim dbType1 As String Dim dbPath2 As String Dim dbType2 As String '**...

Modified Report Modified with VBA (Blocked)
Hi to all, I have a the message "Modified Report Modified with VBA (Blocked)" on Maintenance of customizations screen on each VBA modifed report. This does not allow me to Export the report. What should I do to remove the "Blocked" message? Regards This happens when you have references in your project and its lost. For example you can have ADO "reference, or RetrieveGlobals reference. You only have to reasign the reference and the "Blocked" message disapears. Regards, Takashi "Jeantex" <Jeantex@discussions.microsoft.com> escribi� en el...

Confused with disclaimers on Exch2003.
After reading all the posts here and the web links included with some of them and checking Q317680, I am still no nearer getting the disclaimers to work on our system. I followed the instruction precisely as outlined in Q317680 as it seemed more straight forward than Q317327 with no success. I seemed to get all the "success messages generated by the script". I didn't go for the Q317327 option as I didn't want to install VB on the exchange server just to create a dll file for our disclaimer. Nor do I wish to create a secondary virtual SMTP server as suggested on a border ser...

Exchange relay blocked
Hello, I have exch2003 server running on 2003 SBS. We have a SQL server that has to send email reports to the an SBS server email account. I have allowed relay access to the IP of the SQL server, The subnet, even allowed anyone - which I turned off later and It still blocks. I can test by 'telnet exchsrv 25' which won't even give me a banner. From any other PC on the network I connect and send email. Just this one server is blocked. Any ideas where to look?? Thanks in advance, Kevin -- RHCE, Linux+ and MCP Does the SQL Server have more than 1 NIC? May be an issue with Multih...

Outlook Blocked access to the following potentionally unsafe attachments:AbetterSubForm.mdb.
An associate sent me a MS Access data base file and I received the following message: "Outlook Blocked access to the following potentionally unsafe attachments:AbetterSubForm.mdb.", so I was unable to view the attachment. Is there a way to set Outlook to be able to receive MS Access files? See http://www.slipstick.com/outlook/esecup/getexe.htm for more information. "DaveB" <Burrone50@msn.com> wrote in message news:41513D53-41B5-4B55-8A46-3B7C6E5E6A94@microsoft.com... > An associate sent me a MS Access data base file and I received the following message: "Out...

Exit For statement in nested For ... Next blocks
I have the block of code below in a Sub that tidies up a copied workshee= t = by removing unused rows. It=E2=80=99s contained in a With =E2=80=A6 End = With block that = refers to the copied sheet. I=E2=80=99d like to know how the Exit For st= atement is = handled if it=E2=80=99s in a For ... Next block that is nested within an= other For = =E2=80=A6 Next block. I.e. will it cause the code to exit only the inner= For =E2=80=A6 = Next block and proceed with the If VacRowClear =3D True Then statement, or will it exit the outer For =E2=80=A6 Next block? I use...

Totally confused now Help please
I have been having problems with Outlook 2003 not saving passwords so I created 3 different profiles each with one e mail account in each. Tim 1 = Personal Tim 2 = Business Tim 3 = webmaster I set my archive to autoarchive after a set period of time and everything seems to work fine however I now have about 10 personal folders and archive folders in outlook folder. I want to be able to backup each account using the outlook backup tool and save all folders to my external HD, but I am losing the plot a bit. Can anyone explain (in simple terms please) how I can acheive this. Please just ...

Drag and Drop: Implementing a Drop Target
Hi, I have a CSplitter with multiple windows. I am trying to implement a Drop Target in one of them. In the View Class .h file I declare a variable of type COleDropTarget COleDropTarget m_dropTarget; In the OnCreate() member function I try to register. BOOL b = m_dropTarget.Register(this); While stepping through, It fails in oledrop2.cpp at the following function. BOOL COleDropTarget::Register(CWnd* pWnd) { (...) //the object must be locked externally to keep LRPC connections alive if (CoLockObjectExternal(lpUnknown, TRUE, FALSE) != S_OK) return FALSE; (...) } I don'...

"one or more of the selected senders could not be added to your blocked senders list"
Hey all. I get this when ever I try to killfile anyone at all, regardless of who they are. I'm on OE6 in XP Home and this has only just started happening. My KF only has a few spammers in it so I don't think it's full. Does anyone know what's going on? Thanks "Ayatollah of rock 'n' roller" <thisisf@lse.co.ck> wrote in message news:gGxDn.40$Bm7.30@newsfe29.ams2... > Hey all. I get this when ever I try to killfile anyone at all, regardless of > who they are. I'm on OE6 in XP Home and this has only just started > happen...

Blocking spam
Can I block receiving messages to non-existing accounts so the Exchange 2003 does not send back the "Delivery Status Notification(Failure)"? Thanks in advance. Antonin what you say is not the exact deffinition for spam the spam is the incoming mails to you not the wrong addresses you send mails to but any way to be able to block spam you have to do the following block the relay in your server ( check the web for help material ) use one spam filter such like GFI mailessential or Surfcontrol advise your users to not reply any unknown message Maged N. Roshdy "Antonin Koud...

Outlook blocking attachments #2
Hi there, we have recently migrated our Exchange Server to 2003. Most of the users have Outlook 2000. Since the migration, users cannot open attachments that Outlook considers to be unsafe. We have applied the patch locally as specified in KB:318515 and on the Exchange Server as specified in KB:263297 but users are still having attachments blocked. Is this a problem of using OL2000 with EX2003? I've just discovered a temporary solution for this. Log on to your mail account using Outlook Web Access. It lets you save any attachments that are blocked in the Windows versio...

Best DNSBL to block e-mail Spam?
Hi! I'm using DNSBL to filter spam. These are sbl.spamhaus.org, spam.dnsbl.sorbs.net and bl.spamcop.net . SpamHaus is the best since it has a big percentage of spam emails that it filters. But as I've notice, still we receive spam. Can you suggest other DNSBL host? Btw, I'm using MS Exchange 5.5 SP4. Me No DNS RBL is going to block 100% of the spam you receive. I consider myself lucky if it catches 80%. We use X/Wall in front of our Exchange server, and it has an option forward the messages or mark them in the subject as spam and let them go on to the mailbox (rather t...