Find byte pattern in file (still desperate for solution)

I'm still desperately looking for a solution to this question.  I need a 
very intelligent way to find an array of bytes within a file, but not only 
that preferably a regex expression of sorts.
For example my database has around 50,000 regex entries such as 
"FFFF.*BB0403.*FF" that I want to turn into a byte array then search for.  I 
understand how to go about writing my own code for this, however it is a 
little over my head when we're dealing with searching so many patterns 
against many files while trying to be speedy about the operation.  Does 
anyone have any good example code that I can start with, or know of a 3rd 
party .NET Component that will do this type of searching?

Thanks in advance, happy holidays!

-Joe 

0
Joseph
12/22/2009 5:31:39 AM
dotnet.languages.vb 830 articles. 0 followers. Follow

2 Replies
995 Views

Similar Articles

[PageSpeed] 31

I don't remember right now how it is called but there is basically a state 
based solution for searching multiple strings at once. A research gave me 
http://tomasp.net/articles/ahocorasick.aspx which is the algorithm I was 
thinking of.

http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=769&ns=16291 
could be a starting point...

--
Patrice


"Joseph N. Stackhouse" <junkmauler@hotmail.com> a �crit dans le message de 
news:9F777703-D56E-42A5-8AC8-21796699332C@microsoft.com...
> I'm still desperately looking for a solution to this question.  I need a 
> very intelligent way to find an array of bytes within a file, but not only 
> that preferably a regex expression of sorts.
> For example my database has around 50,000 regex entries such as 
> "FFFF.*BB0403.*FF" that I want to turn into a byte array then search for. 
> I understand how to go about writing my own code for this, however it is a 
> little over my head when we're dealing with searching so many patterns 
> against many files while trying to be speedy about the operation.  Does 
> anyone have any good example code that I can start with, or know of a 3rd 
> party .NET Component that will do this type of searching?
>
> Thanks in advance, happy holidays!
>
> -Joe 

0
Patrice
12/22/2009 11:05:02 AM
"Joseph N. Stackhouse" <junkmauler@hotmail.com> wrote in
news:9F777703-D56E-42A5-8AC8-21796699332C@microsoft.com: 

> I'm still desperately looking for a solution to this question.  I need
> a very intelligent way to find an array of bytes within a file, but
> not only that preferably a regex expression of sorts.

how big are the files?

There are plenty of commmand line tools that can search for bytes in a 
file. Searching for 50,000 different byte types takes time, however. But 
with a fast enough tool, you can do it. And, it is an option if you are 
cataloging and not doing this all the time. Or, after initial run, you 
can run the new files one at a time.

If the files are small enough, "caching" the regex "strings" as bytes 
and then looping is not too time consuming, but with larger files, it 
will bog down the system, esp if there is a risk of accidentally 
splitting these strings into two consecutive buffers. This might still 
be an option for a once in awhile pass, but not for a tool that runs on 
a regular basis.

Peace and Grace,

-- 
Gregory A. Beamer (MVP)

Twitter: @gbworld
Blog: http://gregorybeamer.spaces.live.com

*******************************************
|      Think outside the box!             |
*******************************************
0
Gregory
12/22/2009 9:07:48 PM
Reply:

Similar Artilces:

suggested ways to manipulate XML files in .Net?
Hi everyone, I'm going to manipulate some XML files in my next project-- crawl through the tags, compare if file X has the same nodes as file Y, and if not, what the differences are and who's missing what. I'll also be comparing values between nodes in the two files, for when both documents do have the same named nodes. What .Net classes should I be looking at to accomplish these tasks? Thanks a bunch. Steve Hershoff wrote: > I'm going to manipulate some XML files in my next project-- crawl through > the tags, compare if file X has the same nodes as file Y, and...

Excel 2007 and Word 2007 Hangs when using the File -> Open command
I have a couple users that when they have Excel 2007 or Word 2007 open, and click the File -> Open option, and choose a file, it either takes about 3 minutes to open the file or the program will crash. If they were to double click on the documents or spreadsheets, the doc or sheet will open right up. This is happening on the user's Hard drive as well as network drives. Once you get a file opened in Word or Excel, however, the next time you go to File -> Open, it's really fast to open the file. ...

Integration Manager--log file name?
Is there any way to know the name of the log file (we're writing our logs to text file) from within one of the "After" integration events? If we're successful, we need to send the user an email telling them that the event is successful. We're currently cycling through the whole folder to find the latest one, but if there were a variable in the Integration Manager VBScripts that I could use, that would be much better. Thanks. PSC: If I recall correctly, IM has a Document Error and Integration Error event in the integration properties. Rather than looking at the...

Saving chart as an image file
I want to create a pie chart in Microsoft Excel, then save it as an image file (.jpg or .gif) so I can put it on a website. How do I do this? Secondly, as I create my pie chart, how do I set my data labels? I want to change them from the default (numbers) to my own description for each slice. Jon Peltier has some instructions in his Excel Charting FAQ article for exporting a chart in gif format: http://pubs.logicalexpressions.com/Pub0009/LPMArticle.asp?ID=209#jon008 To add your own labels to the chart, you can use Rob Bovey's XY Chart Labeler. It's a free add-in that yo...

File Sharing between XP Pro and XP Home
I have a few questions concerning file sharing between XP Pro and XP Home in a Workgroup environment. I would really appreciate any help. 1)From what I'm gathering on the internet (I've read multiple articles) you cannot even connect a Windows XP Home computer to a domain. I'm I interpreting this correctly? 2)I have a new server with XP Pro and 5 other computers with XP Home. I only have 3 computers (which includes the new XP Pro server) that really need access to everything (or the main folders) on the server. The other 3 computers need limited access. If I purchase X...

Solution
Currently we have our CRM templates (to use for mail merging) saved locally. We would like to move these onto a shared drive on our network. However, the problem is that when they are shared, we are experiencing a normal.dot issue because it is being accessed by more than one person at a time. We are told there is no way around the normal.dot issue. So we are fine storing the templates locally. However, we would like to set up a process for updating these templates periodically, in the event that changes/updates need to be made. Since they are stored locally, not everyone would rec...

PDF file #5
I have Publisher 2002 and would like to insert a .pdf file into a document I am working on. How do / Can I do this? Many thanks, Ger ...

Can't find CD/DVDdrive
My DVD drives have disappeared. they don't show as hardware in control panel. Working fine yesterday - no software changes made. Both have power to them. I've unplugged everything and put a new lead from them to the motherboard. Doesn't appear to be anything physical. Followed various posts on this site. Mr Fixit recognises the problem but can't fix it. Went into registry. Higher filters there but no lower filters. Any suggestions? On 06/20/2010 05:13 AM, Philx5 wrote: > My DVD drives have disappeared. they don't show as hardware in control panel. ...

linked files
When I open a file it says it is linked to another file. When I check the links it is linked to a file I no longer have. How do I find where the link is coming from and get rid of it. Going to the Edit Links doesn't seem to get rid of the file. Charlie C Try Bill Manville's excellent "FindLink" utility .. It's available at Stephen Bullen's page: http://www.oaltd.co.uk/MVP/Def�ault.htm -- Rgds Max xl 97 --- GMT+8, 1� 22' N 103� 45' E xdemechanik <at>yahoo<dot>com ---- "Charlie 5280" <Charlie 5280@discussions.microsoft.com>...

"Find an event" feature in the Windows Live Mail/Calendar
In the Windows Live mail it is quite simple to "Find a message" as we type something in the search box and enter. But, I do not see any search box with the Calendar. Are there any ways to "Find" something in the Windows Live Calendar? Looking forward to hear from someone. Thank you. ...

Retrieve .pst files
My PC at work is upgraded from windows NT to Windows 2000. Before upgrade I backed up all the outlook 2000 .pst files. Now I have outlook 2003. I unable to retreive .pst files. viji.pst file is available under C:\Documents and Settings\jviji\Local settings\Applicationdata/microsoft/outlook. From outlook menu file -> data file management ->add -> outlook 97-2002 personal folder.pst -> viji.pst prompts to enter passwd. Please reply how do I retrieve the passwd protected .pst file. This are very important emails. Please reply. Thanks Viji Viji Jayaraman <jviji@ems.att.co...

Bytes in Data Page Row
I am reading a book titled "Microsoft SQL Server 2008 Internals". In Chapter 6, which was authored by Kalen Delaney, Kimberly Tripp, and Paul Randall, the following table and Clustered Index is created: CREATE TABLE Employee( EmployeeID INT NOT NULL IDENTITY, LastName NCHAR(30) NOT NULL, FirstName NCHAR(29) NOT NULL, MiddleInitial NCHAR(1) NULL, SSN CHAR(11) NOT NULL, OtherColumns CHAR(258) NOT NULL DEFAULT 'Junk') GO ALTER TABLE Employee ADD CONSTRAINT EmployeePK PRIMARY KEY CLUSTERED (EmployeeID) GO It states on page 321 of the aforementione...

How can I access a file in Outlook Public Folder by VBscript
It does not have to be VBscript.. I need sync the file with my local harddrive... Thanks. Nix wrote: > It does not have to be VBscript.. > I need sync the file with my local harddrive... > > Thanks. Why does it need to be a script at all? You can add your public folder to your PF\Favorites (drag it there) and include it in your send/receive settings for offline use with your OST file. ...

moving files
How do I move files from publisher 98 to publisher 2000? The programs are installed separately. If you are talking about the documents you created, you wouldn't have to. But if you wanted to - copy and paste. -- JoAnn Paules MVP Microsoft [Publisher] "dunefarm" <dune@discus.micro.com> wrote in message news:3854FB0D-CB6B-4D3A-A922-0A982890D3FD@microsoft.com... > How do I move files from publisher 98 to publisher 2000? The programs are > installed separately. ...

open excel 2007 refresh query from batch file
Hello all, I have a simple script saved as "C:\refresh_excel.vbs" that opens an Excel file, and does a refresh for a query on Sheet 1. I created a scheduled task to run this every day. It worked fine for previous versions of Excel, but since I have upgraded to Excel 2007 it doesn't work. Any help it helping me figure this out would be appreciated. /*contents of refresh_excel.vbs*/ Set objExcel = CreateObject("Excel.Application") objExcel.Visible = True Set objWorkBook = objExcel.Workbooks.Open("C:\reports\UPS.xls") objWorkbook.Sheets("...

Can OnPrint output be saved to a file?
Can OnPrint output (icluding fonts,rectangles,brushes...) be saved to a file? I am currently writing output in rtf file format for this purpose and it is more time consuming then producing a report via OnPrint. Thanks. How about using standard OnPrint code and mess with the Print to file option in the Print dialog box? -- Jonathan Wood SoftCircuits http://www.softcircuits.com Available for consulting: http://www.softcircuits.com/jwood/resume.htm "Charles C." <binstar@aei.ca> wrote in message news:%23m9f8B3GEHA.3940@tk2msftngp13.phx.gbl... > Can OnPrint output (icludin...

Arrears Help Files
Hello all, I don't seem to have the help file for the Arrears transactions. I can't find it on my system don't see a reference to it in the main GP 10 help file. We have GP 10 SP 4. I looked in KB see no reference to that particular problem anywhere. TIA Anyone? On Feb 10, 8:12=A0am, Brian <nixtoo...@hotmail.com> wrote: > Hello all, I don't seem to have the help file for the Arrears > transactions. =A0I can't find it on my system don't see a reference to it > in the main GP 10 help file. =A0We have GP 10 SP 4. =A0I looked in KB see= no...

Message-Data file not properly closed???
Upon boot of Outlook, EVERYTIME, I get this message--Data file did not close properly the last time it was used and is being checked for problems. Performance might be affected by the check in progress. When I leave Outlook, I have made sure to close all files and exit properly. Is there some reason this happens? I even go back to my main screen now. Is there something more I should be doing. Did the file checking finish before you closed Outlook? If not, it will continue/restart the next time you start Outlook. If it still does this after the checking has completed, the...

"Can't Create Backup FIle. Save [filename] Without Backp?
I get the following message every so often when running Excel 2003 under Win XP: "Can't create backup file. Save [finename] without backup?" I respond: "OK" "Your changes could not be saved to [filename] because of a sharing violation. Try saving to a different file." I respond: "OK". It presents me with the standard window allowing me to enter a new filename, which I do, then press OK. Then I get: "Do you want to save the changes you made to [fileename]? I respond: "Yes" "Document not saved." This is VERY annoying. I am t...

Money Backup File Security
I regularly take backup of my money file. I secure my money software with a password also. My question is regarding the security of the backup file. When I take a backup do I need to remember the password also? Let's say I took a backup a month ago and after that I changed my money password. And for reasons unknown I forgot the password which I had set when I took the backup. In the event of failure when I try to restore it from that backup will it ask me for a password? If yes, which one? If not - then is it not a security breach considering the fact that the backup CD might be sto...

Still can't post Attachments. Someone please Help !
SSBjYW4gZ2V0IGVtYWlscyB3aXRoIGF0dGFjaG1lbnRzLCBhbmQgc2VuZCBhdHRhY2htZW50cyBp biBlbWFpbC4gDQpJIGdldCBuZXdzZ3JvdXBzIHdpdGggYXR0YWNobWVudHMsIGJ1dCBjYW4ndCBz ZW5kIG9uZS4gSSBnZXQgYW4gZXJyb3IgbWVzc2FnZS4NCg0KVGhpcyBpcyB0aGUgc2FtZSBlcnJv ciBtZXNzYWdlIHdoZXRoZXIgaXRzIE91dGxvb2sgMjAwMCBvciBPdXRsb29rIEV4cHJlc3MuDQoN Ck91dGxvb2sgRXhwcmVzcyBjb3VsZCBub3QgcG9zdCB5b3VyIG1lc3NhZ2UuICBTdWJqZWN0ICdP dXRsb29rIHdvbnQgbGV0IG1lIHBvc3Qgd2l0aCBhbiBhdHRhY2htZW50PycsIEFjY291bnQ6ICdu ZXdzZ3JvdXBzLmJlbGxzb3V0aC5uZXQnLCBTZXJ2ZXI6ICduZXdzZ3JvdXBzLmJlbGxzb3V0aC5u ZXQnLCBQcm90b2NvbDogTk5UUCwgU2VydmVyIFJlc3BvbnNlOiAnNDQxIFBvc...

Problem with Money finding files after hard drive crash
My hard drive was corrupted and crashed. I was able to preserve data files, including the Money account file and Money backup file. The hard drive was replaced, and I copied the Money data files back onto the hard drive, as well as the Money program. When I try to set up Money again, it cannot find or connect with the data files, despite my specifying the drive location of the files. Is there a way to get that to happen? I don't know if this is part of the problem, but the description of my data files calls them Money 2004 files. The recovery version of Money is Money 2005, wh...

"Find and Replace" in Sheet names
Is there a way (with code perhaps) to search and replace sheet names? For example, I have 20 sheet names with "Direct (2)" in the name, and I'ld like to replace teh "Direct (2)" in the sheet name to "Net" in each instance. Thanks, Jim One way: Public Sub Direct2ToNet() Const sRepl As String = "Direct (2)" Dim ws As Worksheet Dim nPos On Error Resume Next For Each ws In ActiveWorkbook.Worksheets With ws If .Name Like "*" & sRepl & "*" Then _ ...

IF ISNUMBER FIND in query design expression
I use the following formula in Excel and it works...I now want to do this in a query. Can I add it as an expression in the design grid in a seperate column and how does the syntax change? I will also want to use TRUE/FALSE for the result. =IF(ISNUMBER(FIND("TRANSF",H2)),"*", "") Try this -- My_Output: IIF(InStr([YourFieldName],"TRANSF")>0,"*", "") -- Build a little, test a little. "gator" wrote: > I use the following formula in Excel and it works...I now want to do this in > a query. C...

How to: Open *.MSG File from Hard Drive via Code
I need to : Open *.MSG File from Hard Drive via Code and extract the file attachment. I have hundreds of these emails that we need the attachments extracted from. But the .msg files are on a hard drive. Someone help me out here.... I need the code to open and access the mailitem.... how do I open a .msg file? Use the CreateItemFromTemplate() method for that. -- Ken Slovak [MVP - Outlook] http://www.slovaktech.com Author: Professional Programming Outlook 2007. Reminder Manager, Extended Reminders, Attachment Options. http://www.slovaktech.com/products.htm "B...