I have some texts files, which i want to read in each line, and then
write back each line to a new text file.

So for example, I want to read in the 2 lines below:


"C:\Data\=D3=EA=D2=B9=D0=C7=BF=D5",6,10,3,3,8
"",0,0,0,318.592,83.04552


I store each line of data in the following vector:


std::vector<CString> FileData;


So FileData[0] =3D ""C:\Data\=D3=EA=D2=B9=D0=C7=BF=D5",6,10,3,3,8"
FileData[1] =3D """,0,0,0,318.592,83.04552"


(the data is in the vector correctly)
Then I go to write the data to a new text file...


FILE * stream;
stream =3D _wfopen(filename, _T("w"));
for(j=3D0;j<FileData.size();j++)
  fwprintf(stream, _T("%s"), FileData[j]);
fclose(stream);


However, when I look at the text file, it's gone wrong:
C:\Data\,0,0,0,318.592,83.0455


I'm doing this in unicode. Is there any obvious problem?

You could use a CStdioFile to do this easily, not sure about the odd 
characters showing though, but CStdioFile works with Unicode so if you're 
using that you'll be safe with just about anything.

Tom

"hamishd" <Hamish.Dean@gmail.com> wrote in message 
news:1176341512.504862.224870@w1g2000hsg.googlegroups.com...
I have some texts files, which i want to read in each line, and then
write back each line to a new text file.

So for example, I want to read in the 2 lines below:


"C:\Data\��ҹ�ǿ�",6,10,3,3,8
"",0,0,0,318.592,83.04552


I store each line of data in the following vector:


std::vector<CString> FileData;


So FileData[0] = ""C:\Data\��ҹ�ǿ�",6,10,3,3,8"
FileData[1] = """,0,0,0,318.592,83.04552"


(the data is in the vector correctly)
Then I go to write the data to a new text file...


FILE * stream;
stream = _wfopen(filename, _T("w"));
for(j=0;j<FileData.size();j++)
  fwprintf(stream, _T("%s"), FileData[j]);
fclose(stream);


However, when I look at the text file, it's gone wrong:
C:\Data\,0,0,0,318.592,83.0455


I'm doing this in unicode. Is there any obvious problem?

Looks like its skipping the first element. Can you verify that its being
written by stepping in. Also, you can use CStdioFile in MFC to read/write
text.

--
Ajay Kalra [MVP - VC++]
ajaykalra@yahoo.com


"hamishd" <Hamish.Dean@gmail.com> wrote in message
news:1176341512.504862.224870@w1g2000hsg.googlegroups.com...
I have some texts files, which i want to read in each line, and then
write back each line to a new text file.

So for example, I want to read in the 2 lines below:


"C:\Data\��ҹ�ǿ�",6,10,3,3,8
"",0,0,0,318.592,83.04552


I store each line of data in the following vector:


std::vector<CString> FileData;


So FileData[0] = ""C:\Data\��ҹ�ǿ�",6,10,3,3,8"
FileData[1] = """,0,0,0,318.592,83.04552"


(the data is in the vector correctly)
Then I go to write the data to a new text file...


FILE * stream;
stream = _wfopen(filename, _T("w"));
for(j=0;j<FileData.size();j++)
  fwprintf(stream, _T("%s"), FileData[j]);
fclose(stream);


However, when I look at the text file, it's gone wrong:
C:\Data\,0,0,0,318.592,83.0455


I'm doing this in unicode. Is there any obvious problem?

Hi Hamish,

I just looked at this again and I think the problem is your use of double 
quotes...  I'm surprised this would compile without complaining.  Of course, 
the Unicode string got a little messed up in the message so it's difficult 
to see the original content.  You may want to just try a standard ASCII 
string for testing to see if that helps as well.

see below:

Tom

"hamishd" <Hamish.Dean@gmail.com> wrote in message 
news:1176341512.504862.224870@w1g2000hsg.googlegroups.com...
I have some texts files, which i want to read in each line, and then
write back each line to a new text file.

So for example, I want to read in the 2 lines below:


"C:\Data\��ҹ�ǿ�",6,10,3,3,8
"",0,0,0,318.592,83.04552

"C:\Data\��ҹ�ǿ�",6,10,3,3,8
"",0,0,0,318.592,83.04552

I store each line of data in the following vector:


std::vector<CString> FileData;


So FileData[0] = ""C:\Data\��ҹ�ǿ�",6,10,3,3,8"
***
Change to (note use of backslashes and additional quotes)

"\"C:\Data\��ҹ�ǿ�\",6,10,3,3,8"
***

FileData[1] = """,0,0,0,318.592,83.04552"

***
Change to (note use of backslashes and additional quotes)

FileData[1] = "\"\"",0,0,0,318.592,83.04552"

***


(the data is in the vector correctly)
Then I go to write the data to a new text file...


FILE * stream;
stream = _wfopen(filename, _T("w"));
for(j=0;j<FileData.size();j++)
  fwprintf(stream, _T("%s"), FileData[j]);
fclose(stream);


However, when I look at the text file, it's gone wrong:
C:\Data\,0,0,0,318.592,83.0455


I'm doing this in unicode. Is there any obvious problem?

> I just looked at this again and I think the problem is your use of double
> quotes...

Yikes. I missed that too.

---
Ajay

Easy to miss... that Unicode rewrite (done by the newsreader during paste no 
doubt) made it kind of like reading an eye test :o)

Tom

"Ajay Kalra" <ajaykalra@yahoo.com> wrote in message 
news:1176387005.117490.6290@q75g2000hsh.googlegroups.com...
>> I just looked at this again and I think the problem is your use of double
>> quotes...
>
> Yikes. I missed that too.
>
> ---
> Ajay
> 

On Wed, 11 Apr 2007 20:13:36 -0700, "Tom Serface"
<tom.nospam@camaswood.com> wrote:

>You could use a CStdioFile to do this easily, not sure about the odd 
>characters showing though, but CStdioFile works with Unicode so if you're 
>using that you'll be safe with just about anything.

To me, it seems that CStdioFile is *not* working with Unicode, but
maybe I'm missing something...

However:
I wrote a simple MFC dialog-based app with a list-box.
I put some Unicode strings into the list-box, and then I save them to
file.

Using both OP's method and CStdioFile fail (when you open the file,
the Chinese Unicode characters aren't there).

Instead, I converted the Unicode (Windows UTF-16) to Unicode UTF-8,
and then wrote the UTF-8 buffer to file.
In this way, I can reopen the file with a text editor, and I can see
the Chinese symbols, too.
(BTW: I wrote "Chinese symbols", but I don't know if they are Chinese
or Japanese or something else from Far East :)


Here's the code main code:

<CODE>
void CTestUnicodeTextDlg::OnBnClickedButton1()
{
    //
    // Put some Unicode strings in list
    //
    std::vector<CString> FileData;
    FileData.push_back(L"Ciao");
    FileData.push_back(L"C:\\Test");

    // UTF-16
    WORD chineseChars[] = 
    { 
        0x4E09,
        0x4E0B,
        0x4E11,
        0x0000, 
    };
    FileData.push_back(CString((LPCWSTR) chineseChars));


    //
    // Show strings
    //
    m_List.ResetContent();
    for ( size_t i = 0; i < FileData.size(); i++ )
    {
        m_List.AddString( FileData.at(i) );
    }

    //
    // Write strings to file
    //

    // *** FAILS ***
    // ...Just writes ANSI...??
    //
    CStdioFile file( 
        _T("test_cstdio.txt"), 
        CFile::modeCreate | CFile::modeWrite | CFile::typeText 
    );
    for( size_t i = 0; i < FileData.size(); i++)
    {
        CString line = FileData.at(i);
        line += L"\n";
        file.WriteString( line );
    }

    //
    // Save in UTF-8 format
    //
    FILE * fout = _wfopen( L"test_utf8.txt", L"wb" );
    for( size_t i = 0; i < FileData.size(); i++)
    {
        // Put the new line (\r\n)
        CString line(FileData.at(i));
        line += L"\r\n";

        // Convert from UTF-16 to UTF-8
        CW2U utf8String( line );

        // Send byte data to output file
        fwrite( (LPSTR) utf8String, 1, strlen(utf8String), fout );
    }
    fclose(fout);
    fout = NULL;
}

</CODE>

And here's the CW2U converter:


<CODE>
//----------------------------------------------------------------------------
// Class: CW2UEX
// Descr: Convert from Unicode UTF-16 (WideChars) to Unicode UTF-8
//----------------------------------------------------------------------------
template< int t_nBufferLength = 128 >
class CW2UEX
{
public:
    CW2UEX( LPCWSTR psz ) throw(...) :
        m_psz( m_szBuffer )
    {
        Init( psz );
    }

    ~CW2UEX() throw()
    {
        if( m_psz != m_szBuffer )
        {
            free( m_psz );
        }
    }

    operator LPSTR() const throw()
    {
        return( m_psz );
    }

private:
    void Init( LPCWSTR psz ) throw(...)
    {
        if (psz == NULL)
        {
            m_psz = NULL;
            return;
        }
        int nLengthW    = lstrlenW( psz )+1;

        // One Unicode UTF-16 character could be converted 
        // up to 4 UTF-8 characters
        int nLengthUtf8 = nLengthW * 4;

        if( nLengthUtf8 > t_nBufferLength )
        {
            m_psz = static_cast< LPSTR >( malloc( 
                      nLengthUtf8*sizeof( char ) ) );
            if (m_psz == NULL)
            {
                AtlThrow( E_OUTOFMEMORY );
            }
        }

        if (::WideCharToMultiByte( CP_UTF8, 0, psz, nLengthW, 
            m_psz, nLengthUtf8, NULL, NULL ) == 0)
        {
            AtlThrowLastWin32();
        }
    }

public:
    LPSTR m_psz;
    char m_szBuffer[t_nBufferLength];

private:
    CW2UEX( const CW2UEX& ) throw();
    CW2UEX& operator=( const CW2UEX& ) throw();
};

typedef CW2UEX<> CW2U;


</CODE>

MrAsm

On 11 Apr 2007 18:31:52 -0700, "hamishd" <Hamish.Dean@gmail.com>
wrote:

>However, when I look at the text file, it's gone wrong:
>C:\Data\,0,0,0,318.592,83.0455
>
>
>I'm doing this in unicode. Is there any obvious problem?

I did some test, and I get the same result (Unicode characters were
"killed", also using CStdioFile [*]).

You might consider my post below in this thread, in which I posted
some code that seems to work (Unicode characters are preserved). I
used Unicode UTF-8 for this.

MrAsm

[*] I'm not a Unicode expert, and maybe I missed something from
CStdioFile or somewhere else.
To be sure about CStdioFile and Unicode not working correctly, better
waiting some expert's post like Mihai's sentences.

CStdioFile will work with Unicode if your program is compiled with Unicode. 
It won't work with UTF-8 at this point.

Tom

"MrAsm" <mrasm@usa.com> wrote in message 
news:ngjs13d3itofk9h8qlp7tcsfjs4l1f1aij@4ax.com...
> On Wed, 11 Apr 2007 20:13:36 -0700, "Tom Serface"
> <tom.nospam@camaswood.com> wrote:
>
>>You could use a CStdioFile to do this easily, not sure about the odd
>>characters showing though, but CStdioFile works with Unicode so if you're
>>using that you'll be safe with just about anything.
>
> To me, it seems that CStdioFile is *not* working with Unicode, but
> maybe I'm missing something...
>
> However:
> I wrote a simple MFC dialog-based app with a list-box.
> I put some Unicode strings into the list-box, and then I save them to
> file.
>
> Using both OP's method and CStdioFile fail (when you open the file,
> the Chinese Unicode characters aren't there).
>
> Instead, I converted the Unicode (Windows UTF-16) to Unicode UTF-8,
> and then wrote the UTF-8 buffer to file.
> In this way, I can reopen the file with a text editor, and I can see
> the Chinese symbols, too.
> (BTW: I wrote "Chinese symbols", but I don't know if they are Chinese
> or Japanese or something else from Far East :)
>
>
> Here's the code main code:
>
> <CODE>
> void CTestUnicodeTextDlg::OnBnClickedButton1()
> {
>    //
>    // Put some Unicode strings in list
>    //
>    std::vector<CString> FileData;
>    FileData.push_back(L"Ciao");
>    FileData.push_back(L"C:\\Test");
>
>    // UTF-16
>    WORD chineseChars[] =
>    {
>        0x4E09,
>        0x4E0B,
>        0x4E11,
>        0x0000,
>    };
>    FileData.push_back(CString((LPCWSTR) chineseChars));
>
>
>    //
>    // Show strings
>    //
>    m_List.ResetContent();
>    for ( size_t i = 0; i < FileData.size(); i++ )
>    {
>        m_List.AddString( FileData.at(i) );
>    }
>
>    //
>    // Write strings to file
>    //
>
>    // *** FAILS ***
>    // ...Just writes ANSI...??
>    //
>    CStdioFile file(
>        _T("test_cstdio.txt"),
>        CFile::modeCreate | CFile::modeWrite | CFile::typeText
>    );
>    for( size_t i = 0; i < FileData.size(); i++)
>    {
>        CString line = FileData.at(i);
>        line += L"\n";
>        file.WriteString( line );
>    }
>
>    //
>    // Save in UTF-8 format
>    //
>    FILE * fout = _wfopen( L"test_utf8.txt", L"wb" );
>    for( size_t i = 0; i < FileData.size(); i++)
>    {
>        // Put the new line (\r\n)
>        CString line(FileData.at(i));
>        line += L"\r\n";
>
>        // Convert from UTF-16 to UTF-8
>        CW2U utf8String( line );
>
>        // Send byte data to output file
>        fwrite( (LPSTR) utf8String, 1, strlen(utf8String), fout );
>    }
>    fclose(fout);
>    fout = NULL;
> }
>
> </CODE>
>
> And here's the CW2U converter:
>
>
> <CODE>
> //----------------------------------------------------------------------------
> // Class: CW2UEX
> // Descr: Convert from Unicode UTF-16 (WideChars) to Unicode UTF-8
> //----------------------------------------------------------------------------
> template< int t_nBufferLength = 128 >
> class CW2UEX
> {
> public:
>    CW2UEX( LPCWSTR psz ) throw(...) :
>        m_psz( m_szBuffer )
>    {
>        Init( psz );
>    }
>
>    ~CW2UEX() throw()
>    {
>        if( m_psz != m_szBuffer )
>        {
>            free( m_psz );
>        }
>    }
>
>    operator LPSTR() const throw()
>    {
>        return( m_psz );
>    }
>
> private:
>    void Init( LPCWSTR psz ) throw(...)
>    {
>        if (psz == NULL)
>        {
>            m_psz = NULL;
>            return;
>        }
>        int nLengthW    = lstrlenW( psz )+1;
>
>        // One Unicode UTF-16 character could be converted
>        // up to 4 UTF-8 characters
>        int nLengthUtf8 = nLengthW * 4;
>
>        if( nLengthUtf8 > t_nBufferLength )
>        {
>            m_psz = static_cast< LPSTR >( malloc(
>                      nLengthUtf8*sizeof( char ) ) );
>            if (m_psz == NULL)
>            {
>                AtlThrow( E_OUTOFMEMORY );
>            }
>        }
>
>        if (::WideCharToMultiByte( CP_UTF8, 0, psz, nLengthW,
>            m_psz, nLengthUtf8, NULL, NULL ) == 0)
>        {
>            AtlThrowLastWin32();
>        }
>    }
>
> public:
>    LPSTR m_psz;
>    char m_szBuffer[t_nBufferLength];
>
> private:
>    CW2UEX( const CW2UEX& ) throw();
>    CW2UEX& operator=( const CW2UEX& ) throw();
> };
>
> typedef CW2UEX<> CW2U;
>
>
> </CODE>
>
> MrAsm 

On Thu, 12 Apr 2007 08:56:40 -0700, "Tom Serface"
<tom.nospam@camaswood.com> wrote:

>CStdioFile will work with Unicode if your program is compiled with Unicode. 

Of course, the program was compiled with Unicode.

MrAsm

On Thu, 12 Apr 2007 08:56:40 -0700, "Tom Serface"
<tom.nospam@camaswood.com> wrote:

>CStdioFile will work with Unicode if your program is compiled with Unicode. 
>It won't work with UTF-8 at this point.

To make it clear: my program was compiled with Unicode (= UTF-16), of
course.
And it seems that CStdioFile did not work with Far East symbols (you
have the code, you can see yourself if you want - maybe I missed
something? But it is not the Unicode compilation :).

And I can have UTF-8 also with Unicode builds. In fact, UTF-8 is just
a byte array, and I wrote that array to file just using a raw C
fwrite() to send the byte buffer to file.

MrAsm

Hi,

I always have a problem when I don't open the file in the correct mode.
binary/text rd/wr etc.

I noticed something similar in the way you were opening the file.
It seems it need a Unicode specifier.

fopen("newfile.txt", "rw, ccs=3D<encoding>");=20
stream =3D _wfopen(filename, _T("w"));

http://msdn2.microsoft.com/en-us/library/yeby3zcb(VS.80).aspx

P.S. I am not sure if "w" needs be unicode.

HTH,

"MrAsm" <mrasm@usa.com> wrote in message 
news:8mns13t396jbk6kr5h602i7ut03r0m7epn@4ax.com...

> To make it clear: my program was compiled with Unicode (= UTF-16), of
> course.
> And it seems that CStdioFile did not work with Far East symbols (you
> have the code, you can see yourself if you want - maybe I missed
> something? But it is not the Unicode compilation :).

I played with this a while back.  I seem to remember that under UNICODE 
CStdioFile still reads files containing the old single byte characters, but 
puts them in strings of wide characters (presumably using the default 
codepage of the computer).  Is this not what it is supposed to do?

Dave
-- 
David Webber
Author MOZART the music processor for Windows -
http://www.mozart.co.uk
For discussion/support see
http://www.mozart.co.uk/mzusers/mailinglist.htm

Asm,

Is that a good email address, if not send me a note at mine, minus the 
..nospam part and I'll send you a couple of files with a replacement for 
CStdioFile that I use that will do Unicode or UTF-8 based on a create flag.

Tom

"MrAsm" <mrasm@usa.com> wrote in message 
news:8mns13t396jbk6kr5h602i7ut03r0m7epn@4ax.com...
> On Thu, 12 Apr 2007 08:56:40 -0700, "Tom Serface"
> <tom.nospam@camaswood.com> wrote:
>
>>CStdioFile will work with Unicode if your program is compiled with 
>>Unicode.
>>It won't work with UTF-8 at this point.
>
> To make it clear: my program was compiled with Unicode (= UTF-16), of
> course.
> And it seems that CStdioFile did not work with Far East symbols (you
> have the code, you can see yourself if you want - maybe I missed
> something? But it is not the Unicode compilation :).
>
> And I can have UTF-8 also with Unicode builds. In fact, UTF-8 is just
> a byte array, and I wrote that array to file just using a raw C
> fwrite() to send the byte buffer to file.
>
> MrAsm 

On Thu, 12 Apr 2007 21:27:29 -0700, "Tom Serface"
<tom.nospam@camaswood.com> wrote:

>Asm,
>
>Is that a good email address, 

Yes my e-mail address is good.


>if not send me a note at mine, minus the 
>.nospam part and I'll send you a couple of files with a replacement for 
>CStdioFile that I use that will do Unicode or UTF-8 based on a create flag.

Thank you.

And what about this?

http://www.codeproject.com/file/stdiofileex.asp

So am I right about the MFC CStdioFile error?
:)


MrAsm

Hi, I got your email and I'll send my revised version of that class you 
mentioned below in a few minutes via email.  I needed to support Unicode or 
UTF-8 or ANSI based on BOM in the file.  I don't use the libary version of 
CStdioFile any more, but I could have sworn it was supposed to work with 
Unicode in a Unicode build, but ...

Tom

"MrAsm" <mrasm@usa.com> wrote in message 
news:raeu13hnkunsbd6e4usle63qqop5pq8nfj@4ax.com...
> On Thu, 12 Apr 2007 21:27:29 -0700, "Tom Serface"
> <tom.nospam@camaswood.com> wrote:
>
>>Asm,
>>
>>Is that a good email address,
>
> Yes my e-mail address is good.
>
>
>>if not send me a note at mine, minus the
>>.nospam part and I'll send you a couple of files with a replacement for
>>CStdioFile that I use that will do Unicode or UTF-8 based on a create 
>>flag.
>
> Thank you.
>
> And what about this?
>
> http://www.codeproject.com/file/stdiofileex.asp
>
> So am I right about the MFC CStdioFile error?
> :)
>
>
> MrAsm 

On Fri, 13 Apr 2007 07:37:17 -0700, "Tom Serface"
<tom.nospam@camaswood.com> wrote:

>Hi, I got your email and I'll send my revised version of that class you 
>mentioned below in a few minutes via email.  

Hi Tom, your class does work also with my Chinese test example!

(I've replyed to your e-mail.)

I think your class is the working version to replace the MFC
CStdioFile. Your class could be useful for the OP, too.
And maybe you could write a knowledge-base for Microsoft about that
issue.

MrAsm