- I belong to the school that claims protocols should be human-readable,
  because, well, it opens them up.  They get so much easier to
  manipulate, and even talk about.  Take HTTP as an example, or SMTP.

- I doubt that programmers are that good with hex.  Even if I limit
  myself to unsigned int, I can't tell what 0xbabe is.  Probably 40000
  or so. Or 30000?  Who knows?  There is a reason decimal is the default
  base in pretty much every language I know of ... including assembly
  languages.

...

I think we have been misled a bit here, too. I haven't read the whole
thread, but it started with something like "dump a huge array of
floats to disk, collect it later".  If you take the more common case
"take this huge complex data structure and dump it to disk in a
portable format", you have a completely different situation, where the
non-text format isn't that much smaller or faster.

/Jorgen

Much earlier in the thread, Allnor wrote, "Binary files
are usually about 20%-70% of the size of the text file,
depending on numbers of significant digits and other
formatting text glyphs."  I don't think anyone has
directly disagreed with that statement yet.

Brian Wood
Ebenezer Enterprises
www.webEbenezer.net

There was, in fact, an 'improvment' to the ancient SEG-Y seismic
data format,

http://en.wikipedia.org/wiki/SEG_Y

the SEG-2,

http://diwww.epfl.ch/lami/detec/seg2.html

where a lot of the auxillary (numeric) information was specificed
to be stored on text format. I first saw the SEG-2 spec about ten
years ago, but I have never heard that it has actually been used.
The speed losses involved with converting data back and forth from
text to binary would fully explain why SEG-2 does not gain wide-
spread acceptence among the heavy-duty users.

In fact, now that you've mentionned seismic data, I agree that a
text format is probably not going to cut it.  I've actually
worked on one project in the field, and I know just how much
floating point data they can generate.

I'm getting tired with re-iterating this for people who
are not interested in actually evaluating the numbers.

Look for an upcomimg post on comp.lang.c++.moderated, where
I distill the problem statement a bit, as well as present
a C++ test to see what kind of timing ratios I am talking about.

Hi all.

A couple of weeks ago I posted a question on comp.lang.c++ about some
technicality
about binary file IO. Over the course of the discussion, I discovered
to my
amazement - and, quite frankly, horror - that there seems to be a
school of
thought that text-based storage formats are universally preferable to
binary text
formats for reasons of portability and human readability.

The people who presented such ideas appeared not to appreciate two
details that
counter any benefits text-based numerical formats might offer:

1) Binary files are about 70-20% of the file size of the text files,
depending
   on the number of significant digits stored in the text files and
other
   formatting text glyphs.
2) Text-formatted numerical data take significantly longer to read and
write
   than binary formats.

Timings are difficult to compare, since the exact numbers depend on
buffering
strategies, buffer sizes, disk speeds, network bandwidths and so on.

I have therefore sketched a 'distilled' test (code below) to test what
overheads
are involved with formatting numerical data back and forth between
text and
binary formats. To eliminate the impact of peripherical devices, I
have used
a std::stringstream to store the data. The binary bufferes are
represented
by vectors, and I have assumed that a memcpy from the file buffer to
the
destination memory location is all that is needed to import the binary
format
from the file buffer. (If there are significant run-time overheads
associated with
moving NATIVE binary formats to the destination, please let me
know.)

The output on my computer is (do note the _different_ numbers of IO
cycles in the two cases!):

Sun Nov 08 19:48:54 2009 : Binary IO cycles started
Sun Nov 08 19:49:00 2009 : 1000 Binary IO cycles completed
Sun Nov 08 19:49:00 2009 : Text-format IO cycles started
Sun Nov 08 19:49:16 2009 : 100 Text-format IO cycles completed

A little bit of math produces *average*, *crude* numbers for IO
cycles:

Text:    6 seconds / (1000 * 1e6) read/write cycles =   6e-9 s per r/w
cycle
Binary: 16 seconds / (100  * 1e6) read/write cycles = 160e-9 s per r/w
cycle

which in turn means there is an overhead on the order of of
160e-9/6e-9 = 26x
associated with the text formats.

Add a little bit of other overheads, e.g. caused by the significantly
larger text file sizes in combination with suboptimal buffering
strategies,
and the relative numbers easily hit the triple digits. Not at all
insignificant when one works with large amounts of data under tight
deadlines.

So please: Shoot this demo down! Give it your best, and prove me
and my numbers wrong.

And to the textbook authors who might be lurking: Please include a
chapter on relative binary and text-based IO speeds in your upcoming
editions. Binary file formats might not fit into your overall
philosophies about human readability and universal portability of C++
code, but some of your readers might appreciate being made aware of
such practical details.

Rune

/
*************************************************************************** /
#include <iostream>
#include <sstream>
#include <time.h>
#include <vector>

int main()
{
        const size_t  NumElements = 1000000;
        std::vector<double> SourceBuffer;
        std::vector<double> DestinationBuffer;

        for (size_t n=0;n<NumElements;++n)
        {
                SourceBuffer.push_back(n);
                DestinationBuffer.push_back(0);
        }

        time_t rawtime;
        struct tm * timeinfo;

        time( &rawtime );
        timeinfo = localtime( & rawtime );
        std::string message( asctime (timeinfo) );
        message.erase(message.size()-1);

        std::cout  << message.c_str() << " : Binary IO cycles started"
                    << std::endl;

        size_t NumBinaryIOCycles = 1000;
        for (size_t n = 0; n < NumBinaryIOCycles; ++n)
        {
                for (size_t m = 0; m<NumElements; ++m )
                {
                        DestinationBuffer[m] = SourceBuffer[m];
                }
        }

        time( &rawtime );
        timeinfo = localtime( & rawtime );
        message=std::string( asctime (timeinfo) );
        message.erase(message.size()-1);

        std::cout << message.c_str() << " : " << NumBinaryIOCycles
                << " Binary IO cycles completed " << std:: endl;

        std::stringstream ss;
        const size_t NumTextFormatIOCycles = 100;

        time( &rawtime );
        timeinfo = localtime( & rawtime );
        message=std::string( asctime (timeinfo) );
        message.erase(message.size()-1);

        std::cout  << message.c_str() << " : Text-format IO cycles
started"
                   << std::endl;

        for (size_t n = 0; n < NumTextFormatIOCycles; ++n)
        {
                size_t m;
                for (m = 0; m < NumElements; ++m)
                        ss << SourceBuffer[m];

                m = 0;
                while(!ss.eof())
                {
                        ss >> DestinationBuffer[m];
                        ++m;
                }
        }

        time( &rawtime );
        timeinfo = localtime( & rawtime );
        message=std::string( asctime (timeinfo) );
        message.erase(message.size()-1);

        std::cout << message.c_str() << " : " << NumTextFormatIOCycles
                << " Text-format IO cycles completed " << std:: endl;

        return 0;

Cheers,

Let's see what I have written. From my post

http://groups.google.no/group/comp.lang.c++/msg/1c4004bbac86a046

[RA] > > File I/O operations with text-formatted floating-point data
     > > take time. A *lot* of time.

[JK] > A lot of time compared to what?

[RA] Wall clock time. Relative time, compared to dumping
     binary data to disk. Any way you want.

     ...
[RA] > > The rule-of-thumb is 30-60 seconds per 100 MBytes of
     > > text-formatted FP numeric data, compared to fractions of a
     > > second for the same data (natively) binary encoded (just try
     > > it).

[JK] > Try it on what machine:-).

[RA] Any machine. The problem is to decode text-formatted numbers
     to binary.

     ...
     Here is a test I wrote in matlab a few years ago, to demonstrate
     the problem (WinXP, 2.4GHz, no idea about disk):

     [matlab  code snipped]

     Output:
     ------------------------------------
     Wrote ASCII data in 24.0469 seconds
     Read ASCII data in 42.2031 seconds
     Wrote binary data in 0.10938 seconds
     Read binary data in 0.32813 seconds
     ------------------------------------

     Binary writes are 24.0/0.1 = 240x faster than text write.
     Binary reads are 42.2/0.32 = 130x faster than text read.

     ...
     The timing numbers (both absolute and relative) would be of
     similar orders of magnitude if you repeated the test with C++.
     ...
     The application I'm working with would need to crunch through
     some 10 GBytes of numerical data per hour.

I think these excerpts should be sufficient to sketch what
kind of world I am living and working in.

Do note thet I never - unlike some other paricipants in this
thread - claimed my numbers to be exact. I am fairly certain
my English is good enough that the above would reasonably be
expected to be interpreted by a reader as *representative*
numbers. If you look closely, I also commented that coding
up a program in C++ instead of matlab as I had done, would
result in *different* numbers, but not solve the fundamental
problem.

So I can't see any reason why you attack me for my numbers
being "wrong"; I never stated they were exact.

A few posts further out:

http://groups.google.no/group/comp.lang.c++/msg/0abdc440e78f98d6

  [RA] So what does text-based formats actually buy you?
  [JK] Shorter development times, less expensive development, greater
       reliability...

       In sum, lower cost.

  [RA] As long as you keep two factors in mind:
       1) The user's time is not yours (the programmer) to waste.
       2) The users's storage facilities (disk space, network
          bandwidth etc) are not yours (the programmer) to waste.

  [JK] The user pays for your time.  Spending it to do something
which
       results in a less reliable program, and that he doesn't need,
is
       irresponsible, and borders on fraud.

This one really pissed me off. Here I had explained to you
what application I am working with, made you aware of the users
requirements in the operational situation, and you explicitly
state that paying attention to such concerns is 'borderline fraud'!

So I can not interpret this in any other way than that you will
use text-based formats, come hell or high water. Which essentially
invalidate any otherwise relevant arguments you might have presented
throughout thread.

Such factors govern what problems can be handled by the software
with reasonable effort and within reasonable time.

True, both binary and text-based numerical IO are O(N), but since
text-based numerical IO is orders of magnitude slower, the strategic
impact on design decisions is the same.

This is exactly what I have been arguing for days and weeks already.
What changed?

There are optimizations and optimizations.  Sometimes you do
know up front that you'll need the optimization; if you know
that you'll have to deal with millions of elements, you know up
front that a quadratic algorithm won't do the trick.

In the case of choosing binary, the motivation for doing so up
front is a bit different---after all, the difference will never
be other than linear.  Partially, the motivation can be
calculated: if you know the number of elements, you can
calculate the disk space needed up front.  In many cases,
however, you know that you'll be locked into the format you
choose, so you have to consider performance issues earlier.
Once you start considering performance issues, however, you're
talking about optimization.

"However, reading and writing binary files can have HUGE
performance gains.  I once came across some numerical code
where it would read and write large datasets. These datasets
were 40-100MB.  The performance was horrendus.  Using mapped
files and binary data made the reading and writing virtually
zero cost and it improved the performance of the product by
nearly 10x times and in some tests over 1000x.  Be careful -
this is one application and the bottle neck was clearly
identified.  This may not be where your application spends
its time."

I hope to beef up the C++ Middleware Writer's support
for writing and reading data more generally.  To begin
with I'm going to focus on integral types and assume
8 bit bytes.  Currently we don't have support for uint8_t,
uint16_t, etc.  I guess those are the types I'll start with.
I'm going through the newsgroup archives to find snippets
that are helpful in this area.  If anyone has a link wrt
this, I'm interested.