Discussion:
Error during commit
(too old to reply)
Philip Bennefall
2018-08-06 16:11:33 UTC
Permalink
We have a repository in our organization which stores a number of rather
large binary files. When attempting to commit, we sometimes get
something like this:


ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the
repository


The file in question was not modified during the given commit, but was
checked in months ago. What can cause this, and what steps can we take
to investigate?


We are using Fossil 2.6.


Thanks in advance for any help.


Kind regards,


Philip Bennefall
Philip Bennefall
2018-08-06 16:13:04 UTC
Permalink
Additional information: The repo checksum is disabled locally. When
enabling it, we get:

Segmentation fault: 11

Kind regards,

Philip Bennefall
Post by Philip Bennefall
We have a repository in our organization which stores a number of
rather large binary files. When attempting to commit, we sometimes get
ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the
repository
The file in question was not modified during the given commit, but was
checked in months ago. What can cause this, and what steps can we take
to investigate?
We are using Fossil 2.6.
Thanks in advance for any help.
Kind regards,
Philip Bennefall
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Richard Hipp
2018-08-06 16:33:39 UTC
Permalink
Post by Philip Bennefall
We have a repository in our organization which stores a number of rather
large binary files. When attempting to commit, we sometimes get
ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the
repository
My guess is that you are probably overflowing a 32-bit integer
someplace. If so, I fear that this will not be something easily
fixed.
--
D. Richard Hipp
***@sqlite.org
Philip Bennefall
2018-08-06 16:41:46 UTC
Permalink
Do you have any recommendations for something we could try in order to
get more information, or would you suggest that we switch to another
DVCS if we need to store files of these sizes?


Kind regards,


Philip Bennefall
Post by Richard Hipp
Post by Philip Bennefall
We have a repository in our organization which stores a number of rather
large binary files. When attempting to commit, we sometimes get
ERROR: [largefile.bin is 999378424 bytes on disk but 210746789 in the
repository
My guess is that you are probably overflowing a 32-bit integer
someplace. If so, I fear that this will not be something easily
fixed.
Richard Hipp
2018-08-06 16:50:35 UTC
Permalink
Post by Philip Bennefall
Do you have any recommendations for something we could try in order to
get more information, or would you suggest that we switch to another
DVCS if we need to store files of these sizes?
Regardless of the problem, I don't think *any* DVCS is appropriate for
storing gigabyte sized binary files. You should look into a
centralized VCS such as subversion, methinks.
--
D. Richard Hipp
***@sqlite.org
Philip Bennefall
2018-08-06 16:57:06 UTC
Permalink
OK. I'll investigate. Thanks for the quick response.


Kind regards,


Philip Bennefall
Post by Richard Hipp
Post by Philip Bennefall
Do you have any recommendations for something we could try in order to
get more information, or would you suggest that we switch to another
DVCS if we need to store files of these sizes?
Regardless of the problem, I don't think *any* DVCS is appropriate for
storing gigabyte sized binary files. You should look into a
centralized VCS such as subversion, methinks.
Philip Bennefall
2018-08-06 17:15:22 UTC
Permalink
The following command solved the issue at least for the moment:

fossil rebuild --vacuum --analyze --compress

I'm not yet sure which of the options made the difference, but I wanted
to report back nevertheless.

Kind regards,

Philip Bennefall
Post by Philip Bennefall
OK. I'll investigate. Thanks for the quick response.
Kind regards,
Philip Bennefall
Post by Richard Hipp
Post by Philip Bennefall
Do you have any recommendations for something we could try in order to
get more information, or would you suggest that we switch to another
DVCS if we need to store files of these sizes?
Regardless of the problem, I don't think *any* DVCS is appropriate for
storing gigabyte sized binary files. You should look into a
centralized VCS such as subversion, methinks.
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Philip Bennefall
2018-08-06 17:30:12 UTC
Permalink
I spoke too soon. The first commit went through after running the
rebuild with the options I listed, but then a second commit right
afterwards caused a segfault again. Running:

fossil status -sha1sum

And then commit again worked. I'm not sure if these are related in any
way or if it's just a fluke.

I definitely realize that storing files of these sizes is very unusual.
Fossil seems to handle thousands of smaller files easily, it's only when
a single file starts to get into the hundreds of megabytes range that it
bombs. But I wanted to report our experience in case it is of use to the
developers, and in case doing so could give us some assistance with the
immediate issue.

Kind regards,

Philip Bennefall
Post by Philip Bennefall
fossil rebuild --vacuum --analyze --compress
I'm not yet sure which of the options made the difference, but I
wanted to report back nevertheless.
Kind regards,
Philip Bennefall
Post by Philip Bennefall
OK. I'll investigate. Thanks for the quick response.
Kind regards,
Philip Bennefall
Post by Richard Hipp
Post by Philip Bennefall
Do you have any recommendations for something we could try in order to
get more information, or would you suggest that we switch to another
DVCS if we need to store files of these sizes?
Regardless of the problem, I don't think *any* DVCS is appropriate for
storing gigabyte sized binary files. You should look into a
centralized VCS such as subversion, methinks.
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Richard Hipp
2018-08-06 17:52:32 UTC
Permalink
Post by Philip Bennefall
But I wanted to report our experience in case it is of use to the
developers, and in case doing so could give us some assistance with the
immediate issue.
Thank you for the report. This is definitely something that should be
fixed. But I have a large queue of other issues of relevance to far
more users (ex: support for email alerts and forum) that need to be
fixed first.
--
D. Richard Hipp
***@sqlite.org
Warren Young
2018-08-06 22:06:49 UTC
Permalink
a second commit right afterwards caused a segfault again
In advance of drh getting time to work on this, maybe you could give two debugging steps on your end:

1. If you’re doing this on a platform that will run Valgrind, try running a debug build of Fossil under it against your repository:

$ cd /path/to/fossil/source/tree
$ ./configure --fossil-debug
$ make clean && make && sudo make install
$ cd /path/to/checkout/directory
$ valgrind fossil status

2. If you get a vgcore out of that, load it into GDB and produce a backtrace. If not, cores are probably disabled there, so run Fossil under gdb and get a backtrace that way.

(The first way is more likely to give a clean backtrace, as the failure gets caught before it can cause damage.)
Richard Hipp
2018-08-06 17:12:34 UTC
Permalink
Post by Richard Hipp
Post by Philip Bennefall
Do you have any recommendations for something we could try in order to
get more information, or would you suggest that we switch to another
DVCS if we need to store files of these sizes?
Regardless of the problem, I don't think *any* DVCS is appropriate for
storing gigabyte sized binary files. You should look into a
centralized VCS such as subversion, methinks.
Further information:

Fossil was written for the purpose of managing the SQLite project.
The primary SQLite Fossil repository contains about 5.8 GB of content
when fully expanded, but that is compressed down to 92 MB. In other
words, the whole repository spanning 18 years of development and 20K
check-ins and 78K artifacts is less than 1/10th the size of just one
of your files.

The largest uncompressed artifact in the SQLite repo is 19 MB and the
median uncompressed artifact size is 42 KB. With compression the
largest artifact is 1.8 MB and median artifact size is a mere 152
bytes.

I analyzed a bunch of other Fossil repositories, and they all mostly
have similar stats. Except, usually the maximum uncompressed artifact
size is about an order of magnitude less than 19 MB (SQLite is has a
few exceptionally large artifacts) and in the case of the SQL Logic
Test repository, the median artifact size was 1.5 KB.
--
D. Richard Hipp
***@sqlite.org
Artur Shepilko
2018-08-07 17:32:48 UTC
Permalink
Post by Richard Hipp
My guess is that you are probably overflowing a 32-bit integer
someplace. If so, I fear that this will not be something easily
fixed.
Looks like it's the case at least in one place.
There's int64 => unsigned int overflow at the call to blob_resize()
(blob.c:865 [http://fossil-scm.org/index.html/artifact?udc=1&ln=on&name=b3ef46413b7313ca]
).

Generally this is tied to the way the Blob type is defined, so the fix
indeed may be painful and involve thorough testing.
This only becomes an issue with LARGE file sizes that overflow the
(unsigned int), which as DRH mentioned is somewhat beyond the expected
Fossil use-cases.

From the code I don't see any easy workaround other than addressing
the sizes of the files stored at the client's end.

May be trying to build a 64-bit version of the Fossil and explicitly
force the int to 64-bit size
BUT this may blow in some other place, so I would not trust it even if
it worked.

I also wonder at effectiveness of tracking the binaries, as these
won't diff properly, thus each new version would likely be stored
entirely, which would kinda turns repo into a gigantic zip archive
with a lot of overhead. I'd rather keep track of a __reference__ to
such binaries, while the binaries could be stored at some other bulk
storage location. Thus a project version would fetch an updated
reference and the pull the resp. binary.


-------------------
src/vfile.c:820
[http://fossil-scm.org/index.html/artifact?udc=1&ln=on&name=8c891c745e280c02]

rc = blob_read_from_file(&disk, zFullpath, RepoFILE);

-------------------
src/blob.c:843 [http://fossil-scm.org/index.html/artifact?udc=1&ln=on&name=b3ef46413b7313ca]

sqlite3_int64 blob_read_from_file(
Blob *pBlob, /* The blob to be initialized */
const char *zFilename, /* Extract content from this file */
int eFType /* ExtFILE or RepoFILE - see above */
){
sqlite3_int64 size, got;
...

size = file_size(zFilename, eFType);
...
blob_resize(pBlob, size);

-------------------
src/blob.c:442 [http://fossil-scm.org/index.html/artifact?udc=1&ln=on&name=b3ef46413b7313ca]

void blob_resize(Blob *pBlob, unsigned int newSize){
pBlob->xRealloc(pBlob, newSize+1);
pBlob->nUsed = newSize;
pBlob->aData[newSize] = 0;
}

Loading...