Discussion:
Off-topic faith declarations (was Re: "how to use git to lose data")
(too old to reply)
Marcelo
2014-09-02 12:57:44 UTC
Permalink
10) Considering 9) (above) it's a proof that those serving God, serve
other people as well.
As a atheist I protest that this is deeply unsettling and off-topic.

You believers have almost ALL the other existing forums for
yourselves. Can you keep your exhibitions of faith out of SOME places,
for crying out loud?
Richard Hipp
2014-09-02 13:31:51 UTC
Permalink
Post by Marcelo
10) Considering 9) (above) it's a proof that those serving God, serve
other people as well.
As a atheist I protest that this is deeply unsettling and off-topic.
I think that Gour's comment was appropriate to the subject at hand and in
good taste. Gour's remark was in no way judgmental or condescending and it
contributed (tangentially) to the topic. Marcelo's reply, on the other
hand, loudly proclaims his particular world-view and posits that anyone who
disagrees must remain silent. Marcelo's reply is definitely off-topic.

Marcelo, you are welcomed to contribute and ask questions and comment here
and I hope that you will continue to participate in this forum. However, I
ask that in the future you keep your remarks civil and on-topic and that
you avoid trying to enforce your world-view on other members of the
community.
--
D. Richard Hipp
***@sqlite.org
s***@gmail.com
2014-09-02 14:18:35 UTC
Permalink
Well said and yes, Fossil is a non-tedious, benevolent lifesaver. My
reservation being scalability of large repo support. While I am unaffected,
those professionals charged with release and maintenance of large code
bases look past Fossil and its SQLite core.
Questions:
Will Fossil ever seek to address very large source control?
Can Fossil offer 2 solutions? SQLite based and PostgreSQL(insert big RDB
here)?

Thanks for Fossil!
Post by Richard Hipp
Post by Marcelo
10) Considering 9) (above) it's a proof that those serving God, serve
other people as well.
As a atheist I protest that this is deeply unsettling and off-topic.
I think that Gour's comment was appropriate to the subject at hand and in
good taste. Gour's remark was in no way judgmental or condescending and it
contributed (tangentially) to the topic. Marcelo's reply, on the other
hand, loudly proclaims his particular world-view and posits that anyone who
disagrees must remain silent. Marcelo's reply is definitely off-topic.
Marcelo, you are welcomed to contribute and ask questions and comment here
and I hope that you will continue to participate in this forum. However, I
ask that in the future you keep your remarks civil and on-topic and that
you avoid trying to enforce your world-view on other members of the
community.
--
D. Richard Hipp
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Ron W
2014-09-02 18:02:39 UTC
Permalink
Post by s***@gmail.com
Can Fossil offer 2 solutions? SQLite based and PostgreSQL(insert big RDB
here)?
I think that the only way this will happen would be to fork Fossil into a
new project. This would be because of the overall underlying goals of the
Fossil project vs a "Fossil-saurus" project.

As I understand it, Fossil is intentionally designed around the feature set
provided by SQLite. Therefore, to support DB back-ends other than SQLite
would not just require rewriting SQL queries, but significant re-working of
Fossil's C implementation.

Certainly, a huge scale DVCS like Fossil would be a good thing. A key
question for a potential project team will be: Once you separate out the DB
back-end, where do you draw the line?

As for Fossil itself, it probably scales better than most people think.
Part of the problem is perception. Part of the problem is lack of certain
features that some people consider desirable or essential.

I know many people who are more comfortable with "large scale" systems like
Perforce, because they are used to working in large organizations. And I
know that some of those do/did work on large software teams. The rest are
like me: The overall "software team" is large, but for any given project,
the number of software members is usually 2, sometimes 1 or 3, rarely more
than that. The "high ceremony" system we are supposed to be using is not
helpful to us.
John Long
2014-09-02 18:11:03 UTC
Permalink
Post by Ron W
Post by s***@gmail.com
Can Fossil offer 2 solutions? SQLite based and PostgreSQL(insert big RDB
here)?
I think that the only way this will happen would be to fork Fossil into a
new project. This would be because of the overall underlying goals of the
Fossil project vs a "Fossil-saurus" project.
As I understand it, Fossil is intentionally designed around the feature set
provided by SQLite. Therefore, to support DB back-ends other than SQLite
would not just require rewriting SQL queries, but significant re-working of
Fossil's C implementation.
I'm probably out of my area but it would seem writing a wrapper to transform
the database calls to $BACKEND_OF_CHOICE would be in order unless sqlite
does stuff that is really not appropriate for other databases and cannot
readily be transformed to make sense with other databases.

"Theoretically" it should be possible to do that without changing the API
calls at all. You would "just" link the wrapper instead of sqlite. If you
can get this to work it's obviously the safest, most low-impact way to do a
major change like that while not breaking what already works.

I have done stuff like this on another platform but am not familiar with any
of the parts here i.e. I have no knowledge of C, fossil, etc.

/jl
--
ASCII ribbon campaign ( ) Powered by Lemote Fuloong
against HTML e-mail X Loongson MIPS and OpenBSD
and proprietary / \ http://www.mutt.org
attachments / \ Code Blue or Go Home!
Encrypted email preferred PGP Key 2048R/DA65BC04
David Given
2014-09-02 18:35:17 UTC
Permalink
On 9/2/14, 8:02 PM, Ron W wrote:
[...]
Post by Ron W
I think that the only way this will happen would be to fork Fossil into a
new project. This would be because of the overall underlying goals of the
Fossil project vs a "Fossil-saurus" project.
It may be plausible to simply implement a RDBMS wrapper library which
presents a SQLite-like API. That would neatly encapsulate and translate
all the SQLite-isms used by Fossil, which means you wouldn't have to
change Fossil at all.

I have no idea whether this is feasible or not. I don't really know how
different SQLite's SQL dialect is from other databases --- since
discovering SQLite I haven't really felt a need to get into MySQL or
Postgres --- and there may also be killer non-API assumptions: e.g.,
with SQLite, local processing is cheap. With a remote DBMS, local
processing is expensive, because you have to push all the data over the
wire. So if Fossil's expecting to be able to cheaply enumerate big
chunks of the version graph, that's unlikely to adapt well to a remote
database.

[...]
Post by Ron W
As for Fossil itself, it probably scales better than most people think.
Part of the problem is perception.
I believe NetBSD has one of the bigger Fossil repositories, and they've
complained that some Fossil operations are slow (commits and updates ---
http://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.pdf claims
8-10 seconds for an operation which git and hg do in 1.

I have a vague memory of a discussion here coming to the conclusion that
this is because of large chains of deltas having to be resolved?

(I myself have a 250MB Fossil repository with a number of large files in
it (multi-megabyte jpegs) and have noticed that operations are
noticeably slower there than on small repositories.)

[...]
Post by Ron W
I know many people who are more comfortable with "large scale" systems like
Perforce, because they are used to working in large organizations.
Perforce handles very, very large repositories very well. (Repositories
too large for git.) e.g.:

http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/34459.pdf

Brief summary: it says that Google's VCS, as of 2011, had 200GB of
*metadata* (amount of actual data is unspecified) and did about 25qps.

(Perforce is *weird*. You thought git was weird? Go use Perforce.)
--
┌───  ───── http://www.cowlark.com ─────
│ "Feminism encourages women to leave their husbands, kill their
│ children, practice withcraft, destroy capitalism and become lesbians."
│ --- Rev. Pat Robertson
Ron W
2014-09-02 20:28:51 UTC
Permalink
Post by David Given
Perforce handles very, very large repositories very well. (Repositories
One of my clients uses Perforce. Yes, it can handle huge repositories very
well. The company has a few projects that benefit both from the ability to
handle huge repositories and from the level of management control
Perforce's user interface uses. But most of the projects have tiny (2 to 5
SW devs) teams that don't benefit from those controls.
Post by David Given
(Perforce is *weird*. You thought git was weird? Go use Perforce.)
Yeah, I've used it. Though the above mentioned client has several in-house
plug-ins for Perforce that, while add some capabilities the client finds
helpful, they also prevent use of some built-in features (for example, it
is necessary to manually maintain each project's manifest of files (I did
create a plug-in to partially automate this, but it still some manual
intervention))
Stephan Beal
2014-09-03 15:45:53 UTC
Permalink
Post by David Given
I have no idea whether this is feasible or not. I don't really know how
different SQLite's SQL dialect is from other databases --- since
discovering SQLite I haven't really felt a need to get into MySQL or
Postgres --- and there may also be killer non-API assumptions: e.g.,
with SQLite, local processing is cheap. With a remote DBMS, local
processing is expensive, because you have to push all the data over the
wire. So if Fossil's expecting to be able to cheaply enumerate big
chunks of the version graph, that's unlikely to adapt well to a remote
database.
See my previous reply for some on that. Note, also, that the C APIs are
_exceedingly_ different. The MySQL C API is an absolute nightmare to work
with. Postgres' C API (insofar as i can determine) cannot figure out on its
own how many custom parameters a statement contains - the caller has to
pass that himself, which makes in impossible to use in general-purpose APIs
without writing code which parses SQL and figure out how many times bound
parameters it contains. (Maybe there's a C API for this somewhere - i
couldn't find it).

sqlite's API, OTOH, is an absolute dream to work with, in terms of
client-side effort vs sqlite effort. i.e. you have to type a little bit,
and for that sqlite does a lot. MySQL requires you know how big each result
column will be, in advance, so that you can allocate the memory for it (in
advance, and then manage it yourself). It's truly painful.
Post by David Given
I believe NetBSD has one of the bigger Fossil repositories, and they've
complained that some Fossil operations are slow (commits and updates ---
http://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.pdf claims
8-10 seconds for an operation which git and hg do in 1.
If the pkgsrc repo can do _anything_ in 8-10 seconds, i'd be impressed ;).
AFAIK, it's the biggest repo out there, in terms of # of files/size. There
are several quite massive ones, though (TCL core is the biggest repo i've
played with).
Post by David Given
I have a vague memory of a discussion here coming to the conclusion that
this is because of large chains of deltas having to be resolved?
Partially, yes, and partially do the the repo-cksum which came up earlier
(a.k.a. "the R-card").

(I myself have a 250MB Fossil repository with a number of large files in
Post by David Given
it (multi-megabyte jpegs) and have noticed that operations are
noticeably slower there than on small repositories.)
Applying fossil deltas requires, in general, 2.X copies in memory at once:
the original, the delta, and the output version. That gets expensive
quickly, in terms of memory, for large files. It would be really easy to
run out of memory on smaller systems (e.g. an RPi), and fossil's
out-of-memory policy is to abort() (which, as horrible as that initially
sounds, cuts the amount of implementation code tremendously because we
never have to check for allocation errors - many of the libfossil
counterparts of fossil core algos are 2x as long, solely because of the
additional error checking needed for the lib-style API).
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Stephan Beal
2014-09-03 15:36:03 UTC
Permalink
Post by Ron W
As I understand it, Fossil is intentionally designed around the feature
set provided by SQLite. Therefore, to support DB back-ends other than
SQLite would not just require rewriting SQL queries, but significant
re-working of Fossil's C implementation.
Right. sqlite does a surprising amount of the work for us, and having its
author write most of the Fossil queries gives us a distinct advantage so
long as we use sqlite ;). We also have a huge advantage in the tight
coupling of sqlite, in that we know that any serious issues in sqlite get
fixed immediately ;), instead of having to wait on the next system update
to get the latest Postgres/MySQL/whatever.


Certainly, a huge scale DVCS like Fossil would be a good thing. A key
Post by Ron W
question for a potential project team will be: Once you separate out the DB
back-end, where do you draw the line?
i had to consider some of these points when starting libfossil. One of the
ideas i _idly_ tinkered with, in terms of API structure, but never
seriously considered, was non-db backends for fossil. It turns out, though,
that it's impossible to codify, in advance, all possible data structures
which clients might want/need. So.... the db API has to be exposed to the
client so that they can define their own data structures (in the form of
queries). Creating a storage-agnostic API which "works enough like a DB"
that fossil could make reasonable use of it would be a years-long project
in and of itself.
Post by Ron W
As for Fossil itself, it probably scales better than most people think.
Part of the problem is perception.
Guilty as charged. Can't give you concrete reason, though. Maybe it's my
inherent pessimism (which possibly has its roots in me enjoying the
surprise of, "oh, it CAN be done!").

I know many people who are more comfortable with "large scale" systems like
Post by Ron W
Perforce, because they are used to working in large organizations. And I
know that some of those do/did work on large software teams. The rest are
like me: The overall "software team" is large, but for any given project,
the number of software members is usually 2, sometimes 1 or 3, rarely more
than that. The "high ceremony" system we are supposed to be using is not
helpful to us.
+1
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Continue reading on narkive:
Loading...