Discussion:
Remove flat-view from menu? Was: File age in the tree view
(too old to reply)
Richard Hipp
2014-12-17 01:05:53 UTC
Permalink
Given the much improved tree-view functionality

https://www.fossil-scm.org/fossil/tree?ci=trunk&mtime

Is there really any reason to keep the legacy "Flat-View" on the menu bar?

https://www.fossil-scm.org/fossil/tree?ci=trunk&type=flat

I think the flat-view functionality should be preserved. (Who knows how
many links to the flat-view exist on various wiki pages, tickets, and
check-in comments.) But I don't see a reason to have it using up valuable
menu-bar space.

Comments?
--
D. Richard Hipp
***@sqlite.org
Stephan Beal
2014-12-17 01:16:43 UTC
Permalink
Post by Richard Hipp
Given the much improved tree-view functionality
https://www.fossil-scm.org/fossil/tree?ci=trunk&mtime
Is there really any reason to keep the legacy "Flat-View" on the menu bar?
https://www.fossil-scm.org/fossil/tree?ci=trunk&type=flat
I think the flat-view functionality should be preserved. (Who knows how
many links to the flat-view exist on various wiki
pages, tickets, and check-in comments.) But I don't see a reason to have
Post by Richard Hipp
it using up valuable menu-bar space.
Actually... i use the flat view more often than not. That's probably just
historical momentum (and the way my old menus are all set up). i wouldn't
whine about loss of flat view, but also won't argue for its removal.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Stephan Beal
2014-12-17 01:17:37 UTC
Permalink
Post by Stephan Beal
Actually... i use the flat view more often than not. That's probably just
historical momentum (and the way my old menus are all set up). i wouldn't
whine about loss of flat view, but also won't argue for its removal.
flat mode meaning /dir instead of /tree.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Richard Hipp
2014-12-17 01:31:12 UTC
Permalink
Post by Stephan Beal
Post by Stephan Beal
Actually... i use the flat view more often than not. That's probably just
historical momentum (and the way my old menus are all set up). i wouldn't
whine about loss of flat view, but also won't argue for its removal.
flat mode meaning /dir instead of /tree.
Yes. And remember, I'm not going to make /dir go away - I'm just wanting
to take away its prominent links on the menu bar. The pages will all still
be there for those who want them. But links to those pages won't clutter
the screen for people who do not.
Post by Stephan Beal
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
--
D. Richard Hipp
***@sqlite.org
Baruch Burstein
2014-12-18 11:46:55 UTC
Permalink
Post by Richard Hipp
Post by Stephan Beal
Post by Stephan Beal
Actually... i use the flat view more often than not. That's probably
just historical momentum (and the way my old menus are all set up). i
wouldn't whine about loss of flat view, but also won't argue for its
removal.
flat mode meaning /dir instead of /tree.
Yes. And remember, I'm not going to make /dir go away - I'm just wanting
to take away its prominent links on the menu bar. The pages will all still
be there for those who want them. But links to those pages won't clutter
the screen for people who do not.
Another thought about tree/flat view. Maybe the default for large (by some
definition of large) repos should be flat mode. I just tried loading
http://netbsd.sonnenberger.org/tree?ci=tip and took over 2 minutes!
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Baruch Burstein
2014-12-18 13:23:02 UTC
Permalink
Post by Baruch Burstein
Post by Richard Hipp
Post by Stephan Beal
Post by Stephan Beal
Actually... i use the flat view more often than not. That's probably
just historical momentum (and the way my old menus are all set up). i
wouldn't whine about loss of flat view, but also won't argue for its
removal.
flat mode meaning /dir instead of /tree.
Yes. And remember, I'm not going to make /dir go away - I'm just wanting
to take away its prominent links on the menu bar. The pages will all still
be there for those who want them. But links to those pages won't clutter
the screen for people who do not.
Another thought about tree/flat view. Maybe the default for large (by some
definition of large) repos should be flat mode. I just tried loading
http://netbsd.sonnenberger.org/tree?ci=tip and took over 2 minutes!
I just tested it with the latest fossil trunk (that server is using a
version almost a year old). While the page on the current server is 12+MB,
the same page, if they were to update the fossil version, is 29+MB! I
definitely think that there should at least be an option to make flat view
the default.
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Richard Hipp
2014-12-18 13:51:45 UTC
Permalink
Post by Baruch Burstein
Post by Baruch Burstein
Post by Richard Hipp
Post by Stephan Beal
Post by Stephan Beal
Actually... i use the flat view more often than not. That's probably
just historical momentum (and the way my old menus are all set up). i
wouldn't whine about loss of flat view, but also won't argue for its
removal.
flat mode meaning /dir instead of /tree.
Yes. And remember, I'm not going to make /dir go away - I'm just
wanting to take away its prominent links on the menu bar. The pages will
all still be there for those who want them. But links to those pages won't
clutter the screen for people who do not.
Another thought about tree/flat view. Maybe the default for large (by
some definition of large) repos should be flat mode. I just tried loading
http://netbsd.sonnenberger.org/tree?ci=tip and took over 2 minutes!
I just tested it with the latest fossil trunk (that server is using a
version almost a year old). While the page on the current server is 12+MB,
the same page, if they were to update the fossil version, is 29+MB! I
definitely think that there should at least be an option to make flat view
the default.
How many separate files are in that project?
--
D. Richard Hipp
***@sqlite.org
Baruch Burstein
2014-12-18 13:57:43 UTC
Permalink
Post by Richard Hipp
How many separate files are in that project?
How do I check?

--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Richard Hipp
2014-12-18 14:00:33 UTC
Permalink
Post by Baruch Burstein
Post by Richard Hipp
How many separate files are in that project?
How do I check?
The /stat page. Example: https://www.fossil-scm.org/fossil/stat
--
D. Richard Hipp
***@sqlite.org
Baruch Burstein
2014-12-18 14:22:17 UTC
Permalink
Post by Richard Hipp
Post by Baruch Burstein
Post by Richard Hipp
How many separate files are in that project?
How do I check?
The /stat page. Example: https://www.fossil-scm.org/fossil/stat
304160
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Richard Hipp
2014-12-18 14:39:51 UTC
Permalink
Post by Baruch Burstein
Post by Richard Hipp
Post by Baruch Burstein
Post by Richard Hipp
How many separate files are in that project?
How do I check?
The /stat page. Example: https://www.fossil-scm.org/fossil/stat
304160
Wow. Are you at liberty to share the rest of the "/stat" page with us?
--
D. Richard Hipp
***@sqlite.org
Stephan Beal
2014-12-18 15:32:16 UTC
Permalink
Post by Richard Hipp
Wow. Are you at liberty to share the rest of the "/stat" page with us?
FYI: the 'dbstat' CLI command shows the same info (or essentially the same).
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Baruch Burstein
2014-12-18 17:50:24 UTC
Permalink
Post by Richard Hipp
Wow. Are you at liberty to share the rest of the "/stat" page with us?
I posted the link above. It is the netbsd src repo.
http://netbsd.sonnenberger.org/stat
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Martin Gagnon
2014-12-18 16:12:38 UTC
Permalink
On Wed, Dec 17, 2014 at 2:16 AM, Stephan Beal <
Actually... i use the flat view more often than not. That's
probably just historical momentum (and the way my old menus are
all set up). i wouldn't whine about loss of flat view, but also
won't argue for its removal.
flat mode meaning /dir instead of /tree.
Yes.  And remember, I'm not going to make /dir go away - I'm just
wanting to take away its prominent links on the menu bar.  The pages
will all still be there for those who want them.  But links to those
pages won't clutter the screen for people who do not.
Another thought about tree/flat view. Maybe the default for large (by some
definition of large) repos should be flat mode. I just tried loading http:/
/netbsd.sonnenberger.org/tree?ci=tip and took over 2 minutes!
I just tested it with the latest fossil trunk (that server is using a version
almost a year old). While the page on the current server is 12+MB, the same
page, if they were to update the fossil version, is 29+MB! I definitely think
that there should at least be an option to make flat view the default.
What browser are you using ? I don't know why, but I have a local repo
with ~175000 files and it take like 4.5 seconds to load in Firefox 34,
but almost 30 seconds in Google Chrome 39. (Linux debian 7 64bits)

From firefox Network benchmark, my page is 5.5MB. (not sure if that is
compressed or not)
--
Martin G.
Stephan Beal
2014-12-18 16:23:23 UTC
Permalink
Post by Martin Gagnon
From firefox Network benchmark, my page is 5.5MB. (not sure if that is
compressed or not)
insofar as i see in cgi.c, it looks like output is always compressed if
possible (based on response code and whether or not is_gzippable() returns
true or not).
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Stephan Beal
2014-12-18 16:24:34 UTC
Permalink
Post by Stephan Beal
....
insofar as i see in cgi.c, it looks like output is always compressed if
Post by Stephan Beal
possible (based on response code and whether or not is_gzippable() returns
true or not).
Which doesn't tell you much. The Content-Length header is the compressed
size which fossil sends. FF/Chrome might be reporting that decompressed
size.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Richard Hipp
2014-12-18 16:47:43 UTC
Permalink
Martin & Baruch:

Thanks for stress-testing Fossil! I don't have any repositories that are
quite that big. In fact, yours appear to be about 100x larger than any
that I test with. This means that you are much more likely to hit
performance issues, sooner, than I do.

Please do watch for pages that are slow to generate, and let me know what
those pages are.

On the footer of each page there is (by default - unless you have changed
it) a line that says how much time was required to generate the page on the
server. I'm interested to know how much time was used to generate some of
your multi-megabyte /tree pages.
Post by Stephan Beal
On Tue, Dec 16, 2014 at 8:17 PM, Stephan Beal <
On Wed, Dec 17, 2014 at 2:16 AM, Stephan Beal <
Actually... i use the flat view more often than not.
That's
probably just historical momentum (and the way my old
menus are
all set up). i wouldn't whine about loss of flat view,
but also
won't argue for its removal.
flat mode meaning /dir instead of /tree.
Yes. And remember, I'm not going to make /dir go away - I'm just
wanting to take away its prominent links on the menu bar. The
pages
will all still be there for those who want them. But links to
those
pages won't clutter the screen for people who do not.
Another thought about tree/flat view. Maybe the default for large
(by some
definition of large) repos should be flat mode. I just tried loading
http:/
/netbsd.sonnenberger.org/tree?ci=tip and took over 2 minutes!
I just tested it with the latest fossil trunk (that server is using a
version
almost a year old). While the page on the current server is 12+MB, the
same
page, if they were to update the fossil version, is 29+MB! I definitely
think
that there should at least be an option to make flat view the default.
What browser are you using ? I don't know why, but I have a local repo
with ~175000 files and it take like 4.5 seconds to load in Firefox 34,
but almost 30 seconds in Google Chrome 39. (Linux debian 7 64bits)
From firefox Network benchmark, my page is 5.5MB. (not sure if that is
compressed or not)
--
Martin G.
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
--
D. Richard Hipp
***@sqlite.org
Stephan Beal
2014-12-18 16:56:24 UTC
Permalink
much time was required to generate the page on the server. I'm interested
to know how much time was used to generate some of your multi-megabyte
/tree pages.
i'd be interested to know what those megs are - maybe we can consolidate
symbols, optimize HTML, shorten CSS names, or some such to cut that down.
5MB is quite a lot for a mobile connection, and most of the Free World
doesn't have real flat-rate internet. In Europe the standard is to cut a
phone's internet speed to Edge-network speed (~ISDN) once you go over your
monthly quota, which is anywhere from 300MB to 5GB, depending on
provider/package. For a 300MB monthly quota, 5MB is quite a chunk.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Richard Hipp
2014-12-18 17:34:51 UTC
Permalink
Post by Stephan Beal
much time was required to generate the page on the server. I'm
interested to know how much time was used to generate some of your
multi-megabyte /tree pages.
i'd be interested to know what those megs are - maybe we can consolidate
symbols, optimize HTML, shorten CSS names, or some such to cut that down.
5MB is quite a lot for a mobile connection, and most of the Free World
doesn't have real flat-rate internet. In Europe the standard is to cut a
phone's internet speed to Edge-network speed (~ISDN) once you go over your
monthly quota, which is anywhere from 300MB to 5GB, depending on
provider/package. For a 300MB monthly quota, 5MB is quite a chunk.
Some quick checks show roughly 200 bytes per file (uncompressed) or about
25 bytes/file compressed. Baruch is seeing 2x or 4x more bytes
uncompressed, but perhaps that is because his repository has filenames that
are longer?

Baruch: Can you pull a /tree from your big repo, do a "Save As..." of the
page to a file, check its size, then run gzip across it and check its
compressed size? Also if you can do "grep '<li class=' page.html | wc" to
let us know exactly how many files are in the page, that would be cool too.

The newly added part (the last change times) add probably 50 bytes/line,
but the addition is highly compressible, so I'm guessing it does not
increase the compressed size by very much.

One way to really save space would be to send only the first 10 or 16 bytes
of the SHA1 hash in the hyperlink for each file, instead of the full 40
bytes. The SHA1 hashes do not compress well, so making that change would
perhaps make a big difference even in the compressed size.
--
D. Richard Hipp
***@sqlite.org
Martin Gagnon
2014-12-18 17:43:38 UTC
Permalink
much time was required to generate the page on the server.  I'm
interested to know how much time was used to generate some of your
multi-megabyte /tree pages.
i'd be interested to know what those megs are - maybe we can consolidate
symbols, optimize HTML, shorten CSS names, or some such to cut that down.
5MB is quite a lot for a mobile connection, and most of the Free World
doesn't have real flat-rate internet. In Europe the standard is to cut a
phone's internet speed to Edge-network speed (~ISDN) once you go over your
monthly quota, which is anywhere from 300MB to 5GB, depending on provider/
package. For a 300MB monthly quota, 5MB is quite a chunk.
Some quick checks show roughly 200 bytes per file (uncompressed) or about 25
bytes/file compressed.  Baruch is seeing 2x or 4x more bytes uncompressed, but
perhaps that is because his repository has filenames that are longer?
Baruch:  Can you pull a /tree from your big repo, do a "Save As..." of the page
to a file, check its size, then run gzip across it and check its compressed
size?  Also if you can do "grep '<li class=' page.html | wc" to let us know
exactly how many files are in the page, that would be cool too.
The newly added part (the last change times) add probably 50 bytes/line, but
the addition is highly compressible, so I'm guessing it does not increase the
compressed size by very much.
One way to really save space would be to send only the first 10 or 16 bytes of
the SHA1 hash in the hyperlink for each file, instead of the full 40 bytes. 
The SHA1 hashes do not compress well, so making that change would perhaps make
a big difference even in the compressed size.
For my 5MB that come from firefox Network profiling, it seems to come
from compressed value.. When I save-as the page, I get a 31.7MB html
file. If I gzip this html file, I get a 4.0MB file.
--
Martin G.
Warren Young
2014-12-18 18:26:31 UTC
Permalink
Post by Martin Gagnon
For my 5MB that come from firefox Network profiling, it seems to come
from compressed value.. When I save-as the page, I get a 31.7MB html
file. If I gzip this html file, I get a 4.0MB file.
You should test with gzip -1, not the default gzip -6 or the CPU-hungry gzip-9. On-the-fly web compression generally corresponds more closely to gzip -1.
Stephan Beal
2014-12-18 17:54:16 UTC
Permalink
Post by Richard Hipp
One way to really save space would be to send only the first 10 or 16
bytes of the SHA1 hash in the hyperlink for each file, instead of the full
40 bytes. The SHA1 hashes do not compress well, so making that change
would perhaps make a big difference even in the compressed size.
304160 files * 30 bytes snipped = 9124800(!) bytes saved. Trimming to 16
bytes = 7299840 saved.

Not too shabby.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Baruch Burstein
2014-12-18 19:37:17 UTC
Permalink
Post by Richard Hipp
Baruch: Can you pull a /tree from your big repo, do a "Save As..." of the
page to a file, check its size, then run gzip across it and check its
compressed size? Also if you can do "grep '<li class=' page.html | wc" to
let us know exactly how many files are in the page, that would be cool too.
I will not be at a computer regularly for the next few days, so I suggest
someone else do the testing. IIRC the time to generate the page was about 7
sec, a few secs for rendering, and most of the time was transferring the
2-3MB compressed HTML (Martin - that is why the local test was so much
faster). The uncompressed one was 12+MB. You can see it for yourself at
http://netbsd.sonnenberger.org/. I am neither the maintainer nor even a
user of that repo. I just turn to it occasionally to test things on a
"large" repo. If anyone wants to test them localy, you can download them
from http://ftp.netbsd.org/pub/NetBSD/misc/repositories/fossil/. Don't try
to clone. the rebuild itself is almost 24 hours.

Richard - were you not aware of these repos, or just hadn't checked using
them? I seem to recall some work done on fossil specifically because of the
pkgsrc repo. Maybe it was adding delta-manifests?

Baruch
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
bch
2014-12-19 03:06:31 UTC
Permalink
I have a copy of NetBSD src (and pkgsrc) that I interact with nearly daily
if there are specifics of this repo that are needed.
Post by Baruch Burstein
Post by Richard Hipp
Baruch: Can you pull a /tree from your big repo, do a "Save As..." of
the page to a file, check its size, then run gzip across it and check its
compressed size? Also if you can do "grep '<li class=' page.html | wc" to
let us know exactly how many files are in the page, that would be cool too.
I will not be at a computer regularly for the next few days, so I suggest
someone else do the testing. IIRC the time to generate the page was about 7
sec, a few secs for rendering, and most of the time was transferring the
2-3MB compressed HTML (Martin - that is why the local test was so much
faster). The uncompressed one was 12+MB. You can see it for yourself at
http://netbsd.sonnenberger.org/. I am neither the maintainer nor even a
user of that repo. I just turn to it occasionally to test things on a
"large" repo. If anyone wants to test them localy, you can download them
from http://ftp.netbsd.org/pub/NetBSD/misc/repositories/fossil/. Don't
try to clone. the rebuild itself is almost 24 hours.
Richard - were you not aware of these repos, or just hadn't checked using
them? I seem to recall some work done on fossil specifically because of the
pkgsrc repo. Maybe it was adding delta-manifests?
Baruch
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Warren Young
2014-12-18 18:23:25 UTC
Permalink
much time was required to generate the page on the server. I'm interested to know how much time was used to generate some of your multi-megabyte /tree pages.
i'd be interested to know what those megs are - maybe we can consolidate symbols, optimize HTML, shorten CSS names, or some such to cut that down.
YSlow and PageSpeed give the current tree view pretty high marks.

The biggest thing I expected to find missing — and was surprised to find that it is not — is gzip compression on the HTTP layer. That alone will help tremendously with multi-MB pages. I wonder if the person reporting that is counting the on-the-wire size or the uncompressed size.

PageSpeed reports that Fossil isn’t minifying the skin CSS. Little wonder, since it offers an inline editor for the raw CSS. Nevertheless, it would be good if Fossil created a minified shadow copy of it and served that instead.

YSlow gives a confusing report about favicon.ico not having an Expires header, but what it really means is that Fossil doesn’t provide a favicon.ico, so the browser wastes time trying to fetch it on every page load. If Fossil were to serve such a file along with a far-future Expires header, it would prevent the browser from repeatedly trying, speeding page loads. (Yes, you must serve more content to make things faster. Welcome to the modern Internet.) It should be part of the skinning mechanism, of course.

Other than that, Fossil’s generated tree-view HTML looks pretty good to me. (I speak as a professional web app developer.) It has little unnecessary whitespace, and I found no inline styling; Fossil is delegating all styling to CSS, as far as I can see.

The generated HTML is highly redundant in its form, but gzip takes care of that. I see roughly 3:1 compression on the largest single directory under Fossil’s control here.

The only thing that could be a bit better is that the inline JS could be minified, but that is notoriously difficult to do in an automated fashion. I’ve used JSMin and YUI Compressor for this in my own web apps; it is not something you want to try and roll by hand.

…says he to the one who rolled his own RDBMS, parser generator, and DVCS. :)

Annnnyway, if you do add JS minification to Fossil, it should be part of the build process for Fossil, rather than something it does on the fly.

JSMin has a very “Fossil” sort of feel to it: single C source file, with no external dependencies other than a C compiler. It doesn’t even ship with a Makefile! It’s under a BSDish license, so it could be put into the Fossil source tree and used during the build process.

http://www.crockford.com/javascript/jsmin.html

I used JSMin successfully for years. I only switched to YUI Compressor recently because I wanted CSS minification. Also, JSMin would occasionally do the wrong thing with certain very large 3rd party JS libraries, particularly when concatenated with *other* very large 3rd party JS libraries, in order to reduce the number of HTTP round trips. Fossil doesn’t have either of these problems.

Besides that, YUI Compressor isn’t suited to the nature of Fossil: it ships as a ¾ MB JAR file.
Stephan Beal
2014-12-18 18:48:42 UTC
Permalink
The biggest thing I expected to find missing — and was surprised to find
that it is not — is gzip compression on the HTTP layer. That alone will
help tremendously with multi-MB pages. I wonder if the person reporting
that is counting the on-the-wire size or the uncompressed size.
It should be compressing, at least based on what i see in cgi.c. The
CGI-mode output APIs buffer up the content/body, and compress it as a final
step.
PageSpeed reports that Fossil isn’t minifying the skin CSS. Little
wonder, since it offers an inline editor for the raw CSS. Nevertheless, it
would be good if Fossil created a minified shadow copy of it and served
that instead.
Interesting idea. i don't know if we have enough CSS to make this worth the
effort, though?
YSlow gives a confusing report about favicon.ico not having an Expires
header, but what it really means is that Fossil doesn’t provide a
favicon.ico, so the browser wastes time trying to fetch it on every page
load. If Fossil were to serve such a file along with a far-future Expires
header, it would prevent the browser from repeatedly trying, speeding page
loads. (Yes, you must serve more content to make things faster. Welcome
to the modern Internet.) It should be part of the skinning mechanism, of
course.
Consider a favico added to the TODO list. Can't believe that never came up
before.
Other than that, Fossil’s generated tree-view HTML looks pretty good to
me. (I speak as a professional web app developer.)
!!!
It has little unnecessary whitespace
The reason for that is because it's created via C strings, where
indention/formatting of the HTML isn't all that useful.
, and I found no inline styling; Fossil is delegating all styling to CSS,
as far as I can see.
A few years ago you might have. It's nice to hear we haven't got any left.
The only thing that could be a bit better is that the inline JS could be
minified, but that is notoriously difficult to do in an automated fashion.
I’ve used JSMin and YUI Compressor for this in my own web apps; it is not
something you want to try and roll by hand.
The JS is also in C code. Normally someone develops it in a local JS file
or their browser console, then paste it into C and adds quotes around it.
It's baked into the binary, so minifying would have to be done on the fly
and would have little benefit because we have little JS.

says he to the one who rolled his own RDBMS, parser generator, and DVCS.
:)
... binary-capable diff algo, single-function hashtable interface, ...

i'm continually amazed by little snippets by Richard which i know would
take me 10x as much work to implement. "Oh, hey, we need a little binary
search here..."
Annnnyway, if you do add JS minification to Fossil, it should be part of
the build process for Fossil, rather than something it does on the fly.
it can't be done with the current code b/c the JS is baked in. We (well, HE
;) recently added a mechanism for importing such resources from files, and
we "could" minify any resources stored in that mechanism.
JSMin has a very “Fossil” sort of feel to it: single C source file, with
no external dependencies other than a C compiler. It doesn’t even ship
with a Makefile! It’s under a BSDish license, so it could be put into the
Fossil source tree and used during the build process.
already got it in a couple of trees :)
Also, JSMin would occasionally do the wrong thing with certain very
large 3rd party JS libraries, particularly when concatenated with *other*
very large 3rd party JS libraries, in order to reduce the number of HTTP
round trips. Fossil doesn’t have either of these problems.
Nope - we've got a few snippets of hand-rolled stuff, but so far no notable
3rd-party libs (except maybe the wysiwyg editor).
Besides that, YUI Compressor isn’t suited to the nature of Fossil: it
ships as a Ÿ MB JAR file.
"I told him I needed help opening a jar. He said to install Java."

(It makes more sense in the context of the picture it was written on.)
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Warren Young
2014-12-18 19:38:01 UTC
Permalink
Post by Stephan Beal
It should be compressing, at least based on what i see in cgi.c.
You skimmed that a bit too fast. I was just commenting that I *expected* to find that Fossil didn’t gzip its own content, but was surprised to find that it does.

gzipped content isn’t yet universal on Apache or nginx-served pages, where you can enable it by simply adding a few canned lines to your server’s config file. For it to be in a custom HTTP server for a web app with a rather narrow audience, well, that’s fairly impressive.
Post by Stephan Beal
i don't know if we have enough CSS to make this worth the effort, though?
The far-future Expires time is more important, since that reduces the ~4 kiB of CSS I have here to 0 kiB, after the first page request.

But, minifying with YUI Compressor does get it down to 2.6 kiB, which ain’t nothin’.

I wouldn’t expect a hand-rolled one-off CSS minifier to do quite that well, but even if it got it to 3 kiB, well, 25% savings always feels nice.
Post by Stephan Beal
I found no inline styling; Fossil is delegating all styling to CSS, as far as I can see.
A few years ago you might have. It's nice to hear we haven't got any left.
I only skimmed a single page, so there might still be some left. Nevertheless, it’s nice not to see a bunch of <font> tags and single-pixel transparent GIFs all over the place.

Fossil is still using <table> based layouts, in the timeline view at least. Tsk, tsk. Not that I have the slightest idea how you’d build that page in “pure” CSS instead. Just teasing. :)
Post by Stephan Beal
The JS is also in C code. Normally someone develops it in a local JS file or their browser console, then paste it into C and adds quotes around it. It's baked into the binary, so minifying would have to be done on the fly and would have little benefit because we have little JS.
Actually, you could instead keep the JS file separate in Fossil’s own repo, then minify it and insert it into the C file as part of the build process.

This would have a side advantage, in that your text editor would syntax-color the JS code properly, since it would be in a *.js file now. You wouldn’t need to keep extracting and re-inserting it to achieve that end.

I seem to recall that Fossil already generates some of its own C. Not true?

Even if you don’t already have a C file assembler in place, it’s pretty trivial to implement. It would be a 20-line shell script based primarily on cat(1) and jsmin.
Stephan Beal
2014-12-18 19:48:03 UTC
Permalink
Post by Warren Young
Post by Stephan Beal
It should be compressing, at least based on what i see in cgi.c.
You skimmed that a bit too fast. I was just commenting that I *expected*
to find that Fossil didn’t gzip its own content, but was surprised to find
that it does.
Doh, indeed - speed reading is not a skill of mine.
Post by Warren Young
Fossil is still using <table> based layouts, in the timeline view at
least. Tsk, tsk. Not that I have the slightest idea how you’d build that
page in “pure” CSS instead. Just teasing. :)
Yeah, but if we're gonna working with lync/elinks/etc., tables are a must.
i wouldn't know off-hand how to replace a table CSS, either, and would be
concerned of backlash from users who are stuck with old IE versions.
Post by Warren Young
Post by Stephan Beal
The JS is also in C code. Normally someone develops it in a local JS
file or their browser console, then paste it into C and adds quotes around
it. It's baked into the binary, so minifying would have to be done on the
fly and would have little benefit because we have little JS.
Actually, you could instead keep the JS file separate in Fossil’s own
repo, then minify it and insert it into the C file as part of the build
process.
That infrastructure was only recently added. Over time, more of those
resources will be moved into the file-based system. The build process bakes
those files into the binary, but we could have an intermediate step for
minification and such.
Post by Warren Young
This would have a side advantage, in that your text editor would
syntax-color the JS code properly, since it would be in a *.js file now.
You wouldn’t need to keep extracting and re-inserting it to achieve that
end.
true enough - it's just been a question of effort. We keep the JS to a
minimum (even moreso because it's tedious to add ;).
Post by Warren Young
I seem to recall that Fossil already generates some of its own C. Not true?
Correct. e.g. the various commands, web page routes, and help text are
extracted from marked-up comments in the sources and compiled in.
Post by Warren Young
Even if you don’t already have a C file assembler in place, it’s pretty
trivial to implement. It would be a 20-line shell script based primarily
on cat(1) and jsmin.
Given the new infrastructure, it wouldn't be much work, it just hasn't
caught anyone's attention/muse yet.

i suspect that some of the lower-hanging fruit, like truncating the UUIDs,
would initially give us bigger wins with less effort.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Warren Young
2014-12-18 20:08:58 UTC
Permalink
i suspect that some of the lower-hanging fruit, like truncating the UUIDs, would initially give us bigger wins with less effort.
Yes, I thought that went without saying. I would have mentioned it in my first email, only I didn’t know if Fossil would accept truncated UUIDs in that particular context.

There is one other way to solve it, which would let you keep the tree view without spamming the client with the entire tree on the first /tree page load: Ajax.

That is, on the initial page load, send only the HTML necessary to render the initial view: the root and the first level of files. Then on each click that requires an unfolding of that sub-tree, send an XHR request back to the Fossil server for that sub-tree’s file list.

That would be a considerable amount of work, of course, but it would noticeably speed things even for smaller trees than the NetBSD one Martin Gagnon is letting us play with. My largest tree here takes 175 ms to generate and nearly 200 ms to arrive over the network. This is slow enough to be noticed by a human. You want to get below about 50-100 ms.

Speaking of latency, if you are thinking of objecting that the XHR requests would add noticeable latency vs the current scheme, where all the data is already present and just has to be displayed, I speak from experience when I tell you that you can get an XHR round-trip time down into that double-digit millisecond range. More speed simply isn’t necessary when dealing with human reaction time.
Stephan Beal
2014-12-18 20:19:27 UTC
Permalink
Post by Warren Young
There is one other way to solve it, which would let you keep the tree view
without spamming the client with the entire tree on the first /tree page
load: Ajax.
There's been some experimentation with "alternative UIs" using AJAX and the
JSON API, as well as libfossil script bindings over CGI, but i haven't
hacked on them in some time b/c i'm having too much fun working on the
newer scripting engine:

http://fossil.wanderinghorse.net/repos/fwiki/editor/

http://fossil.wanderinghorse.net/repos/libfossil/cgidemo/

Both are only proofs of concept apps, but i _do_ use a custom fossil
front-end for some of my wikis, where i store the wiki pages in Google Code
wiki format, save them in fossil, but do the rendering on the client side:

http://fossil.wanderinghorse.net/wikis/

those use a 100% custom Fossil UI, using the JSON API for all communication
with fossil. It's not a paragon of web design (i've got no gift for making
sites pretty, and that demo is relatively old, so no require.js), but it
proves that it can be done.

There's _lots_ yet to experiment with in that regard - it's only a matter
of time.

Speaking of latency, if you are thinking of objecting that the XHR requests
Post by Warren Young
would add noticeable latency vs the current scheme, where all the data is
already present and just has to be displayed, I speak from experience when
I tell you that you can get an XHR round-trip time down into that
double-digit millisecond range. More speed simply isn’t necessary when
dealing with human reaction time.
You don't have to convince me, but as you say: it'd be a lot of work. It'd
be easier to do such work in new trees, i think, and my own personal TODO
list in this regard is longer than my hair.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Andy Bradford
2014-12-18 20:29:45 UTC
Permalink
That infrastructure was only recently added. Over time, more of those
resources will be moved into the file-based system. The build process
bakes those files into the binary, but we could have an intermediate
step for minification and such.
Who is the target audience for such efforts? Minification is definitely
useful for highly trafficked websites, but is there really a huge
concern about statistics on web page loads when it comes to Javascript
and CSS for Fossil?

Just asking...

Thanks,

Andy
--
TAI64 timestamp: 400000005493395b
Warren Young
2014-12-18 21:06:15 UTC
Permalink
Post by Andy Bradford
That infrastructure was only recently added. Over time, more of those
resources will be moved into the file-based system. The build process
bakes those files into the binary, but we could have an intermediate
step for minification and such.
Who is the target audience for such efforts? Minification is definitely
useful for highly trafficked websites, but is there really a huge
concern about statistics on web page loads when it comes to Javascript
and CSS for Fossil?
I only brought it up because it would placate PageSpeed/YSlow.

It’s the sort of thing you fix for the same reason that you silence harmless C compiler warnings: so you don’t develop a habit of ignoring problems, which can blind you to real problems later.

Obviously the low-hanging fruit should be plucked first.
Ron W
2014-12-18 20:30:20 UTC
Permalink
Post by Stephan Beal
true enough - it's just been a question of effort. We keep the JS to a
minimum (even moreso because it's tedious to add ;).
Perhaps it is good that it is tedious to add/edit the JS?
Stephan Beal
2014-12-18 20:49:14 UTC
Permalink
Post by Ron W
Post by Stephan Beal
true enough - it's just been a question of effort. We keep the JS to a
minimum (even moreso because it's tedious to add ;).
Perhaps it is good that it is tedious to add/edit the JS?
One can certainly argue that fossil's core features shouldn't rely on it,
but i think web users as a whole have become spoiled by interactive pages a
bit, and probably see fossil as a bit outdated in that regard. Not that
that's a problem, just noting/speculating. There are advantages to an
"old-style" HTML app, with little or no script, but interactivity is nice
to have. Though i would love to see a fully interactive/AJAX fossil client
(and have several kettles on the stove in that regard!), i couldn't bring
myself to argue that a static UI is unnecessary/unwanted.

JS is also used in part of the anti-bot mechanism which delays populating
of hyperlinks until after (IIRC) the first mouse event.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Warren Young
2014-12-18 21:20:40 UTC
Permalink
true enough - it's just been a question of effort. We keep the JS to a minimum (even moreso because it's tedious to add ;).
Perhaps it is good that it is tedious to add/edit the JS?
You might have the wrong impression about the sort of web developer I am. I’m the sort that runs NoScript in Firefox, not the sort that throws whizzy things onto the web because they’re whizzy.

I want my news site to load without needing any JS.

But a web app? Entirely different story. I think a web app should push as much code out to the client side as it can get away with.

I believe this because I miss the days of native client-side code, when a button click could be handled by C/C++ code near-instantly.

We bought ourselves quite a mixed bag of ease and pain when we traded that world for a new platform that made UI development easy, but where each button’s click handler is off on another computer dozens of milliseconds away, where event handlers might now return a completely new UI instead of just the smallest amount of data that allows the UI to update in place.

With today’s fast CPUs, Ajax lets us bring back the native client, for all practical purposes.
Ron W
2014-12-18 21:42:49 UTC
Permalink
With today’s fast CPUs, Ajax lets us bring back the native client, for all
practical purposes.
My concern about JS (and Java) is that it is too powerful. And web browsers
are not properly "sand boxing" active content like JS.

I have done some web development, and have used JS (and Java) *sparingly*
to enhance the usability, but not depending on JS for real functionality.
Warren Young
2014-12-18 22:23:39 UTC
Permalink
Post by Warren Young
With today’s fast CPUs, Ajax lets us bring back the native client, for all practical purposes.
My concern about JS (and Java) is that it is too powerful.
Don’t lump JavaScript and Java together. They have vastly different capability sets, with entirely different lineages. They only share part of a name due to an accident of history, not because they are in any way similar.
Post by Warren Young
web browsers are not properly "sand boxing" active content like JS.
[citation needed]

Two JS files served from different domains cannot communicate or interfere with each other. That means if I use the same password for Fossil as for, say, Gmail, a tab opened to www.bad-actor.ru will not be able to steal my Fossil password and thereby get into my Google account.

That sounds like sandboxing to me.

There have been bugs, but bugs have been fixed. :)

In any case, if you cannot trust JS served by Fossil, you probably can’t trust it with the files you have checked into it, either.
Stephan Beal
2014-12-18 20:33:20 UTC
Permalink
Post by Andy Bradford
Who is the target audience for such efforts? Minification is definitely
useful for highly trafficked websites, but is there really a huge
concern about statistics on web page loads when it comes to Javascript
and CSS for Fossil?
Just asking...
personally i don't feel that minification would gain us much because we
don't have much css/js, but the points about expiry dates and such are
valid. It might be a nice thing to implement on a bored rainy day, though,
or help someone new to the project get their feet wet.
--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
Andy Bradford
2014-12-18 22:27:00 UTC
Permalink
personally i don't feel that minification would gain us much because
we don't have much css/js, but the points about expiry dates and such
are valid.
Yes, the scope of my question was limited to minification only; I think
the expiry is actually more useful because that completely eliminates
fetching of content for a period of time.

Andy
--
TAI64 timestamp: 40000000549354d6
Richard Hipp
2014-12-18 21:37:13 UTC
Permalink
Post by Stephan Beal
The biggest thing I expected to find missing — and was surprised to find
that it is not — is gzip compression on the HTTP layer. That alone will
help tremendously with multi-MB pages. I wonder if the person reporting
that is counting the on-the-wire size or the uncompressed size.
It should be compressing, at least based on what i see in cgi.c. The
CGI-mode output APIs buffer up the content/body, and compress it as a final
step.
Turns out there was a bug. Not in Fossil, but in the minimalist webserver
that runs the http://www.fossil-scm.org/ website. The webserver was not
sending the HTTP_SERVER_ENCODING cgi parameter down to CGI programs, and so
Fossil did not know that it was allowed to compress content, and so it was
skipping that step.

The problem should now be fixed:
https://www.sqlite.org/docsrc/info/3985f4812ed661
--
D. Richard Hipp
***@sqlite.org
Martin Gagnon
2014-12-18 17:37:52 UTC
Permalink
Thanks for stress-testing Fossil!  I don't have any repositories that are quite
that big.  In fact, yours appear to be about 100x larger than any that I test
with.  This means that you are much more likely to hit performance issues,
sooner, than I do.
Please do watch for pages that are slow to generate, and let me know what those
pages are. 
On the footer of each page there is (by default - unless you have changed it) a
line that says how much time was required to generate the page on the server. 
I'm interested to know how much time was used to generate some of your
multi-megabyte /tree pages.
For this particular repo:

The : /tree?ci=tip&mtime=1 page I get:
This page was generated in about 2.057s


Here's the /stat of this repo:
---------------------------------------------------------------------
Repository Size: 2194643968 bytes (2.2GB)
Number Of Artifacts: 78035 (56344 fulltext and 21691 deltas)
Uncompressed Artifact Size: 208841 bytes average, 104499200 bytes max, 16296705044 bytes (16.3GB) total
Compression Ratio: 7:1
Number Of Check-ins: 883
Number Of Files: 147965
Number Of Wiki Pages: 2
Number Of Tickets: 9
Duration Of Project: 829 days or approximately 2.27 years.
Project ID: 20b326b473b08ee3115aeb5005ea65b5cd68e84b
Server ID: 73434698d3f0fec065302eebc544085c93a34c98
Fossil Version: 2014-12-18 09:38:02 [87185aa5dd] (1.30) [compiled using gcc-4.7.2]
SQLite Version: 2014-12-10 04:58:43 [3528f8dd39] (3.8.8)
Repository Rebuilt: 2014-04-11 04:16:28 By Fossil 1.28 [1762a72f0e] 2014-04-10 08:36:18 UTC
Database Stats: 2143207 pages, 1024 bytes/page, 4045 free pages, UTF-8, delete mode
---------------------------------------------------------------------

Talking about stressing fossil, I notice this repo work much faster
under linux compared with windows on comparable machine. Both are
core-i7, but my linux computer have a slower Seagate Barracuda LP 2TB
hard disk while my windows computer use a fast SSD drive.

Example using fossil "status: command:

Linux (debian 7 64bit):
$ time fossil status

real 0m1.220s
user 0m0.288s
sys 0m0.272s


Windows (windows 8.1 64bit 32bit build using mingw):
$ time fossil status

real 0m9.981s
user 0m0.000s
sys 0m0.062s

I wonder why there's so much differences..

On a older core-2 computer with windows, it even take more than 1 minute.
(I don't have a comparison with linux on same computer..)
--
Martin G.
Baruch Burstein
2014-12-18 11:14:29 UTC
Permalink
Post by Stephan Beal
Post by Stephan Beal
Actually... i use the flat view more often than not. That's probably just
historical momentum (and the way my old menus are all set up). i wouldn't
whine about loss of flat view, but also won't argue for its removal.
flat mode meaning /dir instead of /tree.
What is the difference between /dir and /tree?type=flat ?
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Warren Young
2014-12-17 01:32:04 UTC
Permalink
But I don't see a reason to have it using up valuable menu-bar space.
Screen real estate is precious. Nuke that sucker.
jungle Boogie
2014-12-17 05:34:46 UTC
Permalink
Hi Richard,
Post by Richard Hipp
Given the much improved tree-view functionality
https://www.fossil-scm.org/fossil/tree?ci=trunk&mtime
Is there really any reason to keep the legacy "Flat-View" on the menu bar?
https://www.fossil-scm.org/fossil/tree?ci=trunk&type=flat
I think the flat-view functionality should be preserved. (Who knows how
many links to the flat-view exist on various wiki pages, tickets, and
check-in comments.) But I don't see a reason to have it using up valuable
menu-bar space.
Nice job on taking it out. I like it and all the changes you've
committed today. I especially like that there's the shading as you
mouseover in the code section. Without this shading, it may make it
harder for the eyeballs to travel over to the right modify date.

When reviewing the flat view: https://www.fossil-scm.org/index.html/dir

I'm wondering what order things are sorted/displayed in. Is it
alphabetical, last modified, etc? I understand the CAPITAL letter file
names no sorting correctly because that's unix but make is before (if
you're reading left-to-right) auto.def and even ci_cvs.txt I also
understand the dot files preceding everything else as well. The
shading also means you didn't need to modify the alignment.


If you're looking for other suggestions, then I suggest the rss link
somewhere at https://www.fossil-scm.org/fossil/timeline
I don't necessarily think it needs to have its own menu button but
some place on the page so projects can be tracked easy, even in the
footer would be good.

It's already great there is an rss feed with fossil because I don't
know how people using other SCM track changes, maybe they just update
and see if there's changes.

If the timeline isn't the ideal place, there's always the code section.
Post by Richard Hipp
Comments?
--
D. Richard Hipp
Best,
jungle
--
-------
inum: 883510009027723
sip: ***@sip2sip.info
xmpp: jungle-***@jit.si
Baruch Burstein
2014-12-18 10:26:55 UTC
Permalink
Post by Richard Hipp
I think the flat-view functionality should be preserved. (Who knows how
many links to the flat-view exist on various wiki pages, tickets, and
check-in comments.) But I don't see a reason to have it using up valuable
menu-bar space.
One such link that should probably be fixed is in the header of the
Download page ;)
--
˙uʍop-ǝpısdn sı ɹoʇıuoɯ ɹnoʎ 'sıɥʇ pɐǝɹ uɐɔ noʎ ɟı
Continue reading on narkive:
Loading...