Is HD having problems or is it just my end.

[Prairiepiss]

I haven't had any problems for a long time now. But a couple days ago I started getting logged out. And the new posts search acts like there aren't any. Like HD was reset or something? It happened to me again today. I deleted my cookies to see if that would help. Again had to log back in? So is it my end or HD?

By the way its been working great for a while now. You guys are doing a great job keeping going.

[Odin]

PP,

I do not have this problem. I dó have this problem with the Dutch sister site of HD's though. Only way I can dial in is by responding to a "you have got mail" message!

Odin.

[rad14701]

Aside from the occasional MySQL error I haven't had any major performance problems...

[Bull Rider]

Send more money Prairie...



Bull.

[Prairiepiss]

If I had it I would.

[LWTCS]

Any issues on my side of the globe have very temporary. Site has been really stable.

[Husker]

The biggest stability issue we have right now, is the 'phpbb3_sessions' table. That table gets hit VERY VERY hard. It would usually not be a deal, except when we get a site sucker (teleport, HTTrack, etc). This idiots, try sucking the forum, which is absolutely horrible for the forum, and STUPID for the idiot user. There are 230000 posts (and only 20500 topics), but most site suckers will get (230000-20500)*20500 (plus some additional) pages downloaded when they try to download the site, because they will get a page with each post being the 'top' of the topic. Also, they will likely get more than this, since this will also probably happen in multiple ways, for topics that span more than one page.

All in all, the site suckers are terrible. A lot of them will fire up 4, or 8 or more threads, and hit the site as fast as they can. This simply beats the hell out of the forums, and strains the DB to the point it can be broken.

I watch for site suckers as much as possible,and ban them (IP or IP range level) as soon as I see one. They are NOT welcome on the forum (or wiki). They are not that bad for the parent site. They will suck it once, and pull a couple dozen (or hundred MB), and be done. But the forums and wiki (or almost any 'generated' pages), do not work for these type tools.

H.

[rad14701]

Husker, is there a way to enable throttling and, if so, can it be set to not cause issues with major search engine spiders...??? With the rate that unknown site suckers try to pull pages they "might" be kept at bay or even give up due to lag times... The throttling would only need to be set a several seconds at most in order to make a big difference... Not sure if phpBB has such a beast available or whether the server already has the capabilities...

[Husker]

I do not think phpbb has anything like this. I bet this would be something to be done either at the apache level, or through iptables (firewall), or whatever FreeBSD uses as the FW.

Both of those are things I have not setup before. I might want to conference in Jesse on this, and see what he has to say about it.

Getting a little OT, btw. We might want to take any more of this off the public threads.

H.

[Prairiepiss]

Thanks Husker. I think its just my end. We will see if it comes back.

Keep up the good work. I really do appreciate what you guys do for us. Thank you.

[Just4funguy30]

Okay, I apologize in advance for my ignorance. I'm curious though, what is a "site sucker", & why do some topics have the little image on the left look like it's scrolling, but others don't?
Thanks for all the info you great people are putting up here for us to read & learn from!!

[Husker]

A 'site sucker' is a type of program, which will download pages from the web, parse them, finding all of the links from them, and then visit those links, downloading the pages, finding all of the links on them, etc, etc. It downloads to whatever limits were set when it started, such as follow links up to 4 deep, do not follow links other than site XYZ.com, etc. All of the downloaded pages are stored on the local PC, and usually 'fixed' so that the links point to the directory structure on the local pc. What you end up with (or are supposed to), is a copy of the web site on your local hard drive, that you can access with a browser, but do not have to be connected to the internet. The whole web site 'should' work.

Well, for a site such as a forum, where it is simply a set of PHP scripts (a programming language), and a set of databases full of information, there can 'appear' to be billions of pages in existance. This is not really the case, but it 'appears' that way to a site sucker type application. Thus they will repeatedly download the same, or 'almost' the same pages again and again.

This is 'somewhat' of how the web spider programs (googlebot, bingbot, etc), work. However, for them, phpbb is smart enough to strip down the web pages, removing all links that are non needed, strips most eye candy graphics, and serves up pages that are small and contain much fewer links and are usually text only. phpbb does this by recognising these bots by certain agent strings within the request messages. Doing this keeps the bots 'under control'. We actually have bot records for the known site suckers. However, many of them are able to hide their agent strings, and simply 'appear' like IE, or firefox, etc. Also, they hit the wiki, which is also a php process around a database. Now, we may be able to keep them at bay on HD forums, but the database for the wiki, is run using the same database engine as we use on HD, thus putting more CPU load, and strain on the DB, and also on the Apache web server. Thus in general, these type applications are BAD news for the web site.


As for the scrolling icon, it means a 'hot topic'. I think that gets set at 15 followups, or possibly 25.

H.

[MudDuck]

Just how big would this site be on someones hard drive? Would releasing it on a CD for a donation be somewhere in the realm of possibility? I'm sure most of the site suckers are probably being run by people who worry about this sites info staying available, and with the not so legal nature of the info here that could be a valid concern.

[Husker]

There are some wiki's which release a database dump as a replication method. We 'could' look into this for the main parent site. There would need to be many things changed to make this work.

The user records are in the database. All of that data would need to be sanitized. All personal information, user login passwords, etc would need removed. There is also the PM database. That would need to be expunged. There likely are other things which would need to be looked into and removed, to make sure that only information that is 'public' information is removed. There are also some non-public forums where the information contained within them could not be delivered, and must be fully removed.

Now, what this would build, is a database dump. The forums are NOT a set of pages. They are a set of MySQL tables, that are on a system with a mysql database engine running. Also on that system must be a web server (we run apache), and a php engine. The forum itself, is a set of php scripts (phpbb), which use the mysql engine to read/write data to those tables, and use some flat files for certain things (avatars, images, etc). All together, this is used to render a html page, which your browser sees. So, the flat files would also need to be part of any release like this.

This would probably need to be a multi-step process if done. A full site backup would be made. This that installed into another mirror site (local PC likely). Then numerous SQL delete/update commands would have to be built, to remove all non-public data. All non-public files would need to also be removed.

All in all, this is NOT anywhere near an easy process. And it is something I am not very comfortable with doing, due to the fact that it would be all too easy to release some non-public information, due to the fact that no one, other than the phpbb team really has a absolute set of knowledge about what parts of the database are public and required, and what parts are private information. I 'think' I could clean stuff up, but I would not be 100% sure.

H.

[rad14701]

Just how big would this site be on someones hard drive? Would releasing it on a CD for a donation be somewhere in the realm of possibility? I'm sure most of the site suckers are probably being run by people who worry about this sites info staying available, and with the not so legal nature of the info here that could be a valid concern.

As Husker mentioned, this is just not practical... I have the software it takes to make a static copy of this or any other site and it is both hard on the server as well as consuming far more storage space than the original database-centric site occupies... It spiders a site and saves static copies complete with localized relative links... I have tested this on one of the servers in my office and even for a small amount of data the resulting volume of space required is massive... And it takes a loooong time...

Making a trimmed down version of the databases wouldn't work well because it takes server side software to render the pages...

Copying and Pasting only what you deem pertinent into a text file is the best option...

[MudDuck]

So in a nutshell it would be an excessive amount of work to produce something that your average user here wouldn't have the resources to use anyway, it all seems so easy until you find out how it actually works. Thanks for the replies.