Webmaster's Blog

Moderator: Moderators

Do you think the site and forum are moving in the right direction

Yes
10
100%
No
0
No votes
 
Total votes: 10

User avatar
GDII
Forum Moderator - MR2OCNZ
Posts: 5925
Joined: Sun Jul 10, 2011 5:13 pm
Stomping Ground: Wellington
Prime Mover: SW20
First name: Phill

Re: Webmaster's Blog

Post by GDII »

Thanks Michael! It did have a different error this time vs the previous ones so it does appear to be something else.
1990 SW20 MR2 G-Limited (GEN4 3SGTE)
2000 AE111R Corolla Wagon NZ New Daily
1996 AE101R Corolla Sprint NZ New (Sold)
1990 EP81 Starlet XL (Sold)
1990 EE90 Corolla XL (Sold)
Instagram https://www.instagram.com/sw20glimited/

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

Forum went down, but fix was easy, just needed to clear the cache

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

Hmm started getting weird logging in behaviour, so I disabled a bunch of CDN features to see if that resolves the issue

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

I've enabled uploading *.heic and *.heif images, but none of the browsers support viewing them so you have to download them to see them.

You can convert them to JPEG pictures on your phones:
https://www.howtogeek.com/666363/how-to ... -and-ipad/

Samsung devices still need to use a third-party converter in the meantime, but you can disable using HEIF/C in the meantime.

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

Sorry, had to get home to fix the DB log file index. It got corrupted as it probably auto-rebooted on a crash. If this happens again I'll update the filesystem mounting to be stricter, and potentially automate fixing the index.

User avatar
Benckj
Forum Moderator - MR2OCNZ
Posts: 7161
Joined: Thu Nov 24, 2005 12:44 pm
Stomping Ground: Otago
Prime Mover: More than one MR2
First name: Jim

Re: Webmaster's Blog

Post by Benckj »

Was starting to wonder why it was down for so long. Thnx for fixing issue.
Jim Benck
90 rev 1 parts car
98 rev 5 GT- all the mods

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

We paid 44.23 USD more than usual for hosting last month because of far higher bandwidth usage. The CloudFlare dashboards showed Singapore was 90% of requests and the AWS billing showing about ~450GB transferred. We only have about ~60GB of files. Looking at the access logging, it's almost entirely Bytedance's (TikTok) crawler. They are known for using Singapore data centres including AWS to get around blocks, a good chunk of traffic was going to AWS Singapore. CloudFlare also presented many anti-bot challenges last month, entirely in Singapore. This robot doesn't respect robots.txt and I don't think we have many MR2 owners in the PRC nor Singapore so I've blocked Bytedance/Bytespider and the two countries.

Source: https://wordpress.org/support/topic/psa ... -blocking/

Now that it's blocked we should see a reduction in our bill back to normal this month, but I'll keep an eye on it to make sure it's working. I can see the blocking rule is already doing something. I can escalate by rate-limiting them instead to a very low value if they try crawling from somewhere else and by throwing the anti-bot page from CloudFlare more often.

I've also lowered the billing alert so I get notified sooner if we're going to exceed our normal spend.

Thanks Gary for noticing it and messaging me.

EDIT: For reference, in April, we only transferred ~1.5GB in and ~24GB out which is within the global free tier of AWS.

User avatar
Benckj
Forum Moderator - MR2OCNZ
Posts: 7161
Joined: Thu Nov 24, 2005 12:44 pm
Stomping Ground: Otago
Prime Mover: More than one MR2
First name: Jim

Re: Webmaster's Blog

Post by Benckj »

Good job Michael. Was wonderful what was going on with the below average posts.
Jim Benck
90 rev 1 parts car
98 rev 5 GT- all the mods

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

Been getting hammered by the bots recently. I had to rollback to a couple of days ago to right before the DB got corrupted

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

I moved the SSDs to the newer set while it was down anyway. Both cheaper and faster, should theoretically increase capacity, but realisitically the constraint is probably the limited number of hardware threads.

User avatar
Benckj
Forum Moderator - MR2OCNZ
Posts: 7161
Joined: Thu Nov 24, 2005 12:44 pm
Stomping Ground: Otago
Prime Mover: More than one MR2
First name: Jim

Re: Webmaster's Blog

Post by Benckj »

Thought there was something going on in the backgraound. Thnaks for sorting out.
Jim Benck
90 rev 1 parts car
98 rev 5 GT- all the mods

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

Bots at it again, I've been blocking them proactively, but they keep finding new ways.

In the meantime I'd tuned the database to only use as much RAM as is available so the operating system wouldn't kill it and thus cause data corruption. This means that the website has been smart enough to just refuse serving traffic when there's been too much load.

This is still better than before where it would occasionally require my manual intervention to go fix the DB corruption and then restart the server. The database is by far the bottleneck and it'll just say "too many connections".

The search performance did suffer a little though, so I have a little bit of further tweaking to do.

Trotbags
Club Member - MR2OCNZ
Posts: 109
Joined: Sun Jan 10, 2016 10:51 am
Stomping Ground: Hawkes Bay
Prime Mover: SW20
First name: Bevan

Re: Webmaster's Blog

Post by Trotbags »

Appreciate you keeping this going. It is one of the reliable go-to places for MR2 knowledge.
91 Turbo, 93 G, 94 G Limited, 96 Sonic Shadow.

mknz
Site Admin - MR2OCNZ
Posts: 1188
Joined: Thu Apr 03, 2014 1:17 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Michael
Contact:

Re: Webmaster's Blog

Post by mknz »

Oh man, LLMs have figured out the Q&A question of user signups, which means I've had a flood of user signups. The question was already a bit esoteric for new owners. Switching back to using reCaptcha which isn't perfect, but hopefully harder.

I approve new accounts manually, so none of them got through

User avatar
MIKEP
Area Coordinator - MR2OCNZ
Posts: 1267
Joined: Thu May 22, 2014 11:34 am
Stomping Ground: Auckland
Prime Mover: More than one MR2
First name: Mike

Re: Webmaster's Blog

Post by MIKEP »

So any one called Ima Bot wont get on. Good one thanks Mike
86 AW11 S/C,
81 Porsche 924S
02 VW Golf GTI
Minisprint powered by 4AGE 73D
99 ZZW30 Spyder
04 VW Passat V5
04 Mercedes ML270Cdi
90 SW20 N/A
2022 Club President
Auckland AC-Shows,Motorsport

Post Reply