Webmaster's Blog
Moderator: Moderators
- GDII
- Forum Moderator - MR2OCNZ
- Posts: 5925
- Joined: Sun Jul 10, 2011 5:13 pm
- Stomping Ground: Wellington
- Prime Mover: SW20
- First name: Phill
Re: Webmaster's Blog
Thanks Michael! It did have a different error this time vs the previous ones so it does appear to be something else.
1990 SW20 MR2 G-Limited (GEN4 3SGTE)
2000 AE111R Corolla Wagon NZ New Daily
1996 AE101R Corolla Sprint NZ New (Sold)
1990 EP81 Starlet XL (Sold)
1990 EE90 Corolla XL (Sold)
Instagram https://www.instagram.com/sw20glimited/
2000 AE111R Corolla Wagon NZ New Daily
1996 AE101R Corolla Sprint NZ New (Sold)
1990 EP81 Starlet XL (Sold)
1990 EE90 Corolla XL (Sold)
Instagram https://www.instagram.com/sw20glimited/
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
Forum went down, but fix was easy, just needed to clear the cache
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
Hmm started getting weird logging in behaviour, so I disabled a bunch of CDN features to see if that resolves the issue
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
I've enabled uploading *.heic and *.heif images, but none of the browsers support viewing them so you have to download them to see them.
You can convert them to JPEG pictures on your phones:
https://www.howtogeek.com/666363/how-to ... -and-ipad/
Samsung devices still need to use a third-party converter in the meantime, but you can disable using HEIF/C in the meantime.
You can convert them to JPEG pictures on your phones:
https://www.howtogeek.com/666363/how-to ... -and-ipad/
Samsung devices still need to use a third-party converter in the meantime, but you can disable using HEIF/C in the meantime.
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
Sorry, had to get home to fix the DB log file index. It got corrupted as it probably auto-rebooted on a crash. If this happens again I'll update the filesystem mounting to be stricter, and potentially automate fixing the index.
- Benckj
- Forum Moderator - MR2OCNZ
- Posts: 7161
- Joined: Thu Nov 24, 2005 12:44 pm
- Stomping Ground: Otago
- Prime Mover: More than one MR2
- First name: Jim
Re: Webmaster's Blog
Was starting to wonder why it was down for so long. Thnx for fixing issue.
Jim Benck
90 rev 1 parts car
98 rev 5 GT- all the mods
90 rev 1 parts car
98 rev 5 GT- all the mods
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
We paid 44.23 USD more than usual for hosting last month because of far higher bandwidth usage. The CloudFlare dashboards showed Singapore was 90% of requests and the AWS billing showing about ~450GB transferred. We only have about ~60GB of files. Looking at the access logging, it's almost entirely Bytedance's (TikTok) crawler. They are known for using Singapore data centres including AWS to get around blocks, a good chunk of traffic was going to AWS Singapore. CloudFlare also presented many anti-bot challenges last month, entirely in Singapore. This robot doesn't respect robots.txt and I don't think we have many MR2 owners in the PRC nor Singapore so I've blocked Bytedance/Bytespider and the two countries.
Source: https://wordpress.org/support/topic/psa ... -blocking/
Now that it's blocked we should see a reduction in our bill back to normal this month, but I'll keep an eye on it to make sure it's working. I can see the blocking rule is already doing something. I can escalate by rate-limiting them instead to a very low value if they try crawling from somewhere else and by throwing the anti-bot page from CloudFlare more often.
I've also lowered the billing alert so I get notified sooner if we're going to exceed our normal spend.
Thanks Gary for noticing it and messaging me.
EDIT: For reference, in April, we only transferred ~1.5GB in and ~24GB out which is within the global free tier of AWS.
Source: https://wordpress.org/support/topic/psa ... -blocking/
Now that it's blocked we should see a reduction in our bill back to normal this month, but I'll keep an eye on it to make sure it's working. I can see the blocking rule is already doing something. I can escalate by rate-limiting them instead to a very low value if they try crawling from somewhere else and by throwing the anti-bot page from CloudFlare more often.
I've also lowered the billing alert so I get notified sooner if we're going to exceed our normal spend.
Thanks Gary for noticing it and messaging me.
EDIT: For reference, in April, we only transferred ~1.5GB in and ~24GB out which is within the global free tier of AWS.
- Benckj
- Forum Moderator - MR2OCNZ
- Posts: 7161
- Joined: Thu Nov 24, 2005 12:44 pm
- Stomping Ground: Otago
- Prime Mover: More than one MR2
- First name: Jim
Re: Webmaster's Blog
Good job Michael. Was wonderful what was going on with the below average posts.
Jim Benck
90 rev 1 parts car
98 rev 5 GT- all the mods
90 rev 1 parts car
98 rev 5 GT- all the mods
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
Been getting hammered by the bots recently. I had to rollback to a couple of days ago to right before the DB got corrupted
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
I moved the SSDs to the newer set while it was down anyway. Both cheaper and faster, should theoretically increase capacity, but realisitically the constraint is probably the limited number of hardware threads.
- Benckj
- Forum Moderator - MR2OCNZ
- Posts: 7161
- Joined: Thu Nov 24, 2005 12:44 pm
- Stomping Ground: Otago
- Prime Mover: More than one MR2
- First name: Jim
Re: Webmaster's Blog
Thought there was something going on in the backgraound. Thnaks for sorting out.
Jim Benck
90 rev 1 parts car
98 rev 5 GT- all the mods
90 rev 1 parts car
98 rev 5 GT- all the mods
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
Bots at it again, I've been blocking them proactively, but they keep finding new ways.
In the meantime I'd tuned the database to only use as much RAM as is available so the operating system wouldn't kill it and thus cause data corruption. This means that the website has been smart enough to just refuse serving traffic when there's been too much load.
This is still better than before where it would occasionally require my manual intervention to go fix the DB corruption and then restart the server. The database is by far the bottleneck and it'll just say "too many connections".
The search performance did suffer a little though, so I have a little bit of further tweaking to do.
In the meantime I'd tuned the database to only use as much RAM as is available so the operating system wouldn't kill it and thus cause data corruption. This means that the website has been smart enough to just refuse serving traffic when there's been too much load.
This is still better than before where it would occasionally require my manual intervention to go fix the DB corruption and then restart the server. The database is by far the bottleneck and it'll just say "too many connections".
The search performance did suffer a little though, so I have a little bit of further tweaking to do.
-
- Club Member - MR2OCNZ
- Posts: 109
- Joined: Sun Jan 10, 2016 10:51 am
- Stomping Ground: Hawkes Bay
- Prime Mover: SW20
- First name: Bevan
Re: Webmaster's Blog
Appreciate you keeping this going. It is one of the reliable go-to places for MR2 knowledge.
91 Turbo, 93 G, 94 G Limited, 96 Sonic Shadow.
-
- Site Admin - MR2OCNZ
- Posts: 1188
- Joined: Thu Apr 03, 2014 1:17 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Michael
- Contact:
Re: Webmaster's Blog
Oh man, LLMs have figured out the Q&A question of user signups, which means I've had a flood of user signups. The question was already a bit esoteric for new owners. Switching back to using reCaptcha which isn't perfect, but hopefully harder.
I approve new accounts manually, so none of them got through
I approve new accounts manually, so none of them got through
- MIKEP
- Area Coordinator - MR2OCNZ
- Posts: 1267
- Joined: Thu May 22, 2014 11:34 am
- Stomping Ground: Auckland
- Prime Mover: More than one MR2
- First name: Mike
Re: Webmaster's Blog
So any one called Ima Bot wont get on. Good one thanks Mike
86 AW11 S/C,
81 Porsche 924S
02 VW Golf GTI
Minisprint powered by 4AGE 73D
99 ZZW30 Spyder
04 VW Passat V5
04 Mercedes ML270Cdi
90 SW20 N/A
2022 Club President
Auckland AC-Shows,Motorsport
81 Porsche 924S
02 VW Golf GTI
Minisprint powered by 4AGE 73D
99 ZZW30 Spyder
04 VW Passat V5
04 Mercedes ML270Cdi
90 SW20 N/A
2022 Club President
Auckland AC-Shows,Motorsport