WordPress All In One Wp Security (AIOS) locking cloudflare out

I’m going on a trip tomorrow and not gonna bring my computer, so I decided to take one last check at my WordPress site making sure everything is doing fine. Then I saw recently all the bot like high count visitors are from a narrow range of IP from Singapore. I thought it’s just some good old crawler draining up my server, so I casually added the range to All In One Wp Security (AIOS)’s blacklist manager.

Then, without refreshing, I realized I recently setup cloudflare for all my site cuz my other site was DDoS’ed the other day. Couldn’t this happen to be cloudflare’s proxy server right? So I hit refresh. 403. At the same time, my Better Uptime incident alert came.

Bummed by my own stupidity, I think I should just be able to ssh into the server, disable all the plugins as usual, and back to the game. Well, after renaming both all-in-one-wp-security-and-firewall and all plugins at /var/www/html/wp-content/plugins/, I’m still getting 403. Usually all the plugin related issue got solved at this step without needing to restart service, I’m not sure why this time it didn’t work.

So I start to Google something like AIOS self lockout, trying to find where does it store the blacklist that I can quickly manually edit by command line. No luck.

I tried to disable cloudflare proxy at cloudflare dashboard, no luck. And I’m too lazy to switch DNS provider back to where I was using before.

I started to poke around more in the plugin’s folder, and eventually found the file that did the trick: /var/www/html/wp-content/plugins/all-in-one-wp-security-and-firewall/admin/wp-security-list-locked-ip.php.Seems like the IPs are stored at a table called AIOWPSEC_TBL_PERM_BLOCK. I just commented out the entire line here in the prepare_items() function. Refresh, back online, removed the IP range from blacklist, uncomment the line.

So, that was just a small stupid incident on a random Thursday night 9pm before a trip. 乁། * ❛ ͟ʖ ❛ * །ㄏ

If you find this blog useful and want to support my blog, feel free to:

Become a Patron!

迁移 Object Storage 时候发生的一些愚蠢事件以及解决方案

TL;DR: 我用 rclone 迁移 object storage 的时候脑一残就把目的 bucket access 设成了 private,导致迁移之后长毛象无法访问新的 bucket 所有媒体文件都无法显示,最后只好使用 s3cmd recursively set public acl 了一遍所有媒体文件。

对事情经过不感兴趣想直接看的请点此直接跳转到 Debug 步骤解决方案部分。

Continue Reading

Mastodon 中文全文搜索失效 debug 日记及修复方法

我的长毛象实例最近全文搜索失效,debug 了一下失效原因。简而言之,elasticsearch 自动更新了,所以之前装的中文全文搜索 plugin 本来是给 6.8.12 built 的,现在 6.8.14 运行不了,elastisearch 就卡死了。


1. 在根目录下运行
/usr/share/elasticsearch/bin/elasticsearch-plugin remove analysis-ik
/usr/share/elasticsearch/bin/elasticsearch-plugin remove analysis-stconvert
来卸载旧版本 plugin。

2. 运行如下两个命令下载为新版本编译的两个中文 plugin。
/usr/share/elasticsearch/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/

/usr/share/elasticsearch/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-stconvert/releases/download/v6.8.14/elasticsearch-analysis-stconvert-6.8.14.zip

3. restart elasticsearch,运行 sudo systemctl restart elasticsearch.

以下是 debug 过程,不想深入研究的可以不看。

  1. 今天早上本站用户说搜索坏了,搜不到嘟文只能搜到用户名和 tag,我当时在手机上以为是之前搜索不全的老问题。
  2. 今天晚上自己测试了一下,确实任何全文搜索都搜不到,打开 sidekiq 发现从四月初开始 failure 暴增,应该是有问题。
  3. failed message Faraday::ConnectionFailed: Failed to open TCP connection to localhost:9200 (Connection refused - connect(2) for "localhost" port 9200) ,Google 此 message 可知是 elasticsearch (后简称 ES)的问题。
  4. Google 如何查看 active ports(没错这玩意儿天天用但是永远记不住,but why bother though),用 sudo netstat -tulpn | grep LISTEN 发现 9200 not active.
  5. Google 如何重启 ES,运行 sudo systemctl restart elasticsearch ,但发现 sidekiq 失败 message 并没有下降。Google 如何查看 ES 状态,运行 sudo systemctl status elasticsearch,发现重启失败,最后一行报错是 JAVA_HOME not set.
  6. export JAVA_HOME=/usr/bin/java ,重启 ES,发现没有解决问题,Google 之后发现有人说可能这并不是错误所在。
  7. Google 如何查看 ES log,运行 less /var/log/elasticsearch/elasticsearch.log ,发现报错 uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [analysis-ik] was built for Elasticsearch version 6.8.12 but version 6.8.14 is running
  8. 在 mastodon doc(https://docs.joinmastodon.org/admin/optional/elasticsearch/ )里找到当初安装的俩中文 index plugin repo,进入 repo 找到安装方法,Google remove elasticsearch plugin 方法卸载旧 plugin,安装 6.8.14 相对应新版本的俩 plugin。
  9. 重启 ES,查了几次 status 发现运行稳定,sidekiq 里 failed retry queue 稳定下降,在 Mastodon 里随便搜了几个中文出现了过去的嘟文,确认问题解决。


Become a Patron!


Scaleway Object Storage 不稳定和 Mastodon 迁移备份笔记

我的长毛象实例 douchi.space 一直采用 Scaleway 的 Object Storage,因为它们前 75G 免费,对于这种小服务很划算。但是最近一个月连续发生两次故障,因为是免费 tier 所以客服修复的时间也没有保证。本文记述了故障可能的产生原因、debug 过程、修复方法和以后可能采取的 precaution messures。

Continue Reading