<![CDATA[Hotaru CMS - Blogs - Nick's Hotaru CMS Blog by Nick]]> http://hotarucms.org/blog.php?u=1 Hotaru CMS is an open source social bookmarking platform written in PHP. en Mon, 08 Feb 2010 23:05:40 GMT vBulletin 60 http://hotarucms.org/images/misc/rss.jpg <![CDATA[Hotaru CMS - Blogs - Nick's Hotaru CMS Blog by Nick]]> http://hotarucms.org/blog.php?u=1 Time to Get Zipping http://hotarucms.org/blog.php?b=69 Thu, 04 Feb 2010 17:31:10 GMT I think it's a good time to zip up all the files and offer a download version of Hotaru CMS. We used to have everything zipped up, but it became a real chore to maintain when so many updates were being made across so many files. Now, though, things have changed somewhat. I don't expect any major overhauls of the core system, and plugins can be dealt with separately. But what exactly is involved in "zipping up" the files?

There are currently 46 plugins for Hotaru. So each forum thread for each plugin needs to be updated to match the version number, requirements and description stated in the plugin itself. Then, each plugin needs to be zipped and added as an attachment. Ideally, a screenshot will be added, too.

Once that's done, and if my hand hasn't fallen off with a Repetitive Strain Injury, I'll choose the most important plugins and wrap them up in a "social bookmarking suite". After that, the Theme Downloads section needs updating, too.

Finally, the core files need to be zipped, and an announcement posted about its availability. Oh, and I still have a load of updates that need making in the Documentation section.

Still, it's all good fun, eh? :D ]]> Nick http://hotarucms.org/blog.php?b=69 Bugs Abound! http://hotarucms.org/blog.php?b=68 Wed, 03 Feb 2010 01:05:07 GMT It's typical that a few hours after announcing 1.0.5 that some rather nasty bugs revealed themselves, mostly to do with removing user content. I had one user that was upset at people criticizing the quality of his submission. First he edited his comments so they were blank. That made the Latest Comments and Activity widgets look broken. Then he edited the post description with *ADMINS - PLEASE DELETE ACCOUNT ENTIRELY*, and from then on things went really bad as Hotaru failed to "clean" his history completely from the database, and for good measure, emptied the comments table completely! 14,000+ comments over more than two years gone! Naturally, I reverted to the previous day's backup. More bugs followed such as not being able to close the comment form and not being able to change a user's name (Edit: both fixed at SVN trunk rev. 1103)

So I've got my work cut out to identify what's causing these and get them fixed ASAP. Unfortunately, my son is also buggy - throwing up and poo'ing everywhere - so he's off school today and I'm taking care of him which takes priority for now. ]]> Nick http://hotarucms.org/blog.php?b=68 Bob, On the Job! http://hotarucms.org/blog.php?b=67 Mon, 01 Feb 2010 15:00:56 GMT My son's a fan of Bob the Builder, a British kids show about a team of construction vehicles and their leader, Bob. In every episode, all the characters chant "Can we fix it? Yes, we can!"

So I've had the theme song for that in my head this last week while doing my best to fix all the bugs that have surfaced since putting JapanSoc on Hotaru. There are still a few more bugs and loose ends I want to sort out, and that's delaying things a bit, but I have the whole day free tomorrow and hope to get v.1.0.5 on the SVN trunk so you can play with it.

Oh, nearly forgot! I upgraded the server to a dedicated one so you should find these forums and the demo site running much quicker than before. If you'd like to contribute $3 a month towards the running costs, please click here! :o ]]> Nick http://hotarucms.org/blog.php?b=67 Anti-Spam Progress http://hotarucms.org/blog.php?b=66 Sat, 30 Jan 2010 09:05:04 GMT Since moving JapanSoc to HotaruCMS about 48 hours ago, I've added 32 spammers to the StopForumSpam database and have four pages of them on my "Blocked List". I usually get hit by the same spammers again and again, so hopefully once they're all banned, things will quieten down a lot.

Now that I'm getting some real spam to test Hotaru's numerous anti-spam features on, I'm uncovering a few issues that makes zapping spammers less fun than it should be. First of all, the email notifications of pending posts and comments had no information in them other than a link to the post edit page or Comment Manager. That would be fine if you wanted to go to those pages and click "Delete", but what we really needed was a link to the User Manager page so you could delete and ban the user along with their posts. I've now done that by including all details in the email. Here's an edited example of the new "pending post" notification email:

Hi Nick

A new post has been submitted at JapanSoc

Post Status: pending
Post Title: Title
Post Content: Content
Post Page: Link
Edit Post: Link
Post Management: Link to posts with this status, in this case "pending"

User Name: user name
User Role: member
User Email: User email address
User Account: Link
User Management: Link to users with this status, in this case "member"

Thank you,
JapanSoc Admin
That means I can easily go to User Management and zap the user, having seen the post content in the email body. Alternatively, of course, if it's not spam, I can go to the Post Management page and approve the post.

So while that helps a lot, there's still another problem. What if you come back to your machine and find 30 spam posts waiting for you, submitted by 30 different users? Of course, you can set all new users to "pending" when they register, but at that stage you don't yet know if they are spammers or legitimate users. Assuming you're putting the first X posts into moderation, what you really need is to be able to check the box next to all the spammers in a list and zap them all in one go. The User Manager has such a list, but you need the Post Manager page open in another tab so you can see which users submitted spam, and which submitted ham. Checking back and forth like that is somewhat tedious, so I'm going to work on that and see what I can improve. The biggest hurdle is that User, Post and Comment Managers are all independent plugins, so if we want to tie them in with each other, we have to do it with plugin hooks, and that's quite a challenge.

This is where my focus is right now, but I'll try to find time on Sunday to move what we've got sitting on the SVN 1.0 branch to the trunk. ]]> Nick http://hotarucms.org/blog.php?b=66 JapanSoc Down, Hotaru Standing By http://hotarucms.org/blog.php?b=65 Mon, 25 Jan 2010 15:26:11 GMT Yep, it's true...

JapanSoc was plagued by spam. You never saw it, but I got it in my email box and manually deleted it everyday for the last two years and three months. As a result, the database had grown to ugly proportions, filled with the remnants of deleted users, stories, votes, comments, tags, and more. In fact, it was too big for phpMyAdmin’s export feature, so I decided to run a database cleanup module I made ages ago. That module worked, and I was able to backup the database after all the spam had been wiped. Unfortunately, I then realized I couldn’t log in, and I got an email from one of our top users whose account had been deleted! I guess my module was overly keen.


I’m taking this as a signal to scrap everything and start again. Start again fresh with a new CMS, one which I’ve been developing myself, a clean database, and a spam free inbox to wake up to every morning.


To those of you who have given so much to JapanSoc over the last couple of years, I’m really sorry that all your content will be wiped. I could try moving it to the new system, but there are a lot of complications migrating from JapanSoc to Hotaru CMS, specifically with how user accounts are mapped with RPX, the provider that lets you sign in with Google, Twitter, Facebook, etc. I’m not a big business with a tech support department. I’m just a guy at home in the mountains of Gifu, playing with the internet.


Some of you might be reluctant to sign-up again on the new JapanSoc in case I wipe everything again, and that is totally understandable. I hope, however, that you re-join us anyway!
http://blog.japansoc.com/2010/01/25/...-from-scratch/

So it looks like Hotaru will take to the stage earlier than expected. This can only be a good thing as it will get some serious testing on JapanSoc.com. Wish me luck! ]]> Nick http://hotarucms.org/blog.php?b=65 <![CDATA[Oh Lovely. PhpMyAdmin's Gone Screwy]]> http://hotarucms.org/blog.php?b=64 Sat, 23 Jan 2010 12:19:10 GMT Here's phpMyAdmin as I look at it now:

Click image for larger version Name: phpMyAdmin_screwed_up.png Views: 39 Size: 89.4 KB ID: 372

No idea what's causing that. Will lose the next few hours trying to figure it out, no doubt. :(

Edit: Fixed by upgrading to phpMyAdmin 3.2.5 (Phew!) ]]> Nick http://hotarucms.org/blog.php?b=64 Speed Tests http://hotarucms.org/blog.php?b=62 Fri, 22 Jan 2010 05:06:14 GMT I recently blogged about my concerns over Hotaru's loading speed, but before I attempt Jon's suggestions of using Expires Headers and gzip, I thought I'd run some tests to see exactly what is consuming those valuable microseconds.

The first thing to note is the total time:
22 Jan 2010 12:52:22: INDEX TEMPLATE START - 1264132342.8745
22 Jan 2010 12:52:23: INDEX TEMPLATE END - 1264132343.9985
It took about 1.12 seconds which is well over my desired time of around 0.5 seconds. I know even 1.12 seconds sounds reasonable, but this test was done on my own computer using WampServer, so it's quicker than what you see on the demo site.

Let's get into the nitty-gritty. The full output is too long to post here, but here's an excerpt. Can you spot which plugins are eating up the time?

22 Jan 2010 12:52:22: INDEX TEMPLATE START - 1264132342.8745
22 Jan 2010 12:52:22: 1264132342.8762 pluginHook: theme_index_top (sb_base)
22 Jan 2010 12:52:22: 1264132342.8857 pluginHook: sb_base_theme_index_top (search)
22 Jan 2010 12:52:22: 1264132342.8882 pluginHook: sb_base_theme_index_top (tags)
22 Jan 2010 12:52:22: 1264132342.8904 pluginHook: sb_base_theme_index_top (media_select)
22 Jan 2010 12:52:22: 1264132342.8928 pluginHook: sb_base_theme_index_top (categories)
22 Jan 2010 12:52:22: 1264132342.8993 pluginHook: sb_base_functions_preparelist (search)
22 Jan 2010 12:52:22: 1264132342.9001 pluginHook: sb_base_functions_preparelist (tags)
22 Jan 2010 12:52:22: 1264132342.9008 pluginHook: sb_base_functions_preparelist (media_select)
22 Jan 2010 12:52:22: 1264132342.9016 pluginHook: sb_base_functions_preparelist (users)
22 Jan 2010 12:52:22: 1264132342.9051 pluginHook: sb_base_functions_preparelist (categories)
22 Jan 2010 12:52:22: 1264132342.9988 pluginHook: theme_index_top (submit)
22 Jan 2010 12:52:23: 1264132342.9995 pluginHook: theme_index_top (submit_light)
22 Jan 2010 12:52:23: 1264132343.0075 pluginHook: theme_index_top (vote)
22 Jan 2010 12:52:23: 1264132343.0108 pluginHook: theme_index_top (widgets)
22 Jan 2010 12:52:23: 1264132343.0247 pluginHook: theme_index_top (tweet_this)
22 Jan 2010 12:52:23: 1264132343.0272 pluginHook: theme_index_top (activity)
22 Jan 2010 12:52:23: 1264132343.0313 pluginHook: theme_index_top (video_inc)
22 Jan 2010 12:52:23: 1264132343.0333 pluginHook: theme_index_top (comments)
22 Jan 2010 12:52:23: 1264132343.042 pluginHook: theme_index_top (rpx)
22 Jan 2010 12:52:23: 1264132343.0462 pluginHook: theme_index_top (user_signin)
22 Jan 2010 12:52:23: 1264132343.0511 pluginHook: theme_index_top (users)
It's not obvious, but the second Categories function takes very close to 0.1 seconds. Of the 300+ lines of total output, if we can isolate other functions that take 0.1 seconds, then we've got a full second right there! Let's keep looking...

The header_include hook takes a combined 0.15 seconds, but that's not bad for 36 iterations so we'll leave that...

Ah! Here again, we've got Categories eating up another 0.08 seconds:
22 Jan 2010 12:52:23: 1264132343.2432 pluginHook: header_end (categories)
22 Jan 2010 12:52:23: 1264132343.3253 pluginHook: category_bar_end (media_select)
So that's 0.18 seconds already for just two iterations of Categories....

... and that's it. The rest seems to just be the result of over 300 pluginHook calls for my sample index page. One interesting observation though is that the avatar hooks are getting triggered 54 times out of the total 301. Looking at them closely, each one takes about 0.002 seconds, so that would total about 0.1 seconds.

Hmm... so if we disabled Categories and Gravatar, we should save nearly 0.3 seconds. Obviously we don't want to do that, but just for fun let's see what the total would be...

As expected, the number of pluginHook calls is cut to under 250, and the total time is 0.84 seconds:
22 Jan 2010 13:19:00: INDEX TEMPLATE START - 1264133940.0368
22 Jan 2010 13:19:00: INDEX TEMPLATE END - 1264133940.8732
So by disabling those two plugins, we've shaved 25% off the load time. The next challenge will be to speed up the Categories plugin so we can keep it. It must be getting caught up in its recursive functions that look for sub categories. Hmm... perhaps that can be cached... :confused:

AFTER LUNCH...

Now this is very interesting. I've just noticed that even though I disabled categories, it was still showing up in the log:

22 Jan 2010 13:45:44: 1264135544.7432 pluginHook: sb_base_theme_index_top (media_select)
22 Jan 2010 13:45:44: 1264135544.7456 pluginHook: sb_base_theme_index_top (categories)
22 Jan 2010 13:45:44: 1264135544.7538 pluginHook: sb_base_functions_preparelist (search)
22 Jan 2010 13:45:44: 1264135544.7547 pluginHook: sb_base_functions_preparelist (tags)
22 Jan 2010 13:45:44: 1264135544.7554 pluginHook: sb_base_functions_preparelist (media_select)
22 Jan 2010 13:45:44: 1264135544.7561 pluginHook: sb_base_functions_preparelist (users)
22 Jan 2010 13:45:44: 1264135544.7587 pluginHook: sb_base_functions_preparelist (categories)
22 Jan 2010 13:45:44: 1264135544.8612 pluginHook: theme_index_top (submit)
Looking at the pluginHook function, I can see that I'm writing to the log file just before skipping disabled plugins, so there's nothing out of the ordinary here... or is there? If the plugin is disabled, why is it still taking such a long time to ...erm... not run? Looking at the output above, we can see that Categories is the last plugin to be check at each hook. I wonder, if we move it up the plugin list, will the results change? Let's move it above Media Select and see...

22 Jan 2010 13:57:22: 1264136242.4893 pluginHook: sb_base_theme_index_top (tags)
22 Jan 2010 13:57:22: 1264136242.4916 pluginHook: sb_base_theme_index_top (categories)
22 Jan 2010 13:57:22: 1264136242.4916 pluginHook: sb_base_theme_index_top (media_select)
22 Jan 2010 13:57:22: 1264136242.5045 pluginHook: sb_base_functions_preparelist (search)
22 Jan 2010 13:57:22: 1264136242.5056 pluginHook: sb_base_functions_preparelist (tags)
22 Jan 2010 13:57:22: 1264136242.5063 pluginHook: sb_base_functions_preparelist (categories)
22 Jan 2010 13:57:22: 1264136242.5063 pluginHook: sb_base_functions_preparelist (media_select)
22 Jan 2010 13:57:22: 1264136242.507 pluginHook: sb_base_functions_preparelist (users)
22 Jan 2010 13:57:22: 1264136242.6088 pluginHook: theme_index_top (submit)
Bingo! That's more like it. The Categories plugin has been disabled so it takes no time at all, and now the Users plugin is last and appears to take 0.1 seconds to run, which we now know is not true! The most time consuming task must be happening somewhere between sb_base_functions_preparelist and theme_index_top.

That raises the question of why my earlier results shaved 0.3 seconds off the total time if Categories wasn't the problem? That, I don't yet know. Perhaps a caching thing... :confused:

Anyway, I will investigate this further and see if I can isolate the true cause of that 0.1 second delay! ]]> Nick http://hotarucms.org/blog.php?b=62 Yet Another Update! http://hotarucms.org/blog.php?b=61 Mon, 18 Jan 2010 03:52:44 GMT Since there wasn't much happening, I figured I may as well let everyone have the latest changes, even if they come just two days after the last batch! It must be a pain for those of you uploading files to your server, but it shouldn't take more than a minute to upgrade if you're testing locally with WampServer, etc.

I'm kind of torn between putting out frequent updates vs. fewer updates. I guess since we're still not *officially* released (i.e. no download files), it's better to keep pushing forward until we're all satisfied Hotaru works well enough to use on a live site. ]]> Nick http://hotarucms.org/blog.php?b=61 <![CDATA[All's Quiet At HotaruCMS]]> http://hotarucms.org/blog.php?b=60 Sun, 17 Jan 2010 06:32:34 GMT I'm supposed to be flooded with bug reports right now, but everything's been answered in the forums, and I've passed the only remaining ticket back to Carlo, so ... I'm just twiddling my thumbs. Either Hotaru isn't getting a thorough testing, or everything is already ship-shape!

Perhaps we should wrap it up in .zip files and let more people get their hands on it? ]]> Nick http://hotarucms.org/blog.php?b=60 Thinking About Speed http://hotarucms.org/blog.php?b=59 Fri, 15 Jan 2010 16:40:36 GMT I've updated the demo site to v.1.0.2 (coming soon to the SVN trunk), but it's feeling really sluggish compared to these forums. I'm just not sure why.

Hotaru uses caching quite extensively, especially from the second page view. Here are the stats from the home page of the demo site:

52 database queries and a page load time of 1.8 seconds. This page used 6.9 MB bytes of memory. [Hotaru CMS v.1.0.2]
But it feels more like 6 seconds. Two processes which you would expect to strain the server are (1) checking plugin hooks, and (2) finding and merging all the CSS and JavaScript files from plugins. But...

1) When a plugin is installed, the hooks it uses are saved in the database's pluginhooks table. The pluginHook function queries this table and caches the results for 12 hours, unless a new plugin is installed. So using the cached results should be lightning fast.

2) Hotaru uses PHP's file_exists function to find the CSS and JavaScript files and check their last modified dates. If a file has been modified, all the files are merged into one and archived as a single file which is used until anothermerger is necessary. Apparently, file system access is faster than using a database, so this should be pretty quick, too...

Hmm....

Hold on! How about this:

Even though we know which plugin functions to call at each pluginHook, Hotaru still has to create a new object for every plugin function called. Come to think of it, even the main Hotaru class itself creates new objects for all the functions it redirects to. Perhaps this is the cause of the sluggishness?

Any ideas? Or maybe the demo loads pretty fast for you? ]]> Nick http://hotarucms.org/blog.php?b=59 A Few Goodies in the Works http://hotarucms.org/blog.php?b=58 Thu, 14 Jan 2010 08:01:11 GMT You might have noticed the number of commits to the SVN has dropped off since Hotaru 1.0.1. That's because I'm working on 1.0.2 on my local computer and haven't commited any changes yet because you'll need to run the upgrade script. So I'm holding off until I'm done.

So far, I've made a "Recent Visitors" plugin, a "Who's Online" plugin, added the ability to clear all cache folders in one go, and my favorite, a little form on the admin Maintenance page which you can use to add a site-wide announcement to the top of every page. The biggest fix is Hotaru's SmartCache which isn't working in 1.0.1 (did you even notice? :rolleyes:).

I still want to add the ability to edit default permissions and user settings before Hotaru 1.0.2 goes public.

One minor setback is my mouth. I've just had a bottom wisdom tooth pulled out and it's really uncomfortable! I've had nothing but milk for the last 36 hours and I'm getting really hungry! :eek: ]]> Nick http://hotarucms.org/blog.php?b=58 Looking Back At My Things-To-Do List http://hotarucms.org/blog.php?b=57 Tue, 12 Jan 2010 13:35:52 GMT Back in November, I posted my Things-to-do list. I thought I'd take another look at it and see how my priorities have changed:
  • Daily submission/comment limit options [DONE]
  • Ability to edit your profile [DONE]
  • Option to post articles without a link [DONE]
  • External Vote Button (EVB) [DONE]
  • A way to alert mods about rule-breaking comments
  • Ability to "friend" or perhaps "follow" other users
  • Option to limit the number of urls allowed in posts/comments [DONE]
  • A plugin to let you email some or all users [DONE]
  • A way to allow users to "message" each other
  • New Vote plugin with additional options
  • Pageview voting
  • A plugin to support RPX [DONE]
  • An RSS Importer plugin
  • A settings page for users to customize their site preferences [DONE]
So I did 8 out of 14. Not bad.

Of the ones I didn't do, none of them seem as pressing as these:
  • Ability to change default permissions for roles (e.g. "moderator" or "member" groups)
  • Ability to change default "user settings" for roles
  • Filtering options for users (e.g. so users can hide selected categories, users they don't like, or posts from specified domains)
  • A Who Voted? plugin
  • Error logging
  • A settings page for the Posts Widget so you can specify how many posts to show in the widget
  • A way to alert mods about rule-breaking comments (Ha! Okay, this was from my original list :o)
The number one priority, though, is to get any bugs fixed. That comes first, features come second. ]]> Nick http://hotarucms.org/blog.php?b=57 Stop Spam Plugin Stretches Its Legs http://hotarucms.org/blog.php?b=56 Tue, 12 Jan 2010 06:00:26 GMT I've just managed to fix the problems I was having submitting spammers to the Stop Forum Spam database, and I have to say that this is just totally brilliant! The demo site got its first spammer, so I went to User Manager, checked the box next to the spammer, checked "Add to Blocked List", checked "Add to StopForumSpam.com", selected "Killspam" and voila, the guy is double-banned and his spam post deleted.

What do I mean by double-banned? Not only will his record on the demo site's Blocked List prevent him from registering on the demo site again, but his IP, username and email address are now listed at StopForumSpam.com, so he won't be able to spam any other Hotaru sites either!

I'm a bit disappointed that Akismet didn't catch his post in the first place, but perhaps Akismet just needs more training with the kind of spam that social bookmarking sites attract. Fortunately, when you delete a post in Hotaru, Akismet is notified and learns from it for next time. We shall see...

For the time being, I've enabled another anti-spam option as well: All first-time posts go into moderation and will need approving before the user can post freely. Fortunately, as site admin, I get emailed immediately of any such submissions so can approve things pretty quickly. If I wanted to, I could delegate the approval task to moderators. ]]> Nick http://hotarucms.org/blog.php?b=56 Changing the PostVotes Table http://hotarucms.org/blog.php?b=55 Mon, 11 Jan 2010 14:32:19 GMT One thing that I meant to do before releasing Hotaru 1.0 was to change the way votes were stored in the PostVotes table in the database. In the... One thing that I meant to do before releasing Hotaru 1.0 was to change the way votes were stored in the PostVotes table in the database. In the current version, votes are either "positive", "negative" or "alert", but that doesn't leave much room for flexibility.

So I'm working on changing this to use integers instead. In order for the Pligg Importer to import sites that use Pligg's Star Rating system, I've chosen values between -10 and +10, and -999 for flagged posts. The current Vote plugin uses Digg-style votes, so all votes will be +10. For Reddit-style voting, you could use -10 for negative votes. Star rating-style systems can use a range of numbers, most likely 2, 4, 6, 8 and 10 so that sites imported from Pligg can still function.

The challenge for me was writing the correct SQL queries in the Hotaru upgrade script that would change the current strings in the PostVotes table to their integer equivalents, and change the column data type from ENUM to SMALLINT. Not surprisingly, I screwed up on the first attempt and lost all my test data! :rolleyes:

Tomorrow I'll try it on the demo site, and if it works there I'll move the changes to the SVN trunk. ]]> Nick http://hotarucms.org/blog.php?b=55 And so it begins... http://hotarucms.org/blog.php?b=54 Sat, 09 Jan 2010 16:01:15 GMT Nine days into 2010, and Hotaru CMS is now publicly available (http://hotarucms.org/showthread.php?336-Hotaru-CMS-1-0-Available-for-Testing) from our... Nine days into 2010, and Hotaru CMS is now publicly available from our SVN on Assembla.

Quote Originally Posted by Nick View Post
As promised, Hotaru 1.0 is now available for testing on the SVN trunk. Thank you for your patience over the last month.
What a month! I made 127 commits to the SVN between December 12th and January 9th and am feeling thoroughly knackered. Although I expect users will find a bucket load of bugs, I'm very happy with this latest version of Hotaru. My goals were to make plugin and theme development easier and I think I've done that. Using the Hotaru class as a library of shortcuts to other class methods works really well for simplifying code. It's not perfect, but pretty close to my original plan. The SB Base plugin also serves its purpose by giving some logical order to how page names and types are determined, therefore untangling the knots in my brain that debugging previous plugins caused.

Now I'll sit back and wait for bug reports to come in. I'd like to extend the documentation, but will need feedback from people who are actually making themes and plugins so I know what kind of help I can give them.

With that said, I'm off to bed. Maybe I'll wake up and everyone will be playing with Hotaru... or perhaps no-one will and the demo will be hacked! :p ]]> Nick http://hotarucms.org/blog.php?b=54