Average response time per country and a couple of fixes

During the last weeks we have worked on a couple of minor changes you won’t see because most of them are under the hood or not released yet. However, two of the released ones are probably important to you.

Uptime chart - response timesAverage response time chart

This was something we were planning to do, but there was never enough time to make it done.

So we finally did it in the middle of our work for a major change of the whole downtime alert system (which is going to be released soon) to get a brake and freshen up our minds.

As you can see on the screenshot, we simply added additional data to the existing uptime chart. Those data represent the average response time for your website, grouped by country. You can find it on the test report page.

The blue line has not changed, it’s still an uptime percentage per day. The additional lines are yellow, red and orange. Those show the average response time per day for each country (US, FR and PL).

We also added a tooltip to each line to give you more detailed information about what you see on the screen.

This is the first version of this feature. We are going to enhance it in the nearest future.

No more e-mail flood

While working on the new e-mail notification system, we found out that sometimes something bad happens with our crawlers, and sometimes more than one robot grabs a tests to do an uptime check.

For some (thankfully) edge cases it caused more than one downtime alert.

The edge case was: the robots behave nastily and grab already reserved test to be checked and the website is actually down. Then, two or three robots detect this downtime, and all of them send the downtime notice.

This should not happen again, since we have finally found the real bug in our code.

Realtime uptime widget

If you’re our superuser you probably noticed you have an option to create a public page of your tests.

To make this functionality more interactive and custom, we created a possibility to add a widget wherever you want with the realtime uptime chart.

Adding a widget to your site is really simple, and if you have a opportunity to create a <script> tag to your website, you’ll definitely handle it without any problems.

Step 1 – create a public uptime monitoring page

First thing you need to do is create a public uptime monitoring page. To do that, simply click on the yellow “create hash” button, next to a test name.

Create hash button

When you clicked the button, a new hash is created and now you have access to two different buttons: public page and reset.

Single test on list

Public page is a public link with a realtime uptime monitoring chart. Anytime our robot access your site and report the uptime status, you’ll see a green dot on a chart.

If you hover the dot, you’ll also get an information about the single test proceeded

Realtime uptime monitoring - test site

Remember the page is as public in a same way as your dropbox share links. It is not protected by password or your account session. Anyone who knows the link, can access the site.

If you’d like change the link, you can always do that clicking the “reset” button on a tests list.

Step 2 – copy the secret part

Your new public realtime page URL looks similar to this one:

http://agentslug.com/public/realtime/d5ffc4218f/104

Now, write down (copy) two last parts of the URL. You’ll need it for widget configuration.

For this example, we need to copy this: “d5ffc4218f/104”

Step 3 – insert a <script> to your site.

Now it’s time for the last step. Simply copy the html below and paste it wherever you want (for example on you blog sidebar, just like we did here).

<div id=”uptime_graph” data-title=” ” data-hash=”d5ffc4218f/104″></div>
<script src=”http://agentslug.com/ps/realtime_widget.js” data-target=”uptime_graph”></script>

The only thing you need to customize to make it work for your test is to change the “data-hash” attribute. Paste there the two last one parts of the URL you copied on step 2.

You can also change the “data-title” attribute. If you leave it empty, there will be no title at all (just the chart and your logo), just like we did on this blog.

I’m a web developer, how to customize it

If you’re a web developer, you probably need to customize it.

You can do it both way:

By configuration

The script has basic configuration options. You can change the title (see above) and height of the iframe which displays the widget (default is 300px, max is 500px).

By making your own javascript

If you like to, you can always make your own script, using our as an example.

Source code of the script, extended information can be found on our GitHub repository.

 

 

Freezing curl causing a lot of troubles

Not-so-infinite loops

Our website monitoring mechanism is actually quite simple.  We use curl to make a connection with the website and then we grab some data to check if your site is up or down (actually, curl has more power and we use it in our new features we develop right now as well).

To make sure we are able to test your site in almost equal time intervals, we need to have a couple of testing agents working at the same time.

The job of these agents is to sit and wait for available tasks for some period of time. If there are any, agent reserves one of them and does the actual website uptime test.

However, since we started the Ukrainian and Russian government websites monitoring site (lots of downtimes), we found out that some of the scripts sometimes freeze for no good reason.

It wasn’t a serious problem, because it didn’t influence the uptime monitoring service reliability. However, the zombie agents were taking our precious server resources to do nothing.

Curl has two timeout options

And then, there comes the time when we decided to fix this issue. While debugging we found out that the solution is quite trivial.

As you probably know, we assumed that if your site is not responding in 10 seconds after our call, it means that the website is down. Nobody is waiting 10 seconds for first document request anyway. Even if your server works, the user nearby our agent’s server location might not be able to notice this.

These 10 seconds of timeout are set as a curl option (CURLOPT_CONNECTTIMEOUT). It means “if the server is not responding in X seconds, do not wait more and return connection error [http 0]”.

What we didn’t notice is the other curl option – CURLOPT_TIMEOUT and it was the cause of our problems. The option is more important in this case since it forbids curl to freeze. Connection timeout is related to the connection only. If connection is made quickly, the clock for the whole script is not ticking any more.

Summary (tl;dr)

If you plan to use a curl mechanism inside a do-while loop with iteration limit, remember to set a curlopt_timeout or your loop won’t die sometimes.

 

 

Tutorial for custom maintenance mode in WordPress

Maintenance mode in WordPress

Maintenance mode is the “be right back” state of your website.  You should use it any time you are changing the WordPress code or making anything else with a non-zero possibility to fail and affect a user experience.

You’ve probably noticed that WordPress is actually turning the built-in maintenance mode while upgrading the WP core, or adding new plugins.

In this article, you’ll learn how it works, how to turn the maintenance mode on manually, and how to make it look better.

How the built-in maintenance mode works

When you click the “upgrade WordPress” button,  Wordpress is inserting the .maintenance php file into the site’s root directory with $upgrading variable set to time().

Wordpress maintenance mode function

Every single time someone enters your site, WordPress is looking for this particular file and if it exists, WordPress is comparing the $upgrading variable with the time() value. If the difference is less than 600 seconds (10 minutes), the maintenance mode function is looking for a custom maintenance.php inside wp-content directory.

If it exists, WordPress loads that file and exits the compilation process.

If not, the maintenance function compiles the built-in maintenance landing page and ends with the compilation process. What’s important, WordPress handles the maintenance mode well with a proper http code sending to a user.

Find out why the proper http code is important.

How to make a custom maintenance mode landing page?

To make a custom maintenance mode landing page, you just need to make your own php script and save it into wp-content directory as maintenance.php file. Every time your site is in the maintenance mode, WordPress will look for this file and compile it, ignoring the built-in landing page.

To make it easier for you, we’ve made an example script. You can grab it from our GitHub repository and customise it any way you like.

How to turn on the maintenance mode manually?

If you would like to turn on the maintenance mode manually, without any plugin you just need to insert the .maintenance file into your WordPress main folder.

We made the file for you, extending it with an IP white list and secret key white list support. You can grab it from our GitHub repo as well as the maintenance.php example file.

 

Common HTTP codes list with meanings

HTTP? What a heck is that?

Long story short, HTTP is an abbreviation of “Hypertext Transfer Protocol”. This protocol defines how machines like your computer or mobile phone should talk to any other machine.

Of course, there are plenty of other protocols with different purposes. You’re probably familiar with some of them like FTP, or SSH.

The main and most common purpose of HTTP is to connect a client machine with a server machine to get or post some data.

Here,  we are focused mostly on HTTP protocols. Mostly because it’s the most common protocol which we deal with on AgentSlug.com while we monitor your sites.

You can find more about http protocol in “how machines talk to each other” article.

HTTP codes list

You can find full list of http codes on the Wikipedia.  If you’re familiar with the server maintenance, you probably already know them. If not, you should check the Wikipedia list anyway.

In this article I’d like to tell you more about most common http codes, and their meanings related to AgentSlug.com service (or any other uptime monitoring service as well). The article is meant to be a quick manual for those who are not familiar with technology, but need to know the most important things for their website.

HTTP 200 OK

Your server should respond to our request with this code when everything is fine. When server sends that code to a client with a document header, it means “everything is OK, you can find the document you requested below”.

However,  AgentSlug.com treats this kind of response as a downtime anyway in one case – when you set a keyword to be found in document body and it’s not there.

We thought about this case a lot and as a result we decided to make this that way.

As far as we know, sometimes websites doesn’t respond with proper http code when they are down. For example, there are lots of buggy WordPress themes which return http 200 OK even when a site is under the maintenance mode.  We can not change the world, but we can deal with it giving you a tool to detect the downtime anyway.

Besides the buggy maintenance mode, this feature is also considered a deface detection trigger.

The other downtime case with the http 200 code is when your site is hacked (cracked) and defaced.  Most often, the cracker is going to change the entire site to get the attention. If you set a must-have keyword to your test, there’s quite good chance to detect the problem and send you a notice right away.

HTTP 500 Internal Server Error

In the perfect world we wouldn’t see this code in any production environment.  It means that something really wrong happened while compiling a website code or there is some problem with your hosting.

If you see a downtime notice from AgentSlug.com with this error, you should check your website as soon as possible.

If you’re able to check error logs on your server, you should do that as well.

However, if you can’t find any bug on your website, and downtimes are still happening from time to time – you should consider changing your hosting provider.

HTTP 503 Service Unavailable

This code is a common maintenance mode server response. It means that your site is temporary unavailable, and all visitors (including Google spiders) should check it back in a couple of minutes.

It’s important to set that code when you’re working on production environment and your work can possibly affect user experience with the site.

For example, if you’re upgrading your WordPress, the site is turning a maintenance mode for the entire upgrading process with that code.  Any user coming to your WordPress site at that exact time would see the “come back later” notice.

Why is it so important?

If your “maintenance mode” sends HTTP 200 OK code, there is possibility that in that exact time some search crawlers are going to visit your site to make a copy to its index. I bet you don’t want to get your site indexed while it’s in the maintenance mode.Miniclip.com - maintenance mode with http 200 code indexed

HTTP 0, “not available”

It’s not a real http code. You won’t find it on the Wikipedia list, and your server is not responding with that code anytime.

However we use this code as a “no response” code.

If your server is not responding as quickly as any user is willing to wait (~10–15 seconds) it means your site is down. AgentSlug.com treats this connection as failed as well and sends you a downtime notice with this code in description.

HTTP 404 Not Found

This code simply means that given URL doesn’t match any resource on the server. If you see this code on your downtime message and you’re sure that the URL you set on your test is valid, you should check your site as soon as possible.

HTTP 301, 302, 303 – redirections

All of these codes mean that the resource matches some URL but it’s on another location right now.

Some of the developers use http 302 code as a temporary redirect for maintenance mode. It’s not a good practice, since the code means something else. It’s also not an evil practice, since search crawlers won’t probably download the site with this code.

However, the same developers often redirect user with that code to some /maintenance page which sends http 200 OK code which is actually evil as devil itself (see miniclip.com example above).

403 Forbidden

If you see this code on your downtime message you should check your site immediately. It means that resource behind the given URL is not meant to be shown to public/anonymous user.

Here are some possible causes of this kind of behavior.

  • someone changed your website/app directory permissions (“chmod”) on server and your app can’t be compiled/executed
  • someone/something deleted all your app files and your server can’t find any
  • the app server stopped like the AgentSlug.com and treats it as an abusing service (firewall sends us back home)

That’s all for now. As always, any comments are welcome below. Feel free to ask for explaining any other code you’d like to have explained.

Low account and zero account notifications tweak

Fresh, spring update

Tuning up the notifications

Recently we were working on another feature that may greatly improve your user experience with our service.

In order to meet your expectations, we’ve changed our low account and zero-account notifications.

How to disable notifications?

In fact it’s quite simple. You need to pause or delete all your active tests and then you’ll no longer receive any message.

Secondary email, quick settings link and more

New features you’ll love

Last couple of weeks we’ve been tuning up the AgentSlug.com to make it even better, faster and more reliable. We’re also preparing ourselves to do a big pivot of the app by adding some cool functionalities.

Aside of planning and wireframing the big v 2.0, we’ve already made some new features which seem to be a must-have as soon as possible.

All features listed below are available since today.

Secondary emails

After the upgrade, every website test (monitoring) can be labeled with additional, secondary (alternative) email.

When site is down (or up), AgentSlug.com will send a notification to the secondary email. Even if you set “email notifications” off in your account settings. Secondary email attribute is fully independent.

User story

I’m monitoring (personally) dozen of websites within AgentSlug.com uptime monitoring service. Not all of them are important to me nor my client. Some are just triggered to grab some data from the world.

For example, after we had deployed Ukraine-Russia cyber war monitoring I got flooded with up/down notifications from monitored websites. There were so many emails, that I was forced to turn the email notifications off to my whole account to not get flooded to death.

Now, with a secondary email feature, I still have email notifications turned off as a global setting. However, I set a secondary email to every important website.

Turn off email notifications within notification itself

If you’re a hardcore AgentSlug.com user, there is some probability that you get a lot of messages from the service.

Sometimes you’re aware of a problem with your site and don’t want to get more messages anymore.

Now, you can turn them off by clicking a “turn off” link. The link is valid up to two days and works even if you’re logged out.

Turning off the email alerts

Email notification turned off alert

Just in case you forget that you turned off email alerts, we will send you an email to remind you about that.

It would be sent no more often than once a month and only if one of your sites is down. If none of your site is down, you won’t get any message. Ever.

 

Next step

Feedback is crucial

AgentSlug.com is getting more and more feedback from our first users.  Most of it is positive and that makes me happy, but what’s more important to me is that some of it is really constructive and learning.

As a developer who has decided to make a tool based on “scratching own itch” problem, I can’t be sure what are others’ needs and why they are actually using the app.  Feedback is crucial. Many thanks to everyone for their effort.

Next step

From the first deployment date we are tuning the uptime monitoring module to make it better and more optimized. However, we are about to go further than that.

AgentSlug.com is an uptime monitoring tool for now, but we want to build on it a complementary automatic webmaster tool focused on website monitoring in many dimensions:

  • full content monitoring
    • dead links detection
    • html validation
    • spell checker
    • WAI problems detection within the content (like no [alt] attributes on images)
  • anomalies detection – if your homepage suddenly grows or shrinks it might be hacked (defaced)
  • real users’ load time
  • resources load time testing

That’s just a shortlist of the ideas that we are fully agreed are important for every webmaster or website owner.