Recently I went looking for a solution for backing up from a Plesk server to S3: what I settled on was surprisingly simple.
I started with a simple list of criteria, but as I went looking for a solution, and as I continued to find no good ones, my list got longer. I have a tendency to be quite OK with the bare bones if I'm going to be using an existing system, but if I have to build it myself, I'm normally happy to add more features.
I started with basically "I want to be able to backup files and databases", but the solution I ended up with also gave me the following features:
- backup multiple domains
- along with files and databases, backup the actual domain configuration and mail if required
- rotate backups automatically
- define the frequency of backups and the number of backups to keep before rotating
- open source
- a simple interface right in Plesk
So what did I do?
I don't know, really, if this is a super smart way to do it, or just a cop-out, but basically I realised that hey, the Plesk backup manager already let's us do all of the above... except for the S3 part. All that I ended up doing was installing s3cmd from http://s3tools.org and setting it up to do the syncing to S3, looking at the location on the server that Plesk puts it backups.
So basically, users (or I) define backup rules for each domain as needed (via the Plesk UI) and then s3cmd runs with the sync option once a day.
With s3cmd located in my /root/cli-tools directory, and assuming s3://example.com is the name of the bucket I will use, the actual cron tab entry I use is as simple as:
cd /var/lib/psa/dumps; /root/cli-tools/s3/s3cmd -c /root/cli-tools/s3/s3cfg --delete-removed -H --no-progress sync domains s3://example.com/backups
UPDATE: as per a comment by Rutger below, you may actually want to use:
cd /var/lib/psa/dumps;
/root/cli-tools/s3/s3cmd -c /root/cli-tools/s3/s3cfg --delete-removed -H
--no-progress sync clients s3://example.com/backups
instead of, or possibly in association with, the above line.
When I commission a new Plesk server, I just copy the s3cmd directory over, create a new bucket and I'm done.
The only downsides I see, really, are that if I wanted to just have a single rule for all of the domains, I couldn't. Also, I'm assuming that all of the backups have been run when the cron job runs once a day. Not that that matters too much, as I could just bump the cron job up to hourly if I liked and I wouldn't see much difference.
I think the biggest negative to this approach is that I'm pushing backups explicitly even if what has been backed up actually hasn't changed. That is to say, if Plesk does a backup everyday, then I push a new backup every day... even if nothing has changed since the last backup.
Anyway, I hope this helps someone as for me it was completely obvious once I realised it, but it took me an embarrassingly long time to get to it.
Monday, June 10, 2013
Sunday, June 9, 2013
On having nothing to hide...
Obviously, there's a lot of discussion right now about what information should be considered private, and whether or not people really have nothing to hide. I believe that people have a lot of information that they want to keep private: even when they say they don't.
When I meet people that say they have nothing to hide, I 'play' one (or more) of three cards.
The first thing I will do is ask them how often they masturbate. I know it's a fairly taboo question, but that's the point. It's not illegal, it doesn't hurt anyone, but not many people want to answer the question and I will push them a bit before I change tack. A lot of people think that this is a stupid question because it doesn't make a difference and that it's meaningless. Apparently that makes it somehow mean that they still have nothing to hide, but anyway.
Sometimes I choose to play the "we don't know the future" aspect. One of my cousins is gay. That is not illegal where we live, but there are places where it is and let's face it, there are no doubt people even in our community that would prefer for it to be illegal. In the future, what if it became illegal? All of a sudden, all of that knowledge of what I didn't need to hide becomes a problem.
Let me reiterate that: right now, my cousin has nothing to hide, and I have nothing to hide in knowing that they're gay. In the future, if it became illegal, there's a ton of information that suddenly we may wish to have been kept secret. Unfortunately, that stuff that we didn't have any reason to hide suddenly becomes something we may wish we were able to.
Finally, depending on who I'm talking to, sometimes I will play the family card. If the person I'm talking to has a child (especially a daughter) I like to ask them how they would feel if in the future that child was dating a good government employee that had access to all sorts of information. Then I begin to wonder, what would happen if they went through a messy break-up? I wonder, what if that ex turns out to be a little jealous and maybe a bit of a stalker? Would you support, then, there being a heap of information on your child, being easily accessible to this ex?
I guess a lot of my arguments come down to this: you don't know the future. In this, you have no idea what you do at the moment that may be considered dodgy behaviour in the future.
Even if you think that that's not a problem, there's a very good chance that right now, you do stuff that you don't want people to know and as far as I'm concerned, the very fact that you haven't considered that means that you haven't spent nearly enough time thinking about it. And to be honest, that frightens me just as much.
When I meet people that say they have nothing to hide, I 'play' one (or more) of three cards.
The first thing I will do is ask them how often they masturbate. I know it's a fairly taboo question, but that's the point. It's not illegal, it doesn't hurt anyone, but not many people want to answer the question and I will push them a bit before I change tack. A lot of people think that this is a stupid question because it doesn't make a difference and that it's meaningless. Apparently that makes it somehow mean that they still have nothing to hide, but anyway.
Sometimes I choose to play the "we don't know the future" aspect. One of my cousins is gay. That is not illegal where we live, but there are places where it is and let's face it, there are no doubt people even in our community that would prefer for it to be illegal. In the future, what if it became illegal? All of a sudden, all of that knowledge of what I didn't need to hide becomes a problem.
Let me reiterate that: right now, my cousin has nothing to hide, and I have nothing to hide in knowing that they're gay. In the future, if it became illegal, there's a ton of information that suddenly we may wish to have been kept secret. Unfortunately, that stuff that we didn't have any reason to hide suddenly becomes something we may wish we were able to.
Finally, depending on who I'm talking to, sometimes I will play the family card. If the person I'm talking to has a child (especially a daughter) I like to ask them how they would feel if in the future that child was dating a good government employee that had access to all sorts of information. Then I begin to wonder, what would happen if they went through a messy break-up? I wonder, what if that ex turns out to be a little jealous and maybe a bit of a stalker? Would you support, then, there being a heap of information on your child, being easily accessible to this ex?
I guess a lot of my arguments come down to this: you don't know the future. In this, you have no idea what you do at the moment that may be considered dodgy behaviour in the future.
Even if you think that that's not a problem, there's a very good chance that right now, you do stuff that you don't want people to know and as far as I'm concerned, the very fact that you haven't considered that means that you haven't spent nearly enough time thinking about it. And to be honest, that frightens me just as much.
Wednesday, February 13, 2013
A Friendly Reminder About Something I'd Already Paid For
A few weeks ago I purchased a product online called Texture Packer Pro. The purchase went completely smoothly and without a hitch.
Imagine my surprise, then, when I received an email about my purchase the other day. To be honest, I was a little concerned.
Opening the email was a pleasant surprise. Simply put, the email was a timely reminder about the fact that I should expect a certain payee on my credit card statement, telling me how much it was for, who the payee was and, especially, what the purchase was for.
Such a simple little thing, at the right moment, and there will no longer even be a second of doubt when I look at my statement. I loved it, and intend to do a similar thing as soon as I start selling online, too.
And for what it's worth, if you're looking for a tool to resize and place images into a texture map (and let's face it, who isn't looking for one?) I can totally recommend it!
Imagine my surprise, then, when I received an email about my purchase the other day. To be honest, I was a little concerned.
Opening the email was a pleasant surprise. Simply put, the email was a timely reminder about the fact that I should expect a certain payee on my credit card statement, telling me how much it was for, who the payee was and, especially, what the purchase was for.
Such a simple little thing, at the right moment, and there will no longer even be a second of doubt when I look at my statement. I loved it, and intend to do a similar thing as soon as I start selling online, too.
And for what it's worth, if you're looking for a tool to resize and place images into a texture map (and let's face it, who isn't looking for one?) I can totally recommend it!
Tuesday, January 29, 2013
How FLOSS Software Became More Easily Accepted at Work
This was a long time ago now, but it was at a time when things were really starting to get big in the place where open source / free software and the Internet collided. It was also the time that we were looking to replace our in-house built, desktop based, bug tracking system.
It wasn't great, but it did the job. Even so, I started looking at a web-based bug tracking system that just happened to be licensed under the GPL.
I understood the requirements that were on us. I knew what it meant for us to use and change the code, and I knew what our obligations were for doing so.
Even still, a few people had concerns and I wasn't able to persuade them that we were OK. I suggested that we run the license past our (very technical) legal guy: if he knew what we intended to do, he would also know what was expected of us. My theory was, if he approves the license for our use case, then who really could argue differently?
The use of the GPLed software was approved and we went on our way. If the story ended there, it wouldn't be much of a story, though.
As time went on, we found ourselves gravitating to software licensed under GPL. But why? Quite simply, we knew two things: (a) legal had approved it; and more importantly (b) because the GPL itself is copyrighted, the very fact that legal had already approved that license meant that we didn't have to worry about fine print anymore!
All of a sudden, we didn't have to wonder "does this randomly drawn license for this particular software package allow us to do what we want?" and we didn't need to get 're-approved' to use the GPL (assuming we stayed within the parameters that we were given to begin with).
It was a weight off of our shoulders: almost overnight, we were confident that our understanding of the license requirements was correct, as we knew exactly what that license represented. We no longer had to try and understand hundreds of lines of legal terminology... we'd already done that once!
As time went on, we got a better understanding of a few more licenses (especially LGPL and MIT) and with that, we were able to make better and faster decisions about the libraries of code that we were looking at using.
It all really came down to the simple fact that because the licenses we were looking at were themselves copyrighted, any project that claimed to be using that license was not going to have some strange 'twist' to the license.
Yes, there were the odd times that the project tried to add a rider to the license, but even then, we were no worse off as legal would have had to look at it anyway.
To cut a long story short, these licenses made our lives easier because we all knew what we were able to do without ever really having to specifically analyse the license, simply because we knew that it couldn't have been modified from what we already understood.
And that made us happy.
It wasn't great, but it did the job. Even so, I started looking at a web-based bug tracking system that just happened to be licensed under the GPL.
I understood the requirements that were on us. I knew what it meant for us to use and change the code, and I knew what our obligations were for doing so.
Even still, a few people had concerns and I wasn't able to persuade them that we were OK. I suggested that we run the license past our (very technical) legal guy: if he knew what we intended to do, he would also know what was expected of us. My theory was, if he approves the license for our use case, then who really could argue differently?
The use of the GPLed software was approved and we went on our way. If the story ended there, it wouldn't be much of a story, though.
As time went on, we found ourselves gravitating to software licensed under GPL. But why? Quite simply, we knew two things: (a) legal had approved it; and more importantly (b) because the GPL itself is copyrighted, the very fact that legal had already approved that license meant that we didn't have to worry about fine print anymore!
All of a sudden, we didn't have to wonder "does this randomly drawn license for this particular software package allow us to do what we want?" and we didn't need to get 're-approved' to use the GPL (assuming we stayed within the parameters that we were given to begin with).
It was a weight off of our shoulders: almost overnight, we were confident that our understanding of the license requirements was correct, as we knew exactly what that license represented. We no longer had to try and understand hundreds of lines of legal terminology... we'd already done that once!
As time went on, we got a better understanding of a few more licenses (especially LGPL and MIT) and with that, we were able to make better and faster decisions about the libraries of code that we were looking at using.
It all really came down to the simple fact that because the licenses we were looking at were themselves copyrighted, any project that claimed to be using that license was not going to have some strange 'twist' to the license.
Yes, there were the odd times that the project tried to add a rider to the license, but even then, we were no worse off as legal would have had to look at it anyway.
To cut a long story short, these licenses made our lives easier because we all knew what we were able to do without ever really having to specifically analyse the license, simply because we knew that it couldn't have been modified from what we already understood.
And that made us happy.
Friday, January 11, 2013
Where's Lad Vampire and Muguito When You Need Them?
Another week, another call from "Windows IT Support" trying to get me to download some malware. I do my best to keep them on the line for a few minutes at least, but all it does is upset me, which leads to me calling them names and then fuming for a bit :)
Artists Against 419 is an awesome site that maintains a database of fake bank sites. Along with actively reaching out to the hosting companies etc., trying to get them to take the sites offline, they also used to have an application called Lad Vampire that was (I believe) replaced by another application called Muguito. These applications participated in what were called "virtual sit-ins". Basically they continually downloaded images and so on from the websites, helping to diminish the available bandwidth to the site and therefore making it less likely that a victim will actually be able to use it.
Some people may call it a DDoS, and to be perfectly honest, I might too. It didn't stop me running it though.
After my last call with "random IT support company" I wondered: at some point in the process of them 'helping' you, there must be something downloaded from somewhere. I'm not talking about the remote access software, but the malware that is intended to run in the background.
So where is the list of the servers that are hosting these files and where's the tool to continually download them? Unfortunately I'll never have the self control to get far enough into one of those calls to find out, but I tell you right now: if there's one use of my fibre-op connection I could get behind, it's downloading that malware as often and as quickly as possible.
For what it's worth, I know that random three letter acronym agencies are trying to deal with these companies, but in the meantime, surely there's something else we can collectively do?
Artists Against 419 is an awesome site that maintains a database of fake bank sites. Along with actively reaching out to the hosting companies etc., trying to get them to take the sites offline, they also used to have an application called Lad Vampire that was (I believe) replaced by another application called Muguito. These applications participated in what were called "virtual sit-ins". Basically they continually downloaded images and so on from the websites, helping to diminish the available bandwidth to the site and therefore making it less likely that a victim will actually be able to use it.
Some people may call it a DDoS, and to be perfectly honest, I might too. It didn't stop me running it though.
After my last call with "random IT support company" I wondered: at some point in the process of them 'helping' you, there must be something downloaded from somewhere. I'm not talking about the remote access software, but the malware that is intended to run in the background.
So where is the list of the servers that are hosting these files and where's the tool to continually download them? Unfortunately I'll never have the self control to get far enough into one of those calls to find out, but I tell you right now: if there's one use of my fibre-op connection I could get behind, it's downloading that malware as often and as quickly as possible.
For what it's worth, I know that random three letter acronym agencies are trying to deal with these companies, but in the meantime, surely there's something else we can collectively do?
Monday, December 17, 2012
On Using Javascript to Block Spam... Or Not
A while into one of my previous positions, I was reviewing some of the code behind our main web site, all hand coded by one of my colleagues.
As I came to the contact us form, I noticed the typical spam blocking technique of asking the user to do a small mathematical equation... you know, the normal "What's 2 + 5?" type question.
Interestingly, the form's submit button was not enabled to begin with. As soon as the equation was answered correctly, however, the button became active and the user could submit the form.
Taking a quick look at the code in the browser, it was obvious that this form was entirely using JS to 'enforce' the spam protection. That is to say, JS was required to enable the submit button, and there was no actual server side validation of the spam protection.
Knowing that this is really not the way to do this at all, I asked the developer about it. His answer was that "of course it works... we haven't been getting spammed, have we?"
The answer reminded me very much of Homer Simpson buying Lisa's tiger repelling rock.
Anyway, I tried to make headway with the developer, trying to discuss bots not usually running JS, and so on, but it made no sense to them... as far as they were concerned, it worked as expected.
I finally realised I was just hitting my head against a brick wall and continued with the rest of my work. It was true, we weren't being spammed, and if worse came to worst I could resolve it quickly myself if ever we began to be spammed.
The developer left us not long after, but it was before another colleague and I discovered Google's Skipfish. Without thinking too critically (though we were sure to make sure we weren't going to try anything destructive!) we set it on the site. Things were going really well, until it found our contact us form... and hit it with 100s of different tests. That was the day that we found out that the contact us form also did something else interesting: it sent the email to about a dozen different staff members. By the time it was stopped, I think they each got around 1000 emails, which was really embarrassing.
But at least, I guess, we weren't attacked by any tigers.
As I came to the contact us form, I noticed the typical spam blocking technique of asking the user to do a small mathematical equation... you know, the normal "What's 2 + 5?" type question.
Interestingly, the form's submit button was not enabled to begin with. As soon as the equation was answered correctly, however, the button became active and the user could submit the form.
Taking a quick look at the code in the browser, it was obvious that this form was entirely using JS to 'enforce' the spam protection. That is to say, JS was required to enable the submit button, and there was no actual server side validation of the spam protection.
Knowing that this is really not the way to do this at all, I asked the developer about it. His answer was that "of course it works... we haven't been getting spammed, have we?"
The answer reminded me very much of Homer Simpson buying Lisa's tiger repelling rock.
Anyway, I tried to make headway with the developer, trying to discuss bots not usually running JS, and so on, but it made no sense to them... as far as they were concerned, it worked as expected.
I finally realised I was just hitting my head against a brick wall and continued with the rest of my work. It was true, we weren't being spammed, and if worse came to worst I could resolve it quickly myself if ever we began to be spammed.
The developer left us not long after, but it was before another colleague and I discovered Google's Skipfish. Without thinking too critically (though we were sure to make sure we weren't going to try anything destructive!) we set it on the site. Things were going really well, until it found our contact us form... and hit it with 100s of different tests. That was the day that we found out that the contact us form also did something else interesting: it sent the email to about a dozen different staff members. By the time it was stopped, I think they each got around 1000 emails, which was really embarrassing.
But at least, I guess, we weren't attacked by any tigers.
Wednesday, December 5, 2012
Medium vs Regular?
"Run Lola Run" is an awesome movie. It's one of very few films I've seen more than once at the cinema. It's definitely the only one I've seen 5 times in one week.
The funny thing is, of all of that movie going, only one thing sticks out in my mind: my "discussion" with the owner of the place.
Firstly, understand that this was not at some mega-chain-cinemaplex... it was a small, privately run place. The owner was known to be quite self righteous and the only being more important than her was her little dog, that used to walk around the place and snarl at anyone it didn't like. Secondly, being in Australia, know that there are a certain number of people that dislike any form of Americanisms.
Anyway, it was the middle of the movie and I was sorely in need of a refreshment. I briskly walked to the counter and asked the person behind it for "a regular Coke". Note that the cashier behind the counter was not the owner... the owner was at the other side of the foyer at this time.
Apparently my request got the owner riled up, as not a split second after I asked for this "regular Coke", she called out (no doubt over the head of the dog but still down her own nose) "Regular? What size is regular? In Australia, we only have small, medium and large... I have no idea what size you are asking for".
A lot of the time I ignore this kind of useless banter. Other times, I bite. This was one of those times.
I looked at her in disbelief and said "firstly, it doesn't matter if you know what size it is, as I wasn't asking you for it. Secondly, regular would be whatever size is most regularly ordered here... I also don't know what size regular is, as I don't know what size is ordered most, but that's the size I want. I want whatever size is most regularly ordered."
The short "fine then" meant that I was free to continue the transaction apparently.
The best part of the whole interaction was the look on the cashier's face as they tried to stop laughing at their boss.
Every now and again I am reminded of this discussion, generally when I hear one of two phrases... both of which are relatively common in the software industry. The first is any reference to the Robustness Principle ("Be conservative in what you send, liberal in what you accept"). The second thing I hear a lot that reminds me of this incident, of course, is "Shut up and take my money".
Anyway, it only slightly soured the night. Thankfully, I had the rest of "Run Lola Run" to make up for it...
The funny thing is, of all of that movie going, only one thing sticks out in my mind: my "discussion" with the owner of the place.
Firstly, understand that this was not at some mega-chain-cinemaplex... it was a small, privately run place. The owner was known to be quite self righteous and the only being more important than her was her little dog, that used to walk around the place and snarl at anyone it didn't like. Secondly, being in Australia, know that there are a certain number of people that dislike any form of Americanisms.
Anyway, it was the middle of the movie and I was sorely in need of a refreshment. I briskly walked to the counter and asked the person behind it for "a regular Coke". Note that the cashier behind the counter was not the owner... the owner was at the other side of the foyer at this time.
Apparently my request got the owner riled up, as not a split second after I asked for this "regular Coke", she called out (no doubt over the head of the dog but still down her own nose) "Regular? What size is regular? In Australia, we only have small, medium and large... I have no idea what size you are asking for".
A lot of the time I ignore this kind of useless banter. Other times, I bite. This was one of those times.
I looked at her in disbelief and said "firstly, it doesn't matter if you know what size it is, as I wasn't asking you for it. Secondly, regular would be whatever size is most regularly ordered here... I also don't know what size regular is, as I don't know what size is ordered most, but that's the size I want. I want whatever size is most regularly ordered."
The short "fine then" meant that I was free to continue the transaction apparently.
The best part of the whole interaction was the look on the cashier's face as they tried to stop laughing at their boss.
Every now and again I am reminded of this discussion, generally when I hear one of two phrases... both of which are relatively common in the software industry. The first is any reference to the Robustness Principle ("Be conservative in what you send, liberal in what you accept"). The second thing I hear a lot that reminds me of this incident, of course, is "Shut up and take my money".
Anyway, it only slightly soured the night. Thankfully, I had the rest of "Run Lola Run" to make up for it...
Subscribe to:
Posts (Atom)