For those who haven't heard of the hierarchy of needs, this was a theory proposed in the 1940's by a social psychologist called Abraham Maslow. This theory revolves around the idea that the behaviour of all human beings revolves around 5 basic innate needs and wants. These are:

1. Physical Needs - Food, Warmth, Shade, Shelter etc
2. Security Needs - Regular food, Warmth, Shade, Shelter etc
3. Social Needs - Being part of a team, Social events etc
4. Recognition Needs - Increased responsilbity and status
5. Self-Fulfilment needs - Personal Interests, Challenges

With some of the stories running in the press recently (NoTW I am looking at you) I started to wonder where about the need for an employer to "appear" moral would fit in the above pyramid.

My initial thoughts are that it would come under the social need, as who would want to be known to work for a company who did some pretty despicable things, I then pondered recognition as that is linked to status, which I am sure Andy Coulson would understand is very linked to the morality of ones employer. And then finally self-fulfilment, could I see myself being fulfilled whilst working for a questionably immoral employer? Personally I don't think I could.

I am not sure I know 100% the answer to the question but I would love to hear from people on what their thoughts are on this post as I am sure there are sides to the argument I haven't thought of or idea's that I haven't considered. I have seen some interpretations of Maslow's hierarchy include Morality under security and self-fulfilment, which would tend to back up my points above.

If you want to read more on this theory, the wikipedia page Maslow's hierarchy of needs is a great place to start.

Disclaimer: This isn't meant to be an attack on anyone who works for NoTW or any other company you may consider morally questionable, its aimed to provoke debate and discussion only. I also understand that there is criticism of Maslow's hierarchy of needs, this post doesn't aim to debate the validity of that theory, only that the theory exists.

Well now a challenge has been set by Russell Smithers to blog more frequently and this is a challenge I have accepted, so prepare for a number of posts over the next few weeks/months on a wide range of topics.

I'll be writing the first of my real blog posts in the coming days and if anyone has some suggestions on what they would like to see blogged about post them here and I'll be sure to work through them.

I had been pondering PHP 5.3 for quite some time towards the end of 2009 and into 2010, and had some discussions with various open source developers during the Jump in Camp about what their plans were for PHP 5.3. We had also been discussing it internally as if we were going to adopt PHP 5.3 then it was best to take full advantage of the new namespace support to ensure the core framework was a clean as possible.

In May 2010 I released a blog post titled PHP 5.3 and the Future to try and gauge wider community feedback on the issue.

This kicked off a number of discussions in the background about where PHP was heading and how we could best make osCommerce 3.x as flexible as possible to ensure that we wouldn't need to rebuild from the ground up in a few years. From talking with many people from other open source projects who were also all heading towards the PHP 5.3 minimum requirement it seemed like a no-brainer. Once we'd had chance to experiment with the namespace implementation and could see just how much better our framework would be, developing it for PHP 5.2 seemed to be a very costly mistake.

We didn't just jump into this blindly however, we fully understood what this meant in terms of osCommerce 3.x adoption and even last year there were fewer hosting companies offering PHP 5.3 support than there are now. The outcome from all of that was that we decided to jump to PHP 5.3 as the benefits far outweighed the negatives.

One of the concerns that many people from the community have expressed is the lack of support from hosting companies for PHP 5.3, I personally don't think that its a problem when you take the bigger picture into account.

As late as December last year it was announced that PHP 5.2 had reached End of Life, I had been expecting this to happen at some point so it didn't come as a huge surprise. What this means is that PHP 5.2 would no longer be receiving bug fixes (PHP 5.2.17 aside) or security updates. This would then leave hosting companies with a choice, if they would like to maintain the ability to offer PCI compliant hosting then they would need to move to a version of PHP which was supported and updated. This all pointed to the likelihood that adoption of PHP 5.3 for hosting companies would gain traction at an increasingly rapid pace, of course this wasn't guaranteed so there was an element of risk involved.

It should also be fairly obvious that the landscape of the hosting industry has changed, back in 2001 pretty much everyone was using shared hosting packages as these were really the only ones that were affordable to the average user. The hosting companies were in full control and moving to a different hosting company was incredibly painful experience.

These days however the situation is radically different, Cloud / VPS hosting solutions are now widely available and at a price point that makes them affordable even for the newest of start up online stores. Both of which are able to support PHP 5.3 without affecting any other user. It is now also much easier for hosting companies to offer multiple versions of PHP on the same server, something which also wasn't as easy (if not almost impossible) back in 2001.

The end result is the user now has a lot of choice on where to host their sites and moving between hosts is now incredibly simple (compared to 2001). In my personal opinion any hosting company that won't offer PHP 5.3 will simply get left behind. Users can now vote with their feet and move to a company that offers them the solution they need which has reversed who is really in the driving seat when it comes to adoption.

The reverse option would of been to hold back the whole community just so that we could support users who were using hosting companies that would end up in the minority, I personally think this is a very short-sighted view when you are trying to build a framework that will still hopefully be in use in another 11 years.

Now that the core framework is out there and is being actively worked on, I think deciding to use PHP 5.3 as the minimum requirement was one of the best decisions we ever made.

The reason we have decided to include the need to sign a CLA before contributions can be accepted into the osCommerce core is to ensure that the entire community can understand the terms under which the code has been contributed to the project. Signing the CLA does not change your rights to use your own Contributions for any other purpose and only applies to contributions which you want to be considered for inclusion into the "Core" framework. The CLA does not apply to community addons which are available from the addons site.

The CLA is here to protect the rights of the community to ensure that any code which is included as part of the core framework is removed from licensing and patent restrictions.

Q. Do I need to agree to the CLA to create community addons for osCommerce 3.x?

A. No. Only contributions which you would like to be considered for the "core" framework require the CLA to be agreed to.

Q. If my contribution is accepted in the "core" framework for osCommerce 3.x what does that mean?

A. By agreeing to the CLA you are providing osCommerce a non-exclusive, irrevocable, worldwide, royalty-free, sublicenseable, transferable license.

Q. What does this license allow?

A. You agree to let osCommerce use, copy, prepare derivative works of, distribute and publicly perform and display on any licensing terms your contribution to the "core" framework, including without limitation be distributed under open source licenses such the the GNU General Public License / BSD License and also be included in binary only / commercial projects.

Q. Can I still use any contribution I make to the osCommerce "core" framework for other projects?

A. Yes. Agreeing to the CLA and allowing your code to be considered for inclusion into the "core" framework in no way affects your rights to use your code in any way you like.

In summary, the CLA is there to help make clear that any contributions included into the "core" framework are without restriction and that anyone contributing code has the right to do so, free of any restriction such as intellectual property or patent and that they have the legal right to agree to the terms of the CLA.

I am sure that we can encourage developers to contribute to the framework and by doing so, they can help us make it as stable and as flexible as possible. Now is the time for community to embrace open source and get involved.

Q. Is the BSD License used compatible with the GNU General Public License?

A. Yes. osCommerce 3.x is using the 3-clause BSD license which is fully compatible with the GNU General Public License

Q. What versions of osCommerce does the BSD License apply to?

A. The BSD License is for osCommerce 3.x

Q. Will osCommerce 2.x still be released under the GNU General Public License?

A. Yes, the license for osCommerce 2.x has not changed

Q. What licence are addons for osCommerce 3.x released under?

A. Developers now have the choice to release their Add-Ons under a BSD compatible license or to continue using the GNU General Public License.

Q. Why change the license?

A. The GNU General Public License has served us well for 11 years however we want to create even more business possibilities and opportunities for store owners, developers, and our community. Changing the licence helps make that possible.

Q. Where can I find out more information about the different licenses?

A. There is an interesting article / video from Redhat which is worth watching http://www.redhat.com/magazine/006apr05/features/licenses/. You can also check out Wikipedia

Q. Can I use osCommerce 3.x addons released under the GNU General Public License in a binary only/commercial closed source project?

A. No. The change to using the BSD licence for the core of osCommerce doesn't remove the viral part of the GNU General Public License. If you use GNU General Public Licensed code in your project the source code must be provided to comply with the licence. If you want to release a project as binary only then you will not be able to use any addons that are licenced under the GNU General Public License

Q. Can developers release osCommerce 3.x under the GNU General Public License?

A. No. Only osCommerce has the right to change the licensing of the core framework

Q. Can developers include BSD licensed community addons in projects released under the GNU General Public License?

A. Yes. However including BSD code in a GNU General Public License project doesn't change the licence of the code. It is still released under the BSD license.

This has worked pretty well for me but now I want to automate this with a cron job. Since I am human and quite forgetful automating this will ensure that I never again fail to complete a backup of some critical data. A problem now arises however in how to secure this. I don't want to have to store a password or passphrase in the crontab so I need to find a new way.

After some googling and some experimentation I finally have working something which I am sure real sysadmins have been aware of for a while and that is the ability to restrict a ssh via public/private key to specific commands.

In order to get this to work for myself I did the following steps.

Firstly I created a new passphrase-less rsa key

ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa): /home/username/.ssh/rsync_backup
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/username/.ssh/rsync_backup.
Your public key has been saved in /home/username/.ssh/rsync_backup.pub.

Next I copied the public key to the server I wanted to backup and added this to the /home/remoteuser/.ssh/authorized_keys

I then tested to make sure that I could login to the remote server using ssh

ssh -i /home/username/.ssh/rsync_backup remoteuser@remotehost

Once I had confirmed that it was all working with ssh the next step was to lock down the key to a specific IP address. Since this was an internal system I knew which IP address would be used to connect.

I edited the /home/remoteuser/.ssh/authorized_keys to look like the following

from="192.168.1.50" ssh-rsa <public key>

I then tested ssh login again to make sure I didn't mess anything up

ssh -i /home/username/.ssh/rsync_backup remoteuser@remotehost

All was still working. To play devils advocate I also tried to connect from a different IP and my connection was denied successfully.

Next up I needed to check on connection what command was being called to ensure that only rsync was permitted. After a lot of googling around I came across the following

#!/bin/sh
logfile=/var/log/rsync_backup.log
case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo `date` "- SSH connection rejected" >> $logfile
;;
*\(*)
echo `date` "- SSH connection rejected" >> $logfile
;;
*\{*)
echo `date` "- SSH connection rejected" >> $logfile
;;
*\;*)
echo `date` "- SSH connection rejected" >> $logfile
;;
*\<*)
echo `date` "- SSH connection rejected" >> $logfile
;;
*\`*)
echo `date` "- SSH connection rejected" >> $logfile
;;
*\|*)
echo `date` "- SSH connection rejected" >> $logfile
;;
rsync\ --server*)
{
  echo `date` "- SSH connection accepted" >> $logfile
  $SSH_ORIGINAL_COMMAND
}
;;
*)
echo `date` "- SSH connection rejected" >> $logfile
;;
esac

This script checks the SSH command to be run and rejects it if anything other than rsync is being used.

I once again edited the /home/remoteuser/.ssh/authorized_keys to add the command restriction so that it looked like the following

from="192.168.1.50", command="/home/remoteuser/check-rsync.sh" ssh-rsa <public key>

I then checked to see what would happen if I tried to login using ssh

ssh -i /home/username/.ssh/rsync_backup remoteuser@remotehost

This time my connection was denied. Checking my logfile I saw the following entry

Sun Feb 6 00:51:58 GMT 2011 - SSH connection rejected

Now the final test... could I use the public/private key to connect and run the rsync command. To do this I ran

rsync -avz -e "ssh -i /home/username/.ssh/rsync_backup" remoteuser@remotehost:/home/remoteuser/datatobackup /home/username/remotehost

And low and behold my files were downloaded

receiving file list ... done
home/remoteuser/datatobackup/file1.txt
home/remoteuser/datatobackup/file7.txt
sent 98 bytes  received 2331 bytes  1619.33 bytes/sec
total size is 32663  speedup is 13.45

Job Completed. I can now just set this up as a cron job to run every night and sleep easy in the knowledge that all the valuable data I need is safely backed up.

I would love to hear any feedback on this solution and any tips for ways to improve it.

We all know about the EXPLAIN statement which looks something like

EXPLAIN SELECT statement

This is used to display information from the optimizer about the query execution plan for the SELECT statement including information about how tables are joined and in which order. There is also a second statement called EXPLAIN EXTENDED which will provide even more information about what is happening under the hood.

To show you how this works lets take a common query from the current development version of osCommerce 3.x

EXPLAIN SELECT DISTINCT p.products_id FROM osc_products p LEFT JOIN osc_product_attributes pa ON (p.products_id = pa.products_id) LEFT JOIN osc_templates_boxes tb ON (pa.id = tb.id AND tb.code = "Manufacturers"), osc_products_description pd, osc_categories c, osc_products_to_categories p2c WHERE p.products_status = 1 AND p.products_id = pd.products_id AND pd.language_id = 1 AND p.products_id = p2c.products_id AND p2c.categories_id = c.categories_id

You should get some output similar to the following

As you can see there is lots of useful information there to help understand just what is happening. But it seems the MySQL engine has some other things it would like to tell us. And we get to this extra information by adding the keyword EXTENDED in combination with SHOW WARNINGS to get information about how query looks after transformation as well as any other notes the optimizer may wish to tell us.

Here is the same query again but with some extra information.

You can see there is a new column here called "filtered" which was added in MySQL 5.1.12. This column indicates an estimated percentage of table rows that will be filtered by the table condition.

Now if we send the statement

SHOW WARNINGS \G

We get back exactly how the optimizer has changed the query internally before executing it

SELECT DISTINCT `osc3`.`p`.`products_id` AS `products_id` FROM `osc3`.`osc_products` `p` LEFT JOIN `osc3`.`osc_product_attributes` `pa` ON(((`osc3`.`p`.`products_id` = `osc3`.`pd`.`products_id`) AND (`osc3`.`p2c`.`products_id` = `osc3`.`pd`.`products_id`) AND (`osc3`.`pa`.`products_id` = `osc3`.`pd`.`products_id`))) LEFT JOIN `osc3`.`osc_templates_boxes` `tb` ON(((`osc3`.`tb`.`code` = 'Manufacturers') AND (`osc3`.`tb`.`id` = `osc3`.`pa`.`id`))) JOIN `osc3`.`osc_products_description` `pd` JOIN `osc3`.`osc_categories` `c` JOIN `osc3`.`osc_products_to_categories` `p2c` WHERE ((`osc3`.`c`.`categories_id` = `osc3`.`p2c`.`categories_id`) AND (`osc3`.`pd`.`language_id` = 1) AND (`osc3`.`p`.`products_id` = `osc3`.`pd`.`products_id`) AND (`osc3`.`p2c`.`products_id` = `osc3`.`pd`.`products_id`) AND (`osc3`.`p`.`products_status` = 1))

As you can see to get better performance the engine has changed the order of the query. Using this knowledge it is possible for the developer to understand exactly what the optimizer is doing and possibly find a better way to achieve the same results.

Hopefully I am not the only person in the world who didn't know about this EXTENDED option

The first subject of the day was about porting SQL queries to T-SQL. Transact-SQL (T-SQL) is Microsoft's proprietary extension to SQL and varies from the MySQL version of SQL which most open-source developers are used too. For information on the differences and how to convert MySQL to T-SQL Microsoft provide a useful guide to the most common issues developers will have.

The guide can be downloaded from Guide to Migrating from MySQL to SQL Server 2008 and as you can see there are a number of differences which need to be dealt with by the developer. Most issues can be dealt with and both MySQL and SQL Server can supported simultaneously as long as you are prepared to think about how you write SQL statements to take into account the nuances.

The other path open to developers is to use something like Doctrine which will abstract these differences away from the developer. SQL Server support has also added to Kohana and Lithium so if you use either of those frameworks, adding support for SQL Server should be much less painful.

Just as I started to have panic attacks on just how much work it would be to get osCommerce to support SQL Server we were shown a rather cool tool which is called SSMA (SQL Server Migration Assistant) this will take a MySQL database and will convert its data structure and also the data itself and convert it into a format which SQL Server can understand. I tested this tool against the current 3.x database structure which is being used in osCommerce and because we hadn't really ever supported any databases apart from MySQL I expected a number of problems to be flagged up.

After running through the tool however I was pleased to see that the entire database structure and all of the sample data we provide was migrated to SQL-Server without a hitch. Kudos to the SSMA team for their hard work in making it light work for me! If you are interested in trying SSMA out you can download it from Microsoft SQL Server Migration Assistant 2008 for MySQL

Much discussion was had about how we could best support SQL Server in osCommerce and we decided that a new database layer was needed in order to provide support for all of the database platforms osCommerce will officially support and also allow extensibility so that other platforms could be added without needing to re-write large swathes of core code. This work is currently in progress and hopefully the fruits of our labour can be commited into the Github repository soon.

After lunch came a series of interesting talks about how to optimize the T-SQL once you had it working. Most developers these days should know about the EXPLAIN statement in MySQL which will tell you how MySQL has executed a SELECT statement.

SQL Server has a pretty awesome graphical tool which is built into SQL Server Management Studio and provides more than enough information to help developers decipher what is going on under the hood to get the best performance possible out of SQL Server. I won't go into details on how to use the tools I have mentioned here but I will be blogging on the specifically very soon with a walkthrough on converting the current osCommerce 3.x database structure to SQL Server and also how to use the SQL Server Management Studio to discover how your T-SQL is performing.

After an afternoon of discussions and talks about optimisation of T-SQL and SQL Server in general it was time for us to prepare for a small road-trip to the Space Needle where we would be spending the evening having dinner and relaxing.

I think this had to be one of the highlights of the whole Jump in Camp as the view from the top was amazing. The food and drink in the Restaurant was also top notch and I couldn't find fault with anything apart from the Wind which was absolutely freezing when out on the observation deck.

The evening was rounded off with some good debates in the hotel bar over a few beers about what interoperability really meant and of course some really good banter about getting a Linux version of the SQL Server driver

Was then time for sleep in readiness for the fun of Day 4

The first job for the day after breakfast was to head over to the Microsoft Redmond Campus. A short coach ride and we were there. I knew the campus was big but didn't quite realise how big until I was really there.

Our first visit stop was Building 35 which is the home of SQL Server. After getting our security passes organised we where escorted upstairs to a meeting room where we were introduced to Ted Kummert and Nigel Ellis where an overview of the history of SQL Server was given and also some discussion on SQL Azure.

The expected question soon arose of which I am sure Microsoft cringe every time someone asks it... that being "When we there be a *nix version of the SQL Server driver". Much discussion ensued about the many reasons the open-source community and some commercial companies wanted it and the many reasons it wasn't yet available from Microsoft. I sure hope that one day it will be available but at this moment in time I am not expecting anything to be done about it in the near future.

Next up we had chance to visit the Microsoft store and Visitor centre where we had chance to see some of the many different products that Microsoft had worked on over the years. After some shopping and playing with the Xbox Kinect it was time to head back to the hotel for some lunch before the afternoons activities.

The first thing up in the afternoon was a surprise session of which we knew nothing about and involved us drumming and playing various musical instruments (and there was even some bad dancing at one-point). The idea being to try and get us to understand how different people working together can contribute in harmony even if the tools they are using are different.

Once we were all loosened up it was time to crack out the laptops and get down to the real reason we were all there... coding!

The first challenge was to work on porting our various applications to either the sqlsvr (native) driver or the PDO_SQLSVR driver. This was obviously easier for some projects that had abstraction layers in place already, for osCommerce however it was a little bit more work so progress was initially slow.

Towards the end of the afternoon my brain was getting pretty fried, the timezone difference and lack of sleep was certainly taking its toll. Luckily the evening didn't involve going too far with dinner in the hotel and then some xbox gaming.

Along with the rest of the hotel the food they provided was always excellent, I think I ate so much I thought I was going to burst! It certainly hindered me for the next part of the evening. Harald's flight had also landed by this time so I was looking forward to catching up with him again.

So onto the gamimg... In one of the various rooms in the hotel had been setup 2 Xbox 360's with the new Kinect controller after spending some time getting them calibrated the game was on.. the winner was to get to take a Kinect controller home with them. As you can imagine there was some frantic competition and the game settled on was "Ralleyball" which is part of the Kinect adventures game.

This short video should give you an idea of the game

After numerous rounds and also numerous beers the eventual winner was none other than Paul Reinheimer by the end I was totally exhausted from all the cheering (and drinking) so it was time to hit the hay and try and do a better job and sleeping through the timezone differences, ready for what was install for Day 3!