People like using the word "done" and with good reason - if something is "done" it means we no longer have to worry about it and we can get on with the next task - its a decisive and positive word. But this word is usually misleading and in the real world, software development tasks are very rarely "done".
To say that a software development task is "done" tends to send a message to the project manager and client that the specific user story or feature will never need any revision and is completely free from errors. This is almost certainly not the case. It's a good assumption that new code will contain errors, even if you haven't found them yet and you can be pretty sure that when the new feature is demonstrated to the client or as soon as real people start to use the system, changes will be required.
And at the engineering level, well organized code will generally adhere to the DRY (dont repeat yourself) principal. This means that engineers will try to 'abstract' code, functions and structures to be as general as possible so that they are used as often as possible. This implies that although a feature may appear unchanged from one version to the next, the engineers really know that actually a lot of the underlying code may well have changed and although unit testing can 'positively' test and confirm consistent behaviour, a good dose of manual testing will do no harm.
But this article is not about code organization or testing techniques, it is about why we should try not to use the word "done". When we say this word, we probably really mean: "done to the point where another task is of higher priority". If this definition of "done" is generally accepted then project managers and clients can continue as usual, but should not be alarmed when something marked as "done" contains bugs, needs changing or the code requires rework.
Wednesday, March 21, 2012
The Internet - and what it could be
The Internet is still basically a mass of insular websites that do not interact with each other. Websites are still primerily designed to be read/browsed by a human, regardless of whether it is done from a PC, mobile or tablet device. Most websites are still just like books in a library which do not communicate and share information with one another.
Websites have got prettier and include more media, like music, videos and animations but essentially, websites have not changed and the Internet as a whole as not significantly changed for many years.
But the Internet has an amazing potential, behind all of these insular websites are computers capable of complex data processing and hi-speed communications, but for the most part, this potential power is just used to serve up pre-defined static content.
I think it is now time to build the REAL "web 2.0". For me, this implies websites that communicate and share information automatically with other websites and services in (pseudo) real-time. The end goal of which is to turn the Internet into something really powerful - a single massively parallel computer. The hard part has been done for us with technologies like HTTP, DNS, TCP/IP, JSON etc already defined and well supported by tried and tested technologies - all the building blocks of a new Internet are in place.
But to create this new Internet, we must create a mechanism such that all data currently formatted and published for humans (the majority) is also available to other web services. Consider my own personal website:
http://tomcarnell.com
If you visit this website from a standard web browser, you would be presented with a mixture of graphics, text and hyperlinks organised to be human readable and formatted to be attractive (like any other website). Unfortunately, this type of content is completely unsuitable for a third party web service to make use of, let alone use in a meaningful way. So, lets consider the following two proposed URL's:
http://tomcarnell.com/services.json
http://tomcarnell.com/data.json
Now, a third party web service can read the contents of 'services.json' and can learn that the 'tomcarnell.com' site has a service called 'contact_me', a programatic version of the 'contact form' (http://tomcarnell.com/contact_me) which allows the service to submit a contact request. The services.json file will also detail how the 'contact_me' service should be called, what type of data it returns and maybe some human readable text to explain what the service does and how it should be used. We could also implemenent a 'subscribeToService_xxx' service which means that before allowing just anything to use the 'contact_me' service, one must first subscribe. The rules and details of how these services work are completely up to the website owner.
Our third party web service could also decide to read the contents of 'data.json' which contains semantically structured data. This data might contain the details of my portfolio, my curriculum, my contact details, links to social profile pages and any of the text found on the human readable website. It could also detail daily contractor rates, availability, location and phone number, which a third party recruitment service might find useful when searching for an available web developer in the Madrid area with experience with jQuery for example. Again, the contents of 'data.json' are completely up to the website owner.
Online shops could employ the same standard to publish special offers and could provide services to allow 3rd application to check an order status or product stock levels.
A news website could publish their latest headlines in 'data.json' and maybe provide a search service in 'services.json' which could return a list of news articles that fulfill the search criteria.
But the fun really starts when we start to combine these services. We could have language translation services, hotel reservation services, transport reservation systems, services that interact with social networks, services to access and query government and official data and all this opens us up to the possibility of creating sophisticated 'problem specific' services, for example a service that knows how to plan an entire wedding and knows which other services it needs to locate available venues, check guests calendars, send and recieve invitations, book a chauffeur driven car, obtain estimates for food, gift lists etc...
It would be really great to move in this direction, but it will take the support of lots of people. However, as a stepping stone, website owners can start by using http://livedirectory.org/ - a service which provides the functionality to publish and manage an arbitrary set of data and allow 3rd party applications to subscribe to this data, which means that LiveDirectory will send change notifications to all subscribers when this data is updated. For example, I have implemented the equivilent of the 'data.json' file as a LiveDirectory profile:
http://livedirectory.org/profile/tom-carnell
The above LiveDirectory profile details information about me and allows anybody to 'subscribe' to any part of that data structure. Another example is the jQuery profile:
http://livedirectory.org/profile/jquery
Where services can subscribe to be notified whenever a new version of jQuery is released, which is specifically located here:
http://livedirectory.org/profile/jquery/latest_version
The LiveDirectory.org aims to provide a simply but very powerful API based data manangement, retrieval and subscription system to help the Internet take one step toward becoming a more useful tool for everyone.
Websites have got prettier and include more media, like music, videos and animations but essentially, websites have not changed and the Internet as a whole as not significantly changed for many years.
But the Internet has an amazing potential, behind all of these insular websites are computers capable of complex data processing and hi-speed communications, but for the most part, this potential power is just used to serve up pre-defined static content.
I think it is now time to build the REAL "web 2.0". For me, this implies websites that communicate and share information automatically with other websites and services in (pseudo) real-time. The end goal of which is to turn the Internet into something really powerful - a single massively parallel computer. The hard part has been done for us with technologies like HTTP, DNS, TCP/IP, JSON etc already defined and well supported by tried and tested technologies - all the building blocks of a new Internet are in place.
But to create this new Internet, we must create a mechanism such that all data currently formatted and published for humans (the majority) is also available to other web services. Consider my own personal website:
http://tomcarnell.com
If you visit this website from a standard web browser, you would be presented with a mixture of graphics, text and hyperlinks organised to be human readable and formatted to be attractive (like any other website). Unfortunately, this type of content is completely unsuitable for a third party web service to make use of, let alone use in a meaningful way. So, lets consider the following two proposed URL's:
http://tomcarnell.com/services.json
http://tomcarnell.com/data.json
Now, a third party web service can read the contents of 'services.json' and can learn that the 'tomcarnell.com' site has a service called 'contact_me', a programatic version of the 'contact form' (http://tomcarnell.com/contact_me) which allows the service to submit a contact request. The services.json file will also detail how the 'contact_me' service should be called, what type of data it returns and maybe some human readable text to explain what the service does and how it should be used. We could also implemenent a 'subscribeToService_xxx' service which means that before allowing just anything to use the 'contact_me' service, one must first subscribe. The rules and details of how these services work are completely up to the website owner.
Our third party web service could also decide to read the contents of 'data.json' which contains semantically structured data. This data might contain the details of my portfolio, my curriculum, my contact details, links to social profile pages and any of the text found on the human readable website. It could also detail daily contractor rates, availability, location and phone number, which a third party recruitment service might find useful when searching for an available web developer in the Madrid area with experience with jQuery for example. Again, the contents of 'data.json' are completely up to the website owner.
Online shops could employ the same standard to publish special offers and could provide services to allow 3rd application to check an order status or product stock levels.
A news website could publish their latest headlines in 'data.json' and maybe provide a search service in 'services.json' which could return a list of news articles that fulfill the search criteria.
But the fun really starts when we start to combine these services. We could have language translation services, hotel reservation services, transport reservation systems, services that interact with social networks, services to access and query government and official data and all this opens us up to the possibility of creating sophisticated 'problem specific' services, for example a service that knows how to plan an entire wedding and knows which other services it needs to locate available venues, check guests calendars, send and recieve invitations, book a chauffeur driven car, obtain estimates for food, gift lists etc...
It would be really great to move in this direction, but it will take the support of lots of people. However, as a stepping stone, website owners can start by using http://livedirectory.org/ - a service which provides the functionality to publish and manage an arbitrary set of data and allow 3rd party applications to subscribe to this data, which means that LiveDirectory will send change notifications to all subscribers when this data is updated. For example, I have implemented the equivilent of the 'data.json' file as a LiveDirectory profile:
http://livedirectory.org/profile/tom-carnell
The above LiveDirectory profile details information about me and allows anybody to 'subscribe' to any part of that data structure. Another example is the jQuery profile:
http://livedirectory.org/profile/jquery
Where services can subscribe to be notified whenever a new version of jQuery is released, which is specifically located here:
http://livedirectory.org/profile/jquery/latest_version
The LiveDirectory.org aims to provide a simply but very powerful API based data manangement, retrieval and subscription system to help the Internet take one step toward becoming a more useful tool for everyone.
Wednesday, September 14, 2011
Why database column names should be globally unique
After years of working with relational databases (RDBMS's), I have just started to realise the importance of globally unique names for database columns. Generally I use the term 'globally unique' to mean unique within the database schema, but I it could potentially extend to being unique across all schemas.
The problem that arises from using repeating column names is when two or more tables are used in a single query. For example, if we have two tables: 'customers' (which has a column 'id') and 'customer_orders' (which also has the column 'id') when we perform the following query:
"SELECT * FROM customers c, customer_orders o WHERE c.name = 'tom';"
The "*" clause means we will return every column of both tables. So considering a single row returned from the query, what would be the value of 'id'? Would it be the 'customers.id' field or the 'customer_orders.id' field? What if our application needed to use both values? Of course, we could change our query and give aliases to the columns, which would also mean we would need to explicitly list the columns we wish to be returned:
"SELECT c.id customer_id, o.id order_id, c.name, o.value FROM customers c, customer_orders o WHERE c.name = 'tom';"
So we can now reference either the customers.id value and the order.id value. But this solution makes our query fragile and susceptible to schema changes - what if we later rename the column 'c.name' to 'c.first_name'? The query would then break. In addition, and for the purposes of consistancy of any 'data interface' we might define, we would be forced to use these explict column aliases in all our queries, which would reduce the readability of the SQL and introduce the same problems associated with any form of code duplication.
In reality, the first SQL query would be perfectly good had we used unique column names. For example, 'customers.cst_id' instead of 'customer.id' and 'customer_orders.cso_id' instead of 'customer_orders.id'. Here I have chosen a three character 'prefix' for column names that gives some clue to the name of the containing table. Therefore 'customers.name' would also become 'customers.cst_name'.
Implementing a simple column naming convention across the entire database schema will greatly improve the effectiveness of SQL queries and will reduce the likelihood of application errors that may occur by accidentally referencing the wrong data within an application.
The problem that arises from using repeating column names is when two or more tables are used in a single query. For example, if we have two tables: 'customers' (which has a column 'id') and 'customer_orders' (which also has the column 'id') when we perform the following query:
"SELECT * FROM customers c, customer_orders o WHERE c.name = 'tom';"
The "*" clause means we will return every column of both tables. So considering a single row returned from the query, what would be the value of 'id'? Would it be the 'customers.id' field or the 'customer_orders.id' field? What if our application needed to use both values? Of course, we could change our query and give aliases to the columns, which would also mean we would need to explicitly list the columns we wish to be returned:
"SELECT c.id customer_id, o.id order_id, c.name, o.value FROM customers c, customer_orders o WHERE c.name = 'tom';"
So we can now reference either the customers.id value and the order.id value. But this solution makes our query fragile and susceptible to schema changes - what if we later rename the column 'c.name' to 'c.first_name'? The query would then break. In addition, and for the purposes of consistancy of any 'data interface' we might define, we would be forced to use these explict column aliases in all our queries, which would reduce the readability of the SQL and introduce the same problems associated with any form of code duplication.
In reality, the first SQL query would be perfectly good had we used unique column names. For example, 'customers.cst_id' instead of 'customer.id' and 'customer_orders.cso_id' instead of 'customer_orders.id'. Here I have chosen a three character 'prefix' for column names that gives some clue to the name of the containing table. Therefore 'customers.name' would also become 'customers.cst_name'.
Implementing a simple column naming convention across the entire database schema will greatly improve the effectiveness of SQL queries and will reduce the likelihood of application errors that may occur by accidentally referencing the wrong data within an application.
Friday, July 29, 2011
Delivering software projects: Drupal, Joomla or just go bespoke?
Imagine you are building a house and the time has come to install the electrics. Which of the following do you do:
Importantly, the second option works. An expert is hired to use their skills, experience and tools knowledge to deliver exactly what a client needs - nothing more and nothing less. This is the general approach taken not just in small-scale or private engineering, but also in large scale commercial engineering projects. And this is also the reason why 'off-the-shelf', 'one-size-fits-all' systems are simply not available for purchase - nobody builds them because nobody would buy them.
So why is the world of software engineering projects so obsessed with using off-the-shelf one-size-fits-all software like Drupal and Joomla which almost certainly do not do what the client wants out of the box and will almost certainly need extensive customization and tailoring? Why not follow the rest of the engineering world and deliver client software by using expert engineers to build tailored applications that do exactly what is required using the most appropriate tools and techniques available?
I believe there are three reasons to explain the software engineering obsession with using 'off-the-shelf' systems:
It is unlikely that designing and building an RDBMS from scratch would be a necessity for a given software project, therefore it is likely to be a good idea to use an off-the-shelf solution for this type of specific system component. The same goes for other system components like computer languages, operating systems, communication protocols etc. However, depending on client needs, the design and implementation of a new type of RDBMS maybe exactly what is required. This is to say that even with a bespoke project a vast amount of functionality can be provided by pre-built technologies. However, if a key requirement can not be serviced by pre-built technology then it will have to be built from scratch - drupal or no drupal.
There are many great and often free tools that software engineers can use to build high-quality applications quickly and effectively. An experienced and expert software engineer should be able to choose which of these tools and technologies best fits a clients needs and deliver a bespoke system faster than using a 'one-size-fits-all' product.
I wonder if as the software engineering industry matures that we will see fewer 'off-the-shelf' 'one-size-fits-all' packages and move toward boutique software consultancies that delier high-quality solutions quickly and cost effectively. And more importantly that the software engineering industry follows in the foot steps of far more mature engineering industries.
- Buy an 'off-the-shelf', 'one-size-fits-all' complete home eletrical system and then hire an electrician to customize and tailor the system to fit your needs, or
- Hire an electrician straight-off to build the electrical system for you?
Importantly, the second option works. An expert is hired to use their skills, experience and tools knowledge to deliver exactly what a client needs - nothing more and nothing less. This is the general approach taken not just in small-scale or private engineering, but also in large scale commercial engineering projects. And this is also the reason why 'off-the-shelf', 'one-size-fits-all' systems are simply not available for purchase - nobody builds them because nobody would buy them.
So why is the world of software engineering projects so obsessed with using off-the-shelf one-size-fits-all software like Drupal and Joomla which almost certainly do not do what the client wants out of the box and will almost certainly need extensive customization and tailoring? Why not follow the rest of the engineering world and deliver client software by using expert engineers to build tailored applications that do exactly what is required using the most appropriate tools and techniques available?
I believe there are three reasons to explain the software engineering obsession with using 'off-the-shelf' systems:
- Building systems like Drupal and Joomla is actually pretty easy, which explains why there are hundreds, if not thousands of these types of 'one-size-fits-all' systems on the market. And the sheer number of these systesm gives an overall impression that we should be using them.
- Self preservation - if a software project fails, it is difficult to blame the guy that chose to use Drupal, whereas it is probably much easier to blame the guy that chose to build a bespoke system.
- In a few cases, for simple systems like very basic websites an 'off-the-shelf' system can help speed things along if very minimal customization is required - which of course is simply not the majority of high-value business systems.
It is unlikely that designing and building an RDBMS from scratch would be a necessity for a given software project, therefore it is likely to be a good idea to use an off-the-shelf solution for this type of specific system component. The same goes for other system components like computer languages, operating systems, communication protocols etc. However, depending on client needs, the design and implementation of a new type of RDBMS maybe exactly what is required. This is to say that even with a bespoke project a vast amount of functionality can be provided by pre-built technologies. However, if a key requirement can not be serviced by pre-built technology then it will have to be built from scratch - drupal or no drupal.
There are many great and often free tools that software engineers can use to build high-quality applications quickly and effectively. An experienced and expert software engineer should be able to choose which of these tools and technologies best fits a clients needs and deliver a bespoke system faster than using a 'one-size-fits-all' product.
I wonder if as the software engineering industry matures that we will see fewer 'off-the-shelf' 'one-size-fits-all' packages and move toward boutique software consultancies that delier high-quality solutions quickly and cost effectively. And more importantly that the software engineering industry follows in the foot steps of far more mature engineering industries.
Monday, July 25, 2011
Web usability and HTML: Uploading files and photos
More and more people use online document and photo sharing websites (dropbox, picasa, facebook etc). However, I am still amazed at the number of websites that require files to be uploaded and that do not offer a way to link to an online resource.
A classic example is adding a photo to an online profile or attaching a document to an application form. It would be really helpful to be able to either upload a file, or simply enter a URL to an online file. This is something applications developers could easily implement.
However, the ideal solution would be to change the HTML5 upload file selector to allow the selection of either a disk-based resource/file or a publically available web-based resource (via a url). This change would immediately make ALL web applications that perform file uploading more flexible with zero effort required from developers.
An obvious progression from this would be to allow browsers to access secured resources (private picasa galleries or files on ftp sites etc) - of course, only with the owners permission and with appropriate authentication.
A classic example is adding a photo to an online profile or attaching a document to an application form. It would be really helpful to be able to either upload a file, or simply enter a URL to an online file. This is something applications developers could easily implement.
However, the ideal solution would be to change the HTML5 upload file selector to allow the selection of either a disk-based resource/file or a publically available web-based resource (via a url). This change would immediately make ALL web applications that perform file uploading more flexible with zero effort required from developers.
An obvious progression from this would be to allow browsers to access secured resources (private picasa galleries or files on ftp sites etc) - of course, only with the owners permission and with appropriate authentication.
Tuesday, April 26, 2011
Premium services and content - the price point is just wrong.
How much would you pay per month for access to Wikipedia? $100 per month? $10 per month? $0.10 per month? Nothing?
Wikipedia is probably not a great example because it recieves money via donations, not via subscriptions but the point is that most people would probably not pay $10 per month for Wikipedia. But when pushed people might agree that paying a few cents per month is so reasonable that even the most infrequent readers could justify the cost.
So if people are only prepared to pay a few cents per month for one of the worlds most innovative and used information resources, how can some services or websites charge significantly more? $10's, or even $100's per month? For example, the Wall Street Journal subscription starts at €1.50 - this is not a lot of money, but when compared with the vast numbers of free news websites and alternative sources, maybe this price point completely exceeds what a casual reader would want to pay?
However, great content can not be free. Currently there are two typical financing models for this content:
I think the world of premium services and content could be revolutionised with micro-payments. I could see a service whereby I could see all of my website subscriptions listed in one place and how much I am paying for each. In addition, this service could act as a way to manage my subscriptions - both allowing me to cancel existing subscriptions and subscribe to new sites at the click of a button.
If such a system existed, it could also allow users to give feedback to the content providers, allow users to 'recommend' websites and share opinions about them.
This would be great for web service and content providers because they would not have to care about handling payments. And of course such a system would be great for consumers - having one place to manage all of their subscriptions.
Wikipedia is probably not a great example because it recieves money via donations, not via subscriptions but the point is that most people would probably not pay $10 per month for Wikipedia. But when pushed people might agree that paying a few cents per month is so reasonable that even the most infrequent readers could justify the cost.
So if people are only prepared to pay a few cents per month for one of the worlds most innovative and used information resources, how can some services or websites charge significantly more? $10's, or even $100's per month? For example, the Wall Street Journal subscription starts at €1.50 - this is not a lot of money, but when compared with the vast numbers of free news websites and alternative sources, maybe this price point completely exceeds what a casual reader would want to pay?
However, great content can not be free. Currently there are two typical financing models for this content:
- Somebody else pays - commonly via advertising
- The consumer pays - commonly via a subscription or donation
I think the world of premium services and content could be revolutionised with micro-payments. I could see a service whereby I could see all of my website subscriptions listed in one place and how much I am paying for each. In addition, this service could act as a way to manage my subscriptions - both allowing me to cancel existing subscriptions and subscribe to new sites at the click of a button.
If such a system existed, it could also allow users to give feedback to the content providers, allow users to 'recommend' websites and share opinions about them.
This would be great for web service and content providers because they would not have to care about handling payments. And of course such a system would be great for consumers - having one place to manage all of their subscriptions.
Thursday, September 23, 2010
Google makes way for Femtoo.com - online tracking and notifications
Google have recently announced that it will be removed the tracking features of Google Reader.
In fairness it never worked very well and the alternatives, for example Femtoo.com is more sophisticated, allows far more user control and is just much cooler than Google Reader!
If you are interested in tracking changes to product prices, job postings, financial or world news, sports results and fixtures, house flat prices every 30 minutes - try Femtoo.com.
In fairness it never worked very well and the alternatives, for example Femtoo.com is more sophisticated, allows far more user control and is just much cooler than Google Reader!
If you are interested in tracking changes to product prices, job postings, financial or world news, sports results and fixtures, house flat prices every 30 minutes - try Femtoo.com.
Subscribe to:
Posts (Atom)