I’m writing this as a short(-ish) post with an overview of the Tech, how it is deployed, and other higher level views of the web. I don’t know if this will be helpful to anyone, but I feel like there is alot of uncertainty, even among developers, about what solutions are useful or what some of the best practices are. Keep in mind this is a huge field of endeavor, there are many pet projects and strong opinions, and no one is an expert on everything web development. If you don’t know a term, look in Describing the Web Stack
In the past 10–15 years, the web has expanded to include everything from online publishing of books, magazines, and other media, to interactive web applications, to ecommerce and core business applications. This statement may seem obvious and non-controversial today, but 20 years ago it would have sounded revolutionary. There is a strange dichotomy between developers, who have had to run at breakneck speed just to keep up with the pace of change, and the end users, who see an interface of text, links, and images that has improved, but may not seem fundamentally different on the surface.
But the software underlying it all has stretched, bloated in some place, and like a cell culture, has divided and mutated many times in recent years. To try and get a handle on this, you need to skip past individual species of tech and look at the larger ecosystems.
Media publishing was the first “killer apps” for the web. A ton of work has been poured into this field to the point where almost anyone can publish whatever they want to the entire world without needing to understand any code at all. Just signup for Wordpress, or another Platform, and add your content.
The underlying stack is the same as it has been since the late 90s:
- Linux (or another operating system).
- Apache (or another free web server).
- MySQL (or your favorite Platform’s database).
- PHP (or your Platform’s favorite language).
- Wordpress (or another favorite Platform).
Developers have been able to make a living by simply understanding how their Platform and customizing 4. or 5. to meet their clients’ needs. And for the vast majority of companies, this is all you need.
What’s really interesting is how resistent this stack has been to change. Adobe, Microsoft, even Apple would love to swap out multiple levels of this stack with their own pieces of software but haven’t gained the traction to really take over. I think it comes down to the fact that their best ideas end up just being coopted into the open web standards and the integration with the rest of their products (especially Adobe’s) is trumped by the level of overall support given to the Free Software solutions.
While Media publishing in general has become cheaper, more open, and more of a commodity, Social Media has turned into a system of walled gardens with may different, opaque, software solutions and all costs being borne through advertising and mining of user data. This would probably seem like a dystopian nightmare for some of the Free Software pioneers, but most people don’t care. And the reason why is because the benefits of free conversation between hundreds of millions of people, even if too much of it is toxic, outweighs so many other concerns.
And the scale problems, plus the problems associated with distributing these systems across so many platforms, has attracted alot of good developers. What’s interesting for me is that, while their work maybe hidden away on Facebook or other servers, many of the solutions that are applicable to other fields likely escape into Free Software without us even realizing it. So it’s hard to judge the kind of impact this work has made overall.
Software as a Service
This is probably the second “killer app” of the web, and much of it so ubiquitous that it seems hidden to most users. Businesses that used to have to set up a paper trail to manage something have turned it into a web app. The efficiencies from this have likely killed alot of solid middle class jobs, but they’ve also made things faster and cheaper for everyone. This is also one of the few places where you can truly be a “full stack” developer as other applications have either replaced pieces of the stack with a commodity piece of software, or have a large enough scale that everyone working on it is forced to specialize.
The underlying stack used to look much like the typical LAMP, but has gotten quite a bit taller since.
The server side:
- Linux (or another OS).
- Apache (or another web server).
- MySQL (or another DB).
- PHP (or Your Favorite Web Framework’s language).
- Your Favorite Web Framework. This can serve web pages and/or an API with JSON or XML.
The client side:
- Bootstrap, or another family of front end goodies.
- jQuery (optional, often part of the above).
- Custom CSS for your site. Often using Sass to build.
Much of this stack is actually a product of cross-fertilization between the Media Companies and the Web Software developers. The cynical observer would say this was about SaaS developers copying the work of these larger scale companies and misapplying it. Often causing unnecessary bloat. The really cynical observer might point out how often a developer uses a project to learn some of this tech and then applies to a better job somewhere else.
But many of these front end solutions do deliver a better experience for the end user, if implemented correctly. And, if you can fit your solution into the format required by these products, they also reduce the time required for development. So it’s a tradeoff for most developers.
This could be considered a subset of the Software as a Service family, but this is a really diverse group of systems that separate out based on the scale of system you need and how closely this is connected to the business implementing it.
Consumer and Small business solutions
There are a large number of companies that make their money as the middleman for internet transactions. See: Paypal, eBay, Gumtree, Etsy, etc. Some of them are geared to be super friendly for consumers and require little development skills. A few (like magneto) are basically Wordpress for eCommerce and require varying levels of developer knowledge.
The main advantages to these solutions are:
- You (or your developer) doesn’t have to become a security expert.
- Your clients don’t have to trust you, as long as they trust the third party, so they are more likely to buy something.
- You have someone else to blame if something goes wrong (not a huge advantage).
Midrange level solutions
I don’t know much about the difference between these. Basically, it could be anything from a higher level version of one of the other consumer level solutions (like Paypal or Magneto), or paying a smaller chunk of money to an Enterprise company to set this up. Or hiring a developer that knows how to integrate this with a credit card or other payment system. This has been expanding but kind of in a muddled growth pattern.
Enterprise level solutions
Basically, if you pay a large company (IBM, Sitecore, Magento, etc) a good chunk of money, they will set this up for you and it will have the capability to handle many more transactions than these other solutions. I know next to nothing about this, other than the fact that many of the Fortune 500 type software companies that used to do this have been spinning it off or shuffling it into the smaller (but still very large) companies as it is less profitable than it was.
Describing the Web Stack
Less technical minded readers may have heard these software terms but don’t know what they mean or how they work. Google is your friend for some of these, but I’m just going to describe each piece referred to here so you don’t have to look up to much to get the gist of the idea. Remember that these pieces may or may not all run on the same server. They are often divided up based on budget, expected numbers of users, and other development needs.
Linux (or other OS)
This is the machine (or virtual machine in some cases) that everything runs on. Linux is a good example of the type of OS developers love as it was written by developers for developers and has a number of specialized tools that are included for free. Depending on the company, you can substitute out Windows or even Mac OS for this, with varying levels of success.
Apache (or other Web Server)
This is the software that serves up requests from the user for a given web page. Apache is the most common (still) and is very mature. There are a number of other options that may be faster or better for a specific need. In most cases, this is one of those pieces of the stack that you configure once and forget about unless the user base grows to the point where you have to reconfigure it.
MySQL (or other database)
All of the data specific to a web site is stored and updated by a central piece of software called a database. MySQL is commonly used as it is free, open source, and doesn’t require too much work to configure. However, you can substitute Postgres, Oracle, Microsofts SQL Server, or even other NoSQL solutions like Mongo. It partially depends on your developers and partially on the needs of framework or whatever programming tools you use.
PHP (or another language)
This is where most of the actual development occurs, at least on the server side. This used to be a simple addon module to the web server that would allow for interactive scripts to be ran and serve up some content. It has since grown into a hotbed of activity based on different business and content needs and the scale of people accessing the website.
As websites have shifted from static pages being served up based on files stored on the computer to more dynamic content generated using a scripting language, the set of scripts has morphed into an entire family of software. The major advantage to this is that many common tasks, like interacting with the database, parsing the request sent by the server, and checking the user’s identity to allow them to login, can all be handled in a consistent manner by the framework. This leaves the developer to handle the specific logic needed to handle the server request which allows for faster and more consistent development with reuse of much of the common code.
There are tons of these frameworks out there for pretty much any common language, and some of the less common ones. Most of them work using a similar pattern called “Model, View, Controller” and so mastering one gives a developer a head start on learning another.
Wordpress, Drupal, etc. These are whole stack solutions that include a wide range of features, have been throughly tested in a number of situations, and can be customized to your needs. If your site is basically “serve up text and other goodies” and doesn’t require a large amount of customized user action or interaction with business data and logic, these are the ideal solution.
There are many reasons why development has moved in this direction:
3. This allows for a single web server to handle traffic from mobile apps or other client types. Probably the core reason why this pattern has expanded for alot of sectors. Basically what was a web request becomes more of a web API with different client types, both web and mobile, sending the same URL and getting the same JSON or XML response and then using that to build the view the user sees.
There are a number of tools, and whole libraries (like Bootstrap), that your development team can pull in and use instead of developing most of a site’s look on its own. This speeds things up and allows you to pull in a number of cool features without having to develop them on your own. However, it can blot the site a bit, and allow for a bit of a cookie cutter look if you don’t rein in some of this. Or, if your team is smart, you can use these tools to create a proto-type and then customize it later on.