Changing our mindset and approaches

Known to some as “The Danboe Manifesto,” this blog post summarizes where I think we are in terms of web development today, discusses why I feel things are the way they are, and suggests some things that need to change. My hope in writing this is to push for much lighter, consistent and semantically relevent markup in our web pages. Everything here is strictly my opinion, as someone who has been working on delivering extremely large scale web sites for over six years.

Update, Nov 4 2004: according to Faisal, some are now simply referring to it as the “Danifesto”. So be it.

The road behind

Our sites are currently in a state of crisis regarding issues such as user experience, page weight, performance and consistency across differing browsers and devices. We’re also under significant pressure to increase the end user relevancy of our sites, primarily by the introduction of content, features and services that are highly personalized and customizable.

To accomplish this successfully, we need to reach a model where we can increase the testability, reuse, reliability and predictability of our markup, the very medium over which these features, services and content are delivered.

We’re in this position today because in many ways, we still approach web development like we did in 1998. We place great value on the heavy-hitting, highly-optimized server development, reviewing and optimizing how the bits are spit out the pipe with fierce devotion. Unfortunately, less effort and value has been placed on precisely what bits we’re spitting, and why. This needs to change. There’s a growing movement in the industry around standards-based development, that the separation of presentation and behavior from content is the Holy Grail in achieving the goal of reusable content and predictable experiences.

HTML was not originally a presentational language, it was structural. The Browser Wars changed that, introducing elements such as <font>, <b>, <i>, <blink> and <marquee>, which provided presentational instruction for the display of content, not semantic meaning about the content itself. The Browser wars also introduced DHTML, but in proprietary rather than consistent implementations. This muddied the waters, and resulted in bloated, less efficient content authoring, because content, presentation and behavior became indissolubly linked in the minds and code of many. Worse, it created an environment which promoted targeting of content to specific browsers, thus introducting a significant barrier between the content being authored and the audiences it was authored for.

We need to pull away from the approaches we have taken in the past, hand-picking a number of different “supported” browsers, and striving to make the experience as close as possible between them. We need to stop developing from a “hack mindset” that encourages significant investments being made in areas such as conditional markup, behavior and styles. We need to build a solid, defensible core experience, and we need to stop revising the code that generates this in attempts to solve specific rendering problems in specific browsers or devices. These hacks significantly complicate the code and experience, and reduce our confidence that a predictable experience will be given to new browsers/devices as they come. Here are some examples:

• Trying to make a set of pages look identical in a variety of browsers, when the real problem is designing a page that most importantly works correctly and renders completely in any browser despite display variations.
• Forcing a fixed point size to protect layout and decorative features, when the real problem is a design mindset inappropriately based on a magazine-cover precision that ignores the reality of the web and the constraints of screen displays, browsers and devices that are becoming more predominant and disregards the diversity of media types that are forthcoming.
• Trying to force or constrain text-flow, when the real problem is creating text interesting enough to induce readers to scroll and using layouts based on methods appropriate to the medium instead of those adopted from print or Photoshop.
• Relying entirely on client script rendering techniques to decrease download times, unfortunately also reducing the predictability and usability of the site for users and browsers that wait for the download but do not support scripting.
• Changing three frames automatically in succession, when the real problem is how to structure the site’s navigation to make pages directly accessible.
• Trying to disable the browser’s built-in back and forward navigation, when the real problem is giving the visitor logical and easily discoverable options to work through the documents contained in a site.
• Trying to disable default browser navigation, refresh or caching to stop a form from being submitted twice, when the real problem is programming the server to detect and handle multiple submissions.
• Trying to reinvent the default semantics and rendering behavior of an XHTML tag, such as making text in a <pre> wrap, when the real problem is a lack of familiarity with core XHTML and the intent and default rendering characteristics of its often semantically valid tags in the first place.
• Trying to disable view source capabilities when the real problem is that there is nothing there worth hiding, and even if there were, there is no possible way of hiding it from those who may value its “secrets”.
• Worrying about no-script cases that bypass form validation, when the real problem is how to reproduce the validation code in the server-side script.

The chief problem with this hack mentality is that it encourages us to actively search for complicated implementations. Seek and they shall be found, indeed, but choosey developers choose simple solutions. It’s a parasite that must be eradicated to move forward. These complex approaches do not help, and they won’t go away until we decide to stand up and shout for a simpler, more predictable approach. The Web is an uncertain place—we’ll never be certain that our sites will work in the exact way we wish them to, not even after applying all hacks and insights from CSS, accessibility and usability.

Such hacks have lulled us into a false sense of security and unbelievably, a sense of pride over their complexity. We must shed this false comfort, and openly and commonly accept uncertainty as a basic principle in our business. Once we strip off the hacks, what we’ll find is a common, standards-driven core that will work in predictable ways when used in a valid and compliant manner. We must stop further complicating our client script routines. Indeed, we should consider removing some of them entirely.

Where we are today

We are currently in a crisis regarding the size, weight and complexity of our pages. As this document is being written, we are exploring a reduction in content on our home page to lessen the sting of the first-time experience. Consider that for a moment. By removing content from the page, we make our page lighter, but we also arguably make it less compelling. I assert that this is only necessary for one of two reasons:

• The markup we currently deliver is too bloated because of the hack mentality from which it was built. The ratio of presentation and behavior to content in the bits we send over the wire is too high, but removing content is considered safer than simplifying the already fragile presentation markup that surrounds it.
• The content being removed was never very compelling in the first place, but we failed to challenge it before because we were too focused on trying to get the experience to match the pixel precision of the Photoshop comp.

Unfortunately, the hack mentality is deeply engrained in our development process. How do we build the sites, pages, components and services that our users rely on every day? Here’s a rough outline of a site development process that may have characterized the past several years for someone you know:

• Start with a live version of the site in production. It’s not documented terribly well. It’s running on a platform whose end-of-life is already scheduled. Users are using it.
• Design and planning unveil a new “comp” that shows the new “vision”. It looks cool, but we have questions.
• A spec is created from the “comp”. It provides some answers, and identifies possible directions that the “vision” implies, could imply or was meant to imply. Clarity or more vagueness, depending on the last version of the comp you saw.
• Development starts on the markup and back-end.
• Our underlying platform is also being rebuilt at the moment, so we’re “prototyping” on that because it could get us out the door with the new version quicker.
• Additionally, we decide to “migrate” all of the stuff from the old platform to the new one as a part of this as well. At the same time; just rolling it in, adding more flour if it gets too sticky and starts to clump.
• Finally, we decide to do it right once for all uses, markets and sites. The version we ship will be the standard that everyone can use.
• Initial drops of the markup and back-end are considered not testable because they do not create pages that look like the “comp”.
• We begin adding style, even though the underlying markup it’s hooked to changes every day. Our time is split equally between creating and breaking style/markup bindings as both the markup and CSS are revised in tandem.
• Visual bugs begin to get entered.
• Churn the code, fix the bugs.
• We realize that the data which the “comp” suggests is available is not after all. We can’t slip.
• Churn the code, fix the bugs.
• We slip. We cut or postpone documentation from the schedule to hit our date.
• Churn the code, fix the bugs.
• We ship.
• The documentation, where existing, is already out of date.
• Users complain.
• Key competitors have changed, and areas of investment in the newly shipped version are now considered liabilities.
• Rumors circle of a new “comp” floating around.

The road ahead

The complexity and diversity of our existing markup implementations are quite frankly costing us too much in terms of bits across the wire, client time to render, development effort, testing effort and documentation. The good news is that we’re more aware of this today than we’ve ever been. We can change, but we need to be careful about how we change.

Clearly, we need a better, more workable, consistent process around this. What follows are some thoughts along these lines.

So how should we build the site going forward? By building pages. How should we build them? By building components. How do we build the components? Simple. Just like this:

• Define the data. Before deciding it needs to work in a 3-column layout or support resizing or total less than 500 bytes, decide what it is. In fact, define what it is. What is the data we’re talking about here? What is its intent? Where does it come from? Do we already have it, if not where and how can we get it? Can we look at it? What are the pieces of sub data? How are they grouped, organized?
• Build a schema for the data. What are the rules? How can we define it? How can we validate it? What does the data look like in this schema? Try different cases against the data and try to sneak something by the schema. Fix the wholes as they are found. Let the content inform the structure.
• Build a semantic XHTML structure for this data. Don’t style it. Don’t script it. No classes. No ID’s. Just render the semantic XHTML. Work with it. In a text browser. With a screen reader. Try printing it. Does it work? Is it structured well and accessible? Does it make sense? Can it be understood? Treat this structure just like you would the schema. Again, let the content inform the structure.
• Build a data provider that returns this data in both of these schemas.
• The point up to here is that we’re not viewing this as a visual layout yet.
• Validate the XHTML structure against the standard markup.
• Validate the data representation against the schema.
• Prototype enhancements against the XHTML structure, with a focus on validating that the structure is fundamentally correct and complete. Examples of enhancements are:
• Different visual/print/aural style guides that can be applied to the structure to provide more desirable presentations of this data. Also known as CSS. Strive for flexibility at this stage – write several variant stylings, not just one.
• Considerations where the client can be better leveraged when working with this component for validation, a more intuitive or responsive interace, etc. Also known as scripting or behaviors.
• Revisit “Comp” and Spec.
• Push back on the “comp” in areas where it has shown to be inconsistent with the structure and nature of the data now understood very well. Suggest alternatives that may not have been considered based on this better understanding.
• Push back on feature sets that are not feasible or appropriate now that more is understood, suggest alternatives that are.
• Close “Comp” and Spec.
• Revise the structure with ID’s, classes and semantic changes that are necessary to support the enhancements.
• Finalize the data provider.
• Validate the XHTML structure against the standard markup.
• Validate the data representation against the schema.
• Lock the schemas and document them.
• Build the enhancements and document them.
• This is now being done against a locked and known schema.
• It should be noted that every component need not have enhancements. Some will, and some won’t.

The core experience and its enhancements

So you’re probably wondering about the browser support story here. Actually, it’s very simple. Every user, regardless of browser/device, gets the core structure/experience, without enhancements. Period. In the exact same markup. XHTML 1.0 Strict. They get all of the data and all of the functionality. No one gets blocked from the key experience.

Additionally, each browser/device receives the enhancements that it can use—some (like current browsers) will get really nice enhancements, some (like v4 browsers and earlier) will get no enhancements, just the core experience. It won’t look as pretty. It won’t behave as richly. But it will be fully functional.

This is a big shift in thinking, but the time has come

Instead of building from the user experience in, we’re building from the core data out. For example,

• Everyone gets the data in its simplest presentation. It may look nicer in CSS, but it’s complete without it. This can be taken pretty far – even things like images could be moved from XHTML to CSS where it doesn’t add value, and we can do this consistently and really start saving in page weight.
• Everyone gets navigation, but it does not rely on behavior to render. It may take up less screen real estate or work better with behavior, but everyone gets functional navigation.
• Current browsers get the richness of experience they expect without cluttering the experience for older or other browsers/devices.

Site goals focused on the medium the site is delivered by

Each of our sites is a collection of content and services delivered to our customers over a consistent medium of markup, style and behavior. Our sites have clear goals around this content and these services, but have you ever noticed that we don’t have a tendency to establish goals around the medium itself?

What if we were to do this, using something like the following as a starting point? These ought to be commonplace and unremarkable, but unfortunately, given our history, current situation and direction, they are far from commonplace and need to be actually stated and agreed upon by development, test and management.

Top 10 site goals focused on the medium:

1. The site must be as usable in non-graphical environments as it is in the best and latest browsers. Its content and basic functions must be available to any browser or device; its layout and behavioral enhancements must work in reasonably-supportive browsers.
2. Markup must validate against the XHTML 1.0 Strict spec. The use of elements that are presentational in nature must be avoided. We will separate structure from presentation, which means we not use non-semantic <span> or <div> tags when <p> or heading tags suffice. It means no spacer images, no <table> tags except those to present tabular data, no outdated or invalid attributes, no misuse of <blockquote> for formatting, no use of <br> tags when a structural element can create the same layout while also conveying meaning. It means no <nowrap>.
3. CSS must validate and should be as compact and as logically arranged as possible.
4. The site should strive to be seamlessly accessible, and will be tested against the accepted accessibility standard of US. Section 508 (as an example, of course worldwide this would need to be more exhaustive).
5. Text should be easy and pleasurable to read and should be made the focus of the site, whether for reading, actions or navigation.
6. The site will deliver a recognizably branded look and feel without wasting user time or bandwidth on bloated markup, excessively complex scripts or needless images. The site will not waste resources, but it must possess style. It need not “wow” the user, but it should feel like a place. It should also feel like the recognizable evolution of a familiar place rather than a radical departure from previous incarnations. It will have a visual panache, but it must also work in non-visual environments.
7. Navigation and advertisements should be clear, intuitive and obvious. Visitors who access the site in a linear fashion (for instance, screen readers) should be able to skip right to core content or services.
8. Whenever possible, equivalents to dynamic elements will be provided for the benefits of those using text browsers or other non-traditional devices. Interactivity will not be accepted as an excuse for poor accessibility.
9. The site should provide customization options without losing its brand in the process.
10. The site should offer visual interactivity (some perhaps even playful) so as to seem like a living thing.

A paradigm for decisions

Separation of structure from presentation and behavior. Building from the inside out rather than the outside in. Letting the content inform the schema. In the end, these are just paradigms that create an opportunity to step back and consider how each core experience and its enhancements should be approached, while helping to ensure we don’t cut off one user’s needs because of a decision we made for another’s.

It’s not easy, it will take hard work, determination and focus initially. While it’s difficult to consider how the components, pages and sites operate and interoperate when stripped bare to their XHTML, it’s downright painful to look at how the current version of these components, pages and sites are treated today by different browsers and devices.

Next post (newer)

The Rolling Stone's top 500 albums of all time

Previous post (older)

40GB click-wheel iPod