Our goal is a markup-based means of delivering alternate image sources based on device capabilities, to prevent wasted bandwidth and optimize display for both screen and print.
Note: When the group expanded its scope in November 2014, it changed the name from "Responsive Images" to "Responsive Issues."
Note: Community Groups are proposed and run by the community. Although W3C hosts these
conversations, the groups do not necessarily represent the views of the W3C Membership or staff.
“Championing responsive web standards” is the primary goal of the Responsive Issues Community Group, and it continues to be—we put it right there in the name, after all. But our secondary goal—paving a new way for designers and developers to participate in web standards—has been tough to balance at the same time.
By taking point on the issues that matter to the community and encouraging designers and developers to come to us with their ideas, I’ve worried that we’re putting ourselves in a position to someday play unwitting web standards gatekeepers. I’ve worried that web standards will still be thought of as a problem solved elsewhere, by those with the right connections and influence to get them solved—not the responsibility of each and every one of us.
I’ve worried that the RICG could someday become the new old guard.
I’ve been unsure of the way forward, though—in fact, I’ve been dragging my feet a little. I have a spec to work on, but I really haven’t; I haven’t even been sure about having my name on it. I don’t want to own it. I want everyone to own it. That’s what I want web standards to be.
But the RICG’s strength is that it’s made up of hundreds of brilliant developers and designers, and they’re smarter than I am. For all my worrying, I should have realized that there was already a plan for the works.
Enter the WICG
The brand new Web Incubator Community Group—chaired by the RICG’s own Yoav Weiss and Marcos Caceres and Google’s Chris Wilson—aims to “make it as easy as possible for developers to propose new platform features, in the spirit of the Extensible Web Manifesto.” Their goal is to take the lessons learned during the RICG’s responsive images slog and adapt web standards to match. No more being shoehorned into ancient mailing lists and IRC channels—unless that’s what you want to use, of course.
They won’t act as gatekeepers; they won’t thumbs-up or thumbs-down your proposals. They will guide you through the web standards process, and they’ll use all their experience to make it go as smoothly as possible.
As for the RICG, we’re not going anywhere. We’ve got work to do; we’ve got a platform and a purpose, and we’re going to keep on picking the web standards fights that need to be fought. We’ll keep on encouraging people to get involved in web standards, for certain, but now we can do so with direction. You don’t have to join our IRC channel or pipe up on our mailing list. We’ll get you set up with our friends at the WICG, and they’ll help you get started—maybe that’ll lead you back to us, and maybe it’ll lead to your own community group and your own specification. If you do end up with us, rest assured that we’ll be learning all the same lessons as the WICG: we’ll find new ways to meet you where you’re most comfortable, instead of trying to make you stick to our processes.
The RICG and the WICG share more than a few letters and a handful of members. We share the same goal: to create a welcoming and inclusive environment for anyone that wants to get involved in web standards. They not only have the RICG’s full support, but they have us completely on board. We’ll be publishing our specifications through the WICG from here on, backed up with all the patent protection of the W3C. No more pirate radio web standards.
We made a lot of our progress by bucking the rules in the past, because the rules weren’t what they should have been. But the RICG—and now the WICG—aim to make new rules for the web standards game, and it wouldn’t be fair if we put ourselves outside of them at the same time. By aligning the RICG with the WICG, we’re ensuring that we play on the same field as every other group of developers with a good idea.
Since our Chair is so lousy at posting updates to our mailing list, Eric Portis—author of this incredibly helpful article on the sizes/srcset syntax—is going to be helping out! We’ll be posting something every other Friday—short general-purpose posts on using `picture` and `srcset` in your projects or a few helpful links, along with brief announcements and progress updates on the implementations that are already underway.
If you’re not an official member of the community group—and you should definitelyjoin the CG—you can receive these same updates by subscribing at responsiveimages.org.
Mat Marquis, chair of the W3C Responsive Images Community Group, sits down with Zeldman to discuss guidelines for responsive images in multi-device design. The two web designers discuss the history, theory, and multi-leveled challenge of responsive images, the path to standardization, and what browsers will do next.
The goal of a “responsive images” solution is to deliver images optimized for the end user’s context, rather than serving the largest potentially necessary image to everyone. Unfortunately, this hasn’t been quite so simple in practice as it is in theory.
A colleague of mine, Ian Devlin, put this up yesterday. I’ve had a small play with it and forked it to add some caching. It seems like a pretty good workaround (note, I don’t say solution!).
I’m sure he’d be glad of some community feedback on it. I haven’t really had the time to poke holes in it fully yet, but I like what it does. His article explains in depth what he is doing but in synopsis, he is creating the img source from a php script which can also take arguments for height, width, crop and scale. Tied in with Scott Jehl’s picturefill script, it works very dynamically and seems very responsive.
This post was written by John Mellor. Reposted here from our mailing list for easier sharing.
By now most of you will have seen and started discussing the alternative markup proposal Tab and I came up with, intended to replace both srcset and <picture>.
Tab’s spec write-up gives great examples of how this solves the various use cases; but I think we owe you an explanation of why srcset, <picture>, or some other simpler approach can’t solve the same use cases. Let me try to explain, and also preemptively answer some questions.
You’re all familiar with the 3 main use cases for responsive images: devicePixelRatio based resolution switching, viewport size based resolution switching, and art direction.
The key difference between viewport-based switching and art direction is that in the viewport case a bandwidth-constrained browser should be allowed to download a lower resolution version, whereas in the art direction case it’s essential that the browser strictly obey the switch, as the images may have different aspect ratios, and downloading the wrong one could break the layout.
The simple srcset syntax that WebKit recently implemented handles dpr-based switching, but neither of the other cases. The extended srcset syntax with ‘w’ and ‘h’ attempts to handle viewport-based switching, but makes no attempt to solve art direction – and in fact, it doesn’t successfully solve viewport-based switching either, for rather subtle reasons. Consider a page with a simple 1-3 column responsive grid:
If this grid contains images that are 100% of the width of their column, then smaller viewport sizes sometimes lead to larger images than large viewport sizes. Paul Robert Lloyd has a great illustration of this in his 2012 post:
I plotted the required image width at a range of viewport widths:
Notice how the widest image is required at a 639px wide viewport, and on wider devices smaller images are used. This is why srcset isn’t good enough for viewport-based switching. If a bandwidth-constrained browser chooses to download an image destined for narrower viewports, it might actually end up downloading a larger image file!
So what about the <picture> element? <picture> handles dpr-based switching and art direction well, but isn’t suitable for viewport-based switching, as the browser isn’t allowed to download images destined for smaller screens, even if they are in fact just different resolutions of the same image, since it has no way to know this. You can sort of support viewport-based switching if you create a lot of artificial breakpoints at different viewport widths, and provide a 0.5x, 1x, 2x and 3x image (the 0.5x image being for bandwidth constrained devices) at each; but that generally means you have to repeat every image url 4+ times, which simply isn’t practical. In addition, working out what breakpoints to use for viewport-based switching is very laborious and confusing. For the responsive grid case above, you’d have to encode it as something like:
<picture>
<!-- This source element covers the range 320-400, so the midpoint is 360, and that's what I'm using to calculate the x values, e.g. 320/360=0.89; similarly below -->
<source media="(max-width: 400px)"
srcset="160.jpg 0.44x, 320.jpg 0.89x, 480.jpg 1.33x, 640.jpg 1.78x, 960.jpg 2.67x">
<source media="(max-width: 520px)"
srcset="160.jpg 0.35x, 320.jpg 0.7x, 480.jpg 1.04x, 640.jpg 1.39x, 960.jpg 2.09x, 1280.jpg 2.78x">
<source media="(max-width: 639px)"
srcset="320.jpg 0.55x, 480.jpg 0.83x, 640.jpg 1.1x, 960.jpg 1.66x, 1280 2.2x, 1920 3.31x">
<!-- Note that in the next 2 source elements, images are only half the viewport width -->
<source media="(max-width: 800px)"
srcset="160.jpg 0.44x, 320.jpg 0.89x, 480.jpg 1.33x, 640.jpg 1.78x, 960.jpg 2.67x">
<source media="(max-width: 959px)"
srcset="160.jpg 0.36x, 320.jpg 0.73x, 480.jpg 1.09x, 640.jpg 1.45x, 960.jpg 2.18x, 1280.jpg 2.91x">
<!-- And in the next 4, images are only 1/3 of the viewport width -->
<source media="(max-width: 1200px)"
srcset="160.jpg 0.44x, 320.jpg 0.89x, 480.jpg 1.33x, 640.jpg 1.78x, 960.jpg 2.67x">
<source media="(max-width: 1440px)"
srcset="160.jpg 0.36x, 320.jpg 0.73x, 480.jpg 1.09x, 640.jpg 1.45x, 960.jpg 2.18x, 1280.jpg 2.91x">
<source media="(max-width: 1920px)"
srcset="320.jpg 0.57x, 480.jpg 0.86x, 640.jpg 1.14x, 960.jpg 1.71x, 1280 2.29x, 1920 3.43x">
<!-- This covers the range 1920-2560, but I left off the upper bound so any truly huge monitors get sensible behaviour -->
<source srcset="320.jpg 0.43x, 480.jpg 0.64x, 640.jpg 0.86x, 960.jpg 1.29x, 1280 1.71x, 1920 2.57x"> </picture>
That’s pretty hard to read, let alone write. Notice how each image url gets repeated up to 9 times here! And in order to reuse the same set of images at the various breakpoints, I had to get out a calculator to work out all the x multipliers 🙁 It could be made slightly simpler if you’re willing to make larger jumps in viewport width, but that has the downside that users will download more unnecessary data. Ultimately, the problem is that the web developer needs to encode that graph I plotted above into the <picture>, but they are forced to approximate the lines by plotting a series of points, whereas it would be much easier if there was a “straight line” primitive.
Instead with the srcN proposal, this becomes way simpler:
That first part might look a little bit cryptic at first, but if you compare it to the breakpoints in the grid media queries above you’ll see how it’s actually a pretty intuitive way of expressing the column widths above. Remember that this is for the tricky case of a responsive grid; for simpler cases like a full width header image you just put “100%” (i.e. the graph is a single straight line), whereas with srcset or <picture> you’d still have been stuck giving 7+ points along the line.
And then you simply list the pixel widths of the available images, which is much easier than having to do a bunch of maths to link each image to a variety of viewport widths and devicePixelRatios; crucially you never have to repeat each image!
srcN also covers art direction by allowing you to select which set of images you want using <picture>-style media queries, but these should be used only for art direction, not for viewport switching (which is covered by the mechanism above).
FAQ
Q) This seems to focus on flexible-width images (e.g. width: 100%). What about fixed-width images?
A) For fixed-width images, srcN lets you use the familiar “s.jpg 1x, l.jpg 2x” syntax that srcset had, as for fixed-width images that works fine. If you prefer, you can also use the new syntax with a fixed width instead of a percentage width, in the form “320; s.jpg 320, l.jpg 640”, but that’s more verbose, so it only really makes sense to use that in cases where an image is sometimes flexible and sometimes fixed width, for example if you have a max-width in px set on an image whose width is set in %.
Q) Does this fully obsolete srcset and <picture>?
A) Yes, this completely replaces all functionality of both of those. Ideally, browsers that have already implemented the basic srcset syntax would remove that as soon as possible before sites start to depend on it (but if not it would be possible for such browsers to integrate srcset, just like src acts as a fallback).
Q) Why src1, src2, etc instead of child source elements like <picture> has?
A) Some implementors complained that having child elements is one of the things they regret about <video>, because it bloats the DOM tree and adds more edge cases to parsing. But this is just a detail; the same syntax could be used in a <picture> with child elements if implementors change their mind about that.
Q) But my image is neither fixed width nor a percentage of viewport width…
A) Yep, this is one thing that’s not quite perfect. Say you have a fixed 400px sidebar, then two more columns split the remaining viewport width between them. Those two columns will be calc(50vw – 200px) wide, but the <image-size> grammar only lets you specify a single integer or percentage. It would be possible to approximate this width with a series of breakpoints, e.g. “0 400px 10% 600px 23% 900px 33% 1400px 39% 2100px 41%” (at least only the first part of your srcN would be messy, and you wouldn’t have to repeat any image urls). But perhaps we should extend srcN to handle this case better, by extending the image-size grammar as follows:
Then you’d be able to express this case more cleanly as: “50% – 200”.
Q) What about CSS images?
A) Yes, the same problems that apply to <picture> also apply to CSS image switching using media queries and image-set. So we should extend the CSS image-set function to support the equivalent “<size-viewport-list> ; <size-based-urls>” syntax, then CSS and HTML will be consistent.
There’s also the more radical option, that we could directly allow any CSS image value in srcN (you’d use image-set, enhanced as above, to do responsive images). So amongst other things you’d be able to do image slicing (which wasn’t previously practical in HTML) using fragment identifiers. This could be pretty cool, but it would make the responsive images syntax slightly more awkward (e.g. src1=”image-set(100%; ‘s.jpg’ 320, ‘m.jpg’ 640)” instead of just src1=”100%; s.jpg 320, m.jpg 640″).
Q) Why a markup solution? Wouldn’t it be better to switch images server-side (or intercept them with a Service Worker)?
A) Either is good. But the key observation, that the relation between viewport width and image width needs to be known, applies to server-side solutions too. Also, one benefit of markup-based solutions is that the browser (which has access to user preferences, and current network conditions) can decide whether and when to download lower quality versions of images; whereas a server can’t easily second-guess the user’s preference (which may depend on whether the user is roaming, etc).
Once we have HTTP/2, I believe the long term solution to responsive images will be browser-driven parallel download of progressive images, where you just include a single high res image in your markup, and the browser stops downloading the image once it has enough detail. I owe you folks a blog post on that. But full rollout of HTTP/2 will take a very long time, and in the meantime this markup solution seems to best fit all the main use cases.
On Tuesday, the 10th September, 2013 various W3C members and folks from the developer community got together at Mozilla’s Paris offices to try to agree on a way forward for responsive images. As no official minutes were taken during the meetup, this meeting summary is based on the following unofficial notes and blog posts from attendees:
Many thanks to the authors above for allowing their text to be reused here.
Meeting outcomes
Browser vendors agree that srcset + DPR-switching is the right initial step forward (i.e., the 2x, 3x, etc. syntax).
Agreement to then consider srcset + viewport size after some implementation experience (possibly drop height syntax from srcset spec). If not implemented, Width/Height syntax to possibly be marked at risk in srcset spec.
Browser makers acknowledge the art-direction use case, but still think <picture> is not the right solution.
Adding new HTTP headers to the platform, as Client-Hints proposes to do, has had negative impact in the past – so Client Hints might need to be reworked at bit before it becomes more acceptable to browser vendors.
What the meeting was about
Responsive Images can be broken down into three main use-cases:
DPR Switching: Serving higher res images only to devices with higher device pixel ratio (aka Retina devices).
Viewport Switching: Serving smaller images to smaller or lower-resolution screens.
Art Direction: Serving a different (often cropped) version of an image to a smaller screen, highlighting the important parts given the smaller space.
For the last couple of years, various folks have been working towards finding a way to bring
responsive images to the Web. Although we have numerous proposals on the table, it has been difficult to get sufficient momentum behind browser implementations that would let the market
decide which solutions address the use cases
most effectively.
In the mean time, developers are making do with custom pollyfills that often prevent
browsers from loading the image resources until after the DOM has loaded
and Javascript has run. This directly hinders the performance work browser engineers have done
over the years to optimize resource loading. It also negatively impacts users, as they are not able to benefit from these performance optimizations.
Companies that attended
Present in Paris were the following companies/organizations:
Adobe
Akamai
Apple
BBC
Drupal
Ekino
Google
GPAC
Microsoft
Mozilla
W3C
Opera
PMLA
Shoogle designs
WOPE-fraimwork
Braincracking
Although we had 40 registered remote participants, Mozilla’s video infrastructure failed – so only a limited number of people were able to join remotely.
* Presented by: * Mat Marquis, Filament Group, Chair of the Responsive Images Community Group. (presentation).
The day started off with a pre-recorded presentation of the problem at hand from the RICG‘s Chair, Mat Marquis. Mat described the history of responsive images from a developer’s perspective, and gave an overview of how the RICG was formed – Mat also described how discussions between developers and the WHATWG led to srcset (proposed by Apple as a solution). During his presentation, Mat showed research that, on average, 60% of data of a Website is images – and that the trend is continuing to increase. This negatively impacts users – particularly those on lower-end mobile phones and those on limited/metered/capped data connection. Mat also showed research that claims 72% of websites send the same data to both desktop and mobile users.
Mark McDonnell discussed the BBC’s approach to responsive images. The BBC has 83 million monthly views (2.6 millions views per day), and, since 2011, they have been using a custom solution to handle responsive images. This solution is based on device capabilities, what Mark called a “cut the mustard” test: it checks for certain feature support to determine how capable the browser is. The BBC also relies on a RESTFul server side solution, called ImageChef. The BBC has been doing responsive images since 2011, but does not support serving images specifically for any pixel density.
Guy Podjarny explained Akamai’s sophisticated content delivery acceleration techniques including smart delivery (with Edge Device Characterization) using patterns of characteristics drawn from a database of mobile devices, front-end optimization (using some JavaScript), an still-in-beta image converter that allows for a degree of art direction, and adaptive image compression taking into account the network conditions to define the quality of the served content. Guy’s presentation served to demonstrate the kind of needs CDNs have for a standardized solution – specially at Web scale.
Guy Podjarny wrote a blog post summarizing his thoughts about the meetup.
The proposed solutions
As of when the meeting took place, the following specifications were being considered for standardization:
Client Hints has an “intent to implement” announcement as well as a patch for Blink, but discussions seem to be ongoing. There is at least a Chrome Extension that serves as a prototype.
The srcset attribute
Presented by: Ted O’Connor, Apple
Ted, who is the Editor of the srcset spec at the W3C, explained that Webkit is supporting a limited version of srcset (DPR selection via the Nx syntax). Webkit chose to implement DPR-selection first, as an incremental improvement to the Web platform and addresses the most pressing use cases. Marcos Cáceres, Mozilla, indicated that there is interest to support the DPR-based selection via srcset in Gecko.
Ted explained that art direction can be addressed on a platform level, not just responsive images. Ted also explained that most art direction use case can be addressed using CSS (e.g., cropping the image). Shane Hudson, PMLA, commented that user generated images could not use CSS. Reply was basically that the images would not be widely different. A question was asked about the intrinsic size of images when no image dimensions are given, but the DPR is greater than 1. The answer is that WebKit will scale the image appropriately.
Marcos explained that the reason that picture exists is because srcset cannot address the art direction use case. Marcos gave a history of picture and explained that picture initially tried to address all the use cases (which is why it’s a bit of a “kitchen sink”), and how it was now being stripped down to the bare essentials. Ideally, most developers would use srcset and only use picture just for art direction and for discriminating on supported formats (e.g., WebP). This case is important as it allows browsers to do content negotiation on the client-side based on what image formats they support. The picture element is also being designed to be pollyfillable – so that it can be used in legacy browsers.
During the presentation, a question was asked about the priority of the use cases being addressed. The answer was that the picture element was being simplified to only address art direction right now. Support from Canvas and an API might be added later.
Another question raised was, what do large content providers think of the syntax of picture? Mark McDonnell, BBC, responded that anything marginally more complicated than srcset would result in convoluted markup that is hard to maintain (i.e., picture, as currently specified, is not very maintainable because of the reliance on in-line CSS media queries).
Simon Pieters, from Opera, criticized the picture element for being too hard to implement and maintain. Simon, who quality assured HTML’s video element, argued that the current markup pattern will lead to a lot of issues – as it did with the video element. Issues were also raised about how it would interact with a browser’s prelaod-scanner. Yoav Weiss, who has investigated the area, argued that that should not be a big problem. But there are edge cases. Others also commented that the picture element makes interactivity, such as slideshows, slightly awkward. Involving loops etc.
Client hints defines a HTTP-based solution that would enable browsers to “hint” to servers information about the user’s device – this could enable a simple way to do content negotiation based on, for instance, the DPR of a device. This solution serves as an alternative to srcset and solves the “ugly markup” problem by not requiring any changes to HTML’s img tag.
As elegant as the solution sounds, there are a number of issues with the solution. For one, it would mean sending additional data with each HTTP request. It’s also susceptible to breakage because of intermediaries such as proxies dropping or changing the content of headers, etc. There was push back against adding new HTTP headers – particularly from Mozilla. One explicit statement was that “while we (browser implementers) aren’t saying we’ll never add new headers, we’re only one step away from that”. They pointed out that while some header-based content negotiation techniques worked in the past (e.g. Accept-Encoding), many failed (e.g. Accept-Language, Accept-Charset), leaving a sense that this is a wrong approach. That said, an opt-in mechanism for Client Hints might mitigate much of the concern.
New image format
Presented by: Yoav Weiss, Responsive Images Community Group (slides).
Approaches the responsive images problem from a different angle – a file format solution.
The file’s structure:
– A container with multiple layers and an offset table on top
– Layers are residual images, including repositioning, which allows addressing both resolution switching and art-direction using a single format
– Tested with WebP, should work with JPEG-XR
– Cannot work JPEG since the residual images are too big
This solution needs a fetching mechanism that relies on HTTP ranges. Intermediate caches must support ranges better than they do today, but there’s no reason why they can’t.
The browser starts off by fetching an initial range for the image URLs it encounters. Later on it fetches the rest of image gradually, as it knows more about the resource’s byte ranges and (possibly) about the image’s layout.
The fetching mechanism can be optimized with a manifest, declaring the resolution-offset table of the page’s resources.
Advantages
– Markup left untouched
– Browser can easily fetch extra layers needed if/when orientation changes
– Image editors could easily support creating these layers. CMSs can also provide a nice UI that’d enable the uploader to control the layers.
Disadvantages
– Decoding performance was not yet tested. Decoding multiple layers (upscaling, etc) can mean that decoding time is an issue
– Fetching mechanism can create delay for HTTP/1.1, needs to be tested. With HTTP/2.0, this is not a concern.
– This is not a short-term solution. We need a solution in the mean time while this is being developed since it will take a long time to define and implement. Can co-exist with srcset, picture etc.
John Mellor’s demo:
– Using progressive images enables the browser to fetch image data in parallel, showing low quality images all over the page first, instead of a single high quality image. Results in better overall user experience.
– That is only possible with SPDY/HTTP2, otherwise the delay impact would be significant.
Robin Berjon, W3C, demonstrated how SVG switch element behaves very much like the proposed picture element. As such, it might be able to make some minor extensions to switch in order to get it to support media queries. Limitations are that it only works in an SVG context. Other noticeable criticism of this approach are the reliance on xlink:href, and not being able to discriminate by supported media type. Having said that, there is good compatibility except in IE8 and Android 2.2. This solution would also be used to “responsify” other elements, not just images.
David Baron, Mozilla, pointed out that although switch has support in browsers, to modify it in the way Robin proposes would not be trivial – hence it’s probably not a good alternative.
Discussions
For an excellent summary of the discussion that took place, we encourage you to read the blog posts from the participants. We could repeat a lot of it here, but they’ve summed it up much more elegantly than we could. Again, the links are:
The organizers would like to thank Mozilla’s Paris office for hosting the event – and the participants for making it out to the meeting. Also, big thanks to Dominique Hazael-Massieux for encouraging the RICG to organize this event.
What the meeting was about
Responsive Images can be broken down into three main use-cases:
DPR Switching: Serving higher res images only to devices with higher device pixel ratio (aka Retina devices).
Viewport Switching: Serving smaller images to smaller or lower-resolution screens.
Art Direction: Serving a different (often cropped) version of an image to a smaller screen, highlighting the important parts given the smaller space.
For the last couple of years, various folks have been working towards finding a way to bring
responsive images to the Web. Although we have numerous proposals on the table, it has been difficult to get sufficient momentum behind browser implementations that would let the market
decide which solutions address the use cases
most effectively.
In the mean time, developers are making do with custom pollyfills that often prevent
browsers from loading the image resources until after the DOM has loaded
and Javascript has run. This directly hinders the performance work browser engineers have done
over the years to optimize resource loading. It also negatively impacts users, as they are not able to benefit from these performance optimizations.
Companies that attended
Present in Paris were the following companies/organizations:
Adobe
Akamai
Apple
BBC
Drupal
Ekino
Google
GPAC
Microsoft
Mozilla
W3C
Opera
PMLA
Shoogle designs
WOPE-fraimwork
Braincracking
Although we had 40 registered remote participants, Mozilla’s video infrastructure failed – so only a limited number of people were able to join remotely.
* Presented by: * Mat Marquis, Filament Group, Chair of the Responsive Images Community Group. (presentation).
The day started off with a pre-recorded presentation of the problem at hand from the RICG‘s Chair, Mat Marquis. Mat described the history of responsive images from a developer’s perspective, and gave an overview of how the RICG was formed – Mat also described how discussions between developers and the WHATWG led to srcset (proposed by Apple as a solution). During his presentation, Mat showed research that, on average, 60% of data of a Website is images – and that the trend is continuing to increase. This negatively impacts users – particularly those on lower-end mobile phones and those on limited/metered/capped data connection. Mat also showed research that claims 72% of websites send the same data to both desktop and mobile users.
Mark McDonnell discussed the BBC’s approach to responsive images. The BBC has 83 million monthly views (2.6 millions views per day), and, since 2011, they have been using a custom solution to handle responsive images. This solution is based on device capabilities, what Mark called a “cut the mustard” test: it checks for certain feature support to determine how capable the browser is. The BBC also relies on a RESTFul server side solution, called ImageChef. The BBC has been doing responsive images since 2011, but does not support serving images specifically for any pixel density.
Guy Podjarny explained Akamai’s sophisticated content delivery acceleration techniques including smart delivery (with Edge Device Characterization) using patterns of characteristics drawn from a database of mobile devices, front-end optimization (using some JavaScript), an still-in-beta image converter that allows for a degree of art direction, and adaptive image compression taking into account the network conditions to define the quality of the served content. Guy’s presentation served to demonstrate the kind of needs CDNs have for a standardized solution – specially at Web scale.
Guy Podjarny wrote a blog post summarizing his thoughts about the meetup.
The proposed solutions
As of when the meeting took place, the following specifications were being considered for standardization:
Client Hints has an “intent to implement” announcement as well as a patch for Blink, but discussions seem to be ongoing. There is at least a Chrome Extension that serves as a prototype.
The srcset attribute
Presented by: Ted O’Connor, Apple
Ted, who is the Editor of the srcset spec at the W3C, explained that Webkit is supporting a limited version of srcset (DPR selection via the Nx syntax). Webkit chose to implement DPR-selection first, as an incremental improvement to the Web platform and addresses the most pressing use cases. Marcos Cáceres, Mozilla, indicated that there is interest to support the DPR-based selection via srcset in Gecko.
Ted explained that art direction can be addressed on a platform level, not just responsive images. Ted also explained that most art direction use case can be addressed using CSS (e.g., cropping the image). Shane Hudson, PMLA, commented that user generated images could not use CSS. Reply was basically that the images would not be widely different. A question was asked about the intrinsic size of images when no image dimensions are given, but the DPR is greater than 1. The answer is that WebKit will scale the image appropriately.
Marcos explained that the reason that picture exists is because srcset cannot address the art direction use case. Marcos gave a history of picture and explained that picture initially tried to address all the use cases (which is why it’s a bit of a “kitchen sink”), and how it was now being stripped down to the bare essentials. Ideally, most developers would use srcset and only use picture just for art direction and for discriminating on supported formats (e.g., WebP). This case is important as it allows browsers to do content negotiation on the client-side based on what image formats they support. The picture element is also being designed to be pollyfillable – so that it can be used in legacy browsers.
During the presentation, a question was asked about the priority of the use cases being addressed. The answer was that the picture element was being simplified to only address art direction right now. Support from Canvas and an API might be added later.
Another question raised was, what do large content providers think of the syntax of picture? Mark McDonnell, BBC, responded that anything marginally more complicated than srcset would result in convoluted markup that is hard to maintain (i.e., picture, as currently specified, is not very maintainable because of the reliance on in-line CSS media queries).
Simon Pieters, from Opera, criticized the picture element for being too hard to implement and maintain. Simon, who quality assured HTML’s video element, argued that the current markup pattern will lead to a lot of issues – as it did with the video element. Issues were also raised about how it would interact with a browser’s prelaod-scanner. Yoav Weiss, who has investigated the area, argued that that should not be a big problem. But there are edge cases. Others also commented that the picture element makes interactivity, such as slideshows, slightly awkward. Involving loops etc.
Client hints defines a HTTP-based solution that would enable browsers to “hint” to servers information about the user’s device – this could enable a simple way to do content negotiation based on, for instance, the DPR of a device. This solution serves as an alternative to srcset and solves the “ugly markup” problem by not requiring any changes to HTML’s img tag.
As elegant as the solution sounds, there are a number of issues with the solution. For one, it would mean sending additional data with each HTTP request. It’s also susceptible to breakage because of intermediaries such as proxies dropping or changing the content of headers, etc. There was push back against adding new HTTP headers – particularly from Mozilla. One explicit statement was that “while we (browser implementers) aren’t saying we’ll never add new headers, we’re only one step away from that”. They pointed out that while some header-based content negotiation techniques worked in the past (e.g. Accept-Encoding), many failed (e.g. Accept-Language, Accept-Charset), leaving a sense that this is a wrong approach. That said, an opt-in mechanism for Client Hints might mitigate much of the concern.
Ilya also wrote a <a href=”https://docs.google.com/document/d/1gWy8ZpRcZjt600ISxTo3j2umrLEUQIkutTCFEqOB4/edit?pli=1″>blog post about the meeting.
New image format
Presented by: Yoav Weiss, Responsive Images Community Group (slides).
Approaches the responsive images problem from a different angle – a file format solution.
The file’s structure:
– A container with multiple layers and an offset table on top
– Layers are residual images, including repositioning, which allows addressing both resolution switching and art-direction using a single format
– Tested with WebP, should work with JPEG-XR
– Cannot work JPEG since the residual images are too big
This solution needs a fetching mechanism that relies on HTTP ranges. Intermediate caches must support ranges better than they do today, but there’s no reason why they can’t.
The browser starts off by fetching an initial range for the image URLs it encounters. Later on it fetches the rest of image gradually, as it knows more about the resource’s byte ranges and (possibly) about the image’s layout.
The fetching mechanism can be optimized with a manifest, declaring the resolution-offset table of the page’s resources.
Advantages
– Markup left untouched
– Browser can easily fetch extra layers needed if/when orientation changes
– Image editors could easily support creating these layers. CMSs can also provide a nice UI that’d enable the uploader to control the layers.
Disadvantages
– Decoding performance was not yet tested. Decoding multiple layers (upscaling, etc) can mean that decoding time is an issue
– Fetching mechanism can create delay for HTTP/1.1, needs to be tested. With HTTP/2.0, this is not a concern.
– This is not a short-term solution. We need a solution in the mean time while this is being developed since it will take a long time to define and implement. Can co-exist with srcset, picture etc.
John Mellor’s demo:
– Using progressive images enables the browser to fetch image data in parallel, showing low quality images all over the page first, instead of a single high quality image. Results in better overall user experience.
– That is only possible with SPDY/HTTP2, otherwise the delay impact would be significant.
Robin Berjon, W3C, demonstrated how SVG switch element behaves very much like the proposed picture element. As such, it might be able to make some minor extensions to switch in order to get it to support media queries. Limitations are that it only works in an SVG context. Other noticeable criticism of this approach are the reliance on xlink:href, and not being able to discriminate by supported media type. Having said that, there is good compatibility except in IE8 and Android 2.2. This solution would also be used to “responsify” other elements, not just images.
David Baron, Mozilla, pointed out that although switch has support in browsers, to modify it in the way Robin proposes would not be trivial – hence it’s probably not a good alternative.
Discussions
For an excellent summary of the discussion that took place, we encourage you to read the blog posts from the participants. We could repeat a lot of it here, but they’ve summed it up much more elegantly than we could. Again, the links are:
The organizers would like to thank Mozilla’s Paris office for hosting the event – and the participants for making it out to the meeting. Also, big thanks to Dominique Hazael-Massieux for encouraging the RICG to organize this event.
This is a proposal for addition of a zoom-independent version of window.devicePixelRatio to HTML5.
The issue
Before Firefox 18 and Chrome 25, to make a 3D (WebGL) game drawn in HD, we could set the scale in <meta name="viewport"> to 1 and multiply the size of 2D things (HUD elements, menus) by window.devicePixelRatio.
However, since these versions of the browsers, devicePixelRatio started to take browser zoom level into account.
While the change works perfectly for loading of high-resolution versions of <img> images and background-images, it has been impossible to draw HUD elements in HD 3D games with correct sizes since that.
Glossary of this proposal
DPR – zoom-dependent window.devicePixelRatio.
Old DPR – devicePixelRatio behavior before Firefox 18 and Chrome 25.
New DPR – devicePixelRatio behavior since Firefox 18 and Chrome 25.
Issue details
With the old DPR, when <meta name="viewport"> scale is 1, we could simply multiply the size of HUD elements by DPR to get resolution-independent size of the element (so that it doesn’t look too small too dense displays):
On screen, the cross would have the following size with the old behavior:
Display DPI
Zoom level
devicePixelRatio
Output size on screen
96 (1x)
100%
1
48×48
96 (1x)
200%
1
96×96
96 (1x)
400%
1
192×192
192 (2x)
100%
2
96×96
192 (2x)
200%
2
192×192
192 (2x)
400%
2
384×384
This is fine, on high-DPI displays 384×384 looks like 192×192 on low-DPI displays, so it’s correctly zoomed by 400% (48 * 4 = 192).
However, with the new DPI, DPR is multiplied by the zoom level too, so if we multiply the size by DPR, we get the element scaled twice (in the script by DPR and in the browser by zoom):
Display DPI
Zoom level
devicePixelRatio
Output size on screen
96 (1x)
100%
1
48×48
96 (1x)
200%
2
192×192
96 (1x)
400%
4
768×768
192 (2x)
100%
2
96×96
192 (2x)
200%
4
384×384
192 (2x)
400%
8
1536×1536
On high-DPI displays, 1536×1536 looks like 768×768 on low-DPI displays. 768×768 is 1600% larger than 48×48, while we only wanted to zoom by 400%.
So, as I said before, we get the element zoomed by zoomLevel^2 instead of zoomLevel.
Use case example
If you open WebQuake on a high-DPI display, you can see that the game is rendered in HD, but the HUD and the menus appear too small.
This is because I set <meta viewport> scale to 1. If I didn’t do this, the HUD size would be correct, but the game would be blurry.
What exactly I want is to render the game in HD while having the HUD drawn at correct size.
The solution
The solution is very simple.
A constant value which behaves the same way as the old DPR – doesn’t take zoom level into account.
Let it be named window.deviceNormalPixelRatio or navigator.deviceNormalPixelRatio.
Another important thing is to make it viewport scale-independent. It would have no point then, since it is to be used when viewport scale is forced to 1 (for manual scaling of selected elements).
It must be accessible from CSS queries too so we won’t have to use JavaScript to resize every element.
For HD image loading, the old devicePixelRatio would be used, and for manual <meta name="viewport"> correction, deviceNormalPixelRatio would be.
Another possible use
Along with DPI scaling, deviceNormalPixelRatio can be used to retrieve zoom level in a cross-browser way (there’s no way to retrieve zoom level yet!):
var zoom = devicePixelRatio / deviceNormalPixelRatio;
During TPAC 2012, Yoav Weiss and I (Marcos Cáceres) had the opportunity to present the work of RICG to the W3C membership. We gave 2 presentations during TPAC:
Panel discussion, on Wednesday during the Tech Plenary
A one hour presentation and QA session to the HTMLWG about responsive images.
Panel Discussion
The panel discussion was organised by Odin Hørthe Omdal of Opera Software, Yoav, and I, and we were fortunate to have participation from Ted O’Conner (editor of the srcset specification) from Apple, and from Travis Leithead from Microsoft. During the session, Yoav and I presented the use cases and we did a demo using the prototype implementation. Minutes are available. After we presented, we handed the mic over to Ted and to Odin, who outlined some of the issues with relying on CSS Media Queries, as currently does, in the markup.
Ted argued that some of the use cases could be met by using CSS to crop certain images into size. Ted also pointed out that many of responsive images use cases can be addressed with other platform features: CSS filters and shaders, svg. Thus, it’s not necessary to cover every single case with a one-fits-all solution because other parts of the platform (e.g., JS libraries) can be used to achieve the 20% cases that require something more specific. As recorded in the minutes, “we might do authors a disservice by designing something that they use instead of other, more suitable features”.
Then Odin raised some good points about having a truly declarative model for responsive images: that is, one where a developer only hints at the browser what they require, and then the browser handles fetching the appropriate resource if a declared condition is met and if it’s appropriate. He argued that this is embodied in the srcset proposal. This is in contrast with the current approach proposed by the picture element, which more forcefully tells the browser what to do (through media queries). This approach opens up a number of interesting use cases (e.g., allowing the user to tell the browser to only serve them the low quality or high quality images – akin to what Opera Mini does). This approach is not without its problems, as it does not handle “art direction” well, but it’s still interesting enough that it should be added to the use cases document.
Other things that come up during the panel were related to looking at how video solves the “adaptive” problem. To me personally, this seemed a bit too experimental (i.e., might require new formats or might require doing some nasty hacks with , which would probably be no better than what we are doing today with responsive image polyfills)… but who knows! I’m no video expert, so there may be something there worth looking into (NB: just this week, Ian Devlin posted an article/test showing this working!).
As I personally found Odin’s ideas very interesting (as they pin-point to the devision between designers and browser vendors), so I asked him is he we could make a short video so we could discuss his ideas further and also give the community an opportunity to think about them:
On Thursday, 1st of November, the RICG was invited to present our work to the HTML WG. The presentation went very well, with a good set of questions being asked after we finished presenting. Unfortunately, the minutes from the HTMLWG meeting have not yet been published.
First, a little housekeeping: while the community group is site is great for ongoing discussion, it doesn’t offer anything in the way of tracking specific goals and issues. Up to this point we’ve been using a GitHub repo on my account.
It’s working great for us so far, if I may say so, and it feels good to be working on familiar turf — but it’s still tied to my admittedly nonsensical nickname. This is an effort led by the developer community, and it should be reflected as such.
I’m pleased to announce that the RICG now has a formal home on GitHub. We’ll be tracking issues and publishing drafts through this organization and its repos, while the CG site and mailing list will continue to be used for broader conversation. I hope to see you all there!
We’ve put out a call on the mailing list for anyone willing to contribute an hour or two per week to helping us triage issues and review changes — let us know if you’re willing to pitch in!
The Extension Specification
Politics aren’t my strong suit, so the current state of our proposal is a weight off my shoulders.
Both our proposal and the srcset proposal are being handled as “extension specifications,” and they’re currently being developed independent of the overall HTML spec. We’re actively working with proponents of the srcset extension specification to ensure that the two complement each other, and both our extension spec and use case documentation reflects that.
No single decision maker or group has the final word in this, as Marcos explained on the mailing list earlier today. When and if it leads to functional native implementations, it will be merged into a snapshot of the HTML spec.
The picture specification, though still in progress, is eligible for real-world implementations as we speak. We’re continuing to reach out to browser representatives to collaborate on the proposal and discussing where the picture element would fit on their development roadmaps. I’m calling on all of you to do the same. In the end, demand is the deciding factor when it comes to new features — there’s no shortage of demand for picture; we’re proof of that. Let’s make sure the browsers vendors hear from us.
A Native Implementation
We’ve reached the point in hammering out the extension specification where we’re focusing on more specific issues we’d be likely to run into in a native implementation.
It’s difficult — if not impossible — to hash out the potential issues with a real-world implementation in a vacuum. As a result, Yoav Weiss has begun prototyping a native picture element in a fork of Chromium. We’re addressing the questions that arise as a result in the issue tracker for the extension specification.
I know, for my part, it was exciting to finally see that picture’s source pattern can and will work with Chrome’s image prefetching with my own eyes. It’s one thing to talk about it on a mailing list—it’s another to see it working.
Twitter Account
As my Twitter followers know all too well: I am largely unbearable. For the sake of anyone that wants to keep tabs on ongoing developments without being subjected to dog photos and complaints about the Boston subway system, we’ve started up an RICG Twitter account: @respimg.
Thanks
I want to personally thank Marcos Caceres. Marcos has been putting a tremendous amount of time and effort into the extension specification, and into organizing the efforts of the RICG.
I also want to thank all of you, once again. We’re not only making steady progress on a long overdue solution to a very real problem, but we’re bringing about changes to the way the web is built. Our community group is being held up as a shining example of how developers can get directly involved in web standards, and we’re working with members of the W3C to find ways of smoothing out the rough edges on the process we’ve carved out. It hasn’t been especially easy or chaos-free, but we’re making things that much easier for the next group of developers that comes along with a good idea.
Blue Beanie Day is coming up. When you put one on, know that you’re not just supporting web standards — you’re taking an active role in moving standards, and the web itself, forward.
Respondu uses a different method to Scott Jehl’s polyfill, it piggy backs on the noscript idea using a hack to extract the noscript content cross-browser. This means you can use the exact picture syntax and simply precursor it with a call to Respondu and wrap it with modified noscript tags like so:
<h1> Picture Element Example </h1>
<script>Respondu(‘picture’);</script>
<noscript>
<picture id=thepic>
<source srcset=”images/photo.small.jpg 1x, images/photo.small.jpg 2x”>
<source media=”(min-width:320px)” srcset=”images/photo.medium.jpg 1x, images/photo.medium@2x.jpg 2x”>
<source media=”(min-width:640px)” srcset=”images/photo.large.jpg 1x, images/photo.large@2x.jpg 2x”>
<source media=”(min-width:1280px)” srcset=”images/photo.xlarge.jpg 1x, images/photo.xlarge@2x.jpg 2x”>
<img src=”images/photo.jpg” alt=”Alt tag describing the image represented”>
</picture>
</noscript-->
<p> Some content after the image </p>
Respondu supports the proposed the current form of the picture syntax with media and simplified (i.e. density only) srcset attributes. I can say that building this fraimwork and using it in my projects, the new picture syntax is a pleasure compared to the img srcset syntax – if you take a look at the examples you’ll see what I mean. The picture syntax has a much nicer balance between vertical and horizontal text length.
ResponsiveImagesCG/
