Reflections on “HTTPS is Hard”

Over the last few months I’ve been putting together my talk for the year, based on a blog post that is titled “HTTPS is Hard”. You can read the full article on the Yell blog on which it is published. There’s also an abridged version on Medium. It’s been a very long time coming, and has changed over the time I’ve been writing it, so I thought I’d get down a few reflections on the article.

It’s really long, and took a long time to write

This is firstly, the longest article I’ve written (at over four thousand words, it’s a quarter of the length of my dissertation) and it’s taken the longest time to be published. I had a 95% complete draft ready back in September, when I was supposed to be working on my Velocity talk for October but found myself much more interested in this article. Dan Applequist has repeatedly asked me to “put it in a blog post, the TAG would be very interested” – so finally, it’s here.

The truth is that I’m constantly tweaking the post. Even the day before it goes live, I’m still making modifications as final comments and notes come in from friends that I’ve been working with on this. Also, it seems like every week the technology moves on and the landscape shifts: Adobe offers certs for free, Dreamhost gives away LetsEncrypt HTTPS certs through a one-click button, Netscaler supports HTTP/2, the Washington Post write an article, Google updates advice and documentation, and on and on and on… All through this evolution, new problems emerge and the situation morphs and I come up with new ways to fix things, and as I do, they get put into the blog post. Hence, it’s almost a 20 minute read.

A special thank you to Andy Davies, Pete Gasston, Patrick Hamann and the good people at Yell; Jurga, Claire and the UI team (Andrzej, Lee and Stevie) for their feedback throughout this whole process. I’m sure they skipped to the new bits each time.

Is HTTPS really neccessary, for everyone?

Yes.

Every day something silly happens. Today’s was from generally-awesome tech-friendly company Mailchimp. They originally claimed that “Hosted forms are secure on our end, so we don’t need to offer HTTPS. We get that some of our users would like this, though” (tweet has since been deleted). Thankfully, they owned up and showed CalEvans how to do secure forms.

Still, it’s this kind of naivety that puts everyone’s security at risk. A big thumbs up to Mailchimp for rectifying the situation.

If I were to have started today, would HTTPS still be hard?

Yes, though nowhere near as hard. We’d still have gone through the whole process, but it wouldn’t have taken as long (the Adobe and Netscaler bits were quite time-consuming) and the aftermath wouldn’t have gone on for anywhere near as long if I’d have realised in advance about the referrer problem.

If you’d have known about the referrer issue, would you have made the switch to HTTPS?

Honestly, I’m not sure I would have pushed so hard for it. We don’t have any solid evidence to say it’s affecting any business metrics, but I personally wouldn’t like the impression that traffic just dropped off a cliff, and it wouldn’t make me sign up as an advertiser. Is this why Yelp, TripAdvisor and others haven’t migrated over? Who can say…

This is why the education piece of HTTPS is so important, because developers can easily miss little details like referrers, and just see the goals of ranking and HTTP/2 and just go for it.

The point of the whole article is that there just isn’t the huge incentive to move to HTTPS. Having a padlock doesn’t make a difference to users unless they sign in or buy something. There needs to be something far more aggressive to convince your average developer to move their web site to HTTPS. I am fully in support of Chrome and Firefox’s efforts to mark HTTP as insecure to the user. The only comments I get around the office about HTTPS happen when a Chrome extension causes a red line to go through the protocol in the address bar – setting a negative connotation around HTTP seems to be the only thing that gets people interested.

What’s changed since you wrote the article?

I am really pleased to see the Google Transparency Report include a section on HTTPS (blog post). An organisation with the might and engineering power of Google are still working towards HTTPS, overcoming technical boundaries that make HTTPS really quite hard. It’s nice to know that it’s not just you fighting against the technology.

What about “privileged apps” – you don’t talk about that

The “Privileged Contexts” spec AKA “Powerful Features” and how to manage them is a working draft and there’s a lot of debate still to be had before they go near a browser. I like how the proposals work and how they’ve been implemented for Service Worker. I also appreciate why they’re necessary, especially for Service Worker (the whole thread of “why” can be read on github). I hope that Service Worker has an effect on HTTPS uptake, though this will only truly happen should Safari adopt the technology.

It looks like Chrome is going to turn off Geolocation from insecure origins very soon, as that part of the powerful features task list has been marked as “fixed” as of March 3rd. Give it a few months and geolocation will be the proving ground for the whole concept of powerful features – something that I’ll be watching very closely.

AMP – Long-term Harmful

You may have heard about a project that Google has just announced called AMP (Accelerated Mobile Pages), which aims to speed up the mobile web.

Publishers around the world use the mobile web to reach these readers, but the experience can often leave a lot to be desired. Every time a webpage takes too long to load, they lose a reader—and the opportunity to earn revenue through advertising or subscriptions. That’s because advertisers on these websites have a hard time getting consumers to pay attention to their ads when the pages load so slowly that people abandon them entirely.

I completely understand the need for this project – the web is slow and advertising is such a key revenue source for publishers that these adverts are ruining the experience. Facebook’s Instant project is designed to do much the same thing. AMP is not a new idea.

Technically, AMP builds on open standards and allows you to simply replace some parts of your build and some components of your system with AMP versions of those pages. What happens with these pages is the secret sauce – Google knows of these pages, and if they pass the AMP tests it caches versions of those sites on its own servers.

This isn’t too scary – Google has committed to allow  the major advertising networks and pixel tracking so that businesses can maintain their precious statistics. AMP, in general, is a good idea.

Except for one thing.

If you’re not on it, you’re screwed.

Have a look at these two search pages. Both are searches for “Obama” on a Nexus 5 (emulated). The one on the right is the current search page, the one on the left is with AMP articles.

AMP search page comparison

Notice where the first organic listing is in each page. Currently, it’s just below the first page, but with AMP, it’s over two screens of content away! That’s absolutely massive, and for anyone not in the top 3 results, they may as well not be there.

Basically, if you’re not an AMP publisher, expect your traffic to drop, by double-digit percentages.

Short-term benefits, Long-term harm

Short-term, AMP will make publishers sit up and take notice of web performance, it’ll make websites faster across the board, and make more customers use Google. For users, this is a win/win situation.

I’ve come to the conclusion that this project is harmful to the web in the long term. If a publisher isn’t getting traffic because it’s all going to AMP publishers, then the amount of content on Google drops, quality drops and is put into these large content producers. You end up with less diversity in the web, because small producers can’t make money without using AMP. Even in the end when you’ve only got a few publishers who can survive, they will be encouraged to reduce adverts, or be limited to the subset that Google’s DoubleClick deem performant enough to be included in the AMP-friendly set of advertisers. This may block the big-money adverts, which keep these sites going.

It also creates dependencies on Google, and whilst it’s unlikely to go down any time soon, being cached on their servers and allowing for only the most basic tracking mechanism, and only for browsers that Google supports. It creates a two-tier system that the web is firmly against.

This project seems to polyfill a developer/organisation’s lack of time to dedicate to performance. Many very clever people are working on this education problem (hello to Path to Perf, Designing for Performance and PageSpeed) though it will take time. Short-term fixes like AMP and Facebook Instant are encouraging developers to take shortcuts to performance, handing their problems off to Google. This does nothing for the underlying issues, but with Google giving AMP such prominence in its search results, how can publishers resist?

Or is it a temporary solution

I hope that from all of this, developers sit up and take notice of web performance; improve their sites and provide a better experience for all. If we don’t, Google will keep solutions like this around, leaving the only content that gets interaction as the content that Google approves of – and no one wants that.

State of the Browser 5

Dave Letorey at State of the Browser

A short write up on the weekend

Another year, another State of The Browser – now in its fifth year, it’s a conference for the London Web development community. It’s aimed at the masses, we want it to be accessible to all and have really great speakers, and this year was no exception to those rules.

In its inception, SOTB was a chance for the browser manufacturers to get together and talk about the latest and greatest things in their browsers to a wide audience. A lot of this mandate is being done (very, very well) by Edge conference, organised by our friends at London Web Performance. So, this year, we went back to our roots, the community that LWS is built upon, for talks about the browser, new browsers, new technology, and new ways of working. I’m really pleased with how it worked out, and we had a really high quality of submissions (my fellow organiser Morena Fiore-Kirby covers this really well in her write-up) it’s a shame we couldn’t fit more in.

This also meant that we could feature more new speakers and I’m really happy that we did. LWS has a long history of being the place where great speakers have done their first gigs (Pete Gasston, Laura Kalbag, and apparently Jake Archibald did a talk in the very early days – to just name a few) and I hope this tradition will continue. We were very pleased to welcome Martin Jakl (@JaklMartin) and Laura Elizabeth (@laurium) to the stage, mentored by Pete and Solé, they both did a great job with their talks. They’ve both got bright futures so watch out for them.

My highlight

I’m very happy to say that my surprise of the weekend was Chris Heilmann. Other than his incredible generosity in giving away 10 years worth of tech swag (see below) he gave the best talk that I’ve ever had the pleasure to hear him give. He seemed truly passionate about his new product, Edge, and the people that he works with to make the web better, fixing not just the web that you and I see, but also internal websites, massive SAP systems and changing web standards culture in huge corporations. If I hadn’t already been standing, I’d have stood up to applaud him and what the Edge development team have done. Thank you.

Chris' Tech Swag (half of it) Chris Heilmann at State of the Browser 5

Finally

Thanks to everyone who came along and were such a friendly bunch of people. We got lots of great feedback, and I can’t wait to do it again next year!

Check out the photo gallery for the event on Google Photos
For more on the event, check out these articles:

p.s. The whole event was live-streamed, and we have videos of the whole event too – the first half is already up on the @webstandards Vimeo page with the rest to follow shortly. A massive thank you to our live events team Pete Wood and company who always do an amazing job. Thank you!

Data-Driven Performance Breakout at Edge Conference

I was lucky enough to attend Edge Conf in London this year, a day that I always truly enjoy. The main sessions of the conference were streamed live and videos will be available later, but the break-outs weren’t recorded. These were the sessions I enjoyed the most and it’s a shame that people won’t see them without being there – so here’s my notes on what was said to the best of my ability (and with a big hat tip to George Crawford for his notes). Patrick Kettner was the moderator.

Q: How can we use the masses of data that RUM collects to get businesses to care about performance?

Business leaders like metrics from companies that they can relate to (i.e. Amazon, eBay) but these aren’t very useful metrics as the scale is completely different. Finding stats from competing or relevant companies is hard, so how do you make them care?

Introducing artificial slowness is one way to convince people, but not good for business. There’s also the risk that you may not see increase in conversion from speed improvements! Filmstrips are incredibly useful at this point to see what’s going on and these are available in Chrome Dev tools in the super secret area.

Showing videos to business people makes it really hit home – people hate it when they can visibly see their site suck. It’s like making people watch a user test for their site. Shout out to Lara Hogan at Etsy (their engineering blog is awesome) for their great work on this, something that Yell has copied.

Metrics that are useful: first render, SpeedIndex, aren’t available in the browser. Using SpeedCurve can really make business people sit up and take notice of performance because it’s a pretty interface to those things.

All-in-all, the standard metrics are unlikely to be the best for you, so add in user timing markings (and a very simple polyfill) and graph those, including sending them to WebPageTest so you can measure the things that are important to you over time. This was done very successfully by The Guardian (hat tip Patrick Hamann).

Q from Ilya Grigorik: The browser loading bar is a lie, yet users wait for it. What metric should it use?

Basically, developers can put their loading after the onLoad event to hack around the loading spinner. If we stop the spinner at first render, it’s not usable. If we stop it at when the page can be interacted with when would that be? The browser runs the risk of “feeling slower” or “feeling faster” by just changing the progress bar. Apparently there’s one browser that just shows the bar for three seconds, nothing more.

No real consensus was reached here, but it was a very interesting discussion

Q: Flaky or dropped connections are important to know about for performance metrics – what can the room say about their experiences gathering offline metrics?

When the FT tried this with their web app they often exceeded localStorage sizes and sometimes POST sizes (25MB) as users could be offline for a week or more. The Guardian had good success with bundling beacons up into one big post to save money with Adobe Omniture/SiteCatalyst.

The best solution is the Beacon API (sendBeacon) which promises to deliver the payload at some point (which images/XHR don’t right now). It’s implemented in Google Analytics, you just have to enable it in the config, other tracking providers don’t have it right now.

Q: What metrics APIs are missing in browsers?

A unique opportunity to ask Ilya to add APIs into Chrome – not to be passed up

  • Frame Timing API – requested as an ES7 observable (which is unlikely).
  • Performance Observer – a subscribable stream of events that will need processing to be useful. This will give accurate frame-rate
  • Network error logging API – could work like an error reporter that posts to a configurable second origin (via a header like CSP)
  • JavaScript runtime errors without hacking window.onError
  • SpeedIndex, or a proxy for it. There’s a script for this already but it’s not massively accurate. Standardising SpeedIndex would be great.
  • First Paint – according to Ilya it’s not possible and quite subjective browser-to-browser

Wrap-up

I’d have loved to stay and chat more (nice to meet Tim Kadlec in person, shout out to the Path to Performance podcast as well), it’s rare to have a lot of the web performance community in the same room at the same time and should definitely happen more often.

If there’s things I’ve missed, let me know in the comments or on twitter (@steveworkman)